dataframe - Julia: CSV.write very memory inefficient?

Question

I noticed that when saving large dataframes as CSVs the memory allocations are an order of magnitude higher than the size of the dataframe in memory (or the size of the CSV file on disk), at least by a factor of 10. Why is this the case? And is there a way to prevent this? Ie is there a way to save a dataframe to disk without using (much) more memory than the actual dataframe?

In the example below I generate a dataframe with one integer column and 10m rows. It weighs 76MB but writing the CSV allocates 1.35GB.

using DataFrames, CSV

function generate_df(n::Int64)
    DataFrame!(a = 1:n)
end

julia> @time tmp = generate_df2(10000000);
  0.671053 seconds (2.45 M allocations: 199.961 MiB)

julia> Base.summarysize(tmp) / 1024 / 1024
76.29454803466797

julia> @time CSV.write("~/tmp/test.csv", tmp)
  3.199506 seconds (60.11 M allocations: 1.351 GiB)

Answer 1

What you see is not related to CSV.write, but to the fact that DataFrame is type-unstable. This means that it will allocate when iterating rows and accessing their contents. Here is an example:

julia> df = DataFrame(a=1:10000000);

julia> f(x) = x[1]
f (generic function with 1 method)

julia> @time sum(f, eachrow(df)) # after compilation
  0.960045 seconds (40.07 M allocations: 613.918 MiB, 4.18% gc time)
50000005000000

This is a deliberate design decision to avoid unacceptable compilation times for very wide data frames (which are common in practice in certain fields of application). Now, this is the way to reduce allocations:

julia> @time CSV.write("test.csv", df) # after compilation
  1.976654 seconds (60.00 M allocations: 1.345 GiB, 5.64% gc time)
"test.csv"

julia> @time CSV.write("test.csv", Tables.columntable(df)) # after compilation
  0.439597 seconds (36 allocations: 4.002 MiB)
"test.csv"

(this will work OK if the table is narrow, for wide tables it might hit compilation time issues)

This is one of the patterns that are often encountered in Julia (even Julia itself works this way as args field in Expr is Vector{Any}): often you are OK with type unstable code if you do not care about performance (but want to avoid excessive compilation latency), and it is easy to switch to type-stable mode where compilation time does not matter and type-stability does.