r - 'Embedded nul in string' error when importing csv with fread

Question

I have a large file (3.5G) that I'm trying to import using data.table::fread.

It was originally created from an rpt file that was opened as text and saved as a CSV.

This has worked fine with smaller files (of the same type of data-same columns and all. This one is just for a longer timeframe and wider reach).

When I try and run

mydata <- fread("mycsv.csv")

I get the error:

Error in fread("mycsv.csv") : embedded nul in string: 'year'

What does this mean?

Answer 1

We can remove the null terminators on the command line using something like:

sed 's/\0//g' mycsv.csv > mycsv.csv

Or as suggested by @marbel, fread allows you to pass the sed call inside the text. Such as:

fread("sed 's/\0//g' mycsv.csv")