Importing large CSV files

Hi @akt01,

That’s a large file! ?? How do you maintain it?

The plugin streams the file content, it does not read the entire file into memory. So it should be possible from a technical point of view. But it might take some time to process such a large file so you will probably run into a timeout. I would give it a try and then see what happens. If you can send me the CSV file, I can test for you?

Thanks,
Peter

I have a command line tool that does a bunch of things to sanitize a CSV file for MySQL loading. strip blank columns and rows, column name fixup etc.
One of the things it does is splits the CSV file into 200,000 rows per file.

If I find the time I may publish it as a gist (github) file. It’s written in python.

I have posted my python cleanup tool here.

Interesting, thank you @peterschulznl and @charlesgodwin,

@peterschulznl, yes it’s quite large, it’s just a test file that I’m using to test scalability.

I’ve been looking into it more and have had success with using mysqlimport client utility that imported a ~2.5GB CSV file. I needed to reduced the size just because of disk space limitation, but it should work fine for ~5GB or even larger size. Need to run it in the shell via SSH to the database server.

I think this built-in utility runs “LOAD DATA INFILE” internally, which seems to be reasonably efficient at handling large CSV files.

Thanks!