January 27th, 2020

Loads into glitch okay*: supreme-sovereign-mi561jl15e.glitch.me/data/sets?mete… :-) * sqlite-utils is not handling the escaped quotes correctly in tunes.csv, see 7th tune. * (Aside, @simonw, glitch now appears to put dragged files into a CDN assets dir, so your glitch instructions/code need updating, sorry!)

Tab separated is s lot better as a default than CSV as CSV quote escaping is bonkers and often non deterministic

Whether they go into the assets CDN depends on file type — let us know if there should be different handling there.

TSV has no standard; how do you specify e.g. a newline? At least CSV and its quoting is RFC-specified, and I don’t see what is bonkers or non-deterministic about it.

These were some Mb CSV files, so given the space available it probably makes sense for them to be in the CDN, and then the db-setup-script could load/process them from there, I guess, but not an expert at glitch or datasette, will see what @simonw says :)

You use actual tabs and newlines, and use \t and \n \r and \ to escape. You don’t need to escape commas or any kind of quote. Here’s a spec iana.org/assignments/me…

If you think CSV is converged the read this PEP python.org/dev/peps/pep-0…

The spec you link to does not mention escape characters at all… Of course CSV has lots of bad use, I am well aware of that, but it does at least have a well-defined non-bonkers deterministic specification.

I got csvs-to-sqlite to mostly work using the —skip-errors option: