On: Mistakes Developers Make When Using Python for Big Data Analytics

[mkesper]

Even when not using pandas, do yourself a favour and use csv.DictReader/DictWriter, as csv format has quite some quirks.

https://docs.python.org/3/library/csv.html#csv.DictReader

[mercurial → mkesper]

Yes. Never use split(",") or equivalents to read or write a CSV, always use a library. Otherwise you'll get a nasty surprise next time your code encounters a line or row separator in a value.

[t1m → mkesper]

The csv module has it's fair share of quirks to be aware of as well, like not supporting Unicode.

[mazelife → t1m]

...which is why I usually turn to this: https://pypi.python.org/pypi/unicodecsv/0.9.4

It's a "drop-in replacement for Python 2's csv module which supports unicode strings without a hassle."

[juliangregorian → t1m]

Even in Python 3?

[mkesper → juliangregorian]

In Python3 it's completely fine, if you open the files with the correct encoding. For Python2, you've got to use UnicodeReader/Writer (and to open in binary mode): https://docs.python.org/2/library/csv.html?highlight=csv#exa...