Node.js: parsing CSV file and populating a column from Twitter API
Posted on September 12, 2012 | Categories: Technology
I was trying to implement a simple script that would read data from a .csv file, then do a lookup for each row hitting Twitter API, then fill that additional info (API response) into a new column. Sounds pretty simple.
Found this great node-csv module that makes parsing and writing into a .csv file super easy.
Twitter API connection was also an easy part.
But try connecting those two and write into a file. Nada.
The reason it’s not working is because the ‘transform’ function included in node-csv finishes before all the data transform is complete. And in my case I need it to wait for the HTTP response to come back. I read that people have similar issues when writing into a database.
So I went and submitted an issue to the repo owner, who replied within hours, which is very nice. Hope that it’s actually doable. Meanwhile, I tried using another CSV parser module called ya-csv, and ran into the same result. Looks like both use similar methods to read and write data, and do not include support for data transform/write callbacks.
I googled it, and looks like people are creating async loops to get around this issue, but I’m gonna do it in Python first to save time, then try that loop workaround and compare results.
September 13th, 2012 at 3:39 pm
[...] First attempt to do it with node.js (for the sake of speed, plus I love node) did not quite work. So I decided to fallback on Python and write a quick and dirty script to get it done, forgetting about events and async calls. [...]