Node.js: parsing CSV file and populating a column from Twitter API

I was trying to implement a simple script that would read data from a .csv file, then do a lookup for each row hitting Twitter API, then fill that additional info (API response) into a new column. Sounds pretty simple.

Found this great node-csv module that makes parsing and writing into a .csv file super easy.

Twitter API connection was also an easy part.

But try connecting those two and write into a file. Nada.

The reason it’s not working is because the ‘transform’ function included in node-csv finishes before all the data transform is complete. And in my case I need it to wait for the HTTP response to come back. I read that people have similar issues when writing into a database.

So I went and submitted an issue to the repo owner, who replied within hours, which is very nice. Hope that it’s actually doable. Meanwhile, I tried using another CSV parser module called ya-csv, and ran into the same result. Looks like both use similar methods to read and write data, and do not include support for data transform/write callbacks.

I googled it, and looks like people are creating async loops to get around this issue, but I’m gonna do it in Python first to save time, then try that loop workaround and compare results.

Advertisements

One thought on “Node.js: parsing CSV file and populating a column from Twitter API

  1. Pingback: Tanya Nam » Blog Archive » Python > write to a CSV file with Twitter API lookups

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s