I was trying to implement a simple script that would read data from a .csv file, then do a lookup for each row hitting Twitter API, then fill that additional info (API response) into a new column. Sounds pretty simple.

Found this great node-csv module that makes parsing and writing into a .csv file super easy.

Twitter API connection was also an easy part.

But try connecting those two and write into a file. Nada.

The reason it’s not working is because the ‘transform’ function included in node-csv finishes before all the data transform is complete. And in my case I need it to wait for the HTTP response to come back. I read that people have similar issues when writing into a database.

So I went and submitted an issue to the repo owner, who replied within hours, which is very nice. Hope that it’s actually doable. Meanwhile, I tried using another CSV parser module called ya-csv, and ran into the same result. Looks like both use similar methods to read and write data, and do not include support for data transform/write callbacks.

I googled it, and looks like people are creating async loops to get around this issue, but I’m gonna do it in Python first to save time, then try that loop workaround and compare results.