Today I wanted to test out how to fetch data from one of the http APIs and save it to Mongo.
A quick example: get Twitter profile data for a given Twitter username.
In languages like PHP, we’d call curl and pass the URL. With node.js it’s actually really simple, we just call the GET method.
We are setting our request object that will call “GET” with our specified URL and wait to receive the response.
Request also has another method “on” that will listen for response data to come back (‘data’ event).
NOTE: It is not enough to just detect the ‘response’ event, as it looks like data coming back in “chunks”. I created a variable called “completeResponse” that concatenated all chunks into the final complete response (before I added that, my app would throw errors because of incomplete json that I tried running through JSON.parse()). Then wait for the ‘end’ event before actually saving the data into mongo.
After you get your final parsed data object, it’s ready to be saved into mongo.
Working example can be found on github: https://github.com/ntanya/datacoll.git
(right now it’s all running on local machine)