Today I wanted to test out how to fetch data from one of the http APIs and save it to Mongo.
A quick example: get Twitter profile data for a given Twitter username.
In languages like PHP, we’d call curl and pass the URL. With node.js it’s actually really simple, we just call the GET method.
We are setting our request object that will call “GET” with our specified URL and wait to receive the response.
Request also has another method “on” that will listen for response data to come back (‘data’ event).
NOTE: It is not enough to just detect the ‘response’ event, as it looks like data coming back in “chunks”. I created a variable called “completeResponse” that concatenated all chunks into the final complete response (before I added that, my app would throw errors because of incomplete json that I tried running through JSON.parse()). Then wait for the ‘end’ event before actually saving the data into mongo.
Detailed info on http request and this issue
After you get your final parsed data object, it’s ready to be saved into mongo.
Working example can be found on github: https://github.com/ntanya/datacoll.git
(right now it’s all running on local machine)
Tanya Nam 
Sorry to say but i couldnt find the working example github: https://github.com/ntanya/datacoll.git
I had to move this repo to private since it’s a working repo with passwords and such. Gists should give you a pretty good idea on how to put this together
You can do it in a much simpler way by using requestify, which is a great HTTP client for nodeJS + it supports caching.
http://ranm8.github.io/requestify
thanks for the tip Ran
cvhgfuhgjh
Working with http requests in node.js (similar to curl)
ON JUNE 8, 2012 BY NTANYA54IN TECHNOLOGY
Today I wanted to test out how to fetch data from one of the http APIs and save it to Mongo.
A quick example: get Twitter profile data for a given Twitter username.
In languages like PHP, we’d call curl and pass the URL. With node.js it’s actually really simple, we just call the GET method.
We are setting our request object that will call “GET” with our specified URL and wait to receive the response.
Request also has another method “on” that will listen for response data to come back (‘data’ event).
NOTE: It is not enough to just detect the ‘response’ event, as it looks like data coming back in “chunks”. I created a variable called “completeResponse” that concatenated all chunks into the final complete response (before I added that, my app would throw errors because of incomplete json that I tried running through JSON.parse()). Then wait for the ‘end’ event before actually saving the data into mongo.
Detailed info on http request and this issue
After you get your final parsed data object, it’s ready to be saved into mongo.
Working example can be fou