Working with http requests in node.js (similar to curl)

Today I wanted to test out how to fetch data from one of the http APIs and save it to Mongo.

A quick example: get Twitter profile data for a given Twitter username.

In languages like PHP, we’d call curl and pass the URL. With node.js it’s actually really simple, we just call the GET method.

We are setting our request object that will call “GET” with our specified URL and wait to receive the response.

Request also has another method “on” that will listen for response data to come back (‘data’ event).

NOTE: It is not enough to just detect the ‘response’ event, as it looks like data coming back in “chunks”. I created a variable called “completeResponse” that concatenated all chunks into the final complete response (before I added that, my app would throw errors because of incomplete json that I tried running through JSON.parse()). Then wait for the ‘end’ event before actually saving the data into mongo.

Detailed info on http request and this issue

After you get your final parsed data object, it’s ready to be saved into mongo.

Working example can be found on github: https://github.com/ntanya/datacoll.git

(right now it’s all running on local machine)

Advertisements

6 thoughts on “Working with http requests in node.js (similar to curl)

  1. Tanya

    I had to move this repo to private since it’s a working repo with passwords and such. Gists should give you a pretty good idea on how to put this together

  2. Working with http requests in node.js (similar to curl)
    ON JUNE 8, 2012 BY NTANYA54IN TECHNOLOGY
    Today I wanted to test out how to fetch data from one of the http APIs and save it to Mongo.

    A quick example: get Twitter profile data for a given Twitter username.

    In languages like PHP, we’d call curl and pass the URL. With node.js it’s actually really simple, we just call the GET method.

    We are setting our request object that will call “GET” with our specified URL and wait to receive the response.

    Request also has another method “on” that will listen for response data to come back (‘data’ event).

    NOTE: It is not enough to just detect the ‘response’ event, as it looks like data coming back in “chunks”. I created a variable called “completeResponse” that concatenated all chunks into the final complete response (before I added that, my app would throw errors because of incomplete json that I tried running through JSON.parse()). Then wait for the ‘end’ event before actually saving the data into mongo.

    Detailed info on http request and this issue

    After you get your final parsed data object, it’s ready to be saved into mongo.

    Working example can be fou

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s