Working with http requests in node.js (similar to curl)

Today I wanted to test out how to fetch data from one of the http APIs and save it to Mongo.

A quick example: get Twitter profile data for a given Twitter username.

In languages like PHP, we’d call curl and pass the URL. With node.js it’s actually really simple, we just call the GET method.

We are setting our request object that will call “GET” with our specified URL and wait to receive the response.

Request also has another method “on” that will listen for response data to come back (‘data’ event).

NOTE: It is not enough to just detect the ‘response’ event, as it looks like data coming back in “chunks”. I created a variable called “completeResponse” that concatenated all chunks into the final complete response (before I added that, my app would throw errors because of incomplete json that I tried running through JSON.parse()). Then wait for the ‘end’ event before actually saving the data into mongo.

Detailed info on http request and this issue

After you get your final parsed data object, it’s ready to be saved into mongo.

Working example can be found on github: https://github.com/ntanya/datacoll.git

(right now it’s all running on local machine)

Super simple file upload with node.js

Here’s a quick and dirty tutorial on how to upload files with node. I didn’t use any dependencies aside from “fs” that allows us to move the file from the temp location to where we need it to be. We’ll also use express to make it super easy.

First, add all required modules and create our server:

Now let’s render a page with a form:

And add a post method to handle uploads once the form is submitted.

First, check out log to see what you’re getting in res.files:

app.post('/', function(req, res){
console.log(JSON.stringify(req.files)); // this will let you see what you're getting in the files

});

And to complete this, we’ll actually need to save the uploaded file into our desired location, in my case it’s /public/images folder:

Note: in res.files object, make sure that your file name is the same name as the input field.
<input type="file" name="uploadfile" />
var temp_path = req.files.uploadfile.path;

Thanks to hacksparrow for the post on this.

Complete working example on github

Staying current in technology

My favorite tech person and mentor of all times is Mike. And when I’m not having enlightened discussions with him over lunch or in the office, I’m on his blog reading stuff that he somehow manages to churn every day (jealous of his mad writing skillz :)

Today, he writes:

In tech, we’re sort of fucked compared to say a violinist, who can be pretty sure their main tool of the trade isn’t going to change much from the day they pick it up to the day they die, so they can sink all that energy into pure, beautiful mastery. In tech, we don’t even know if our keyboards are going to look the same 5 years from now.

All us, readers of his blog know that his “violin” is a solid combo of Linux/vim/Python/Mercurial – which makes a ton of sense, and if you’re good with that stuff you are golden.

My additional thinking is that javascript, that grew from just something that your browser supports natively into de-facto standard of the coolest new technologies, from advanced client-side frameworks (prototype, jquery, and newer stuff like backbone) to databases (mongo) to server-side (node). So now, the power is in hands of javascript masters: if you know the language, you know how to build up any app, all aspects of it. I definitely has its quirks (callbacks, ubiquituos “this”), but to me they are like delicious puzzles that once solved, promise great rewards. That’s what I’m sticking to for now, and having fun with it.

Deploying node.js/mongodb app on heroku and mongolab

Ok, in my rare downtime at work I’ve decided to put together a quick test app that uses node.js and mongo.

For starters, here’s an excellent resource on node: http://howtonode.org

And a great tutorial on getting started with your first node.js app: http://howtonode.org/express-mongodb

This tutorial covers the app setup, first locally with memory storage, then locally with mongo DB. It also briefly notes the use of Express (node.js framework) and Jade (that allows you to create quick HTML templates, similar to haml for Ruby).

What it doesn’t cover, is deployment of your app into the real world – the internets! The choice of hosting is obvious – heroku provides the best platform out there (in my opinion) for easy deployment of apps (they started with ruby and now are node.js compatible, with their most recent cedar stack).

To get your heroku environment setup, follow this excellent walkthrough: https://devcenter.heroku.com/articles/nodejs
Follow it up until the Postgres part – we will actually need to setup Mongo.

For running Mongo, heroku offers an add-on that you can use for your apps through MongoLab. And the setup is just as easy and straightforward: https://devcenter.heroku.com/articles/mongolab

Remember that you need your MONGOLAB_URI in order to connect your app to your Mongo database hosted on Mongolab.


$ heroku config | grep MONGOLAB_URI
MONGOLAB_URI => mongodb://heroku_app1234:random_password@dbh00.mongolab.com:27007/heroku_app1234

Now, to the interesting part: how to update your app to work on heroku.

We need to update 2 things.

1) Port. Locally, your node.js server was listening on port 3000. With heroku, it will be a different port every time, and to find out the actual port we will make use of the environment variable that’s available:

process.env.PORT

So at the bottom of your app.js file, switch this:


app.listen(3000);

To this:


var port = process.env.PORT || 3000;
app.listen(port, function() {
console.log("Listening on " + port);
});

2) Database connection string. This is a little bit more involved and I had to comb through multiple resources to actually get this to work.
Instead of using multiple variable to define Mongo (db, server, bson, connection and object) we will use “connect” – a package that simplifies that for us.

So replace the top part of your article.js file, by updating this:


var Db = require('mongodb').Db;
var Connection = require('mongodb').Connection;
var Server = require('mongodb').Server;
var BSON = require('mongodb').BSON;
var ObjectID = require('mongodb').ObjectID;

ArticleProvider = function(host, port) {
this.db= new Db('node-mongo-blog', new Server(host, port, {auto_reconnect: true}, {}));
this.db.open(function(){});
};

To this:


var mongostr = YOUR_MONGOLAB_URI_VALUE
var localstr = "mongodb://localhost/node-mongo-blog";

var connect = require('connect');
var mongo = require('mongodb');
var database = null;

ArticleProvider = function() {

mongo.connect(mongostr, {}, function(error, db){
console.log("connected, db: " + db);

database = db;

database.addListener("error", function(error){
console.log("Error connecting to MongoLab");

});
});
};

Note the separation of database into a global variable. Using “this.db” gave me issues (database was coming up “null” or “undefined”)

Final notes:

– In your package.json file make sure you list new dependencies (mongo and connect):


"dependencies": {
"express": "2.5.8",
"jade": ">= 0.0.1",
"stylus": ">= 0.0.1",
"mongodb": ">= 0.9.6-7",
"connect": "1.8.5"
},

– Heroku has a nice interface where you can manage add-ons, so you can manage your MongoLab db right from there. Just click on My Apps -> [name of the app] -> Add-Ons, and select MongoLab

Find the source code on git here: https://github.com/ntanya/exptest.git