Just got back home from a meetup hosted at AppNexus, by AppNexus.

The topic of today’s talk was scaling the complex ad serving and bidding system that supports billions of ad impressions and stores terabytes of data. I have to admit, it was one of the most technical meetup presentations that I’ve attended, and it was excellent!

Mike, the CTO of AppNexus, is a fast-talker, who started with the story of how the company was born, and covered a lot of info from building the company up and adding functionality (ad bidding system was added when ad-selling business demanded it), to hard drive performance, to build vs buy vs outsource topic.

I especially liked the overview of data warehouse and crunching tools: Netezza, Vertica and Hadoop, and how changing requirements dictated choice of tools. Netezza worked great as a single instance, but did not scale with clustering, Hadoop was very hard to learn and configure from scratch, but meets most of the needs now and will suffice for the next 2 years.

There was a good question from the audience about ideas for startups: what are the technological painpoints and gaps that need to be filled. Interestingly enough, Mike named monitoring as one of the areas that’s lacking a great tool. Someone next to me mentioned New Relic, but I was wondering if that’s enough to monitor thousands of servers.

So for me, this talk was full of information on areas that I had little knowledge in, and it’s always great to see who develops breakthrough solutions in technology and how they solve problems (big data problems are very very interesting).

Another bit that got me wondering, since I’m into MongoDB lately, was when to use Mongo vs Hadoop. And sure enough, I found a really good deck from 10gen with not only the answer, but also great practical demos. Yay for the internet and sharp minds in NY tech community! I feel proud to live in this huge tech hub, and humbled because there are just so many things yet to learn.