Friday, December 9, 2011

My journey to big data utopia in the cloud- Telling it like it is...

Like every other IT engineer who is tired of trying to scale, and is trying to squeeze the last drop of performance from relational data bases,  I set out in search of big data utopia on the Cloud.

Being die-hard open source power users, my colleagues and I naturally gravitated towards the two shining stars in the big data galaxy - Apache Cassandra and Apache Hadoop. With a little digging , we quickly began to realize the gold mine of innovation and engineering that underlie these technologies; but it also became apparent that using these new technologies is a non-trivial exercise with a serious learning curve and many challenges, even for experienced open source power users.

We've also noticed that, as with most new revolutions that come along in the open source world, there is a confusing cloud of hype, and an array of proprietary products that sound, look and smell like open source but really aren't! It becomes increasingly difficult for new users of these products to debunk the hype and to uncover those murky lines where seemingly open source paths can suddenly lead you into closed source, vendor locked territory.
  
This forum hopes to create a hands-on, hype free learning zone for disseminating our own experiences around working with Cassandra and Hadoop on the Cloud, and we welcome other users of these products to chip in with their own comments (good, bad or ugly), lessons learned and collectively help to not only learn more and contribute to this great new revolution, but to also  debunk the myths, the hype and the scams!

No comments:

Post a Comment