Multiple instances of Cassandra on the same server?

Now that we kicked the tires and installed Cassandra, it's time to take her for a drive.

Before going there, I'd like to mention that after my earlier post about the problems/issues I had in installing DataStax’s version of Cassandra on SUSE  - I installed it on a RedHat Linux server using their RPM installer and this time it did install smoothly without any issues and works nicely.

So I will say that the RPM packages works as advertised and, if you are a RedHat user (perhaps Debian too – though I haven’t tried Debian myself) - then the DataStax installers could be a good choice for you - they've done a great job packaging these and have tested things well. We prefer to use SUSE, so it is still a problem for us, and at some point, we'll need to go back and see what we can do to get it to work, or even better someone at DataStax hopefully will!

Now for our first test-drive:

You can setup as many instances of Cassandra as you want on a single server - if you don't have the luxury of multiple servers - though it is pretty cheap to do this on EC2 and I would recommend that as a better alternative to what I am going to describe here.

Anyway, lets assume that like me at the moment, you are stuck with one server. You can still do pretty decent work with Cassandra if you make some tweaks in the internal configurations of each instance. Each of these tweaks can also be easily automated into a shell script. so that when you go grab a new version from Apache (and you’ll be doing a lot of this, I promise) – you just run your script and it will copy the apache distro into multiple folders and tweak the settings below in each folder. For purposes of this post, I am keeping my description below simple– with 3 instances, configured manually.

These are the steps :

1) Copied the apache tar into my Linux server, untarred it and copied the contents the three folders called apache-cassandra-instance-1, apache-cassandra-instance-2 and apache-cassandra-instance-3. Three is good enough for initial testing / playing around with and certainly better than just a single instance which will not really allow you to learn much since most of the interesting stuff in Cassandra happens on a multiple node cluster.

2) Make changes to the conf/cassandra.yaml file in each of the 3 new instance folders. Need to change the following 

                                                    

MAX_HEAP_SIZE="XXXM"'- where XXX can be 256 or 512, etc

   

listen_address and rpc_address are set to loopback addresses 127.0.0.1, 2 and 3 - so each instance  uses different ports but loop back to the same server. It's a kludge but it works.

Change RMI ports to three different ports for example, 8080,1 and 2

To startup Cassandra – bin/cassandra –f  will keep it in foreground. Repeat the same for the 3 instances in three separate windows.

You can now use JConsole to inspect Cassandra (at login prompt just enter the address 127.0.0.1:8081 , 127.0.0.2:8082, 127.0.0.3:8083 to inspect the Cassandra instance –for example to look at the Heap size, etc

There’s one last important step. Cassandra uses  Hashing to divide data across the cluster (ring)  of nodes. Each node has an “Initial Token”. This is the node's logical position in the ring. You need to calculate and set these initial tokens in each node. More about this in the next post.

Open Source - and Commercial?

Open Source does not => Free ride.  

I get that.

Most open source projects that we use in our enterprise are complex, rapidly evolving bundles of software. Downloading them and playing around a little is easy and free, but when you begin to talk about serious development, test and production deployment to support mission critical needs, the dynamics change completely. In these cases, an organization can either go out and hire a large group of really smart engineers who can keep on top of the open source software, or can hire a smaller crew in house and outsource the job of open source support to commercial open source shops, that will often sell support subscriptions at a reasonable cost. (Redhat for Linuxand FuseSource for Apache SOA products are familiar examples)

These companies in turn go out and hire some of the brightest and best open source gurus who designed and wrote the stuff in the first place. Great minds need to put bread on their table too. Or as James Strachan, the brilliant founding member of Apache Camel and ActiveMQ said in a recent conference – “we need money for our beers too, you know”. Of course you do. And nobody should grudge these dedicated open source engineers a dime of what they earn.

That’s not the problem with Commercial Open Source.  So what is the problem?

In “The Failure of Commercial Open Source Software” - Rachael King  of Bloomberg Business Week asserts that open source can be successful only “when it’s supported by a broad community of developers, not just one company trying to extract revenue.”

Bingo!

This has been going on since open source first became reality. As the article points out, the seemingly unending cycle of brilliant open source projects being cornered by commercial companies, with profit as its primary driver, has been repeating with almost every major open source project.

It’s a familiar pattern .  Companies looking for venture capital or profit will jump on the next hot open source project to come along, hire all its committers, make noises for a few months (or years, depending on the size and relative level of integrity) about how committed they are to open source, realize they want to make more money to satisfy their ever hungry venture capitalists or stock holders, and eventually end up killing the open source goose that’s been faithfully laying the golden eggs. Sometimes, it’s the founders and lead engineers of the open source projects that go out and start these companies but it’s the same volition – and has the same end result. The death of the open source project.

Lather, Rinse,  Repeat – for a decade or two. And here we are today, seeing the same pattern in the Hadoop and Cassandra world.

But all hope is not lost. The free market and common sense do amazingly prevail.

In “Battle on: MapR, Cloudera pimp their Hadoop products” -  Derrick Harris points out that while only as recently as 8 months ago,  the mighty Apache Hadoop  platform was monopolized by a single company (CloudEra) – today there are four companies in fierce competition to outdo each other on Hadoop offerings.

For me, the average Joe Open Source Power user - this is really awesome. It’s a healthy trend and gives me hope that Hadoop – the open source essence of it – will continue to flourish and thrive in coming years, no matter what folly one or the other of these commercial companies might end up falling prey to.

It’s only a matter of time that we should begin to see this happening with Cassandra. It appears today that Datastax has a monopoly on the Apache Cassandra project (just Google for Apache Cassandra and you’ll probably find Datastax occupying most of the first 10 search result pages!). If Apache Cassandra is going to survive, thrive and flourish,  which I sure hope will be the case, other capable companies will have to emerge in this space, to help take Cassandra to the next level and to make it a long term viable NOSQL database for all of us. And I’m confident this will be the case.

Let me be clear. As I said at the outset – I have nothing against CloudEra, Datastax or any other commercial vendor trying to re-distribute open source or make money of it. This is in fact, an essential element for the success of open source and we welcome and need your expertise and innovation. And I certainly have nothing but the deepest respect and gratitude for the multitude of committed open source engineers who work long hours to create this amazing products, and who may today work in the ranks of such companies to make an honest living – all the power to you.

All we ask is of commercial companies reselling open source is – don’t try to corner and monopolize any open source project -  Apache Hadoop, Cassandra or other wise. Keep the project  open, healthy and growing! Don’t kill the goose that’s laying the golden eggs – for your sake – and for all of ours!

OK – I’ll get off my rant now J Back to more hands on Cassandra and Hadoop in my next post. 

Up and running. Or not...

It’s time to start kicking Cassandra’s tires.

The first (and obvious) place to download it is from the Apache Cassandra project download page.  So I went there and got the latest stable release of Cassandra (1.0.6, released on 2011-12-14).

I also went out looking for commercially supported versions of Cassandra and after a quick Google, found a product offered by DataStax called DataStax Enterprise.  I got the tar ball version for their latest release (dse-1.0-3-bin.tar.gz)

I installed both versions on our SUSE Linux Dev. Server.

To cut a long story short: The Apache Cassandra version unpackaged, installed and started up cleanly in minutes. The DataStax Enterprise Edition did not.

Here are the steps I followed:

1)      With the Apache Version

# tar -xzvf apache-cassandra-1.0.6-bin.tar.gz

# cd apache-cassandra-1.0.6

# bin/cassandra –f

<Cassandra starts up cleanly with a long console output that looks reasonably happy and ends with

 INFO 22:56:38,688 Listening for thrift clients...>

 2) Now the Datastax Version

#tar -xzvf dse-1.0-3-bin.tar.gz

#cd dse-1.0-3

# bin/dse cassandra -f

< Cassandra tries to start but  crashes with an error -

Error occurred during initialization of VM

Could not reserve enough space for object heap

Could not create the Java virtual machine.>

     Oh Brother – the ol’ Java Heap again.

Doesn’t my server have enough memory ? Well - It does  . ~ 8 GB actually

Just to double check, I moved over to our mighty Cloud Test Server(hosted on Amazon EC2) also running SUSE Linux. and many of our powerful enterprise Java apps (for example, ESB's like Apache Servicemix, ActiveMQ, etc).

Repeated the same install steps.

Same results – Cassandra from Apache starts up clean and nice. The DataStax version crashes.

OK - I can feel a rant coming on - about Commercial Open Source – but let me hold that off till the next post -:)

My journey to big data utopia in the cloud- Telling it like it is...

Like every other IT engineer who is tired of trying to scale, and is trying to squeeze the last drop of performance from relational data bases,  I set out in search of big data utopia on the Cloud.

Being die-hard open source power users, my colleagues and I naturally gravitated towards the two shining stars in the big data galaxy - Apache Cassandra and Apache Hadoop. With a little digging , we quickly began to realize the gold mine of innovation and engineering that underlie these technologies; but it also became apparent that using these new technologies is a non-trivial exercise with a serious learning curve and many challenges, even for experienced open source power users.

We've also noticed that, as with most new revolutions that come along in the open source world, there is a confusing cloud of hype, and an array of proprietary products that sound, look and smell like open source but really aren't! It becomes increasingly difficult for new users of these products to debunk the hype and to uncover those murky lines where seemingly open source paths can suddenly lead you into closed source, vendor locked territory.
  
This forum hopes to create a hands-on, hype free learning zone for disseminating our own experiences around working with Cassandra and Hadoop on the Cloud, and we welcome other users of these products to chip in with their own comments (good, bad or ugly), lessons learned and collectively help to not only learn more and contribute to this great new revolution, but to also  debunk the myths, the hype and the scams!