Rackspace supports open-source software development in several ways, the most obvious being OpenStack. Rackspace believes that software should be developed in the open when it makes sense. This post describes some of the work we have been doing within the Node.js and Cassandra communities.

Rackspace has a long history of working within the Cassandra community. Rackspace engineers participated in crucial stages of its development by moving to get Cassandra into the Apache Incubator and were instrumental in further developing it into a top-level Apache project. Cassandra is used today by dozens of companies and is one of the leaders in the “NoSQL” space.

The release of Cassandra Query Language (CQL), initially developed by a Rackspace engineer, coincided with the release of Cassandra 0.8. Rackspace subsequently supported substantial development for the Java, Python and Node.js CQL drivers.

CQL aims to be an easier way to read and write to Cassandra databases. CQL was designed to closely resemble SQL, easing the burden of developers being exposed to Cassandra for the first time.

When Rackspace started developing Cloud Monitoring, we decided to use Node.js for our API service endpoints. These services would need to communicate with a Cassandra storage cluster. We made the decision to use CQL even though it was still under active development and no Node.js driver existed for it at the time. We figured it would be easier to implement the driver than to use the existing Cassandra Thrift client. Also, we knew that we would be able to contribute the driver back to the project so that other Node.js users would benefit.

The main driver repository is on Github. A mirror is also maintained at Apache Extras. Currently CQL 2 is supported, but we plan to support CQL 3 in the future.

node-cassandra-client Examples

The rest of this post is intended as a node-cassandra-client primer of sorts. It contains examples of how to connect, update and select results from your Cassandra database in Javascript using node-cassandra-client.

First we need to create a keyspace. I’m going to use cqlsh, which ships with Cassandra, to do that. I’ve got Cassandra set up to listen to port 19170.

$ cqlsh localhost 19170
cqlsh>

After that, you can paste the following CQL into cqlsh to set up keyspaces and column families.

CREATE KEYSPACE EXAMPLE_KS WITH strategy_class = SimpleStrategy AND \
strategy_options:replication_factor=1;
USE EXAMPLE_KS;

This column family is a simple column family where keys and column names are strings, but column values are expected to be integers:

CREATE COLUMNFAMILY simple_cf (
    KEY ascii PRIMARY KEY
) WITH comparator=text AND default_validation=varint;

Cassandra supports more complex types too. This column family has string keys, UUID column names and binary column values:

CREATE COLUMNFAMILY complex_cf (
   KEY ascii PRIMARY KEY
) WITH comparator=uuid AND default_validation=blob;

Of course, you are free to specify every column as well. This column family specifies string keys with three columns that are obviously typed.

CREATE COLUMNFAMILY very_complex_cf (
    KEY ascii PRIMARY KEY,
    string_col text,
    uuid_col uuid,
    int_col varint
);

The remaining examples all use the very_complex_cf column family.

It is easy to connect to a Cassandra cluster using node-cassandra-client. You simply create an options hash, pass it to the Connection constructor and call connect().

var Connection = require('cassandra-client').Connection; 
function doSimpleConnect(callback) {
 // these are the connection parameters you need to connect to Cassandra.
 var connectionOptions = {
   host: '127.0.0.1',
   port: 19170,
   keyspace: 'EXAMPLE_KS',
   use_bigints: false
 };

 var con = new Connection(connectionOptions);
 con.connect(function(err) {
   // if err != null, something bad happened. 
   // else, assume all is good.  your connection is ready to use.
   if (!err) {
     // close the connection and return to caller.
     con.close(callback);
   } else {
     // no need to close, just return to caller.
     callback(err);
   }
 });
}

NOTE: use_bigints is a flag that tells the driver to wrap numerical results in wrapper objects. This enables you to get around numerical limitations in Javascript.

node-cassandra-client also supports connection pools. After establishing a connection pool, the query API is identical to regular connections, so you will not have to worry about whether you are using one or the other. 

var PooledConnection = require('cassandra-client').PooledConnection; 
function doPoolConnect(callback) {
 var conOptions = { hosts: ['127.0.0.1:19170'],
                    keyspace: 'EXAMPLE_KS',
                    use_bigints: false };
 var con = new PooledConnection(conOptions);
 con.shutdown(function() {
   // no error object by design.
   callback();
 });
}

While the query API between Connection and PooledConnection is identical, there are a few differences in construction and tear-down that should be noted:

1) The constructor hash for Connection accepts a single string for host, while in PooledConnection it expects a list of host:port strings.
2) Terminating a Connection is done via Connection.close(), while terminating a PooledConnection is done via PooledConnection.shutdown(). Each method accepts a callback.
3) PooledConnections have no connect() method. You are free to start executing queries as soon as it is constructed.

Everything else is done using the execute() method of either a Connection or a PooledConnection instance. It takes three parameters:

1) a parameterized CQL statement,
2) an array of the parameters that are appropriately typed,
3) a callback accepting (err) for UPDATEs, or (err, rows) for SELECTs.

Let’s insert some data into very_complex_cf. Some of these examples use the node-async library to avoide deeply nested callbacks.

function doComplexInsert(callback) {
 var conOptions = { hosts: ['127.0.0.1:19170'],
                    keyspace: 'EXAMPLE_KS',
                    use_bigints: false };
 var con = new PooledConnection(conOptions);
 var cql = 'UPDATE very_complex_cf SET ?=?, ?=?, ?=? where KEY=?';
 var params = ['string_col', 'string_value',
               'uuid_col', '6f8483b0-65e0-11e0-0000-fe8ebeead9fe',
               'int_col', 42,
               'complex_insert_row'];
 con.execute(cql, params, function(err) {
   // demonstrates use of a callback.  A simplification would have been:
   // con.execute(cql, params, callback);
   if (err) {
     console.log(err);
   }
   con.shutdown(callback);
 });
}

You can even batch multiple updates in a single statement:

function doComplexBatchInsert(callback) { 

 var conOptions = { hosts: ['127.0.0.1:19170'],
                    keyspace: 'EXAMPLE_KS',
                    use_bigints: false }; 
 var con = new PooledConnection(conOptions);
 con.execute('BEGIN BATCH USING CONSISTENCY ONE \
              UPDATE very_complex_cf SET ?=?, ?=?, ?=? where KEY=?; \
              UPDATE very_complex_cf SET ?=?, ?=?, ?=? where KEY=?; \
              UPDATE very_complex_cf SET ?=?, ?=?, ?=? where KEY=?; \
              APPLY BATCH;',
              [ // first row
                'string_col', 'value for string col in row_1',
                'uuid_col', '6f8483b0-65e0-11e0-0000-fe8ebeead9ff',
                'int_col', 25,
                'complex_batch_row_1',      
                 // second row
                'string_col', 'value for string col in row_1',
                'uuid_col', '6f8483b0-65e0-11e0-0000-fe8ebeeada00',
                'int_col', 26,
                'complex_batch_row_4',

                // third row
                'string_col', 'value for string col in row_1',
                'uuid_col', '6f8483b0-65e0-11e0-0000-fe8ebeeada02',
                'int_col', 27,
                'complex_batch_row_3'
              ],
              function(err) {
                if (err) {
                  console.log(err);
                }
                con.shutdown(callback);
             }
  );
}

A simple SELECT statement works like this:

function doSelectAll(callback) {
 var conOptions = { hosts: ['127.0.0.1:19170'],
                    keyspace: 'EXAMPLE_KS',
                    use_bigints: false };
 var con = new PooledConnection(conOptions);
 con.execute('SELECT * from very_complex_cf', [], function(err, rows) {
   if (!err) {
     console.log(rows.length); // should be 4 at this point.
     rows.forEach(function(row) {
       console.log(row.key);

       // access column names and values by index.
       for (var i = 0; i < row.colCount(); i++) {
         console.log(row.cols[i].name);
         console.log(row.cols[i].value);
       }

       // access column values by hash.
       console.log(row.colHash['string_col']); // it's a string        
       // an instance of require('cassandra-client').UUID
       console.log(row.colHash['uuid_col']);
       console.log(row.colHash['int_col']); // it's a number.
     });
   }
   con.shutdown(callback);
 });
}

The CQL could have been altered to return a single row or individual columns:

function doSelectiveSelect(callback) {
 var conOptions = { hosts: ['127.0.0.1:19170'],
                    keyspace: 'EXAMPLE_KS',
                    use_bigints: false };
 var con = new PooledConnection(conOptions);
 con.execute(
   'SELECT uuid_col FROM very_complex_cf WHERE KEY=?',
   ['complex_batch_row_1'],
   function(err, rows) {
     if (!err) {
       // rows.length should == 1.
       var row = rows[0];
       console.log(row.key);
       console.log(row.colCount()); // should be 1 also.
       // getting the values out is an exercise for the reader.
     }
     con.shutdown(callback);
 });
}

node-cassandra-client also has support for UUID and binary column types. Using the complex_cf columnfamily:

function doComplexColumnNames(callback) {
 var conOptions = { hosts: ['127.0.0.1:19170'],
                          keyspace: 'EXAMPLE_KS',
                          use_bigints: false };
 var con = new PooledConnection(conOptions);
 var updateArgs = ['6f8483b0-65e0-11e0-0000-fe8ebeead9ff',
 new Buffer([1,2,3,4,5]),
                   'uuid_row_1' ];
 async.series([
   function insert(callback) {
     con.execute('UPDATE complex_cf set ?=? where KEY=?', updateArgs, callback);
   },
   function select(callback) {
     con.execute('SELECT * from complex_cf WHERE KEY=?', ['uuid_row_1'], 
     function(err, rows) {
       if (!err) {
         // rows.length should == 1.

         // first access the col by index (when you know it).
         // col name should be an instance of UUID.
         console.log(rows[0].cols[0].name);
         // col value should be <Buffer 01 02 03 04 05>
         console.log(rows[0].cols[0].value);

         // if you don't know the col index, you need to use the col hash.
         // of course, this assumes you know the column name.
         // in this case, it should point to the same buffer already referenced.
         console.log(rows[0].colHash['6f8483b0-65e0-11e0-0000-fe8ebeead9ff']);
       }
       callback(err);
     });
   }
 ],
 function(err) {
   if (err) {
     console.log(err);
   }
   con.shutdown(callback);
 });
}

As demonstrated, when working with results from a SELECT query, you can access columns by index or by column name. If you access columns by name, you need to use the string version of the column name.

NOTE: All the code used in these examples can be found in this gist.

Conclusion

Rackspace understands open-source development. Development on node-cassandra-client is ongoing and patches are always welcome. In fact, we regularly push new releases to the npm registry.

If you think this kind of work would be challenging and fun, and enjoy contributing to open-source projects, we are always looking for talented engineers.

The Apache Cassandra team is in the process of rolling up a release candidate for Cassandra 0.7.  That makes this a good time to highlight some of the new features and changes that will be in this version.

Big Changes

Memory Efficient Compactions (CASSANDRA-16)

Compaction is the process by which Cassandra iterates over its data, removing deleted data forever and combining multiple data files into a single data file.  This results in faster read operations.  One limitation of the implementation was that if a row of data couldn’t fit in memory, then the data file it belonged to could never be compacted.

Since one of the main features of Cassandra is its use of sparse columns, users expect to be able to add columns without penalty until disk runs out.  Cassandra 0.7 solves this by making compactions happen incrementally.

Online Schema Changes (CASSANDRA-44)

Prior to Cassandra 0.7 adding and removing column families and keyspaces required you to first distribute an updated configuration file to each of your nodes and then execute a rolling restart of your cluster.  That was not too bad, but it was manual and required human intervention, so it was possible to make mistakes.

Cassandra 0.7 solves this problem by exposing the ability to create and drop column families and keyspaces from its client API.  Using the same methods there is limited support for updating existing column families and keyspaces (e.g., increasing the replication factor for a particular keyspace).

Additionally, we’ve made it easy to migrate your keyspace definitions from your 0.6 storage-conf.xml configuration file using bin/schematool.  In the future we plan to add the ability to rename column families and keyspaces.  A wiki article contains more detailed information.

Secondary Indexes (CASSANDRA-749)

Cassandra has always been an excellent platform for storing data.  However, it has lacked an expressive, efficient way to query data by any property other than the primary key (you need the answer to “what keys match these particular columns and values?”).  To cope with this, many application developers ended up creating their own secondary indexes that were basically inverted indexes of already existing column families.  This solution worked well, but was tedious to maintain and involved a lot of duplicated effort.

The need for this feature has weighed heavily on our minds for a while.  Work even began on it prior to the release of Cassandra 0.6.  One design decision that required some wrangling over was whether secondary indexes should use node-local or distributed storage.  Each approach had distinct advantages and disadvantages and served different use-cases.  We ended up using node-local storage (the secondary indexes are wholly contained on the same node whose key they index). Distributed secondary indexes may eventually make their way into Cassandra, but not in 0.7.

Using the API for online schema changes, secondary indexes can be added to existing or brand new column families.  An index can be queried on the client side using the get_indexed_slices() API method.

Time to Upgrade?

For these reasons, Cassandra 0.7 is a compelling upgrade from earlier versions.  But wait!  The Cassandra team has been busy fixing bugs and adding more features.  You can get the full details from the changelog, but I’ll highlight a few of them here.

Streaming fixes

The streaming code has been simplified to offer better tracking when files get streamed.  Additionally, the requirement for anti-compaction, the time-consuming process of splitting a data file into multiple parts, has been removed.   Streaming is faster and more predictable now.

Read Performance

Several things have happened to make reads faster overall.  First, the read path has been optimized to take further advantage of the row cache introduced in 0.6 (CASSANDRA-1267 and 1302).  Second, row keys have been switched from strings to byte arrays, picking up some speed by eliminating the overhead required for java string construction.   Third, usage of byte[] has been replaced with java.nio.ByteBuffer. This allowed us to reduce the amount of byte array copying on the read path and eliminated a significant amount of garbage generation (bonus!), easing the pain still felt by stop-the-world garbage collection.

Thrift

We’ve upgraded our Thrift version from 0.2 to 0.5.  This comes with significant performance improvements.  Thrift framed transport is now the default, in contrast to non-framed which was default in 0.6 and earlier.

API

As with every release of Cassandra so far, our client API has fluctuated quite a bit—something we are not necessarily proud of.  A mind-shift is occurring though.  Most Cassandra experts, including myself, have begun recommending that users access Cassandra through one of the available third party libraries (e.g.: Hector, Pelops, Pycassa) instead of directly via Thrift.  The main benefit of this approach is that users will be immune from the sort of API changes that happen from version to version so long as client developers keep up.

There are several things worth noting about the updated API.

1. The keyspace argument has been dropped from nearly every method and has been replaced by a set_keyspace() method that sets the keyspace for the current connection session.
2. Expiring (TTL) columns have been introduced.  This gives you the ability to set the time-to-live for columns so that they will automatically be removed by compaction without any intervention on your behalf.
3. ConsistencyLevel.ZERO is removed.

Configuration
storage-conf.xml has been replaced with cassandra.yaml. The intent is to provide a format that is more human-readable.  We have provided a converter that will allow you to convert your XML configuration into YAML prior to upgrading.  Also, more settings are configurable than ever before.

Other Changes

1. We have continued the trend of exposing more operations and metrics via JMX, including compactions, sstable accesses, endpoint states (gossiper) and TCP messages.
2. The authentication framework and built-in Hadoop support have both received a lot of attention.
3. Introduction of native code (JNA) for snapshots and help with memory swapping.
4. Enhanced CLI experience

Conclusion

Cassandra 0.7 offers significant advantages in terms of features and speed than in earlier releases.  We’ve done our best to make the upgrade process simple: old data files are still compatible, but your cluster must be brought down entirely.

As always, if you would like to learn more about Cassandra, good places to start are the website, wiki and #cassandra on IRC (freenode).  Apache also hosts several mailing lists that are excellent resources.  Additionally, NEWS.txt is a good place to go to find out more about some of the high-level changes in Cassandra.