Rackspace Open Sources Whiskey, A Test Framework

Filed in Product & Development by Tomaz Muraus | March 14, 2012 1:30 pm

Giving back to the technical community is very important to Rackspace so we’ve open sourced a lot of the technology behind Cloud Monitoring.  In this series of blog posts, we’d like to introduce you to these different technologies and why you should care about them.  Primarily we’ll focus on the why, what and how you can use them.

For us (and many others), open source is an enabler to building new products (just like the cloud) and we’re part of strong communities and other projects. However, when we get a chance to open source something, we jump at the opportunity to give back.  And to be honest, we get back something in return.  Peer review, feedback and general discussion around the technologies help us make them better.

We’ve invested a lot of time and effort in Node.js and it has enabled so much velocity in such a familiar paradigm.  We have a ton of modules to share with the community.  The first one we’d like to share is around testing and test automation.  So to kick it off, our first piece of technology is the Whiskey test automation stack named after our team’s favorite adult beverage.

When we first started working with Node.js there weren’t many different test frameworks available. There was expresso and some other BDD frameworks and that was pretty much it. Since we didn’t really follow the BDD practices we decided to go with expresso.

It worked fine for basic async and unit tests, but it didn’t really offer any kind of test isolation and process orchestration which we were looking for. Those are just two of the reasons we decided to write our own test framework called Whiskey.

Test Isolation

In Whiskey test isolation is achieved by running each test file (a test file is a collection of tests) in a separate child process. This model is also more scalable compared to a classical model where all the tests run in the same process. Currently child (worker) processes communicate with a parent (reporter) process over a Unix socket, but in the future we could easily change this to TCP and distribute child (worker) processes over multiple servers.

Process Orchestration

In our product we have a lot of integration tests and services that tests depend on. Those services also have a lot of inner-dependencies, e.g. Zookeeper and Cassandra need to be started before the messenger service and so on. At first, when we only had a few services, we used a custom SCons target that started all the services before running the integration tests. Maintaining custom SCons targets soon became too cumbersome and simply starting all the services one by one in a sequential order was relatively slow. That’s why we added support for process orchestration to Whiskey.

Process runner uses a simple JSON configuration file where you specify all the services and their dependencies. When you start Whiskey with –dependencies <config path> option, the config file is parsed and used to start services in the correct order. If it is possible, multiple independent services are also started in parallel, which speeds up the whole bootstrap process.

If you are interested in more details you can also read a post I wrote on my personal blog called Whiskey Node.js test runner now with more goodness – introducing process runner.

[1]

Process runner starting the processes before running our integration tests.

Test Coverage

One of the important features on our list was also a fully integrated code coverage framework. Expresso did offer basic code coverage support, but it only supported outputting the results to the console, which is not useful for larger files and projects. The code coverage module in Whiskey is similar to expresso using node-jscoverage for actually instrumenting the code, but it offers more flexibility and features, including:

●    three different reporters – text, html and json
●    support for hooking up the coverage for services started with the process runner
●    support for aggregating code coverage reports

Aggregated code coverage reports are especially useful if you have multiple test suites in a single code base. In our case we have two types of tests – simple unit tests which don’t rely on external services and more complex integration tests which rely on different services. We use the JSON reporter when we are generating code coverage reports. This way, Whiskey writes out a JSON file with the code coverage statistics before exiting. In our case we end up with multiple different JSON coverage report files: one for simple tests, one for integration tests and one for each node service that has been started by the process runner. The report files are aggregated with a –coverage-aggregate option and we end up with a nice looking HTML report.

[2]

Code coverage HTML report overview page.

[3]

Code coverage HTML report single file view.

Those are just a few of the cool features of Whiskey. Other features include:

●    support for async tests
●    support for tap output
●    support for reporting variables which have leaked into the global scope
●    support for generating Makefiles with common test targets
●    simple integration with node debugger
●    support for reporting tests timing

That’s it for today. Don’t forget to check the Whiskey github page and check back for a future post when Racker Gary will talk about the Node.js Cassandra CQL client we have open sourced.

As much as we love building these technologies, we’d love for people to use and break them. All of these projects are released under the Apache 2.0 license and available under the github[4].[5]com[6]/[7]racker[8] organization. Would you like to work on project like this? Check us out, we’re hiring.

Endnotes:
  1. [Image]: http://c179631.r31.cf0.rackcdn.com/Whiskey_01.png
  2. [Image]: http://c179631.r31.cf0.rackcdn.com/Whiskey_02.png
  3. [Image]: http://c179631.r31.cf0.rackcdn.com/Whiskey_03.png
  4. github: http://github.com/racker
  5. .: http://github.com/racker
  6. com: http://github.com/racker
  7. /: http://github.com/racker
  8. racker: http://github.com/racker

Source URL: http://www.rackspace.com/blog/rackspace-open-sources-whiskey-a-test-framework/