With the Essex cycle complete, feature lockdowns and release candidates behind us, I wanted to let the community know some of the things the Rackspace Cloud Builders team did to prepare for the Essex release of OpenStack.
There is a lot going on, and I hope to cover a few of our key focus areas over coming months, including a new test suite we developed and made freely available for other cloud operations and engineering teams facing our same challenges.
Lately our engineering team has focused a lot on making sure the command line utilities in the OpenStack ecosystem are working as expected. Ensuring that cloud operators can rely on these tools to function properly is extremely important to the adoption of OpenStack. To that point we needed a way to validate that all of the command line tools are functioning properly on a cloud infrastructure we are supporting. As you can imagine, our customer use cases vary wildly. At any given time, we are supporting a customer who manages their cloud exclusively through the dashboard, while also supporting a different customer who manages their cloud exclusively through the EC2 compatibility command line tools. Ensuring that all of these tool sets function properly, and that any bugs in their operation are being fixed, is a major focus for our Rackspace Cloud Builders engineering team.
Since the middle of the Essex development cycle all commits to the Core OpenStack projects (Nova, Glance, Keystone, Swift and Horizon) are tested by an integration tool (Devstack https://github.com/openstack-dev/devstack) that exercises a set of each project’s command line functionality. While Devstack is able to validate functionality for the latest state of the upstream code bases, we need to validate as much of the command line functionality as possible across multiple states of the OpenStack code base (i.e. Diablo/Stable, Essex, etc). We realized that our needs were different than those of the developer community, which prompted us to begin developing a test suite called Exerstack (https://github.com/rcbops/exerstack ). Exerstack enables us to validate the functionality of existing deployments as well as the current development version of OpenStack. With this tool we have the ability to write simple tests in bash that can validate the command line tool sets from the OpenStack Core projects. Individual tests in the suite can be skipped if they are not supported on a specific version of code (e.g. if it is older than Essex). Alternatively, tests can be skipped if a feature does not work on a given version of an operating system (e.g. Nova suspend/Nova resume on Ubuntu-10.10 and Ubuntu-11.04). Here is a screenshot of the nova CLI tests running on Ubuntu-12.04 using the Essex packages provided by Canonical:
In order to tie all of our testing work together, we make heavy use of the Jenkins continuous integration platform. This enables us to define jobs that can be run automatically given a set of triggers. For example, every time our deployment recipes are updated Jenkins will automatically spin up an Ubuntu-12.04 server, deploy an all-in-one OpenStack system, and automatically run Exerstack against the resulting server. This provides immediate and recurring feedback against the state of our deployment recipes, the state of OpenStack packages in Ubuntu-12.04 and the state of our testing.
Our end goal for all of this testing is to provide confidence to our customers that their OpenStack deployments are working as expected. We hope others in the community can find value in this tool as well, whether they are deploying lots of different clouds or adding features to their cloud deployments. If there are features you would like to see added to Exerstack, I would like to hear from you, and as always patches are happily accepted.