EDITOR’S NOTE: Ken Hui will be joining John Griffith, OpenStack Program Technical Lead for the Cinder Project and Solutions Architect at SolidFire for a webinar Tuesday, April 29 at 11:00 a.m. CDT to talk about OpenStack Block Storage Design Considerations including an interactive panel discussion at the end. Please join John and I by registering at: https://t.co/CRSEOkM5sD.
In part 1 of this series, I provided a basic overview of the OpenStack block storage service project, called Cinder, and highlighted how it is implemented with commodity hardware as well as third-party storage solutions. In this post, I will review some reference architectures and design principles for building OpenStack Cinder solutions using both commodity-off-the-shelf (COTS) hardware as well as third-party storage solutions. The content is based on experience gathered from various OpenStack-powered Rackspace Private Cloud (RPC) deployments.
To review, there are several use cases for Cinder:
Although all of the above use cases can be seen across various RPC deployments, customers typically design their Cinder environment to support the “compute node crash” and “third-party storage solution” use cases. While COTS hardware or third-party storage solutions can support both use cases, there are design principles that dictate which solutions are a best fit for a particular deployment (more on that later).
Before walking though different design options, let’s talk about how to choose the underlying storage that backs Cinder, which is a discussion that is often overlooked by cloud administrators and users. While a scale-out architect could mitigate some of these issues by dispersing I/O requests across multiple instances, there are many cases when that is not possible either because of the workload type or the infeasibility of having enough compute nodes in a particular RPC deployment to satisfy the requests. In those cases, some attention must be paid to drive selection and configuration. For example, as an RPC architect, I work with customers to understand not only their CPU and RAM requirements, but also their storage workload, bandwidth and capacity requirements so I can recommend the best-fit solution.
Since I will be using the Rackspace Private Cloud as the canonical example for architecting Cinder in OpenStack, let me level-set by stating what is supported in RPC. Currently, we use Dell servers with internal and external drive enclosures as COTS Cinder volume nodes, aka virtual storage arrays; these enclosures can host 7200K, 10K and 15K SAS drives as well as SSD drives, all of which can be configured in RAID 10 or RAID 5. RPC also supports using Cinder with third-party storage solutions from vendors such as EMC, NetApp and SolidFire, with each solution supporting its particular mix of supported drive types and RAID configurations. I talk more about application sizing and showcase a storage IOPS calculator on my personal blog.
Cinder with COTS Servers
The majority of Cinder implementations in RPC, and likely in other OpenStack based products, use COTS servers as the Cinder Volume node/virtual storage array (VSA). You have the option of installing all Cinder services on a single node or dispersing them across multiple nodes. By default, RPC places the cinder-api and cinder-scheduler services on the controller nodes and the cinder-volume services on the nodes that will function as the virtual storage arrays, serving out Cinder volumes. Rackspace recommends the following configuration for a Cinder volume node/virtual storage array:
The diagram above was simplified to focus on just the Cinder components (I’ve simplified the networking and the controller nodes above do not show all the services that would normally be running on them in an RPC deployment). Some design considerations to think about when deploying this solution:
In conversations with folks in the Cinder project community, including those on the Cinder technical team, there have been several alternative solutions proposed:
Cinder with Third-Party Storage Back-Ends
For customers who required a third-party storage solution, RPC currently supports solutions from EMC, NetApp and SolidFire, with more to be offered in the future. There are a number of reasons customers may choose these solutions over a COTS approach:
When integrating with third-party storage back-ends, RPC places the cinder-volume service on the controller nodes, along with the cinder-api and cinder-scheduler services. As described in part 1, instead of interfacing with LVM, the cinder-volume service interfaces with the volume driver for the specific third-party back-end to export an iSCSI volume to cloud instances.
Some design considerations to think about when deploying this solution:
A third category of Cinder storage back-ends is software only solutions, such as Ceph and GlusterFS, that can utilize COTS hardware while providing more robust enterprise storage features. These options are not currently supported in RPC but we are actively evaluating them and customers should see of them offered as an option in the near future.
For readers who want to learn more about Cinder with OpenStack, please consult the Rackspace Knowledge Center article on “Configuring OpenStack Block Storage” and the “Block Storage” section of the OpenStack Configuration Reference Guide.
I recently joined SolidFire Cloud Solutions Architect Aaron Delp (@aarondelp) to talk with SolidFire’s John Griffith (@jdg_8) about the role of an OpenStack Program Technical Lead, the evolution of Storage from Icehouse to Juno, Cinder architecture and where block storage fits into OpenStack deployments. You can catch that replay on the Cloudcast at: http://t.co/LRwe1Fxo98.