Every truth has two sides; it is as well to look at both, before we commit ourselves to either. –Aesop
Today, I felt compelled to collect and share my thoughts on the topic of cloud building and open source software. I’m currently working on OpenStack related projects for Rackspace Cloud Builders, but I have other experiences from my roles at Puppet Labs and Cloudscaling, which provide perspective that some might find useful.
It turns out building clouds is hard.
If it was easy, everyone would have one. A cloud in every driveway!
The basic idea of an Infrastructure as a Service (IaaS) cloud is simple: put some http APIs in front of a hypervisor. After that, the exercise quickly turns into a bunch of hard networking and storage puzzles; balancing cost, performance and failure characteristics. Accounting for scale, the happy path proof of concept solution quickly becomes a twisting maze of corner cases that ignored all the hard distributed systems problems.
The control plane provided by something like OpenStack is really the tip of the iceberg to deploy a cloud, let alone to operate the infrastructure. OpenStack has made certain architectural choices. Other people have made different choices. Some lessons have to be learned the hard way and there is an inevitable tension between good enough and good at all. There is also an intrinsic tension between flexibility and usability. If an organization isn’t accustomed to building and managing web infrastructure, and is not well versed in practical devops tools and practices, then deploying and operating a significant cloud is going to be overwhelming. That’s true no matter which stack is chosen.
Does that mean I think OpenStack is perfect? No. Do I think the Linux kernel is perfect? No. Can I have an honest discussion about the strengths and weaknesses of different technology choices? Yes. Can I build and operate a production service on OpenStack and Linux? Yes. People are, and more and more they will. I’m not at liberty to disclose all the details, but the last hurdles to opening the Rackspace OpenStack cloud to the public are not issues with OpenStack. If you are going to be an IaaS service provider, there is plenty of work around the edges that aren’t the core IaaS feature set.
Is it a misrepresentation and intellectually dishonest to say there are no major production OpenStack deployments? I’ll let someone else decide.
Another thing that can be difficult to build and nurture is an open source community.
There are patterns of success but there is not a single formulaic way to build community. Communities form around projects due to the dynamics of personalities, sensibilities and needs. Open source creates a whirling mix of people and technology.
There is an ‘Open Source’ spectrum, ranging from consortiums focused on solving shared problems to projects created by the work of a single author scratching an itch. That’s not a bad thing.
Do I agree with everything that has been done in OpenStack? No. And I have raised these issues publicly and privately. On one hand, I think the project has been too inclusive, in the sense that membership came with little or no commitment. On the other hand, the governance has probably been somewhat over engineered and the dialog around issues like the foundation has tended to submarine for periods. There has definitely been a lot for different organizations to absorb as they got caught up in the fastest growing open source project, which has literally captured global imaginations and resources. There is no handbook on how to manage the interests of so many organizations, developers and users.
On balance, you will have a hard time convincing me the credit shouldn’t far outweigh the criticisms.
Some might question why OpenStack needs a foundation, but does throwing a project into the Apache Software Foundation fix all governance issues and build a community? The facts don’t unequivocally support that conclusion. For every successful software project you can point to as a vibrant top level Apache project, the incubator is littered with orphans. The most successful projects tend to be developed by a consortium solving the same problem or an organization solving a problem really well for itself. Projects originally developed as products have not fared well in Apache.
I have to assume there are people at the long list of companies deploying OpenStack that are capable of evaluating technology. I have to assume they surveyed some of the available options. I have deployed and evaluated a number of open source alternatives, and for what I hope to accomplish a distributed message passing service oriented architecture is superior to a single monolithic service. Your use cases and mileage may vary. If there was a clearly superior alternative available, OpenStack would not have the support that it does.
There is ultimately only one business model: solve problems and create value that people want to pay for. Markets are not won or lost on press releases. The days of old style enterprise software sales based on asymmetric access to information are becoming harder and harder to sustain in a world where that information moves without friction. That is the promise of open solutions. Not everyone is comfortable with openness, including vendors and customers. We’re all learning what open means together. It doesn’t mean the same thing to everyone and the meaning is evolving.
Cloud is a nascent market that will be won by those who solve real problems. No project or product should expect to be handed the crown this early in what will be a long race. Sometimes the best thing that can happen in a contest is to be reminded that you are in one.
Three cheers for open source, now let’s get back to work.