Powering Research (And Cookie Baking Robots) With OpenStack At MIT

Filed in Cloud Industry Insights by Jonathan Proulx | November 5, 2013 10:20 pm

At OpenStack Summit Hong Kong[1], Jonathan Proulx of MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) highlighted how the lab is using OpenStack to power dozens of research projects.

The Massachusetts Institute of Technology (MIT[2]) Computer Science and Artificial Intelligence Laboratory (CSAIL[3]) is the largest research laboratory at MIT and one of the world’s most important centers of IT research. Our lab and its researchers have been instrumental in developments like time-sharing, massively parallel computers, public key encryption, mass commercialization of robots and many of the underlying technologies of the ARPANet and the web.

Learn more about how MIT’s CSAIL uses OpenStack in this video:

It’s sophisticated research – and we need tomorrow’s infrastructure today to make it happen.

In the summer of 2011, we started our journey to the cloud. It was donated infrastructure. The hardware was great, but there was something missing in the software. It was a prerelease turn-key system which meant we didn’t need to do anything to make it go, but it also meant we couldn’t update the system ourselves. All of the vendor resources were going into a major rewrite for release (which turned out to be OpenStack-based, but that’s someone else’s blog post to write). In the meantime, our version fell behind, and at the leading edge of research we can’t wait for others

We needed rapid deployment and the ability to adapt as we go. At the same time, we had a research project that had a strict deadline and we needed a cloud for it – we had to get it up and running quickly.

We looked to OpenStack and deployed it on to the hardware to get the newer Linux kernel we needed. It took about two weeks to go from decision to the research groups getting science done. We did this using our existing tools FAI[4] (a PXE based installer) to get Ubuntu on the bare metal and Puppet[5] (using community maintained modules) to deploy the OpenStack components and configuration on top of that.

We deployed the Essex release of OpenStack in July 2012 and quickly upgraded to Folsom in September of that year. In our private OpenStack cloud, we run 768 physical cores (1,536 vCPUs), 64 nodes and, as of earlier this year, we run at 90 percent utilization with 1:1 virtual to physical resource scheduling.

In August, we updated to Grizzly and we’re looking at deploying Havana soon. With the Grizzly upgrade we moved to Neutron networking to provide direct access to existing data center VLANs and allow users to create and manage their own GRE-based private networks. We also began providing different scheduling zones using host aggregates so some parts of our cloud are still scheduled 1:1 for our high intensity compute needs, but other areas are over committed allowing more virtual resources for more traditional cloud workloads such as web applications, continuous integration testing and development systems. Our Havana plans include adding orchestration through Heat and metering through Ceilometer, some of the newer cool features within OpenStack.

The reaction to OpenStack from our research teams was incredibly positive. Users immediately figured out how to use it and hit the ground running with some groundbreaking research projects.

Currently, we have 114 users running 38 projects on our OpenStack cloud, both of these numbers are growing almost every day. We’ve seen a total of 468,866 instances over the life of our OpenStack cloud, and more than 6.3 million vCPU hours.

Our OpenStack cloud currently powers research for host of key research initiatives, including:

These are just a handful of the 38 exciting projects currently under way on our OpenStack deployment.

And we’re not stopping there. We have a number of future initiatives planned, and other ideas that haven’t quite made it all the way to “plan” yet. We’re looking at moving our Internet-facing applications to OpenStack; that’s a huge move and a testament to OpenStack’s current stability. A number of researchers have expressed interest in getting “under the cloud” to do research on the cloud systems themselves. We’re also examining the idea of “one cloud per student,” which would allocate a certain number of VMs for students and researchers to fool around with without being tied to a specific project or goal. Another idea is “cloud desktops,” or personal VMs in the cloud to which lab members can connect to from any device.

Those are just some of the amazing possibilities OpenStack creates in the research community. We’re excited to be a part of it and look forward to what we’ll be able to do with Havana and future releases.

I’ve been with CSAIL in various incarnations since 2000 and currently hold the title of Senior Technical Architect in The Infrastructure Group (TIG[12]). The opinions expressed here though formed in that context are my own, not necessarily those of my employer.

Check out more videos from Rackspace Training for OpenStack’s visit to MIT[13].

Endnotes:
  1. OpenStack Summit Hong Kong: http://www.openstack.org/summit/hk/
  2. MIT: http://web.mit.edu/
  3. CSAIL: http://www.csail.mit.edu/
  4. FAI: http://fai-project.org/
  5. Puppet: http://www.puppetlabs.com/
  6. ALFA: http://groups.csail.mit.edu/EVO-DesignOpt/groupWebSite/
  7. TREC-KBA: http://trec-kba.org/
  8. Daniel Sanchez’s: http://people.csail.mit.edu/sanchez
  9. LIS: http://lis.csail.mit.edu/
  10. Julia: http://julialang.org/
  11. IJulia: https://github.com/JuliaLang/IJulia.jl
  12. TIG: http://tig.csail.mit.edu/
  13. Rackspace Training for OpenStack’s visit to MIT: http://www.rackspace.com/blog/tag/mit/

Source URL: http://www.rackspace.com/blog/powering-research-and-cookie-baking-robots-with-openstack-at-mit/