Repose Within Rackspace

To understand Repose better, let’s examine how we use Repose here at Rackspace. One of our internal applications of Repose has been to simplify development and use of Atom Hopper, another of our open-sourced products that I’ve blogged about before. We use Atom Hopper as a key tool to collect data from our deployment of OpenStack. We have several internal teams that want to access this data as well as to publish additional data into Atom Hopper. Naturally, all of this requires Atom Hopper to perform basic functions such as authentication, rate limiting and logging. This is where Repose steps in. Repose already has components that can do this for Atom Hopper, or for any other web based service for that matter. This lets the Atom Hopper team focus on just the core pieces of Atom Hopper while Repose deals with general-purpose functions like authentication and rate limiting. This relationship with Repose works for other web services developed by teams within Rackspace: those developers focus on  their own web service and let Repose do the other tasks that almost every web service needs. It’s no longer necessary for every development team to re-invent the wheel for client authentication, rate limiting and such functions.

In the same way, you can use Repose to help you focus on your own web-based service. You can use Repose no matter what programming language you used to develop your web service.

Benefits And How Repose Works

Repose’s design gives it several benefits. Repose is:

• Scalable. Repose is incredibly scalable because it is designed to be stateless, allowing state to be distributed across the web.
• Flexible. Repose can be run as an executable JAR, deployed as a WAR file in any Servlet container or deployed on a separate server. Repose’s configuration allows a user to define which components to use and details for each component.
• Extensible. Repose’s functionality is delivered as separately-configurable components. New components can easily be added to Repose. You can see an ever-growing list of Repose components in our wiki.
• High performance. Repose can handle high loads with high accuracy.
• Configurable. Repose’s components can be configured easily and any changes to the configuration are automatically re-loaded.

Integration patterns are technology-independent designs that are robust and re-usable. Repose is a platform that can be used to implement integration patterns. In particular, Repose uses these two patterns:

• Pipes and Filters: Repose implements the pipes and filters pattern by providing a way of constructing a route by creating a chain of filters. Requests are passed through the chain so that one filter’s output becomes the next filter’s input. This pipeline structure is analogous to the UNIX pipe command. The advantage of the pipeline approach is that it enables you to combine services, some of which can be external to the application, to create more complex forms of message processing.
• Message Router: Repose implements the message router pattern as a filter that consumes messages from a single consumer endpoint and then redirects them to the appropriate target endpoint, based on a specific decision criterion. A message router is concerned only with redirecting messages; it does not modify the message content. Repose acts as an intermediate proxy and routes accordingly. Any component can make Repose route.

One year ago we launched Repose as open source software.  We’ve been very happy with Repose in production and welcome your feedback.

Getting Started With Repose

To learn more about Repose, the Open Repose website is your starting point providing links to the source code in GitHub. This is also the right place to find our documentation, including an FAQ and wiki; the wiki has the most current information. You might also be interested in a walkthrough on setting up Repose to use rate limiting over on our DevOps blog.

What if you could create something as complex as this, with data in multiple formats from multiple origins stored across multiple servers but aggregated for multiple consumers, who could then repackage it for consumers of their own?

If you couldn’t give your client a copy of all your data, you could ask the client to describe the specific data that’s needed and then assemble those items the client needs. However, if you had many clients, each with their own mountains of data, would you have to create a direct path from every consumer to every fragment of data they need?

What you need is to easily create a bridge, integrating any number of data origins with any number of data consumers. Enter in Atom Nuke.

With Atom Nuke, no matter where your data originates and who consumes the data, it could be this simple to think about.

Atom Nuke Simplifies Integration

We created Atom Nuke to give ourselves two kinds of power related to the high volumes of data produced by our Atom feeds.

• fission, making it easy to divide data in new ways
• fusion, making it easy to combine data in new ways

A six-way integration requires eighteen paths, connecting three data origins with three data consumers so each has direct and equal access. Adding one new origin or consumer requires adding many new paths.

Atom Nuke is an open-source collection of utilities built on a simple, fast Atom implementation that aims for a footprint of minimal dependency. The Atom implementation has its own model and utilizes a SAX parser and a StAX writer.

• SAX (Simple API for XML) makes it simple to read existing data
• StAX (Streaming API for XML) makes it simple to stream data to and from applications

With Atom Nuke providing a bridge, a six-way integration requires six paths, one from each of the three origins and three clients, with each path terminating at Atom Nuke. Adding one new origin or consumer requires adding one new path.

We designed our Nuke implementation for immutability, maximum simplicity and memory efficiency. Nuke also contains a polling event framework that can poll multiple sources. Each source may be registered with a configured polling interval that governs how often the source is polled during normal operation. That source may have any number of Atom listeners added to its dispatch list. These listeners will begin receiving events on the next scheduled poll.

Atom as a Building Block

Atom is a self-discoverable and generic syndication protocol. The Internet Engineering Task Force (IETF) describes Atom in several ratified Requests for Comments (RFCs):

• the Atom RFC
• the Atom Paging and Archiving RFC
• the Atom Publishing Protocol RFC

The unique properties of the Atom specification have made it popular as a protocol for generic event distribution, syndication and aggregation. Using Atom as a common interchange format, event publishers add their domain-specific events to an Atom publication endpoint. Downstream, subscribers are notified of events they’ve pre-identified as relevant, controlling what they consume from potentially-vast collections of published data.

Atom Nuke Within Rackspace

Within Rackspace, the Cloud Integration team builds tools for all our software development teams to use. We need to provide high-quality tools but we also need them to be easy to use and work smoothly together so that we can encourage adoption throughout Rackspace.

Using Atom Nuke, we collect data from the Atom feeds supplied by Atom Hopper, another of our open-source tools. We then take that Atom data and feed it into several systems, including those that perform analytics on OpenStack  deployments throughout our data centers. The analytics engine uses Nuke to collect the entire Atom feed data so it can be marshalled into a Hadoop cluster. By combining our Atom Nuke and Atom Hopper tools, we’ve enabled complete portability of data: we can combine Atom events with data from other sources such as Rabbit MQ messages and Flume logs without requiring consumers of that data to deal with the complexities of interacting with those dissimilar sources.

Nuke Makes Working with Atom Easy

Atom Nuke excels as a an Atom feed crawler, since you can poll multiple feeds from multiple endpoints as well as define the polling intervals down to milliseconds. In addition, you can select events in response to specific triggers, such as when a specific Atom entry contains a subscribed category. However, Nuke is much more than a feed crawler, it can create its own Atom feeds if needed.

We built Atom Nuke with Java but we recently extended support to Python. Nuke is licensed under the Apache 2 license and was created by John Hopper, a software engineer on the Rackspace Cloud Integration team. We’ve created some tutorials to get developers started with Nuke.

Building with Boxes, Not Bricks

Writing about a different kind of atom in a world that was just beginning to understand atomic structure and atomic energy, H.G. Wells (1866-1946) imagined a future in which using the power stored within atoms transformed many aspects of human life:

“I feel that we are but beginning the list. And we know now that the atom, that once we thought hard and impenetrable, and indivisible and final and–lifeless–lifeless, is really a reservoir of immense energy. That is the most wonderful thing about all this work. A little while ago we though of the atoms as we thought of bricks, as solid building material, as substantial matter, as unit masses of lifeless stuff, and behold! these bricks are boxes, treasure boxes, boxes full of the intensest force.”

—H.G. Wells, The World Set Free, 1914

We’re now at a similar point with the technology of our time. We have explored enabling technologies, such as Atom, and have begun fully using and building upon their capabilities, putting them to work in new ways to make new things possible. As we begin building with Atom Nuke, we’re using Atom not as a brick, but as a treasure box, containing amazing possibilities for fission and fusion, dividing and combining data to make new applications possible. By making Atom Nuke and some of our other projects such as Atom Hopper available as open source, we hope we are also creating treasure boxes filled with ideas and possibilities.

To learn more about Atom Nuke, visit our project site and check out the source code on GitHub.

Atom Hopper Within Rackspace

Within Rackspace, the Cloud Integration team builds tools for all our software development teams to use. We created Atom Hopper to collect and disperse information that is of interest to multiple audiences, delivering consistent information to everyone who is listening. For example, activity generated by the OpenStack Nova project is important to that team, but just as important to the teams responsible for Usage and Billing.

We use Atom Hopper to capture messages once and store them in a database; teams that want to read the messages can parse the data via an Atom feed according to their own schedules. Consumers can use whatever technology they wish to get the Atom feed produced by Atom Hopper. They can use Atom Hopper to select a specific entry within the feed or to page through the feed from newest entry to oldest or from oldest to newest—whatever suits their needs.

Atom Hopper 1.0

Atom Hopper 1.0 is available now as open source under the Apache License version 2.0, and we invite you to examine and contribute to our GitHub repository at https://github.com/rackerlabs/atom-hopper. Just as we’ve found Atom Hopper to be a valuable tool for groups to manage messages at Rackspace, we hope you’ll find many uses for it in your own environment.

To try Atom Hopper, install it on your own server and then configure it according to your preferences. Our wiki includes instructions and suggestions to help you with installation and configuration. By following the examples provided in our documentation, you can then use Atom Hopper to get feeds, select entries from within feeds, add entries to feeds and add categories to entries.

Pluggable Data Adapter: Choose Your Own Datastore

A key feature of Atom Hopper is its pluggable data adapter, which allows Atom Hopper to communicate with a user-configured back end datastore, including relational databases, NoSQL or other options. By default, Atom Hopper ships with a Hibernate data adapter. We’ve tested it with PostgreSQL, MySQL and H2 databases. With the Hibernate data adapter, you can configure Atom Hopper to allow each Atom feed to store and retrieve data from completely different database servers. For example, Atom Hopper can talk to PostgreSQL on one feed and MySQL on a different feed.

Scalability: Load Balancing Multiple Feeds On Multiple Datastores

Atom Hopper was designed to scale. Atom Hopper holds no state, so it’s able to scale horizontally very well. Inside Rackspace, we use a load balancer to balance several Atom Hopper instances that all share the same configured Atom feeds. With the possibility of multiple Atom Hopper instances managing multiple feeds on multiple datastores, you can create an Atom Hopper configuration to suit a wide range of workloads.

You can use Atom Hopper to turn a large collection of Atom feeds into narrowly-targeted information, usable for a specific purpose and free of entries irrelevant to that purpose.

Fork It

Atom Hopper is available as open source under the Apache License version 2.0; the source code is available on GitHub. Visit our project website at www.atomhopper.org for links to more information about Atom Hopper, including our wiki where we’ve posted the most current information about setting up and using Atom Hopper for your own projects.

Don’t forget to follow Atom Hopper on Twitter!