Rackspace Cloud partner Absolute Performance brings their product System Shepherd into the battle to provide the sysadmin a new weapon to turn the odds in their favor. System Shepherd is a hosted monitoring service targeted at today’s mixed computing environments, improved by Absolute Performance’s work in the space since the late 1990s.

Monitoring Agent

At the heart of System Shepherd is their monitoring agent. This small binary (only a few megabytes of resident memory) provides the hooks into the things that you need monitored on your systems and applications. Being an easily-installable agent, it’s well-suited for including in your custom cloud images. Install once, take a snapshot, and your new systems built off of that image will be set up to connect to your monitoring infrastructure.

One problem that pops up with cloud infrastructure is that servers may be provisioned and removed rapidly to deal with changing loads. As sysadmins monitoring in this space well know, that can lead to an annoying cascade of phantom alerts as monitoring software notices that deleted servers are no longer responding to pings or other monitoring checks. System Shepherd solves this problem by hooking directly into the Rackspace Cloud Servers API to check on the status of your environment. If a server stops responding to the monitoring tests, System Shepherd will query Cloud Servers to query that server. If the server no longer exists in your environment, System Shepherd will note that server as deprovisioned and not alert you. If the server is still provisioned, but not responding, then the alert is sent. Cell phones everywhere breathe a sigh of relief.

Enterprise Technology Built-In

While System Shepherd can be used by deployments large and small, it cut its teeth on larger enterprise setups. This means that it has deeper hooks into enterprise technologies built-in. There are agents for databases like MySQL, Oracle, and SQL Server, applications like Exchange or Handysoft Bizflow, core technologies like Apache, Tomcat, and Websphere. Hardware doesn’t need to feel left out, as Cisco, Nortel, F5, Juniper, and Foundry get agents that talk to their hardware. For companies running their own private clouds, VMWare ESX is currently supported, with Xen, Hyper-V, and Parallels Virtuosos coming soon. And of course, there are hooks for pulling data out of the output of shell commands and SNMP. Finally, the agents support current releases of Windows and all major Linux distributions (plus FreeBSD!)

The Dashboard

All of this data gets aggregated and displayed in the System Shepherd dashboard. The dashboard features drag-and-drop components (or “gadgets” in the System Shepherd nomenclature) to display the information you want to see in the way you want it organized. The dashboard is also where system-wide configuration is managed, saving the hassle of logging into machine after machine to tune settings. There is also an extensive library of metric templates to get you up and running quickly.

If you’d like herd your systems into line, contact Absolute Performance and take a look at System Shepherd! They have a number of different offerings, so find the one that works best for your setup.

Easy To Use UI

Rollbase provides a simple web UI that allows you to model your business data and hook into it. At the core of the application are business objects. These represent things like employees, customers, inventory items, invoices, and the like. They basically map to relational database tables, so you can link objects with the standard array of relationships (one-to-one, one-to-many, etc). You define these objects, along with their attributes, within Rollbase. Here’s a quick snapshot of the UI:

It walks you through the steps to create an object. It allows you to set up advanced features like audit trails, versioning, and the like as well as import attributes from a standard list, such as documents, locations, and workflows, which automatically give you features useful for those types of objects.

The fields in the object also have properties beyond the normal database varchar/integer set. There are fields for email addresses, percentages, checkboxes, radio buttons, select lists, and more. For more complex needs, there are computable fields, which allow you to run calculations on the input value with Javascript expressions.

Once you have your objects and relationships defined, the next step is providing an interface into them. Using the Rollbase WYSWYG drag-and-drop view editor, you can develop the interfaces into your data. Create forms to add or edit data, then build pages to display the data. Of course, as security is important to your app, you can set different access permissions to different data on a per-user or per-group basis. And because the pages are HTML, you can embed custom HTML/Javascript widgets into them to provide more functionality.

Because this is an application designed for business, there are plenty of features a business will need. Rich email and document templates are one, allowing for nice-looking mail and exports into Word and Excel. There is a detailed multi-level charting and reporting engine as well. Additionally, if you want your applications to face externally, you can create a public portal into your data, with external user access available as a control.

For users that want to get started with even less effort, Rollbase has an application directory with more than 20 preconfigured applications, from Customer Relationship Managers to software bug trackers to employee management and recruiting.

Available in Two Forms: for ISVs and Resellers

A hosted white label program makes it easy to develop and deploy business applications under any brand or identity. Hosted resellers run on Rollbase’s Rackspace-powered infrastructure. Rollbase is also offered as a licensed product allowing larger organizations and systems integrators to procure a robust multitenant platform as the foundation for SaaS application development and delivery. Licensees can choose to run Rollbase on dedicated Rackspace physical or Cloud Servers or on servers of their own choosing.

Rollbase offers a 30 day free trial, so if your business or group could use fast, easy, and powerful business applications, give them a try!

UniCloud uses a VPN to create a secure, consistent network between your physical and cloud hosts. This new VPN IP space allows all servers in your environment to have a standardized network that spans multiple datacenters and hosting environments. UniCloud handles this VPN setup automatically, along with providing DNS services for it, saving you the headache of tweaking VPN configurations when you bring new nodes online. This VPN can also be used to transfer reasonably-sized data sets up to compute nodes.

Additionally, UniCloud abstracts representations of compute environments (local, cloud, etc) into various resource adapters. Multiple resource adapters can exist in any given compute environment, making it easy to scale workloads over different local and remote computing providers for a single job. This also means that as new compute environments come to market, Univa can add support for them as a resource adapter and easily expand your options.

Coupled with these resource adapters is the notion of application mobility. With different computing options available, there needs to be a way to move applications and configurations from one environment to another with a high degree of consistency and control. UniCloud’s solution to this problem comes in the form of “application kits.”. These kits provide a self-contained bundle consisting of the application binaries or source, configuration data, and install/uninstall scripts. A kit or collection of kits can be used to encapsulate one particular piece of software, or an entire application stack. Once a kit is created, it can be added to a central software repository and then managed using policy rules. These kits are what allow applications to move from system to system, without a large amount of custom configuration.

So you can see how UniCloud makes it easy to manage compute resources, but its rule engine is what really sets UniCloud apart. As an application that really cut its teeth on the Sun/Oracle Grid Engine, UniCloud was engineered to solve problems involving very large-scale data processing requirements. Being able to solve those types of problems means that it UniCloud also works for many common use cases.

The Python-based rules engine can monitor nearly any aspect of a system or the environment as a whole, and can take actions based on the results. For example, imagine that one UniCloud user has a private physical-server cluster environment where they run an image analysis program through a distributed batch system. When the computing needs exceed the capacity of their local systems, they use UniCloud to “cloud burst” into both Rackspace Cloud Servers and Amazon EC2. This allows them to get the CPU cycles that they need, while avoiding the need for a large CapEx in local servers to handle these peak loads. For this customer, UniCloud dynamically creates a single distributed batch infrastructure on Grid Engine. From their end user’s point of view, nothing changes – they submit jobs as usual and it’s UniCloud on the back-end that allocates cloud nodes and moves jobs around. The UniCloud policy engine can even take a cost metric into account (for the cost and latency differential between local and cloud nodes, or different cloud provider costs) to intelligently spin up the most cost-effective environments.

Such a use case is a great reason to look at cloud computing – being able to seamlessly spin up cloud nodes to handle unusual workloads, while paying only pennies per hour. Add to that other useful features such as synchronized user accounts, push-based file systems, and more, and it becomes clear that UniCloud is a great option for a company that does not want to buy another rack of servers to deal with spikes in computing needs.

Want to find out more? Contact Rackspace Cloud partner Univa about UniCloud. They’ll be glad to help you out!

Kaavo offers free 30-day demo accounts to its IMOD system. Just visit the Kaavo site and look for the “Sign-up for free trial” link.

Once you log into your IMOD account, you’ll see a list of available templates. These are packaged definitions that will set up environments for you. We’re going to to look at the “Linux php-colab 3tier-For-Rackspace” template. That will set up our three-tier architecture, and will install and run the phpCollab project collaboration application as well, so we’ll have a real-world product in place by the end.

Click on that option in the templates list, and you’ll see an XML file appear on the right. Kaavo uses XML to define these environments and control what they can do. A more user-friendly interface to handle common cases is in the works, but you can control nearly any aspect of the deployment from this XML file.

Each of the tiers is defined in a tier section in the XML. You can specify the order in which the tiers are displayed and the order in which they are started. Handy, of course, if your application needs its database online before the web servers attempt to talk to it. You can also specify command actions that will run as each server in that tier comes online. For example, in the above screenshot, you can see that in the web_tier (where the load balancer lives), there are several actions that run. We’ll take a look at how those actions are defined shortly, but the names should be pretty self-explanatory.

Scrolling down the XML, we can see the actual Cloud Servers that make up this tier. We can see that there should be a minimum of one and maximum of two load balancers running at any given point in the autoscaling cycle, and we should start up with 1. We’re using Cloud Server image 187811 (CentOS 5.4), and flavor 1 (256MB server). Those can be adjusted to taste. The other two tiers are set up in a similar manner, though in the db_tier you can see an example of having different server types within one tier.

Next in the XML, we see the actions defined. These are the scripts that will be run on the specified servers to perform the actions that we want. Some of the actions can be executable scripts, and some can just be raw files (config files and the like) that aren’t run. Below, we see both types in the screenshot below – one will install some RPMs, the other just sets up the MySQL cluster manager config file. Note that the execute attribute is set to “true” on the script, and “false” on the config file. Note also we’re specifying the path and filename.

There is also variable substitution available in these actions. In this action, we can see variables (parameters) set statically in the XML file, as well as ones dynamically generated (the IP addresses of the ndbd servers).

Finally, at the bottom of the file, there are event handlers, to run actions when states change, such as tiers are scaled up or down.

Ok, enough XML, let’s see how it works. At the top of the XML listing, fill in a system name and hit Deploy System. You’ll see that system listed under My Systems on the left. Click on that to see a diagram of your system and its tiers. You’ll notice that it’s “unconfigured” – you need to go into each system type and choose the Rackspace Cloud account to run them under, as well as adjusting images and flavors.

Once all systems are configured, you should see the initial configuration (number of servers, etc) as you defined in the XML. Now you can press the “Start System” button and wait. You’ll see spinners pop up in the display as servers are configured in the order that you specified. Enjoy the emails notifying you of the new servers that are being started. We’ll wait.

Grab a snack while you wait.

All done? Great! You should now have green lights and “Running” labels beneath all servers. If you click the blue triangle on your load balancer to get the Public IP, and open a web browser to that http://your-ip/phpCollab, you should see phpCollab! You can also go to http://your-ip/balancer-manager to see the Apache load balancer in action.

Now let’s throw some load at it. Click the “Event” button in IMOD, and choose the app-tier-overloaded event. This simulates what happens when load gets too high. You should immediately see another app server added to the mix. Visit the balancer-manager URL to see that the new server has been added to the mix.

If you’d like to play with the scaling parameters, configure the events through the IMOD UI. You’ll want to do this before starting the system, so that your changes are persisted.

Now, feel free to play around with the system, add traffic, modify the XML, enjoy! Remember, you have 30 days to run the system through its paces and figure out how Kaavo can help you leverage the power of the Rackspace Cloud.

Before I begin, why would you want to store assets in the cloud? First, you get the benefit of massively expandable storage. You don’t need to worry about adding another disk drive to your RAID or LVM pools, you just keep uploading and let the cloud provider worry about the disks. Second, you can gain the speed of a Content Distribution Network (CDN), such as Rackspace’s partner Limelight. This allows your users to get data faster, as the CDN routes their requests to network nodes located closer to the end user. Finally, you’re decoupling dynamic data (uploaded files) from static data (your application code), which will allow you to more easily scale your application servers based on server snapshots.  That comes in handy when scaling out server images on Cloud Servers.

With that in mind, here’s a quick look at the big three plugins. For each of them, I have also provided links to the source code on github, as well as a small sample Rails application demonstrating how to use them. All of these utilities rely on the cloudfiles gem (source).  (As an aside, that gem is due for some refactoring, so if anyone wants to help out with that project, feel free to drop me an email.)

attachment_fu by Rick Olson (Sample Application)

attachment_fu was one of the first solid file attachment plugins for Rails, with first commits happening back in December 2006.  Mike Clark blogged about it in early 2007, and it was included as an example in the Advanced Rails Recipes book.  It was created as the successor to the earlier acts_as_attachment plugin. Uploaded files can be stored to the filesystem, database, Rackspace Cloud Files, or Amazon S3.

In attachment_fu, the uploaded file is stored as a discrete model in and of itself. So for a User with an uploaded Photo, I’ve found it easier to put the user data in the User model, and join that with a Photo model that just handles the uploaded file. I haven’t had much luck adding extra data to the attachment model. Additionally, due to the way the metadata on the object is stored in the database, one model cannot represent multiple types of uploaded files (so the same model can’t handle both a Photo and an Avatar).

attachment_fu can work with a wide variety of image processors, including RMagick, ImageScience, Mini-Magick, GD2, and even CoreImage on the Mac. The attachment_fu README also gives examples of how to use it in scripting, which is handy if you need to bulk-add data, or migrate from a different file upload system. The README is also very thorough documenting the various options to the has_attachment definition, as well as the database tables.

paperclip by Thoughtbot (Sample Application)

(Cloud Files support in paperclip-cloudfiles)

Paperclip is likely the most popular file upload plugin today, with over 2,000 watchers on github and over 300 forks. Part of the great suite of Thoughtbot software (including clearance, hoptoad, factory_girl, shoulda, and more), it allows an easier integration of uploaded files with existing models. You don’t need to dedicate an entire new model to your uploaded files, instead you just add a few new database fields to the model (representing filename, content type, size, and update date), and you can also have multiple files attached to a given model.

Paperclip offers an extensible selection of post-processors for when the file is uploaded, so you can perform tasks such as OCR’ing files into text, transcoding video, and more. There are also pre- and post- hooks that allow you more fine-grained control over handling uploads. For image processing (the most common use case), thumbnailing is built in, and Paperclip uses system-level ImageMagick binaries to handle them (avoiding memory issues with RMagick).

The base installation of paperclip supports storing files on the filesystem or in Amazon S3. Thoughtbot policy is that they don’t add any code into the mainline of their projects unless they actively use it in client projects, so there is no support for Cloud Files in the main branch of paperclip. I maintain a fork of paperclip with Cloud Files support, though, and that gem can be found as “paperclip-cloudfiles” on the gem servers. It’s on my list to refactor that code to be a plugin into the mainline, as opposed to a fork of the entire project.

Carrierwave by Jonas Nicklas (Sample Application)

Carrierwave is the new kid on the block, with first commits coming in early 2009. Interestingly, it is not just a Rails solution, but provides options for Rails, Merb, Sinatra, or other Rack applications. It splits out the uploader code into a class located in app/uploaders for Rails applications. This gives you the benefit of isolating the uploading code (rspec testing macros are built in), while easily allowing your base model to include multiple uploaded assets. Carrierwave can store files locally on the filesystem, on Rackspace Cloud Files, or on Amazon S3 (using AWS-S3 or Right-S3).

Carrierwave provides generators to give you the basic skeleton for your uploader. Because the uploader model is just a Ruby class, it’s very easy to override defaults or add new features – just open up that class and redefine methods as you go! Carrierwave can be configured on a global level (though an initializer) and on a per-uploader basis.

Another interesting thing about Carrierwave is built-in support for not only ActiveRecord, but MongoDB’s GridFS. With NoSQL databases growing in popularity, having out-of-the-box support for MongoDB is a handy way to dip your feet in the waters. For image processors, RMagick, ImageScience, and MiniMagick are supported. Carrierwave provides support for pulling uploads from a remote URL in addition to a direct file upload, which is quite cool. It also easily handles the common use case of removing previously uploaded files with a simple checkbox.

Carrierwave has an extensive amount of documentation in its README file, so peruse that to see what it can do!

The wrap-up

attachment_fu is the old reliable workhorse. It’s not under heavy development at present, so it will be interesting to see if things pick up as Rails 3 gains traction. But you will be able to find a lot of online documentation supporting it, and it’s stable and works well.

Paperclip is leaner, integrates well with the rest of the Thoughtbot toolkit, and is widely popular. It’s tailored to a fairly specific technology set by design, though, so Rackspace customers will need to use a fork of the main project.

Carrierwave is the upstart, taking a different tack on the idea of asset uploads, and is touching on a wider variety of technology than the others. It’s still under active development, so you’ll want to keep an eye on how things change.  I’m very impressed with it.

Play around with all three, find the one that suits your particular feature needs, and easily leverage the power of Cloud Files!  And if I’ve gotten anything completely wrong, feel free to correct me in the comments!

I’ve spun up a basic Fedora 12 server that will be the base image for my farm of web servers in this instance, and have set up this initial server in Scout.  After creating that server in Scout, I get instructions to install one Ruby gem, a server key, and a sample crontab entry.  Easy!

I’ll then go install that gem on my test server, register the server with Scout, and set up the crontab entry so that it will update Scout every minute.

[root@scout-web1 ~]# yum install ruby rubygems

[Omitted for brevity]

Complete!

[root@scout-web1 ~]# gem install scout

Successfully installed elif-0.1.0

Successfully installed scout-5.0.3

2 gems installed

[root@scout-web1 ~]# scout

=== Scout Installation Wizard ===

You need the Server Key displayed in the Server Settings tab.

It looks like:

6ecad322-0d17-4cb8-9b2c-a12c4541853f

Enter the Server Key:

37dc7d3f-290d-408d-9605-c31c727450b9

Attempting to contact the server...

Success!

Now, you must setup Scout to run on a scheduled basis.

If you are using the system crontab

(usually located at /etc/crontab):

****** START CRONTAB SAMPLE ******

* * * * *  root /usr/bin/scout 37dc7d3f-290d-408d-9605-c31c727450b9

******  END CRONTAB SAMPLE  ******

If you are using this current user's crontab

(using crontab -e to edit):

****** START CRONTAB SAMPLE ******

* * * * *  /usr/bin/scout 37dc7d3f-290d-408d-9605-c31c727450b9

******  END CRONTAB SAMPLE  ******

For help setting up Scout with crontab, please visit:

[root@scout-web1 ~]# echo "* * * * *  root /usr/bin/scout 37dc7d3f[...]" >> /etc/crontab

At this point, you can do anything else you need to get this server ready to be a base image for future instances in your web cluster (update packages, install application software, configure centralized authentication, etc).  When you’re done, visit the Rackspace Cloud control panel and create a backup snapshot image for this server.

While that backup is saving, go back over to Scout.  Select this server from the monitored servers list, then choose the “Server Settings” tab from the top-right.  You’ll then check the “Cloud Image” checkbox.  What this does is tell Scout that any other server that checks in with this same key is functionally equivalent to this server, so it should get the same settings, alert thresholds, plugins, and such that you define for this server.

At this point I’ll go into the server definition for this monitored server in Scout and add a couple more plugins to this server’s monitoring setup, to get it like I want it.

Now, it’s back to the Rackspace Cloud control panel to kick off a couple more cloud servers.  Only this time, instead of choosing the generic Fedora 12 instance as a base image, I’ll choose the “My Backups” tab and select the snapshot that I just saved of the customized system.  From there, I’ll create two more web servers.

Now we go back to scout, look at our server tab, and the two new servers have just appeared, replicating the configuration of the original!

If you add new plugins or change thresholds to the primary system’s monitoring configuration, newly created servers will pick them up. You can push out updates to existing servers using Scout’s Copy & Paste functionality.

So just like that, you can quickly and easily have new servers in your Cloud clusters get their monitoring configured at bootup – no more SSH’ing into each server, or having to constantly update a server image.  Now that’s a time-saver.

For more information, visit Scout or talk to us here at the Cloud!  If you can’t wait to try out Scout in your environment, sign up with coupon code “rack” (now until March 3rd) and get $10 off of your first paid month of service.