Rackspace Cloud Files: How To Upload Very Large Objects

Filed in Product & Development by Jerry Schwartz | July 17, 2012 8:30 am

We frequently update our products, and Cloud Files[1] is no exception. In this series of technical posts, we’ll dig into how developers using the Cloud Files API can leverage new capabilities in Rackspace Cloud Files.

If you need to store very large files (e.g. videos, HD movies or backups), Cloud Files accomplishes this by allowing customers to upload multiple file segments and a manifest file to map those segments together. Large files will be downloaded as a single file.

How It Works

Objects that are larger than 5 GB must be segmented, prior to upload. You then upload the segments like you would any other object and create a manifest object telling Cloud Files how to find the segments of the large object. The segments remain individually addressable, but retrieving the manifest object streams all the segments concatenated. There is no limit to the number of segments that can be a part of a single large object.

In order to ensure the download works correctly, you must upload all the object segments to the same container, ensure each object name has a common prefix where their names sort in the order they should be concatenated. You also create and upload a manifest file. The manifest file is simply a zero-byte file with the extra X-Object-Manifest: <container>/<prefix> header, where <container> is the container the object segments are in and <prefix> is the common prefix for all the segments.

It is best to upload all the segments first and then create or update the manifest. With this method, the full object will not be available for downloading until the upload is complete. Also, you can upload a new set of segments to a second location and then update the manifest to point to this new location. During the upload of the new segments, the original manifest will still be available to download the first set of segments.

Example:  Upload Segment Of A Large Object

  PUT /<api version>/<account>/<container>/<object> HTTP/1.1
  Host: storage.clouddrive.com
  X-Auth-Token: eaaafd18-0fed-4b3a-81b4-663c99ec1cbb
  ETag: 8a964ee2a5e88be344f36c22562a6486
  Content-Length: 1
  X-Object-Meta-PIN: 1234
  s

No response body is returned. A status code of 201 (Created) indicates a successful write; status 411 (Length Required) denotes a missing Content-Length or Content-Type header in the request. If the MD5 checksum of the data written to the storage system does NOT match the (optionally) supplied ETag value, a 422 (Unprocessable Entity) response is returned.
You can continue uploading segments like this example shows, prior to uploading the manifest.

Example:  Upload Next Segment Of The Large Object

  PUT /<api version>/<account>/<container>/<object> HTTP/1.1
  Host: storage.clouddrive.com
  X-Auth-Token: eaaafd18-0fed-4b3a-81b4-663c99ec1cbb
  ETag: 8a964ee2a5e88be344f36c22562a6486
  Content-Length: 1
  X-Object-Meta-PIN: 1234
  w

Next, upload the manifest you created that indicates the container the object segments reside within. Note that uploading additional segments after the manifest is created will cause the concatenated object to be that much larger but you do not need to recreate the manifest file for subsequent additional segments.

Example: Upload Manifest

  PUT /<api version>/<account>/<container>/<object> HTTP/1.1
  Host: storage.clouddrive.com
  X-Auth-Token: eaaafd18-0fed-4b3a-81b4-663c99ec1cbb
  Content-Length: 0
  X-Object-Meta-PIN: 1234
  X-Object-Manifest: container/object/segments
                  
  [...]
                    

The response’s Content-Type for a GET or HEAD on the manifest will be the same as the Content-Type set during the PUT request that created the manifest. You can easily change the Content-Type by reissuing the PUT request.

This is just one of many new features in Rackspace Cloud Files that developers can take advantage of. Cloud Files customers are urged to try out this new feature in their next API project or add it to a current application. If you’re not yet using Cloud Files, sign up for an account and check it out. And now, using Cloud Files is more affordable since we reduced the cost of Cloud Files by 33 percent[2].

Endnotes:
  1. Cloud Files: http://www.rackspace.com/cloud/cloud_hosting_products/files/
  2. we reduced the cost of Cloud Files by 33 percent: http://www.rackspace.com/blog/cloud-files-updates/

Source URL: http://www.rackspace.com/blog/rackspace-cloud-files-how-to-upload-very-large-objects/