Using Swiftly to Download an Exported Image


Prerequisites

This article presumes that you have already used a Cloud Images export task to export one of your images from the Rackspace open cloud, that you understand how exported images are stored in Cloud Files as Dynamic Large Objects, and that you have installed the swiftly Cloud Files client. If you need help on any of these topics, please consult the following sources:

Downloading your image

Set some environment variables

Do the following in a bash shell:

CF_USERNAME=      # your Rackspace cloud username
CF_API_KEY=       # your Rackspace cloud API key
CF_REGION=        # 3 char region code for where your exported image is located (e.g., ORD)
LOCALFILENAME=    # what you want the downloaded image file to be named
CONTAINER=        # the container in Cloud Files containing your exported image
IMAGEFILENAME=    # the name of your exported image in Cloud Files

Invoking Swiftly

After you've got your environment variables set, you can invoke swiftly from the command line to perform the download. (You may want to do this in a screen session. If you're not familiar with the GNU screen program, you can find a quick introduction in the article: Installing the Swiftly Cloud Files Client.)

swiftly \
  --auth-url=https://identity.api.rackspacecloud.com/v2.0 \
  --auth-user=$CF_USERNAME \
  --auth-key=$CF_API_KEY \
  --region=$CF_REGION \
  get \
    --output=${LOCALFILENAME} \
  ${CONTAINER}/${IMAGEFILENAME}

Depending on your use case, you may be downloading to a cloud server that's already in the Rackspace open cloud. If that's the case, you'll want to add the --snet option to the command so that the file will be transferred over the internal cloud network. Additionally, if you want swiftly to keep you fully notified about what it's doing as it downloads your image file, you can add the --verbose option. If you add these, your invocation will look like this:

swiftly \
  --auth-url=https://identity.api.rackspacecloud.com/v2.0 \
  --auth-user=$CF_USERNAME \
  --auth-key=$CF_API_KEY \
  --region=$CF_REGION \
  --snet \
  --verbose \
  get \
    --output=${LOCALFILENAME} \
  ${CONTAINER}/${IMAGEFILENAME}

Here's a sample verbose response for the above call. (This image is a little over 2GB.)

VERBOSE 0.00 1 Attempting auth v2 RAX-KSKEY:apiKeyCredentials with https://identity.api.rackspacecloud.com/v2.0
VERBOSE 0.00 1 Establishing HTTPS connection to identity.api.rackspacecloud.com
VERBOSE 0.00 1 > POST /v2.0/tokens
VERBOSE 0.00 1 > {"auth": {"RAX-KSKEY:apiKeyCredentials": {"username": "joeuser", "apiKey": "968f8053a6f34677b70a3787cd3bd18a"}}}
VERBOSE 0.55 1 < 200 OK
VERBOSE 0.55 1 Establishing HTTPS connection to snet-storage101.iad3.clouddrive.com
VERBOSE 0.57 1 > GET /v1/MossoCloudFS_d4fef4ad-71f1-4ec4-b5fa-f15bf3317d0d/exports/9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd User-Agent: Swiftly v2.02  X-Auth-Token: 5c2ddf4d5031452fb2bae430560aa592
VERBOSE 0.93 1 < 200 OK

Checking Your Download

The time it takes for swiftly to complete your download will depend on the speed of your network connection and general network congestion. In some cases, the download may complete prematurely. To verify that this hasn't happened, you can compare the size (in bytes) of the file you've received with the size of the file that exists in Cloud Files. If they're not the same size, you've experienced a problem.

You can find out how big the file you've received is by looking at your local filesystem:

$ ls -l
total 2524008
-rw-rw-r-- 1 joeuser joeuser 2584576512 Apr 28 19:52 downloaded-image.vhd

You can use swiftly to determine the size of the file as it exists up there in Cloud Files:

swiftly \
  --auth-url=https://identity.api.rackspacecloud.com/v2.0 \
  --auth-user=$CF_USERNAME \
  --auth-key=$CF_API_KEY \
  --region=$CF_REGION \
  head \
  ${CONTAINER}/${IMAGEFILENAME}

Your response will look something like this. You're interested in the value of the Content-Length header.

Content-Length:    2584576512
Content-Type:      application/octet-stream
Etag:              "83dce2d37b7046228f72c1e5c688d8c8"
Last-Modified:     Fri, 28 Feb 2014 21:37:36 GMT
X-Object-Manifest: exports/9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd-
X-Timestamp:       1393623455.54221
X-Trans-Id:        txe67683c4407d4e8aa01b8-00535eac8aiad3

Note that the size of the local file in bytes matches the Content-Length of the Dynamic Large Object in Cloud Files. Thus, the download did not end prematurely.

On the other hand, all this tells us is that the correct number of bytes was received. It doesn't tell us that the value of every byte is correct.

If you have reason to suspect that your download was corrupted, or if you're trying to download an image from one side of the earth to the other (for example, you're in SYD and you're trying to download an image from LON), you may want to try an alternative technique.

Alternative Download Method

If your image is very large, or you experience network interruptions, the simple download method outlined above may repeatedly fail. Here's a more hands-on approach you can use. We'll assume you've set all the environment variables described above.

The basic idea behind this technique is that instead of downloading the entire object at once, we'll download the individual segments that make it up. This way, we will have smaller, more manageable pieces to work with, we can verify the checksum for each of the pieces to make sure no part is corrupted, and if a segment is messed up, we can re-download only that faulty segment. After we're sure we've got all the pieces, we can stream them locally into a single VHD file.

Get the DLO Manifest

Use the following command

swiftly \
  --auth-url=https://identity.api.rackspacecloud.com/v2.0 \
  --auth-user=$CF_USERNAME \
  --auth-key=$CF_API_KEY \
  --region=$CF_REGION \
  head \
  ${CONTAINER}/${IMAGEFILENAME}

The response will look something like this:

Content-Length:    2584576512
Content-Type:      application/octet-stream
Etag:              "83dce2d37b7046228f72c1e5c688d8c8"
Last-Modified:     Fri, 28 Feb 2014 21:37:36 GMT
X-Object-Manifest: exports/9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd-
X-Timestamp:       1393623455.54221
X-Trans-Id:        tx020fee01f492491abc2ac-00535d40f7iad3

The indication that this is a Dynamic Large Object is given by the X-Object-Manifest header. The value of this header specifies the container in which the segments are contained and the pattern that's used to find the segments in that container in the format container/pattern. Parsing the example response above, you can see that the container isexports and the pattern is 9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd-. When you request a download of this manifest object, what Cloud Files will do is go to the container in your account named exports, find all the objects it contains whose names match the pattern beginning 9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd-, and stream them out to you in alphabetical order.

Get the Detailed List of Segments

(This might be a good time to review How Exported Images are Stored as Dynamic Large Objects for a reminder of the naming conventions used by the Cloud Images export task.)

OK, now you know the container and pattern that will be used to locate the segments for the Dynamic Large Object you want to download. We can use swiftly to get a detailed list of the segments:

CONTAINER="exports"
PATTERN="9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd-"
swiftly \
  --auth-url=https://identity.api.rackspacecloud.com/v2.0 \
  --auth-user=$CF_USERNAME \
  --auth-key=$CF_API_KEY \
  --region=$CF_REGION \
  get \
    --prefix=${PATTERN} \
    --full \
  ${CONTAINER}

Here's a sample response:

   1048576000 2014-02-28 21:33:46.41 699580384e28b7c2b3b8d916169c11ca  application/octet-stream 9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd-00001
   1048576000 2014-02-28 21:35:43.81 b513404695fc1117c42ee6dc7de91ef1  application/octet-stream 9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd-00002
    487424512 2014-02-28 21:36:55.05 372d9904e2048dc84dea82fdb0ead10e  application/octet-stream 9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd-00003

The way to read the output is like this:

   number-of-bytes   creation-date-time   MD5-checksum   output-type   segment-name

You can see that the DLO consists of three segments. The first two are the same size and the final segment is smaller.

Download the Segments

Next, you can use swiftly to download the individual segments that we've identified to a local directory. (Make sure this directory already exists in your filesystem before attempting the download.)

LOCAL_DIR="my-images"
swiftly \
  --auth-url=https://identity.api.rackspacecloud.com/v2.0 \
  --auth-user=$CF_USERNAME \
  --auth-key=$CF_API_KEY \
  --region=$CF_REGION \
  --verbose \
  get \
    --prefix=${PATTERN} \
    --all-objects \
    --output=${LOCAL_DIR}/ \
  ${CONTAINER}

Here's some sample output with the swiftly verbose option enabled:

VERBOSE 0.00 1 Attempting auth v2 RAX-KSKEY:apiKeyCredentials with https://identity.api.rackspacecloud.com/v2.0
VERBOSE 0.00 1 Establishing HTTPS connection to identity.api.rackspacecloud.com
VERBOSE 0.00 1 > POST /v2.0/tokens
VERBOSE 0.00 1 > {"auth": {"RAX-KSKEY:apiKeyCredentials": {"username": "joeuser", "apiKey": "968f8053a6f34677b70a3787cd3bd18a"}}}
VERBOSE 0.34 1 < 200 OK
VERBOSE 0.34 1 Establishing HTTPS connection to storage101.iad3.clouddrive.com
VERBOSE 0.36 1 > GET /v1/MossoCloudFS_d4fef4ad-71f1-4ec4-b5fa-f15bf3317d0d/exports?format=json&prefix=9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd- User-Agent: Swiftly v2.02  X-Auth-Token: 5c2ddf4d5031452fb2bae430560aa592
VERBOSE 0.37 1 < 200 OK
VERBOSE 0.38 1 > GET /v1/MossoCloudFS_d4fef4ad-71f1-4ec4-b5fa-f15bf3317d0d/exports/9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd-00001 User-Agent: Swiftly v2.02  X-Auth-Token: 5c2ddf4d5031452fb2bae430560aa592
VERBOSE 0.48 1 < 200 OK
VERBOSE 14.53 1 Establishing HTTPS connection to storage101.iad3.clouddrive.com
VERBOSE 14.54 1 > GET /v1/MossoCloudFS_d4fef4ad-71f1-4ec4-b5fa-f15bf3317d0d/exports/9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd-00002 User-Agent: Swiftly v2.02  X-Auth-Token: 5c2ddf4d5031452fb2bae430560aa592
VERBOSE 14.61 1 < 200 OK
VERBOSE 207.58 1 Establishing HTTPS connection to storage101.iad3.clouddrive.com
VERBOSE 207.59 1 > GET /v1/MossoCloudFS_d4fef4ad-71f1-4ec4-b5fa-f15bf3317d0d/exports/9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd-00003 User-Agent: Swiftly v2.02  X-Auth-Token: 5c2ddf4d5031452fb2bae430560aa592
VERBOSE 207.76 1 < 200 OK
VERBOSE 218.22 1 Establishing HTTPS connection to storage101.iad3.clouddrive.com
VERBOSE 218.23 1 > GET /v1/MossoCloudFS_d4fef4ad-71f1-4ec4-b5fa-f15bf3317d0d/exports?format=json&marker=9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd-00003&prefix=9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd- User-Agent: Swiftly v2.02  X-Auth-Token: 5c2ddf4d5031452fb2bae430560aa592
VERBOSE 218.24 1 < 200 OK

Now, let's take a look at the segments that were downloaded:

$ ls -l my-images/
total 1853964
-rw-rw-r-- 1 joeuser joeuser 1048576000 Feb 28 21:33 9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd-00001
-rw-rw-r-- 1 joeuser joeuser  362443328 Feb 28 21:35 9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd-00002
-rw-rw-r-- 1 joeuser joeuser  487424512 Feb 28 21:36 9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd-00003

Do you see the problem? Recall from How Exported Images are Stored as Dynamic Large Objects that each segment should be the same size except for the final segment (which can be smaller). Thus, segment-00002 should be the same size as segment-00001, and you can see that they're not. So segment-00002 is corrupted.

What about the final segment? How do we know it's the correct size? Recall that earlier you invoked swiftly with the --full option to get the full details on all the segments in the  exports container that match the pattern 9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd-. From that sample output (displayed here), you can tell that the final segment should be 487424512 bytes, and if you look at the directory listing above, you can see that segment-00003 is the correct size.

Re-Download A Segment (if necessary)

Let's get a fresh copy of the bad segment:

SEGMENT_NAME="9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd-00002"
swiftly \
  --auth-url=https://identity.api.rackspacecloud.com/v2.0 \
  --auth-user=$CF_USERNAME \
  --auth-key=$CF_API_KEY \
  --region=$CF_REGION \
  --verbose \
  get \
    --output=${LOCAL_DIR}/${SEGMENT_NAME} \
  ${CONTAINER}/${SEGMENT_NAME}

Here's some sample verbose output:

VERBOSE 0.01 1 Attempting auth v2 RAX-KSKEY:apiKeyCredentials with https://identity.api.rackspacecloud.com/v2.0
VERBOSE 0.01 1 Establishing HTTPS connection to identity.api.rackspacecloud.com
VERBOSE 0.01 1 > POST /v2.0/tokens
VERBOSE 0.01 1 > {"auth": {"RAX-KSKEY:apiKeyCredentials": {"username": "joeuser", "apiKey": "968f8053a6f34677b70a3787cd3bd18a"}}}
VERBOSE 0.38 1 < 200 OK
VERBOSE 0.38 1 Establishing HTTPS connection to storage101.iad3.clouddrive.com
VERBOSE 0.39 1 > GET /v1/MossoCloudFS_d4fef4ad-71f1-4ec4-b5fa-f15bf3317d0d/exports/9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd-00002 User-Agent: Swiftly v2.02  X-Auth-Token: 5c2ddf4d5031452fb2bae430560aa592
VERBOSE 0.47 1 < 200 OK

Now, let's take a look at the segments that have been downloaded:

$ ls -l my-images
total 2524016
-rw-rw-r-- 1 joeuser joeuser 1048576000 Feb 28 21:33 9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd-00001
-rw-rw-r-- 1 joeuser joeuser 1048576000 Apr 29 02:59 9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd-00002
-rw-rw-r-- 1 joeuser joeuser  487424512 Feb 28 21:36 9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd-00003

That's better—you can see that the second segment has the correct size.

Verifying the MD5 Checksums

At this point, we know that all segments have the correct size. But is the content of each segment correct? To verify that it is, we can take the MD5 checksum of each segment and compare it to the MD5 checksum of the corresponding segment in Cloud Files. (We already have the list of MD5 checksums of the segments in Cloud Files; they were displayed when you invoked swiftly with the "--full" option above, follow this link to see for yourself.)

$ cd my-images
$ for i in $(ls -1 9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd-*); do md5sum $i ; done 699580384e28b7c2b3b8d916169c11ca 9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd-00001 b513404695fc1117c42ee6dc7de91ef1 9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd-00002 372d9904e2048dc84dea82fdb0ead10e 9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd-00003

The way to read the output from the md5sum command is that the checksum is on the left, and the filename is on the right.  If you compare the checksums displayed here to the ones we got from Cloud Files earlier (displayed here), you can see they match. We've successfully downloaded the segments!

Reconstituting the VHD File

While it's great that all the segments have been downloaded correctly, what we really want is a single VHD file, not a bunch of segments. You can make a single VHD like this:

# still in the "my-images" directory
$ cat 9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd-0000* > 9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd

# let's see what we've got now $ ls -l total 5048020 -rw-rw-r-- 1 joeuser joeuser 2584576512 Apr 29 03:20 9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd -rw-rw-r-- 1 joeuser joeuser 1048576000 Feb 28 21:33 9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd-00001 -rw-rw-r-- 1 joeuser joeuser 1048576000 Apr 29 02:59 9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd-00002 -rw-rw-r-- 1 joeuser joeuser 487424512 Feb 28 21:36 9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd-00003

You can scroll back through this document to find the place where we used swiftly to do a HEAD request on the Dynamic Large Object in Cloud Files (it's here). There we determined that the Content-Length for a request of the DLO would be 2584576512 bytes. You can see that's the size of the reconstituted VHD file.

Should we compute an MD5 checksum of the VHD file? You could, but we'd have nothing to compare it to. Since a DLO isn't a "real" object in Cloud Files (what it "really" is, is a series of segment objects whose content is streamed out in a particular order), Cloud Files doesn't store an MD5 checksum for the DLO. Further, we don't need the checksum: we know the VHD file that we've got locally is made up of 3 segments, each one of which has the correct content, and we've put them together in the correct order. So our VHD file must be identical to the file that was exported from Cloud Images into Cloud Files.

Cleaning Up

Let's take another look at your local filesystem:

ls -lh my-images
total 4.9G
-rw-rw-r-- 1 joeuser joeuser  2.5G Apr 29 03:20 9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd
-rw-rw-r-- 1 joeuser joeuser 1000M Feb 28 21:33 9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd-00001
-rw-rw-r-- 1 joeuser joeuser 1000M Apr 29 02:59 9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd-00002
-rw-rw-r-- 1 joeuser joeuser  465M Feb 28 21:36 9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd-00003

Unlike in Cloud Files, where the segments contain data and the DLO is just a manifest telling Cloud Files what the content of the DLO is, these are all real files. You can see this by asking the operating system to tell you how much space these files are taking up:

$ du -chs my-images
4.9G	my-images
4.9G	total

You can see that we're using up twice as much space as the size of the VHD file. So locally, now that we've used the segments to reconstitute the VHD file, we no longer need the segments around, and you can delete them without affecting your VHD:

$ rm my-images/9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd-0000*

$ ls -lh my-images
total 2.5G
-rw-rw-r-- 1 joeuser joeuser 2.5G Apr 29 03:20 9af8acc8-8189-48b9-b3d6-8152c60074d8.vhd

$ du -chs my-images
2.5G	my-images
2.5G	total

You can see that we've still got our 2.5GB VHD file. But remember, we're working on your local computer here. In Cloud Files, Dynamic Large Objects have a different relationship with their segments. If you delete segments in Cloud Files, you'll corrupt your DLO (and hence, your VHD).

Conclusion

In this article, we discussed two ways for you to use swiftly to download from Cloud Files and image that you've exported from Cloud Images. If you'd like to use swiftly to upload a VHD to Cloud Files so that you can import it into the Rackspace open cloud, we've got an article about that, too: Using Swiftly to upload an image to be imported.



Was this content helpful?




© 2011-2013 Rackspace US, Inc.

Except where otherwise noted, content on this site is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License


See license specifics and DISCLAIMER