Does Cloud Files Support large file transfer?


Yes, the Rackspace Cloud now supports transfer and storage of larger files! Here is a list of frequently asked questions about our large file support:

How does Rackspace support upload of large files to Cloud Files?

Though support for uploading content to Cloud Files through the Control Panel is limited to files below 5GB, we can accommodate the transfer of files larger than 5GB by allowing you to segment your files into multiple file segments. 

How large should my file segments be?

This is a matter of preference and Rackspace does not enforce any lower limits on the file size. You may not exceed file segments of 5 GB, and Rackspace recommends not storing file segments below 100-200MB.

Can I serve my large files over the CDN?

The CDN does come with a few restrictions on file size. At this time, you may not serve files larger than 10GB from the CDN.

Is there a simpler way to use this code?

We have created a tool called the Swift tool to make this process easier.  You can get information about the Swift Tool here and download the Swift tool. The tool will segment your large file for you, create a manifest file, and upload them accordingly. After the file has been uploaded, this tool will manage your file segments for you, deleting and updating segments as needed.

When should I use the API vs the ST Tool?

If you are interested in developing against the Rackspace Large File Support code to incorporate into your application, you will want to work directly with the Cloud Files API. You will be following the steps outlined below:

# First, upload the segments 
curl -X PUT -H 'X-Auth-Token: <token>' \     http://<storage_url>/container/myobject/1 --data-binary '1' 
curl -X PUT -H 'X-Auth-Token: <token>' \     http://<storage_url>/container/myobject/2 --data-binary '2' 
curl -X PUT -H 'X-Auth-Token: <token>' \     http://<storage_url>/container/myobject/3 --data-binary '3'  
 
# Next, create the manifest file 
curl -X PUT -H 'X-Auth-Token: <token>' \     
-H 'X-Object-Manifest: container/myobject/' \     http://<storage_url>/container/myobject --data-binary ''
  
# And now we can download the segments as a single object 
curl -H 'X-Auth-Token: <token>' \     
http://<storage_url>/container/myobject

If you are interested in uploading large files, but not interesting in incorporating our code into an application, you may find it easier to use the ST (Swift Tool) for your uploads and management. You can download that tool here. If you are using the tool, your process will look like this:

This will upload largefile.iso to test_container in 10 segments (in this case 10 MB per segment for a total of 100 MB).

$ st -A https://auth.api.rackspacecloud.com/v1.0 -U glh -K 3a25c2dc74f24c3122407a26566093c8 upload -S 1048576 test_container largefile.iso
largefile.iso segment01
largefile.iso segment06
largefile.iso segment09
largefile.iso segment04
largefile.iso segment05
largefile.iso segment03
largefile.iso segment07
largefile.iso segment08
largefile.iso segment02
largefile.iso segment00
largefile.iso

Now, to download the large file as a single object:

$ st -A https://auth.api.rackspacecloud.com/v1.0 -U glh -K 3a25c2dc74f24c3122407a26566093c8 download test_container largefile.iso
largefile.iso

You can list the segments, and download them separately if you want:

$ st -A https://auth.api.rackspacecloud.com/v1.0 -U glh -K 3a25c2dc74f24c3122407a26566093c8 list test_container_segments
largefile.iso/1300906481.0/10485760/00000000
largefile.iso/1300906481.0/10485760/00000001
largefile.iso/1300906481.0/10485760/00000002
largefile.iso/1300906481.0/10485760/00000003
largefile.iso/1300906481.0/10485760/00000004
largefile.iso/1300906481.0/10485760/00000005
largefile.iso/1300906481.0/10485760/00000006
largefile.iso/1300906481.0/10485760/00000007
largefile.iso/1300906481.0/10485760/00000008
largefile.iso/1300906481.0/10485760/00000009

You can list the main large object itself:

$ st -A https://auth.api.rackspacecloud.com/v1.0 -U glh -K 3a25c2dc74f24c3122407a26566093c8 list test_container
largefile.iso

When using the object name to list large object segments, the segment names will be returned in alphabetical order. Plan segment names accordingly. For example, "name10" would come before "name9" alphabetically, so to preserve proper order the latter segment should be named "name09".

What will the download experience be like?

Once files are segmented and uploaded with a manifest file, your large file will be served as a single file, so the experience will mimic the download or service of any other object retrieval.

Do I keep access to my file segments?

Yes, you may edit your file segments just like any other object within Cloud Files.

How do I ensure my files are linked correctly?

This is done by including your manifest file in your upload. You can change your file name by editing this manifest file as well. Rackspace recommends using prefixing in your file segments in order to easily map your manifest file to the portions of your large file. For example, you could name your segments as so:

Myfavoritemovie-01
Myfavoritemovie-02
Myfavoritemovie-03
..etc..

In this case, you would just point your manifest file to the prefix:

Myfavoritemovie

Can I use this feature from the Control Panel?

At this time, Rackspace has not implemented this functionality into the Rackspace Cloud Control Panel.



Was this content helpful?




© 2014 Rackspace US, Inc.

Except where otherwise noted, content on this site is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License


See license specifics and DISCLAIMER