Does Cloud Files Support large file transfer?
Yes, the Rackspace Cloud now supports transfer and storage of larger files! Here is a list of frequently asked questions about our large file support:
How does Rackspace support upload of large files to Cloud Files?
Though support for uploading content to Cloud Files through the Control Panel is limited to files below 5GB, we can accommodate the transfer of files larger than 5GB by allowing you to segment your files into multiple file segments.
How large should my file segments be?
This is a matter of preference and Rackspace does not enforce any lower limits on the file size. You may not exceed file segments of 5 GB, and Rackspace recommends not storing file segments below 100-200MB.
Can I serve my large files over the CDN?
The CDN does come with a few restrictions on file size. At this time, you may not serve files larger than 10GB from the CDN.
Is there a simpler way to use this code?
We have created a tool called the Swift tool to make this process easier. You can get information about the Swift Tool here and download the Swift tool. The tool will segment your large file for you, create a manifest file, and upload them accordingly. After the file has been uploaded, this tool will manage your file segments for you, deleting and updating segments as needed.
When should I use the API vs the ST Tool?
If you are interested in developing against the Rackspace Large File Support code to incorporate into your application, you will want to work directly with the Cloud Files API. You will be following the steps outlined below:
# First, upload the segments curl -X PUT -H 'X-Auth-Token: <token>' \ http://<storage_url>/container/myobject/1 --data-binary '1' curl -X PUT -H 'X-Auth-Token: <token>' \ http://<storage_url>/container/myobject/2 --data-binary '2' curl -X PUT -H 'X-Auth-Token: <token>' \ http://<storage_url>/container/myobject/3 --data-binary '3' # Next, create the manifest file curl -X PUT -H 'X-Auth-Token: <token>' \ -H 'X-Object-Manifest: container/myobject/' \ http://<storage_url>/container/myobject --data-binary '' # And now we can download the segments as a single object curl -H 'X-Auth-Token: <token>' \ http://<storage_url>/container/myobject
If you are interested in uploading large files, but not interesting in incorporating our code into an application, you may find it easier to use the ST (Swift Tool) for your uploads and management. You can download that tool here. If you are using the tool, your process will look like this:
This will upload largefile.iso to test_container in 10 segments (in this case 10 MB per segment for a total of 100 MB).
$ st -A https://auth.api.rackspacecloud.com/v1.0 -U glh -K 3a25c2dc74f24c3122407a26566093c8 upload -S 1048576 test_container largefile.iso largefile.iso segment01 largefile.iso segment06 largefile.iso segment09 largefile.iso segment04 largefile.iso segment05 largefile.iso segment03 largefile.iso segment07 largefile.iso segment08 largefile.iso segment02 largefile.iso segment00 largefile.iso
Now, to download the large file as a single object:
$ st -A https://auth.api.rackspacecloud.com/v1.0 -U glh -K 3a25c2dc74f24c3122407a26566093c8 download test_container largefile.iso largefile.iso
You can list the segments, and download them separately if you want:
$ st -A https://auth.api.rackspacecloud.com/v1.0 -U glh -K 3a25c2dc74f24c3122407a26566093c8 list test_container_segments largefile.iso/1300906481.0/10485760/00000000 largefile.iso/1300906481.0/10485760/00000001 largefile.iso/1300906481.0/10485760/00000002 largefile.iso/1300906481.0/10485760/00000003 largefile.iso/1300906481.0/10485760/00000004 largefile.iso/1300906481.0/10485760/00000005 largefile.iso/1300906481.0/10485760/00000006 largefile.iso/1300906481.0/10485760/00000007 largefile.iso/1300906481.0/10485760/00000008 largefile.iso/1300906481.0/10485760/00000009
You can list the main large object itself:
$ st -A https://auth.api.rackspacecloud.com/v1.0 -U glh -K 3a25c2dc74f24c3122407a26566093c8 list test_container largefile.iso
When using the object name to list large object segments, the segment names will be returned in alphabetical order. Plan segment names accordingly. For example, "name10" would come before "name9" alphabetically, so to preserve proper order the latter segment should be named "name09".
What will the download experience be like?
Once files are segmented and uploaded with a manifest file, your large file will be served as a single file, so the experience will mimic the download or service of any other object retrieval.
Do I keep access to my file segments?
Yes, you may edit your file segments just like any other object within Cloud Files.
How do I ensure my files are linked correctly?
This is done by including your manifest file in your upload. You can change your file name by editing this manifest file as well. Rackspace recommends using prefixing in your file segments in order to easily map your manifest file to the portions of your large file. For example, you could name your segments as so:
Myfavoritemovie-01 Myfavoritemovie-02 Myfavoritemovie-03 ..etc..
In this case, you would just point your manifest file to the prefix:
Can I use this feature from the Control Panel?
At this time, Rackspace has not implemented this functionality into the Rackspace Cloud Control Panel.
© 2011-2013 Rackspace US, Inc.
Except where otherwise noted, content on this site is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License
See license specifics and DISCLAIMER