Multipart upload allows you to divide a large file into smaller chunks and upload them in parts to S3 Object Storage. When all parts are successfully uploaded, the storage service assembles them into the completed object.
This is a good method to use if you are uploading large videos, archives, backups or other files. If one part fails, it might be possible to retry that part instead of uploading the entire file again.
Typically, multipart upload involves three phases:
Start upload The app creates a new multipart upload with the storage service.
Split and upload the pieces: The file is divided and uploaded in small pieces one by one. Where supported by the client the parts can be uploaded in parallel .
Upload complete: The application then sends a request for completion when all the required parts have been uploaded, to assemble the parts into the final object.
Once the multipart upload is successfully completed, the uploaded file will be available as a completed object.
Exact steps will depend on your storage panel, S3-compatible client or application.
Ensure your storage service and upload tool supports multipart uploads.
Some storage panels handle large uploads automatically; others require an S3-compatible client, SDK, or application that supports multipart uploads.
Select the file you wish to upload, enter the destination Space and object key.
Before you start, make sure you have the right permissions to upload the object, and that you are uploading it to the correct place.
The upload tool splits the file into small chunks. The best size of part is determined by the file size, connection reliability, and capabilities of the storage service.
Small parts might make for more requests. Bigger parts will reduce the number of requests but might take longer to upload one by one.
Some S3-compatible services also have minimum and maximum part sizes and a maximum number of parts.
The client uploads each piece and tracks its progress.
A supported client can retry the failed parts without starting the whole upload again if the connection is lost. If the network and storage service can efficiently handle multiple requests, parallel uploads can increase throughput.
The client then requests completion following a successful upload of all parts.
wait for the multipart upload to finish Then, make sure the object is in your Space and is downloadable/openable.
Why: unstable connection, timeout, failed part.
What to do Check your network connection and the message of error the upload tool gives you. If the client supports resuming multipart uploads, retry the failed upload or part.
Possible causes The specified part size does not comply with the storage service requirements.
What to do Find out the application-specific limits for multipart uploads and set the part size accordingly in your upload tool.
Possible reasons The credentials do not have permission to upload to the object or to complete a multipart upload.
How to: Check credentials, permissions, destination Space, object key. Do not give out secret access keys while troubleshooting.
Possible reasons: The multipart upload was not completed successfully or the object was uploaded to a different destination.
What to do: Verify the target Space and object key, the final completion response. Make sure the upload tool reported success.
Possible reasons The upload was cancelled before the completion request or it was partially uploaded and then cancelled.
What to do: Verify that the existing upload can be picked up by your client. If it is not you might have to cancel the upload that did not complete and start again. “Do this only when you’re sure that the upload is not needed anymore.
Uploaded parts do not necessarily amount to a usable file until the multipart upload is complete.
For certain clients, the failed parts can be retried without re-uploading all the other parts.
Until they are completed or aborted, incomplete multipart uploads can occupy storage space.
The client uploading must maintain the upload ID and information required to complete the upload.
Part sizes and part count will be different service limits.
If possible, do not start multiple uploads of the same large file. This can result in higher network utilisation and duplicate objects being created.
Multipart upload is useful for large files, poor network connections, or applications that need to retry individual parts if they fail. A regular upload will suffice for small files.
If your storage panel doesn’t have controls for multipart upload, check to see if large uploads are automatically handled, or if you need an S3-compatible client or application.