Upload multipart file
Required capabilities:
filesAcl:WRITE
Multipart file upload enables upload of files larger than 5000 MB, using a uniform API on all cloud environments for CDF.
Each file part must be larger than 5 MiB, and smaller than 4000 MiB. The file part for the last uploadURL can be smaller than 5 MiB. Maximum 250 upload URLs can be requested.
The supported maximum size for each file uploaded with the multi-part upload API is therefore 1000 GiB (1.073 TB / 0.976 TiB).
The client should calculate the ideal number of parts depending on predetermined or estimated file size, between 1 and the maximum.
Specify the number of parts in the parts URL query parameter. The parts parameter is required.
The request creates metadata information for a new file, and returns in addition to the file id, also a uploadId, and a list of uploadUrls for uploading the file contents.
To upload a file, send an HTTP PUT request to each of the uploadUrls, with the corresponding part of the file in the request body.
You may use a ‘Content-Length’ header in the PUT request for each part, but this is not required.
A failed part PUT upload can be retried.
If ‘contentEncoding’ is specified in the files/initmultipartupload request body, the full file contents must be compressed with that encoding before uploading the parts as byte ranges of the full content. When the file is later downloaded through a download link, the response will include a ‘Content-Encoding’ header set to this value, and the client is responsible for decompressing the content accordingly.
The client must ensure that the parts of the source file are stored in the correct order, using the order of the uploadUrls as specified in the response.
This to avoid ending up with a corrupt final file.
The parts can optionally be uploaded in parallel, preferably on a subset of parts at a time, for example maximum 3 concurrent PUT operations.
Once all file parts have been uploaded, the client should call the ‘files/completemultipartupload’ endpoint,
with the required file ID (as id or externalId) and uploadId fields in the request body. This will assemble the parts into one file.
The file’s uploaded flag will then eventually be set to true.
A standard sequence of calls to upload a large file with multipart upload would be for example as follows:
POST files/initmultipartupload?parts=8, to start a multipart upload session with 8 parts. Optionally specify thecontentEncodingrequest body property if the file content is compressed with a compression encoding. Use standard CDF project request headers such asAuthorizationandContent-Typefor thisPOSTrequest. Also optionally add aContent-Encodingrequest header to thisPOSTrequest, but note that it would be for the compressed JSON POST request body content, not for the full file content. Expect a 201 CREATED response code, and a response body with information to be used in thePUT <uploadURL>andPOST files/completemultipartuploadrequests.PUT <uploadUrl>, for each of theuploadUrlsin the response from thefiles/initmultipartuploadrequest. Expect a 200 OK or 201 CREATED response for each PUT request. NOTE: You can set theContent-Lengthrequest header for thisPUTrequest. Do not addAuthorization,Content-TypeorContent-Encodingheaders in thePUT <uploadURL>requests for the multipart upload based requests.POST files/completemultipartupload, with request body{ "id":123456789, "uploadId":"ABCD4321EFGH" }. This will assemble the file. Use standard CDF project POST request headers such asAuthorization,Content-Typefor this request. Expect a 200 OK response.
Consider verifying that the file is eventually marked as uploaded with a call to the getFileByInternalId endpoint.
NOTE: The uploadUrls expires after one week. A file that does not have the file content parts uploaded and completed within one week will be automatically deleted.
Request throttling
Please note that this endpoint is subject to the new throttling policy, which imposes limits on both request rate and concurrency. For more details, please refer to the Files resource documentation.
Authorizations
Access token issued by the CDF project's configured identity provider. Access token must be an OpenID Connect token, and the project must be configured to accept OpenID Connect tokens. Use a header key of 'Authorization' with a value of 'Bearer $accesstoken'. The token can be obtained through any flow supported by the identity provider.
Query Parameters
If 'overwrite' is set to true, and the POST body content specifies a 'externalId' field, fields for the file found for externalId can be overwritten. The default setting is false.
If metadata is included in the request body, all of the original metadata will be overwritten. The actual file will be overwritten after a successful upload. If there is no successful upload, the current file contents will be kept.
File-Asset mappings only change if explicitly stated in the assetIds field of the POST json body. Do not set assetIds in request body if you want to keep the current file-asset mappings.
The 'parts' parameter specifies how many uploadURLs should be returned, for uploading the file contents in parts. See main endpoint description for more details.
1 <= x <= 250Body
Fields to be set for the file.
Name of the file.
256The external ID provided by the client. Must be unique for the resource type.
255"my.known.id"
Directory containing the file. Must be an absolute, unix-style path.
512The source of the file.
128File type. E.g. text/plain, application/pdf, ..
256"image/jpeg"
Custom, application specific metadata. String key -> String value. Limits: Maximum length of key is 128 bytes, value 10240 bytes, up to 256 key-value pairs, of total size at most 10240.
1 - 1000 elementsA server-generated ID for the object.
1 <= x <= 9007199254740991The dataSet Id for the item.
1 <= x <= 9007199254740991The number of milliseconds since 00:00:00 Thursday, 1 January 1970, Coordinated Universal Time (UTC), minus leap seconds.
x >= 01730204346000
The number of milliseconds since 00:00:00 Thursday, 1 January 1970, Coordinated Universal Time (UTC), minus leap seconds.
x >= 01730204346000
The security category IDs required to access this file.
100A server-generated ID for the object.
1 <= x <= 9007199254740991A list of the labels associated with this resource item.
10Geographic metadata.
The content encoding used to compress the file contents. When set, the file contents must be compressed with the specified encoding before being uploaded through the returned upload URL(s). When the file is later downloaded through a download link, the response will include a 'Content-Encoding' header set to this value, and the client is responsible for decompressing the content accordingly.
gzip, br, deflate, identity, zstd, compress, x-gzip, x-compress Response
The response for a successful request to initiate upload of multiple parts for a file.