Object Storage Service:Multipart copy using OSS SDK for Python 2.0

Usage notes

• The sample code in this topic uses the region ID cn-hangzhou of the China (Hangzhou) region. By default, a public endpoint is used to access resources in a bucket. If you want to access resources in the bucket from other Alibaba Cloud services in the same region in which the bucket is located, use an internal endpoint. For more information about the regions and endpoints that OSS supports, see Regions and endpoints.

• To copy an object, you must have the read permissions on the source object and read and write permissions on the destination bucket.

• The source bucket and destination bucket must be located in the same region. For example, objects in a bucket located in the China (Hangzhou) region cannot be copied to a bucket located in the China (Qingdao) region.

• Make sure that no retention policies are configured for the source bucket and the destination bucket. Otherwise, the following error message is returned: The object you specified is immutable.

Method

upload_part_copy(request: UploadPartCopyRequest, **kwargs) → UploadPartCopyResult

Request parameters

Response parameters

For the complete definition of the multipart copy method, see upload_part_copy.

Multipart copy process

To upload a local file by using multipart upload, perform the following steps:

1. Initiate a multipart upload task.

   Use the client.initiate_multipart_upload method to obtain a unique upload ID in OSS.

2. Upload parts.

   Use the client.upload_part_copy method to obtain a unique upload ID in OSS.

   Note

   • For parts that are uploaded by running a multipart upload task with a specific upload ID, the part numbers identify their relative positions in an object. If you upload a part and reuse its part number to upload another part, the new part overwrites the original part.

   • OSS includes the MD5 hash of part data in the ETag header and returns the MD5 hash to the user.

   • OSS calculates the MD5 hash of the uploaded parts and compares the MD5 hash with the MD5 hash that is calculated by using OSS SDK for Harmony. If the two hashes are different, OSS returns the InvalidDigest error code.

3. Completes the multipart upload task.

   Combine the uploaded parts into an object by using the client.complete_multipart_upload method.

Sample code

The following sample code provides an example on how to copy an object from a source bucket to a destination bucket by using multipart copy and combine the parts into a complete object:

import argparse
import alibabacloud_oss_v2 as oss

# Create a command line argument parser and describe the purpose of the script: synchronous multipart copy and upload sample
parser = argparse.ArgumentParser(description="upload part copy synchronously sample")

# Add the --region command line parameter, which indicates the region where the bucket is located. This parameter is required.
parser.add_argument('--region', help='The region in which the bucket is located.', required=True)
# Add the --bucket command line parameter, which indicates the name of the destination bucket to which the object is to be uploaded. This parameter is required.
parser.add_argument('--bucket', help='The name of the bucket.', required=True)
# Add the --endpoint command line parameter, which indicates the domain name that other services can use to access OSS. This parameter is optional.
parser.add_argument('--endpoint', help='The domain names that other services can use to access OSS')
# Add the --key command line parameter, which indicates the key of the destination object in OSS. This parameter is required.
parser.add_argument('--key', help='The name of the object.', required=True)
# Add the --source_bucket command line parameter, which indicates the name of the bucket where the source object is located. This parameter is required.
parser.add_argument('--source_bucket', help='The name of the source bucket.', required=True)
# Add the --source_key command line parameter, which indicates the key of the source object in OSS. This parameter is required.
parser.add_argument('--source_key', help='The name of the source object.', required=True)

def main():
    # Parse the parameters provided on the command line to obtain the values entered by the user.
    args = parser.parse_args()

    # Load the authentication information required to access OSS from environment variables for identity verification.
    credentials_provider = oss.credentials.EnvironmentVariableCredentialsProvider()

    # Use the default configurations of the SDK to create a configuration object and set the authentication provider.
    cfg = oss.config.load_default()
    cfg.credentials_provider = credentials_provider
    cfg.region = args.region

    # If a custom endpoint is provided, update the endpoint property in the configuration object.
    if args.endpoint is not None:
        cfg.endpoint = args.endpoint

    # Use the preceding configurations to initialize the OSS client and prepare for interaction with OSS.
    client = oss.Client(cfg)

    # Obtain the metadata of the source object.
    result_meta = client.get_object_meta(oss.GetObjectMetaRequest(
        bucket=args.source_bucket,
        key=args.source_key,
    ))

    # Initialize a multipart upload request and return an upload ID to identify the process.
    result = client.initiate_multipart_upload(oss.InitiateMultipartUploadRequest(
        bucket=args.bucket,
        key=args.key,
    ))

    # Define the size of each part. In this example, the size is set to 1 MB.
    part_size = 1024 * 1024
    total_size = result_meta.content_length  # The total size of the source file.
    part_number = 1  # The part number starts from 1.
    upload_parts = []  # Used to store the information of uploaded parts.
    offset = 0  # The current byte offset.

    # Loop until all data is uploaded.
    while offset < total_size:
        num_to_upload = min(part_size, total_size - offset)  # Calculate the amount of data to be uploaded this time.
        end = offset + num_to_upload - 1  # Determine the end position.

        # Perform the actual multipart copy and upload operation.
        up_result = client.upload_part_copy(oss.UploadPartCopyRequest(
            bucket=args.bucket,
            key=args.key,
            upload_id=result.upload_id,
            part_number=part_number,
            source_bucket=args.source_bucket,
            source_key=args.source_key,
            source_range=f'bytes={offset}-{end}',  # Specify the range in the source object.
        ))

        # Print the status information of the uploaded part.
        print(f'status code: {up_result.status_code},'
              f' request id: {up_result.request_id},'
              f' part number: {part_number},'
              f' last modified: {up_result.last_modified},'
              f' etag: {up_result.etag},'
              f' source version id: {up_result.source_version_id}'
        )

        # Record the information of the successfully uploaded part.
        upload_parts.append(oss.UploadPart(part_number=part_number, etag=up_result.etag))
        offset += num_to_upload  # Update the offset.
        part_number += 1  # Update the part number.

    # Sort all uploaded parts by part number.
    parts = sorted(upload_parts, key=lambda p: p.part_number)

    # Send a request to the OSS service to complete the multipart upload.
    result = client.complete_multipart_upload(oss.CompleteMultipartUploadRequest(
        bucket=args.bucket,
        key=args.key,
        upload_id=result.upload_id,
        complete_multipart_upload=oss.CompleteMultipartUpload(
            parts=parts
        )
    ))

    # Print the detailed results after the upload is complete.
    print(f'status code: {result.status_code},'
          f' request id: {result.request_id},'
          f' bucket: {result.bucket},'
          f' key: {result.key},'
          f' location: {result.location},'
          f' etag: {result.etag},'
          f' encoding type: {result.encoding_type},'
          f' hash crc64: {result.hash_crc64},'
          f' version id: {result.version_id}'
    )

# When this script is directly executed, call the main function to start processing the logic.
if __name__ == "__main__":
    main()  # The entry point of the script. The program flow starts from here.

Reference

• For the complete sample code for multipart copy, see upload_part_copy.py.