Preparations for migrating data from Amazon S3 to Alibaba Cloud OSS - Data Online Migration

This topic describes the preparations required before you migrate data from Amazon Simple Storage Service (S3) to Alibaba Cloud Object Storage Service (OSS).

Step 1: Create an IAM user for migration

Create an Identity and Access Management (IAM) user, grant the user permission to read source data, and create access keys. For more information, see IAM users.

Important

Data Online Migration does not support AWS S3 whitelists. Before you migrate data, remove whitelist restrictions.
The information in the preceding link may be outdated due to changes in the origin server and is for reference only.

Step 2: Create a destination bucket

Create a destination bucket to store the migrated data. For more information, see Create a bucket.

Step 3: Create and authorize a RAM user

Important

This RAM user will be used to create roles and perform migration tasks. For best practice, create this user in the same Alibaba Cloud account as your source or destination bucket.
If you have not created a RAM user, see Quick start: Create a RAM user and grant permissions.

Log on to the RAM console. On the Users page, find your RAM user and click Add Permissions in the Actions column.

System policy: AliyunOSSImportFullAccess (Permissions for managing Online Migration Service).

Custom policy: This policy must include the ram:CreateRole, ram:CreatePolicy, ram:AttachPolicyToRole, and ram:ListRoles permissions.

For more information, see Create a custom policy. The following is an example policy:

{
    "Version":"1",
    "Statement":[
        {
            "Effect":"Allow",
            "Action":[
                "ram:CreateRole",
                "ram:CreatePolicy",
                "ram:AttachPolicyToRole",
                "ram:ListRoles"
            ],
            "Resource":"*"
        }
    ]
}

Step 4: Grant permissions on the destination bucket to a RAM role

Follow the steps that correspond to your destination bucket's ownership.

Same-account destination

Automatic authorization
Use automatic authorization in the Data Online Migration console. For more information, see Step 3: Create a destination data address of the Migrate data topic.
Manual authorization
Note
Use manual authorization in the following scenarios:
- Grant one RAM role permissions on multiple source buckets for centralized management.
- Your account is close to the RAM role limit and you want to avoid creating new roles.
- Automatic authorization is unavailable.
1. Create a RAM role for migration
Log on to the RAM console in the account where you created the RAM user. On the Roles page, click Create Role.
1. Principal Type: Select Cloud Service.
2. Principal Name: Select Data Transport.
3. Role Name: Enter the RAM role name. The RAM role name must be in lowercase.
2. Grant permissions to the RAM role
On the Roles page, find the created RAM role and click Grant Permission in the Actions column.
- Custom policy: Attach a custom policy that includes the oss:List*, oss:Get*, oss:Put*, and oss:AbortMultipartUpload* permissions to the RAM role.
For more information, see Create a custom policy. The following code provides an example custom policy:
Note
Use the following policy as a reference. Replace <myDestBucket> with the destination bucket name.
For more information about RAM policies for OSS, see Common examples of RAM policies.
Important
If the destination bucket uses SSE-KMS encryption, attach the AliyunKMSFullAccess system policy to the RAM role.
```
{
  "Version": "1",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "oss:List*",
        "oss:Get*",
        "oss:Put*",
        "oss:AbortMultipartUpload"
      ],
      "Resource": [
        "acs:oss:*:*:<myDestBucket>",
        "acs:oss:*:*:<myDestBucket>/*"
      ]
    }
  ]
}
```

Cross-account destination

1. Create a RAM role for migration

Log on to the RAM console in the account where you created the RAM user. On the Roles page, click Create Role.

Principal Type: Select Cloud Service.
Principal Name: Select Data Transport.
Role Name: Enter the RAM role name. The RAM role name must be in lowercase.

lQLPKIBPhyQhs7vNAlPNA-mwb_9Zfe8j6sMHtpv2syNfAA_1001_595

2. Grant permissions to the RAM role

Important

Adding a new bucket policy will overwrite any existing policy. Ensure that your new policy includes all statements from the previous policy to avoid breaking existing permissions.

Log on to the OSS console with the Alibaba Cloud account that owns the destination bucket.
In the left-side navigation pane, click Buckets. On the Buckets page, click the name of the destination bucket.
In the left-side pane of the bucket details page, choose Permission Control > Bucket Policy.
On the Bucket Policy tab, click Add by Syntax, and then click Edit. In the code editor, enter the custom bucket policy and click Save.
- Grant the RAM role permissions to list, read, and write all resources in the destination bucket.

Note

Use the following policy as a reference. Replace the placeholders with your actual values before you use it. For more information about RAM policies for OSS, see Common examples of RAM policies.

<otherDestBucket>: The name of the destination bucket.
<otherUid>: The ID of the Alibaba Cloud account that owns the destination bucket.
<myUid>: The ID of your current Alibaba Cloud account (the one used for migration).
<roleName>: The name of the RAM role you created.

{
  "Version": "1",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "oss:List*",
        "oss:Get*",
        "oss:Put*",
        "oss:AbortMultipartUpload"
      ],
      "Principal": [
         "arn:sts::<myUid>:assumed-role/<roleName>/*"
      ],
      "Resource": [
        "acs:oss:*:<otherUid>:<otherDestBucket>",
        "acs:oss:*:<otherUid>:<otherDestBucket>/*"
      ]
    }
  ]
}

3. Configure a policy for a custom key

If SSE-KMS is configured for the destination bucket, attach the AliyunKMSFullAccess system policy to the RAM role.
If the destination bucket is encrypted with a custom KMS key, you must also configure the key's policy:
1. Log on to the KMS console and find the custom key.
2. On the Key Policy tab of the details page, click Configure Key Policy. In the Key Policy panel, enter the ARN of the RAM role in the Cross-account User field. For more information, see Configure a key policy.

Step 5: Restore archived source data

You must manually restore objects from the Archive storage class before creating a source data address and migration task.
Set the restoration period (the number of days the object stays restored) to be long enough to complete the entire migration. This prevents objects from returning to the archived state mid-migration.
You may be charged for the restoration operation. The fee may be relatively high. For more information about the billing methods, contact the service provider that offers the source bucket.

Note

Data Online Migration will not automatically restore archived objects. Objects that are still archived or in the process of being restored cannot be migrated and will be skipped.

For instructions on how to restore objects in Alibaba Cloud OSS, see Restore Objects.

For instructions on restoring archived objects in an Amazon S3 bucket, see Restoring an archived object.

Important

The information in the preceding link may be outdated due to changes in the origin server and is for reference only.

Step 1: Create an IAM user for migration

Step 2: Create a destination bucket

Step 3: Create and authorize a RAM user

Step 4: Grant permissions on the destination bucket to a RAM role

Same-account destination

Automatic authorization

Manual authorization

1. Create a RAM role for migration

2. Grant permissions to the RAM role

Cross-account destination

1. Create a RAM role for migration

2. Grant permissions to the RAM role

3. Configure a policy for a custom key

Step 5: Restore archived source data