:Preparations

Step 1: Upload the list file

An HTTP/HTTPS list file consists of two components: a manifest.json file that defines the schema, and one or more .csv.gz files that contain the actual file list. Each .csv.gz file must not exceed 100 MB.

1. Create a CSV list file

   Create a CSV-formatted list file on your local machine. Each row represents a file. Rows are separated by a line feed (\n), and columns within each row are separated by commas (,).

   Important

   • Key and Url are required. All other fields are optional.

   • Each row must end with a line feed. Missing line feeds cause CSV parsing failures that interrupt the migration task.

   • Required items

     Name	Required	Description	Notes
     Url	Yes	Data Online Migration uses this URL to download the file content (GET request) and retrieve file metadata (HEAD request). Note The Url must be accessible via commands such as curl --HEAD "$Url" and curl --GET "$Url". Data Online Migration does not support URL redirection.	Both Url and Key must be URL-encoded if they contain special characters. Without encoding, the migration may fail. Url: URL-encode the URL so that it remains accessible by command-line tools such as curl (without redirection). Key: URL-encode the intended object name for the file in OSS. Important After encoding, confirm the following. Otherwise, the migration may fail or the file path at the destination may differ from what you expect. The plus sign (+) in the original string is encoded as %2B. The percent sign (%) in the original string is encoded as %25. The comma (,) in the original string is encoded as %2C. For example, the original string a+b%c,d.file encodes to a%2Bb%25c%2Cd.file.
     Key	Yes	The object name in OSS after migration. The final name is prefix + Key value. For example, if the prefix is backup/ and the Key is images/photo.jpg, the object is stored as backup/images/photo.jpg.

     The following example shows a plain (unencoded) CSV file named plain_example.csv with two columns: Url and Key. The URLs are directly accessible using the curl command, and the Key values are the intended object names in OSS.

     https://help-static-aliyun-doc.aliyuncs.com/assets/img/zh-CN/1354977961/p486238.jpg,assets/img/zh-CN/1354977961/p486238.jpg
     https://www.example-fake1.com/url-that-can-be-accessed-only-after-encoding/123.png,url-that-can-be-accessed-only-after-encoding/123.png
     https://www.example-fake2.com/url-that-can-be-accessed-without-encoding/123.png,url-that-can-be-accessed-without-encoding/123.png
     https://www.example-fake3.com/Chinese/Japanese/Korean/123.png,Chinese/Japanese/Korean/123.png

     Important

     Do not use the built-in Notepad application on Windows to edit manifest.json or plain_example.csv. Notepad may add a byte order mark (BOM) (0xefbbbf) to the first 3 bytes of the file, causing parsing errors in Data Online Migration. On Linux or macOS, run od -c plain_example.csv | less to check whether the BOM is present. On Windows, use Notepad++ or Visual Studio Code instead.

     If your URLs or keys contain special characters, encode them before use. The following Python script reads plain_example.csv line by line and writes the URL-encoded output to example.csv.

     # -*- coding: utf-8 -*-
     import sys
     
     if sys.version_info.major == 3:
         from urllib.parse import quote_plus
     else:
         from urllib import quote_plus
         reload(sys)
         sys.setdefaultencoding("utf-8")
     
     # Source CSV file path.
     src_path = "plain_example.csv"
     # URL-encoded file path.
     out_path = "example.csv"
     
     # The sample CSV contains only two columns: url and key.
     with open(src_path) as fin, open(out_path, "w") as fout:
         for line in fin:
             items = line.strip().split(",")
             url, key = items[0], items[1]
             enc_url = quote_plus(url.encode("utf-8"))
             enc_key = quote_plus(key.encode("utf-8"))
             # The enc_url and enc_key vars are encoded format.
             fout.write(enc_url + "," + enc_key + "\n")
     

     After running the script, example.csv contains the following encoded content:

     https%3A%2F%2Fhelp-static-aliyun-doc.aliyuncs.com%2Fassets%2Fimg%2Fzh-CN%2F1354977961%2Fp486238.jpg,assets%2Fimg%2Fzh-CN%2F1354977961%2Fp486238.jpg
     https%3A%2F%2Fwww.example-fake1.com%2Furl-that-can-be-accessed-only-after-encoding%2F123.png,url-that-can-be-accessed-only-after-encoding%2F123.png
     https%3A%2F%2Fwww.example-fake2.com%2Furl-that-can-be-accessed-without-encoding%2F123.png,url-that-can-be-accessed-without-encoding%2F123.png
     https%3A%2F%2Fwww.example-fake3.com%2FChinese%2FJapanese%2FKorean%2F123.png,Chinese%2FJapanese%2FKorean%2F123.png

   • All items

     Name	Required	Notes
     Key	Yes	The object name in OSS after migration. The final name is prefix + Key value.
     Url	Yes	Data Online Migration uses this URL to download the file content (GET request) and retrieve file metadata (HEAD request).
     Size	No	The size of the file to migrate, in bytes. Note This field is used to calculate storage usage for migrated files. If omitted, the storage usage chart in the console will be unavailable.

     Note

     The order of items is not fixed, but must match the order of fields in the fileSchema field in manifest.json.

2. Compress the CSV file

   Compress each CSV file into a .csv.gz file using gzip.

   • Compress a single file

     To compress a single file named example.csv, run:

     gzip -c example.csv > example.csv.gz

     Note

     The gzip command without the -c flag removes the source file after compression. To keep the source file, use gzip -c <source_file> > <source_file>.gz.

     This produces a .csv.gz file.

   • Compress multiple files

     To compress all CSV files in a directory (for example, example1.csv, example2.csv, and example3.csv in the dir directory), run:

     gzip -r dir

     Note

     The gzip command compresses each file in the directory separately and does not package the directory itself. Source files are not retained.

     This produces example1.csv.gz, example2.csv.gz, and example3.csv.gz in the dir directory.

3. Create the manifest.json file

   The manifest.json file defines the schema and references all CSV list files. Configure the following fields:

   • fileFormat: Set to CSV.

   • fileSchema: Lists the column names in the CSV file, in the same order as the columns.

     Note

     The number of columns in the CSV file must match the number of fields in fileSchema. The migration service validates this at task creation.

   • files: An array of CSV file references. Each entry includes:

     • key: The path to the .csv.gz file in the bucket.

     • MD5checksum: A 16-digit hexadecimal MD5 string (case-insensitive). Example: 91A76757B25C8BE78BC321DEEBA6A5AD. Leave blank to skip the checksum check.

     • size: The size of the .csv.gz file.

   Example:

   {
       "fileFormat":"CSV",
       "fileSchema":"Url, Key",
       "files":[{
           "key":"dir/example1.csv.gz",
           "MD5checksum":"",
           "size":0
       },{
           "key":"dir/example2.csv.gz",
           "MD5checksum":"",
           "size":0
       }]
   }

4. Upload the manifest file to OSS or AWS S3.

   1. Upload manifest.json and all .csv.gz files. The file names in the bucket must match the file names referenced in manifest.json.

   2. Record the path to manifest.json. You will need to specify this path when creating the source data address.

Step 2: Create a destination bucket

Create a destination bucket to store the migrated data. For more information, see Create a bucket.

Step 3: Create a RAM user and grant permissions

Log on to the RAM console with your Alibaba Cloud account. On the Users page, find the RAM user that you created and click Add Permissions in the Actions column.

1. System policy: Permissions for managing Online Migration Service (AliyunOSSImportFullAccess).

2. Custom policy: This policy must include the ram:CreateRole, ram:CreatePolicy, ram:AttachPolicyToRole, and ram:ListRoles permissions.

   For more information about how to manage permissions, see Create a custom policy. The following is a sample policy:

   {
       "Version":"1",
       "Statement":[
           {
               "Effect":"Allow",
               "Action":[
                   "ram:CreateRole",
                   "ram:CreatePolicy",
                   "ram:AttachPolicyToRole",
                   "ram:ListRoles"
               ],
               "Resource":"*"
           }
       ]
   }

Step 4: Grant permissions on the manifest bucket

Instructions vary depending on whether the inventory bucket belongs to your account.

Step 5: Grant permissions on the destination bucket

Follow the steps that correspond to your destination bucket's ownership.

{
  "Version": "1",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "oss:List*",
        "oss:Get*",
        "oss:Put*",
        "oss:AbortMultipartUpload"
      ],
      "Principal": [
         "arn:sts::<myUid>:assumed-role/<roleName>/*"
      ],
      "Resource": [
        "acs:oss:*:<otherUid>:<otherDestBucket>",
        "acs:oss:*:<otherUid>:<otherDestBucket>/*"
      ]
    }
  ]
}