Copy local data - Data Transport - Alibaba Cloud Documentation Center

This topic describes how to place an order for a desired Data Transport III device, execute a data migration task, view the logs of the data migration task, read data from a data source, and store data in the Data Transport III device.

Notes

Note the following items when you migrate data using Data Transport:

When you specify the source data address from which the data is migrated to the Data Transport III device, you must enter an absolute path in the Directory To Be Migrated field. The path must start and end with a forward slash (/) and cannot contain environment variables or special characters. Make sure that the path you specify in the Directory To Be Migrated field is valid and exists.
When you use the Data Transport III device to migrate data, the resources in the source data address are used. This may affect the continuity of your business. To ensure business continuity, we recommend that you enable throttling for your migration task on the source or run your migration task during off-peak hours.
If two files with the same name exist in the source data address and destination Object Storage Service (OSS) bucket, the Overwrite All mode is used. This way, the file in the source data address overwrites the file in the destination OSS bucket. If the two files with the same name have different content, you must change the names of the files or back up the files.

By default, the Data Transport III device retains the last modified time attribute of a source file. If lifecycle rules are configured for the destination OSS bucket and the source file is within the validity period of the lifecycle rules after migration, the file may be deleted or converted to a file of a specific storage class within the validity period.
- If you do not need to retain the last modification time attribute, you can contact Alibaba Cloud technical support to perform the corresponding configuration. Make sure that you are fully aware of the risks associated with not retaining the last modified time attribute.

Limits

Character device files, block device files, socket files, and pipeline files at the source data address cannot be migrated.
If the source data address contains symbolic links, the symbolic links are migrated only when the transfer_symlink parameter is set to true during data migration. Otherwise, the symbolic links are not migrated. For more information, see Symbolic links.
After a hard link at the source data address is migrated to the destination data address, the hard link becomes a regular file.
The attributes of parent directories cannot be migrated.
File permissions, such as Set User ID (SUID), Set Group ID (SGID), and Super Block ID (SBID) cannot be migrated.
The name of the directory to be migrated combined with the name of a file under the directory cannot exceed 1,024 bytes in length. The names of directories at each level cannot exceed 200 bytes in length and the names of the files under these directories cannot exceed 255 bytes in length. If the length limits are exceeded, the corresponding files cannot be migrated.
The attributes that can be migrated include Permissions -> X-Oss-Meta-Perms (OSS), Uid:Gid -> X-Oss-Meta-Owner (OSS), mtime -> X-Oss-Meta-Mtime (OSS), atime -> X-Oss-Meta-Atime (OSS), and ctime -> X-Oss-Meta-Ctime (OSS).
Note
- The permissions include nine permissions, such as read, write, and execute.
- Uid indicates the user ID. Gid indicates the ID of the group to which the user belongs. The two parts are separated by a colon (:).
Attributes that cannot be migrated include but are not limited to AccessTime, Attr, and Acl.
Note
The attributes that cannot be migrated include but are not limited to the preceding attributes. Whether other attributes can be migrated is unknown. The actual migration results prevail.

Preparations

1. Decrypt and mount a storage pool

After a Data Transport III device is powered on, the built-in storage pool is locked. You can use the initial secret to decrypt the storage pool or change the initial secret by creating a new secret. Then, you can use the new secret to decrypt the storage pool. To obtain the initial secret, contact Alibaba Cloud technical support.

Run the following command to decrypt the storage pool:
- Run the crypt open <password> command.
Perform the following steps to create a new secret and use the new secret to decrypt the storage pool:
- Run the crypt addsecret <old_password> <new_password> command.
- To delete the initial secret, run the crypt deletesecret <old_password> command.
- To open the storage pool with the new secret, run the crypt open <new_password> command.

Note

We recommend that you specify a new secret that contains letters, digits, and special characters, and must be at least eight characters in length. Example: Hello@12345!.

2. Initialize mgwserver

Important

Only a single bucket can be created in a Data Transport III device. To create multiple buckets, contact Alibaba Cloud technical support.

Create a bucket, an AccessKey ID, and an AccessKey secret for the current data migration task.

Check if the mgwserver service is running correctly. Run the mgwserver status command. If the status is abnormal, contact Alibaba Cloud technical support.
Create a bucket and set the AccessKey ID and AccessKey secret. Run the mgwserver create-bucket <Bucket> <AK> <SK> command.
Note
- Bucket: the name of the bucket. We recommend that you use the name of the OSS bucket as the bucket name. Example: xxx-xxx-data.
- AK: the AccessKey ID. It can contain lowercase letters, digits, hyphens (-), and underscores (_). Example: aliyuncs@2024-xx-xx.
- SK: the AccessKey secret. It can contain lowercase letters, digits, hyphens (-), and underscores (_). Example: xxxxxxxx.

Create migration tasks

Important

You must specify the level of the directory from which you want to migrate data. To migrate all data from the source data address, you need to create only one migration task. If you want to migrate data from N directories in the source data address, you must create N migration tasks. Confirm the number of migration tasks to be created and determine the number of job.cfg files to be copied based on the number of migration tasks. Prepare one job.cfg file for each migration task.

Configure migration task files

Run the command cdmgwclient && cd conf to switch to the specified directory.
Determine the number of tasks to create and copy the corresponding number of demo-job.properties files. Each task requires one demo-job.properties file.
Note
- Distinguish the demo-job.properties files. For example, you can specify different file names, such as job1.properties and job2.properties.
- If you want to migrate data in all directories, you need to create only one migration task.
- If you migrate data in specific directories, you must create one migration task for each directory. In this example, the job1.properties file is used to demonstrate related operations.

Open one of the demo-job.properties files, such as the job1.properties file, and configure the parameters described in the following table.

Parameter	Required	Example value
job_name	Yes	The name of the migration task. Example: example_job.
src_path	Yes	The source data address. It must start and end with a forward slash (/). For example: If the source path is `/mnt/nas/example_dir/`, a file named example.jpg exists in this path, and the destination path is set to `mgw:example_bucket/dest/`, the full path of the file example.jpg after migration is `mgw:example_bucket/dest/example.jpg`.
dest_path	Yes	The destination storage path in Data Transport. The format is `mgw:BucketName/prefix/`. For example: `mgw:example_bucket/dest/`. BucketName: the name of the bucket in which the mgwserver server is initialized. prefix: the prefix of the bucket. The prefix must end with a forward slash (/). Specify a prefix: For example, if the source path is `/mnt/nas/example_dir/`, a file named example.jpg exists in this path, and the destination bucket prefix is set to `example/dest/`, the full path of the file example.jpg after migration is `mgw:example_bucket/example/dest/example.jpg`. Do not specify a prefix: For example, if the source path is `/mnt/nas/example_dir/`, a file named example.jpg exists in this path, and no prefix is set for the destination bucket, the full path of the file after migration is `mgw:example_bucket/example.jpg`.
transfer_dir	Yes	Specifies whether to migrate directories. If you set this parameter to true, directories are migrated. If you set this parameter to false, directories are not migrated. When this parameter is set to true, all directories scanned at the source data address are to be migrated. In addition, the statistics about the directories are included in the log file of the migration task. An empty object whose name ends with a forward slash (/) is created in the destination OSS bucket, and the attributes of the source directories that can be migrated are described in the user metadata of the destination object. When this parameter is set to false, all directories scanned at the source data address are ignored, and the statistics about the directories are excluded from the log file of the migration task. An empty object whose name ends with a forward slash (/) is not created in the destination OSS bucket.
transfer_symlink	Yes	Specifies whether to migrate symbolic links. If this parameter is set to true, symbolic links are migrated. If this parameter is set to false, symbolic links are not migrated. When this parameter is set to true, all the symbolic links at the source data address are to be migrated. In addition, the statistics about symbolic links are included in the values of the Files and Volume of Stored Data parameters of the migration task. The corresponding symbolic links are created in the destination OSS bucket and the attributes of the source directories that can be migrated are described in the user metadata of the symbolic links. If the value of the Target attribute of a symbolic link is an absolute path, the value is first parsed into the shortest equivalent absolute path (AbsTarget) based on the source directory in which the symbolic link is stored. If AbsTarget contains the prefix of the source data address, the prefix is replaced with the prefix of the destination data address. The value after the replacement is the value of the Target attribute of the corresponding symbolic link created in the destination OSS bucket. Note For example, the prefix of the source data address of a migration task is /mnt/nas/example_dir/, The `prefix` in `dest_path` is `example/dest/`, and `/mnt/nas/example_dir/links/a.lnk` is a symbolic link in the source path. Assume that its `Target` property is: The Target value of the source symbolic link is ../data/./a.txt. The value is parsed into the shortest absolute path /mnt/nas/example_dir/data/a.txt. Then, /mnt/nas/example_dir/ in the shortest absolute path is replaced with /example/dest/. Therefore, the final Target attribute value of the corresponding destination symbolic link is /example/dest/data/a.txt. The Target attribute value of the source symbolic link is /mnt/nas/example_dir/verbose/../data/./a.txt. The value is parsed into the shortest absolute path /mnt/nas/example_dir/data/a.txt. Then, /mnt/nas/example_dir/ in the shortest absolute path is replaced with /example/dest/. Therefore, the final Target attribute value of the corresponding destination symbolic link is /example/dest/data/a.txt. If this parameter is set to false, all the symbolic links at the source data address are ignored. In addition, the statistics about symbolic links are excluded from the values of the Files and Volume of Stored Data parameters of the migration task. Important Regardless of whether you migrate or ignore the symbolic links at the source data address, the files and directories to which the symbolic links point are migrated only when the files and directories are included in the data to be migrated.
transfers	No	The number of concurrent transfers for each task. The default value is 16. For large file migration, we recommend that you use the default value. For small file migration, we recommend that you set this parameter to 256. The maximum value is 256.
log_level	No	The log level. The default value is INFO. If you need more detailed logs, you can set this parameter to DEBUG.

Start a migration task

Important

Before you perform the following operations, make sure to run the cd mgwclient command to switch to the correct directory.

Run the bash console.sh start conf/properties_file_name command to start the task. Example: bash console.sh start conf/job1.properties.

View the status of a migration task

Important

All commands related to bash console.sh must be run after you switch to the directory by running the cd mgwclient command.

To view the status of a migration task, perform the following steps:

Run the bash console.sh status <job_name> command to view the task status. Replace <job_name> with the actual task name.
1. If the value of JobStatus is running, files are being migrated and the migration task is in progress. You can view the migration progress as shown in the following figure.
2. If the value of JobStatus is failed, specific files failed to be migrated. You can troubleshoot the issues based on the logs or contact Data Transport technical support for troubleshooting.
3. If the value of JobStatus is succeed, all files are migrated. You can view the total amount of data and the total number of files that have been migrated, as shown in the following figure.

Note

The following items describe the response parameters:

The first Transferred parameter: The values included in this parameter respectively indicate the amount of data scanned and the amount of data migrated, the percentage of migrated data, and the average migration speed.
The second Transferred parameter: The values included in this parameter respectively indicate the number of files scanned and the number of files migrated, and the percentage of migrated files.
Checks: the number of skipped files. This parameter is returned only when files are skipped. If specific files that failed to be migrated need to be migrated again after the first migration task is complete, the files that are migrated are counted in the skipped files.
Errors: the number of files that failed to be migrated. This parameter is returned only when specific files failed to be migrated.
Elapsed time: the total migration duration.
ScanComplete: indicates whether the scanning of data or files is complete. Valid values: true and false.

Warning

If you want to migrate data from many directories, data scanning may require an extended period of time to complete and the migration process changes slowly.

Delete a migration task

If a migration task is started unexpectedly, you can run the following command to delete the migration task and the directory in which the task logs are stored:

Run the bash console.sh delete <job_name> command to delete the specified task. Replace <job_name> with the actual task name. Files that have already been migrated are not deleted.

What to do next

Important

The following operations are performed in the /home/aliyuncs/log/mgwclient/job_name directory.

If the status of the migration task is failed, you can troubleshoot the issues based on the logs or contact Data Transport technical support for troubleshooting.
If the status of the migration task is succeed, you can verify the data consistency between the source and destination data addresses based on the CRC-64 checksum.

View logs

Important

In the following paths and file names, replace job_name with the actual task name.

The /home/aliyuncs/log/mgwclient/job_name directory stores log files that record the file upload status and the task status.

Log file name	Description
job_name-status.log	Records the process of the migration task.
job_name-folder.log	Records the directories that are migrated.
job_name-transfers-error.log	Records the files that failed to be migrated.
job_name-crc64-succeed.log	Records the files that are migrated and passed CRC-64 verification.
job_name-crc64-error.log	Records the files that failed CRC-64 verification.
job_name-crc64-skipped.log	Records the files that are skipped during the migration.

Important

Files that failed to be migrated and failed CRC-64 verification are counted in the total number of failed files.

Task progress logs

The job_name-status.log file records basic information about the migration task progress.
- The first Transferred parameter: The values included in this parameter respectively indicate the amount of data scanned and the amount of data migrated, the percentage of migrated data, and the average migration speed.
- The second Transferred parameter: The values included in this parameter respectively indicate the number of files scanned and the number of files migrated, and the percentage of migrated files.
- Checks: the number of skipped files. This parameter is returned only when files are skipped. If specific files that failed to be migrated need to be migrated again after the first migration task is complete, the files that are migrated are counted in the skipped files.
- Errors: the number of files that failed to be migrated. This parameter is returned only when specific files failed to be migrated.
- Elapsed time: the total migration duration.
- ScanComplete: indicates whether the scanning of data or files is complete. Valid values: true and false.

Migration directory logs

The job_name-folder.log file records logs for uploaded directories. This log is generated only if transfer_dir is set to true in the task configuration file. A log entry is created for each successfully uploaded directory. The file records the following information:
- Column 1: the names of directory paths.

Migration failure logs

The job_name-transfers-error.log file records files that failed to migrate. The log is updated in real time until the task is finished. The file records the following information:
- Column 1: the names of file paths.
- Column 2: migration failure information.

Logs of migrated files that passed CRC-64 verification

The job_name-crc64-succeed.log file records the list of CRC-64 values for successfully migrated files. The log is updated in real time until the task is finished. The file records the following information:
- Column 1: the CRC-64 values of files.
- Column 2: the names of file paths.

Logs of files that failed to be migrated and did not pass CRC-64 verification

The job_name-crc64-error.log file records the list of CRC-64 values for files that failed migration. The log is updated in real time until the task is finished. The file records the following information:
- Column 1: the CRC-64 values of files.
- Column 2: the names of file paths.

Logs of files that are skipped during migration

The job_name-crc64-skipped.log file records skipped files in the task. The log is updated in real time until the task is finished. The file records the following information:
- Column 1: the names of file paths.