All Products
Search
Document Center

Data Online Migration:Migrate data

Last Updated:Feb 04, 2024

This topic describes how to migrate data from Object Storage Service (OSS) to a Network Attached Storage (NAS) file system.

Usage notes

  • Data Online Migration accesses the source data by using the public interfaces provided by the storage service provider of the source data. The access behavior depends on the interface implementation of the storage service provider.

  • When Data Online Migration is used for migration, it consumes resources at the source and destination data addresses. This may affect your business. To ensure business continuity, we recommend that you enable throttling for your migration tasks or run them during off-peak hours after careful assessment.

  • Before a migration task starts, Data Online Migration checks the files at the source data address and the destination data address. If a file at the source data address and a file at the destination data address have the same name, and the File Overwrite Method parameter of the migration task is set to Yes, the file at the destination data address is overwritten during migration. If the two files contain different information and the file at the destination data address needs to be retained, we recommend that you change the name of one file or back up the file at the destination data address.

Step 1: Create a source data address

  1. Log on to the Data Online Migration console as the Resource Access Management (RAM) user that you created.

    Important

    If you want to migrate data across accounts, we recommend that you log on to the console by using the account to which the destination NAS file system belongs. This makes it easier to create a NAS data address.

  2. In the left-side navigation pane, choose Data Online Migration > Data Address. On the page that appears, click Create Data Address.

  3. In the Create Data Address panel, configure the parameters and click OK. The following table describes the parameters.

    Parameter

    Required

    Description

    Data Type

    Yes

    The type of the source data address. Select OSS.

    Data Name

    Yes

    The name of the source data address. The name must be 3 to 63 characters in length. The name cannot contain special characters except for hyphens (-) and underscores (_).

    Data Region

    Yes

    The region in which the source data address resides. Example: China (Zhangjiakou).

    Enable Transfer Acceleration

    No

    If you want to use Data Online Migration together with the transfer acceleration feature of Object Storage Service (OSS), enable the transfer acceleration feature for buckets. The transfer acceleration feature takes effect within 30 minutes after it is enabled. After you enable the feature, wait for 30 minutes before you create a migration job.

    Important

    If you use the transfer acceleration feature to access OSS buckets, you are charged additional transfer acceleration fees. For more information, see Enable transfer acceleration.

    OSS Endpoint

    Yes

    The endpoint of the source data address. Select an endpoint based on the region in which the source data address resides. For more information, see Regions and endpoints.

    Important

    If you select an internal endpoint for the OSS data address and use the OSS data address as the source address when you create a migration job, you can select only a NAS or OSS data address in the same region as the destination address.

    AccessKey Id

    Yes

    The AccessKey pair of the RAM user that you created. The AccessKey pair is used for OSS to check whether the RAM user has the permissions to read the source data.

    Note

    If you want to migrate data across accounts, enter the AccessKey pair of the RAM user that is created by the source Alibaba Cloud account.

    AccessKey Secret

    Yes

    OSS Bucket

    Yes

    The OSS bucket in which the data to be migrated is stored.

    OSS Prefix

    No

    The prefix of the source data address.

    • If you specify a prefix, data under the specified path is migrated. The value cannot start with a forward slash (/) and must end with a forward slash (/). Example: data/to/oss/.

    • If you do not specify a prefix, all data in the source bucket is migrated.

Step 2: Create a destination data address

  1. In the left-side navigation pane, choose Data Online Migration > Data Address. On the page that appears, click Create Data Address.

  2. In the Create Data Address panel, configure the parameters and click OK. The following table describes the parameters.

    Parameter

    Required

    Description

    Data Type

    Yes

    The type of the destination data address. Select NAS.

    Data Name

    Yes

    The name of the source data address. The name must be 3 to 63 characters in length. The name cannot contain special characters except for hyphens (-) and underscores (_).

    Data Region

    Yes

    The region in which the NAS file system resides.

    NAS Type

    Yes

    The type of the NAS file system. Select Alibaba Cloud.

    File System

    Yes

    The NAS file system to which the data is migrated.

    Mount Point

    Yes

    The mount target of the NAS file system.

    Important

    Only the NAS file systems whose mount target resides in a virtual private cloud (VPC) rather than the classic network are supported.

    Sub Folder

    No

    The subdirectory in which the migrated data is stored. If you specify a subdirectory, the migrated data is stored in the specified subdirectory. If you do not specify a subdirectory, the migrated data is stored in the root directory (/) by default.

    Important

    If you specify a subdirectory, make sure that the subdirectory exists on the NAS server. Otherwise, the data address fails to be created.

    Note

    If you select an Extreme NAS file system, the value of this parameter starts with share/ by default. If you select another type of file system, you can enter a value of your choice. If you select a file system that uses the Server Message Block (SMB) protocol, the value starts with myshare/ by default.

    Note

    For information about the status of a data address, see the View the status of a data address section of this topic.

Step 3: Create a migration job

  1. In the left-side navigation pane, choose Data Online Migration > Migration Jobs. On the page that appears, click Create Job.

  2. In the Create Job panel, read the terms of the migration service and select I agree to the above terms and activate the Data Transport service. Then, click Next.

  3. In the Fee Reminder message, click Yes, Go Ahead.

  4. In the Job Config step, configure the parameters and click Next.

    Parameter

    Required

    Description

    Job Name

    Yes

    The name of the migration job. The name must be 3 to 63 characters in length and can contain only lowercase letters, digits, and hyphens (-). It cannot start or end with a hyphen (-).

    Source Data Address

    Yes

    The source data address that you created.

    Destination Data Address

    Yes

    The destination data address that you created.

    Specified Directory

    No

    The directories in which the files and subdirectories are migrated or not migrated. Valid values:

    • Do not filter: migrates all data from the source data address.

    • Exclude: does not migrate the files or subdirectories in the specified directories.

      For example, you want to migrate all the directories in the root_dir/ directory except root_dir/bad_sub1/ and root_dir/bad_sub2/. In this case, you can select this method and specify bad_sub1/ and bad_sub2/.

    • Contain: migrates only the files and subdirectories in the specified directories.

      For example, you want to migrate only root_dir/good_sub1/ and root_dir/good_sub2/ in the root_dir/ directory. In this case, you can select this method and specify good_sub1/ and good_sub2/.

    Note
    • A directory name cannot start with a forward slash (/) or a backslash (\), and cannot contain double forward slashes (//), double periods (..), or double quotation marks ("). The character string that consists of all the specified directory names cannot exceed 10 KB in length.

    • A directory name must end with a forward slash (/). Example: docs/.

    • You can specify up to 20 directories that you want to include or exclude.

    Migration Type

    Yes

    The data migration mode. Valid values:

    • Full: The system migrates all data from the source data address to the destination data address. You must configure the Start Time Point of File parameter. The system migrates all data whose last modification time is later than the specified time. After all data is migrated, the migration job ends.

      If you change the data at the source data address after the migration job is complete, you can submit another job to migrate all data. In this case, the system migrates only the changed data.

    • Incremental: The system runs a migration job based on the values that you specified for the Migration Interval and Migration Times parameters.

      • You must configure the Start Time Point of File parameter. During the first migration, the system migrates all data whose last modification time is later than the specified time. After the first migration is complete, incremental migration is performed based on the value that you specified for the Migration Interval parameter. For each incremental migration, the system migrates only the data that is created or modified after the previous migration starts and before the current migration starts.

      • If you set the Migration Times parameter to N, full migration is performed once and then incremental migration is performed (N - 1) times.

        For example, the current time is 08:00, March 10, 2019 and you set the Migration Interval parameter to 1, the Migration Times parameter to 5, and the Start Time Point of File parameter to 08:00, March 5, 2019. When the first migration starts, the system migrates all files whose last modification time is between 08:00, March 5, 2019 and 08:00, March 10, 2019. In this example, the first migration requires one hour to complete and the second migration starts at 10:00, March 10, 2019. The duration from 08:00 to 10:00 consists of one hour for the first migration process and one hour for the migration interval. During the second migration, the system migrates the files whose last modification time is between 08:00, March 10, 2019 and 10:00, March 10, 2019. The migration job consists of one full migration and four incremental migrations.

    Important

    Before full or incremental migration starts, the system compares the files at the source data address with the files at the destination data address. If a file at the destination data address has the same name as a file at the source data address, the file at the destination data address is overwritten when one of the following conditions is met:

    • The Content-Type values of the files at the source data address and the destination data address are different.

    • The last modification time of the file at the source data address is later than that of the file at the destination data address.

    • The size of the file at the source data address is different from that of the file at the destination data address.

    Start Time Point of File

    Yes

    The time to filter data to be migrated. The system migrates the data that is created or modified after the specified time. Valid values:

    • All: migrates all files.

    • Assign: migrates only the files that are created or modified after the specified time.

      For example, if you set the time to 08:00:00, November 1, 2018, only the files that are created or modified after 08:00:00, November 1, 2018 are migrated.

    Migration Interval

    This parameter is required if you set the Migration Type parameter to Incremental.

    The default value is 1, and the maximum value is 24. Unit: hours.

    Migration Times

    This parameter is required if you set the Migration Type parameter to Incremental.

    The default value is 1, and the maximum value is 30.

    File Overwrite Method

    Yes

    The method that is used to process the file at the source data address whose name is the same as that of a file at the destination data address. Valid values:

    • LastModified: If a file at the source data address has the same name as a file at the destination data address, the system compares the last modification time that is indicated by the LastModified property of the files.

      • If the last modification time of the file at the source data address is later than that of the file at the destination data address, the file at the source data address is migrated and the file at the destination data address is overwritten.

      • If the last modification time of the file at the source data address is earlier than that of the file at the destination data address, the file at the source data address is not migrated and the file at the destination data address is retained.

      • If the two files have the same last modification time, the system checks the values of the Size and Content-Type properties of the files.

        If the two files have the same Size and Content-Type values, the file at the source data address is not migrated. If the two files have different values of at least one property, the file at the source data address is migrated and the file at the destination data address is overwritten.

    • Condition: If a file at the source data address has the same name as a file at the destination data address, the system compares the values of the LastModified, Size, and Content-Type properties of the two files.

      • If all property values of the two files are the same, the file at the source data address is not migrated.

      • If the two files have different values of at least one property, the file at the source data address is migrated and the file at the destination data address is overwritten.

    • All: If a file at the source data address has the same name as a file at the destination data address, the system performs no comparison and directly overwrites the file at the destination address with the file at the source data address.

    • No: If a file at the source data address has the same name as a file at the destination data address, the system performs no comparison and directly skips the file at the source data address during the migration.

  5. In the Data Prediction section of the Performance step, configure the Data Size and File Count parameters.

    Note

    To ensure a successful migration job, we recommend that you estimate the volume of data that you want to migrate as accurately as possible. For more information, see the Estimate the amount of data that you want to migrate section of the "Before you begin" topic.

  6. Optional. In the Flow Control section of the Performance step, configure the Time Range and Max Flow(MB/s) parameters and click Add.

    Note

    To ensure business continuity, we recommend that you configure the Time Range and Max Flow(MB/s) parameters based on the workload peaks and troughs.

  7. Click Create and wait until the migration job is complete.

Step 4: Verify data

Data Online Migration solely handles the migration of data and does not ensure data consistency or integrity. After a migration task is complete, you must review all the migrated data and verify the data consistency between the source and destination data addresses.

Warning

Make sure that you verify the migrated data at the destination data address after a migration task is complete. If you delete the data at the source data address before you verify the migrated data at the destination data address, you are liable for the losses and consequences caused by any data loss.

View the status of a data address

After you create a NAS data address, the data address enters one of the following states:

  • Normal: The data address is created.

  • Creating: The data address is being created. The system requires approximately 3 minutes to create the first NAS data address. If the data address remains in the Creating state for an extended period of time, you can click Refresh in the upper-right corner to check the most recent state of the data address.

  • Invalid: An exception has occurred in the process of creating the address. Check your settings and verify whether Data Online Migration can access the NAS service. If the exception persists, submit a ticket.

References

  • For information about how to migrate data from a NAS file system to OSS, see Migrate data.