File Storage NAS:Migrate data using the rsync tool

Prerequisites

You have an NFS file system with a mount target in a virtual private cloud (VPC).

Billing

Migrating data between NAS file systems incurs the following costs:

• If you use an Elastic Compute Service (ECS) instance as an intermediate node for data transfer, you are charged for the ECS instance based on its configuration. For more information about the billing of ECS, see Billing overview.

• You are charged for the storage used by both NAS file systems. We recommend that you purchase resource plans to offset storage usage fees. For more information about the billing of NAS, see Billing overview.

• If you use Cloud Enterprise Network (CEN) to connect VPCs, you are charged for the transit routers and inter-region connections that you use. For more information about the billing of CEN, see Billing.

Preparations

To migrate data between NAS file systems, the Elastic Compute Service (ECS) instance must be able to access both the source and destination file systems. Therefore, you must ensure that both NAS file systems can be accessed from the same Alibaba Cloud virtual private cloud (VPC).

1. View the mount target information of the source file system.

   Before you migrate data, record the mount target information of the source file system and the VPC where the mount target resides. For more information, see Manage mount targets.

   Note

   If your file system has only a classic network mount target, you must create a mount target in a VPC. For more information, see Add a mount target.

2. Configure the mount target of the destination file system.

   • File systems in the same region

     • If the source and destination mount targets are in the same VPC, obtain the destination mount target information and then migrate the data. For more information, see Migrate data.

     • If the source and destination mount targets are not in the same VPC, prepare the mount targets in one of the following ways:

       • Create a new file system in the destination region and zone. A new mount target is automatically created. For more information, see Create a General-purpose NAS file system in the console.

         Note

         When you purchase a pay-as-you-go General-purpose NAS file system (Capacity, Performance, or Premium) that uses the NFS protocol, select the same VPC and vSwitch as the source mount target. A destination mount target is automatically generated. After the file system is created, you can purchase a resource plan to reduce costs.

       • Create a new mount target for an existing file system. For more information, see Add a mount target.

       • Use Cloud Enterprise Network (CEN) to connect the VPCs of the source and destination mount targets. For more information, see Mount a NAS file system across VPCs in the same region using CEN.

   • If the file systems are in different accounts or regions

     If your source and destination mount targets are in different accounts or regions, you must use CEN to connect their VPCs. For more information, see Mount a NAS file system across accounts and regions using CEN.

Migrate data

After you prepare the source and destination mount targets, create a new ECS instance, mount both NFS file systems on the instance, and then use the rsync tool to copy the data. To migrate the data, perform the following steps.

1. Mount the source and destination file systems.

   Important

   We recommend that you purchase a new temporary ECS instance to perform the migration. If you use an existing ECS instance, the migration task may compete for CPU and network bandwidth resources with your running services.

   Log on to the ECS console and click Create Instance. On the Create Instance page, configure the following parameters:

   • Region: Select the region where the source file system is located.

   • Network and Zone: Select the VPC, zone, and vSwitch where the source file system is located.

   • Instance Type: Select an instance type with the minimum specifications.

   • Image: Select CentOS 7.6.

   • Storage: Click Elastic Ephemeral Disk | Apsara File Storage NAS | Dedicated Block Storage Cluster (Optional) and then click Add File Storage to configure the storage. For more information, see the following figure.

     Note

     • If the source and destination mount targets are in the same VPC, you can configure the NAS mount information on the ECS purchase page. After the ECS instance starts, the source and destination NAS file systems are automatically mounted.

     • If the source and destination mount targets are not in the same VPC, region, or account, configure only the source file system on the ECS purchase page. After the ECS instance is created, you must manually mount the destination file system. For more information about how to manually mount the destination file system, see Mount an NFS file system.

     

   After the ECS instance is created and the source and destination NAS file systems are mounted, run the following command to confirm the mount status.

   mount | grep nas.aliyuncs.com

   If the file systems are mounted, output similar to the following is displayed. The source file system is mounted to the /mnt/volumeA directory, and the destination file system is mounted to the /mnt/volumeB directory.

2. Install the migration tools.

   Run the following command to install the migration tools.

   sudo yum install -y rsync tmux

   Note

   • rsync is a tool used to copy data.

   • tmux is a tool that you can use to view the progress.

3. Migrate historical data.

   Run the following command to synchronize the historical data from the source file system to the destination file system.

   tmux
   sudo rsync -avP /mnt/volumeA/ /mnt/volumeB/

   You can also use rsync to concurrently copy and upload data. The command is as follows:

   threads=<Number of threads>; 
   src=<Source path/>; 
   dest=<Destination path/>; 
   rsync -av -f"+ */" -f"- *" $src $dest && (cd $src && find . -type f | xargs -n1 -P$threads -I% rsync -av % $dest/% )

   For example, if the number of threads is 10, the source path is /abc, and the destination path is /mnt1:

   threads=10; 
   src=/abc/; 
   dest=/mnt1/; 
   rsync -av -f"+ */" -f"- *" $src $dest && (cd $src && find . -type f | xargs -n1 -P$threads -I% rsync -av % $dest/% )

   Note

   • The source path in the rsync command must end with a forward slash (/). Otherwise, the data path cannot be matched after synchronization.

   • The tmux command creates a new tmux session. Running rsync in the tmux session lets you view the progress. If the connection to the ECS instance is disconnected during the migration, you can log on to the ECS instance again and run tmux attach to resume the tmux session and continue to monitor the migration progress.

   • In a test on the source file system, it took 320 minutes to migrate one million 100 KiB files with a total size of 100 GiB using rsync.

4. Migrate incremental data.

   If business applications running on other ECS instances write data to the source file system during the historical data migration, you must synchronize the new incremental data after the historical data migration is complete.

   1. Stop business applications.

      To prevent new data from being written, stop the business applications that use the source file system on all ECS instances and containers before you synchronize incremental data.

      Important

      • After you stop the business applications, do not manually delete any data from the source file system. Otherwise, data loss may occur.

      • Perform these operations during off-peak hours. You can run the fuser -mv <dir> command to find the process IDs (PIDs) of the processes that are reading from and writing to the NAS file system.

   2. Synchronize incremental data.

   Run the rsync command to synchronize the incremental data that was generated after the historical data migration started to the destination file system.

   sudo rsync -avP --delete /mnt/volumeA/ /mnt/volumeB/

   The rsync command first scans the source path. Therefore, the process may take a long time to complete even if there is not much incremental data.

   Warning

   The --delete option deletes data from the destination file system that was deleted from the source file system. Use this option with caution to prevent accidental data deletion.

5. Check the migration result.

   After the migration is complete, run the following rsync command to check whether the data on the destination file system is consistent with the data on the source file system.

   sudo rsync -rvn /mnt/volumeA/ /mnt/volumeB/

   If the data is consistent, the following information is displayed, and no file paths are listed.

Switch applications to the new file system

After the data migration is complete, switch your business from the old file system to the new one. To do this, unmount the old file system and mount the new file system on all relevant ECS instances and containers.

• You can mount a NAS file system directly on an ECS instance.

  1. Run the mount | grep nas.aliyuncs.com command to record the existing NAS mount information. Note the local path <dir> to which the NAS file system is mounted.

  2. Run the fuser -mv <dir> command to find the PIDs of the processes that are reading from and writing to the NAS file system, and then run the kill -9 command to stop all of them.

  3. Run the umount <dir> command to unmount the old file system.

  4. Mount the new file system to the original <dir> path. For more information about mount parameters, see Mount an NFS file system.

  5. Start the processes that access the NAS file system and confirm that read and write operations are normal.

  6. Modify the automatic mount information in /etc/fstab to replace the old mount target with the new one.

• You can mount a NAS file system to containers in a Kubernetes cluster.

  1. Modify the existing YAML configuration file for the dynamic or static persistent volume to replace the old mount target with the new one.

  2. Use the modified configuration file to generate a new pod. Confirm that the new file system is mounted on the pod and that read and write operations are normal.

  3. Recycle all pods that use the old file system.

References

• Release a file system instance

• Delete a network instance connection from a CEN instance