Apsara File Storage NAS:Migrate data by using the Robocopy tool

Billing

The migration between NAS file systems involves the following costs:

• If you use an Elastic Compute Service (ECS) instance as the intermediate node for data transfer, you are charged for the ECS instance based on the instance configuration. For more information about the billing of ECS, see Overview.

• You are charged for the storage usage of the two NAS file systems. We recommend that you purchase resource plans to offset storage usage fees. For more information about the billing of NAS, see Billing overview.

• If you use Cloud Enterprise Network (CEN) to connect VPCs, you are charged for the transit routers and inter-region connections that you use. For more information about the billing of CEN, see Billing rules.

Before you begin

When you migrate data between NAS file systems, the ECS instance must be able to access both the source NAS file system and the destination NAS file system. Therefore, you must make sure that the source and destination NAS file systems can be accessed from the same VPC.

1. View the information about the mount target of the source file system.

   Before you migrate data, you must obtain the information about the mount target of the source file system and the VPC in which the mount target resides. For more information, see View mount targets.

   Note

   If the mount target of your NFS file system is created in the classic network, you must create a mount target in a VPC for the file system. For more information, see Create a mount target.

2. Configure the mount target of the destination file system.

   • The file systems reside in the same region

     • If the mount targets of the destination file system and the source file system belong to the same VPC, you can obtain the information about the mount targets and migrate data. For more information, see Migrate data.

     • If the mount targets of the destination file system and the source file system belong to different VPCs, you can use one of the following methods to prepare the mount targets:

       • Create a file system in the destination region and zone. A mount target is automatically created for the file system. For more information, see Create a General-purpose NAS file system in the NAS console.

         Note

         When you purchase a pay-as-you-go General-purpose NAS file system (Capacity, Performance, or Premium) with the SMB protocol, select the same VPC and vSwitch that are used by the source mount target. A mount target is automatically created for the file system. After you create a file system, you can purchase resource plans to offset the storage usage fees of the file system.

       • Create a mount target for the destination file system. For more information, see Create a mount target.

       • Use CEN to connect the VPCs of the destination mount target and the source mount target. For more information, see Use CEN to mount a NAS file system across VPCs in the same region.

   • The file systems belong to different accounts or reside in different regions

     If the mount targets of the destination file system and the source file system belong to different accounts or reside in different regions, you must use CEN to connect the VPCs. For more information, see Use CEN to mount a NAS file system across accounts and regions.

Migrate data

After you prepare the source and destination mount targets, purchase a new ECS instance, mount the two SMB file systems on the ECS instance, and then use the Robocopy tool to replicate data. Perform the following steps to migrate data:

1. Mount the source and destination file systems on the ECS instance.

   Important

   If you want to migrate data, we recommend that you purchase a new ECS instance. If you use an existing ECS instance to migrate data, the migration task may consume CPU and network bandwidth resources that are allocated to running workloads.

   Log on to the ECS console and click Create Instance. On the Basic Configurations page, configure the following parameters:

   • Region: Select the region and zone in which the source file system resides.

   • Instance Type: Select the minimum specification.

   • Image: Select a Windows Server version. We recommend that you select Windows Server 2019.

   • Storage: Click NAS File System. Then, click Add NAS File System. The following figure shows the detailed configurations.

     Note

     • If the source and destination mount targets reside in the same VPC, you can configure the mount information of NAS file systems in the Storage section of the Basic Configurations page. When you start the ECS instance, the source and destination NAS file systems are automatically mounted on the instance.

     • If the source and destination mount targets belong to different VPCs, regions, or accounts, you only need to configure the source file system on the ECS buy page. After you create an ECS instance, you can manually mount the destination file system on the ECS instance. For more information, see Mount an NFS file system on a Linux ECS instance.

     

   After you create an ECS instance and mount the source and destination NAS file systems on the ECS instance, run the following command to check whether the mount is successful:

   net use

   If the mount is successful, an output that is similar to the following information appears. The source file system is mounted to disk Z, and the destination file system is mounted to disk Y.

   Status     Local      Remote                       Network
   ------------------------------------------------------------------------------
   OK        Y:      \\29e9c24****-eab13.cn-wulanchabu.nas.aliyuncs.com\myshare
                                                MicrosoftWindowgNetwork
   OK        Z:       \\29fe7f4****-txr31.cn-wulanchabu.nas.aliyuncs.com\myshare
                                                MicrosoftWindowgNetwork

2. Migrate data.

   Run the following command to migrate data from the source file system (disk Z) to the destination file system (disk Y):

   robocopy Z:\ Y:\ /e /w:5 /z /mt:32

   Note

   Only the data in the specified directory is migrated. The directory is not migrated.

   The following table describes the parameters. Replace the values of the parameters with the actual values.

   Parameter	Description
   /mt	Specifies the number of concurrent threads. Default value: 8. Valid values: 1 to 128. In this example, 32 threads are used for multi-thread replication.
   /w	Specifies the number of seconds between two consecutive retries caused by errors.
   /z	Enables resumable upload.
   /e	Copies all subdirectories, including empty directories.
   /copyall	Copies all file information. The information includes the following items: Data Attributes Timestamp Access control list (ACL) Owner information Audit information

   Note

   If you want to accelerate the migration of large amounts of data, for example, hundreds of millions of small files larger than 10 TB, you can install the latest Python program on a Windows ECS instance. For more information, see How do I accelerate data migration to an SMB file system?

3. Check the migration result.

   After you complete data migration, run the following command to check whether the data that is stored in the destination file system is consistent with the data that is stored in the source file system.

   ROBOCOPY Z:\ Y:\ /e /l /ns /njs /njh /ndl /fp /log:reconcile.txt

   The following table describes the parameters. Replace the values of the parameters with the actual values.

   • /e: lists only directories, including empty directories.

   • /l: records the differences without modifying or copying files.

   • /fp: includes the full paths of files in logs. This parameter is required only if the /ndl parameter is not specified.

   • /ns: does not include the file size in logs.

   • /ndl: does not include folders in logs.

   • /njs: does not include the job summary.

   • /njh: does not include the job header.

   • /log:reconcile.txt: writes the migration result to the reconcile.txt log file. If the migration result already exists in the log file, the existing content is overwritten.

Switch workloads to the new file system

If you want to switch your current workloads from the original file system to the new file system after you migrate data, unmount the original file system from all ECS instances and containers and mount the new file system on the instances and containers.

• To mount the new file system on an ECS instance, perform the following steps:

  1. Run the net use command to obtain the mount information of the existing NAS file system. Take note of the local drive letter to which the file system is mounted.

  2. Run the following command to unmount the original file system:

     net use Z: /delete

     Replace the drive letter Z: with a drive letter specific to your environment.

     Note

     • You can run the net use * /delete command to unmount each available file system one by one in Windows.

     • You can run the net use * /delete /y command to unmount all the available file systems without any confirmation in Windows.

  3. Mount the new file system to the original drive letter. For more information, see Mount an SMB file system on a Windows ECS instance.

  4. Restart the processes that access the new file system to check whether the processes can perform read and write operations on the file system as expected.

  5. Modify the automatic mount information in the auto_mount.bat file by replacing the original mount target with the new mount target.

• To mount the new file system on a Windows container, perform the following steps:

  1. Modify the YAML configuration file by replacing the original mount target with the new mount target.

  2. Use the modified configuration file to generate new pods. Make sure that the new file system is mounted on the pods and the pods can perform read and write operations on the file system as expected.

  3. Recycle all the pods that use the original file system.

FAQ

How do I accelerate data migration to an SMB file system?

You may need to write large amounts of business data to an SMB file system and accelerate the migration of large amounts of data, for example, hundreds of millions of small files larger than 10 TB (each file about 100 KB). In this case, you can install the latest Python program on a Windows ECS instance. Perform the following steps:

1. Download and install the latest Python program.

2. Set the system environment variable PATH to the Python execution path, for example, C:\Python27.

   set PATH=%PATH%;C:\python27

   You can also run the where python command to view the installation path of Python, as shown in the following figure.

3. Copy the migration.py script to the local directory of the ECS instance, for example, C:\.

   #!/usr/bin/python
   
   import os
   import random
   import string
   import sys, getopt
   import datetime
   import time
   
   def execute_cmd(cmd):
      print('\tExecuting cmd: %s' % cmd)
      count = 0
      rc = 0
      while (count < 3*60):
          rc = os.system(cmd)
          if (rc != 0):
              count += 1
              time.sleep(1)
              continue
          else:
              break
   
      if rc != 0:
          print('\tFailed to execute cmd: %s. rc:%d' %(cmd, rc))
      return
   
   def migrate_subdirs(srcPath, dstPath, rangeBegin, rangeEnd, ignoreFile):
      currTimeStr = datetime.datetime.now().strftime("%Y-%m-%d_%H:%M:%S")
      print('Start to migrate from %s to %s for subdir range[%s, %s] at %s.\n' %(srcPath, dstPath, rangeBegin, rangeEnd, currTimeStr))
      index = 0
   
      for entry in os.listdir(srcPath):
          if os.path.isdir(os.path.join(srcPath, entry)):
              if index >= rangeBegin and index <= rangeEnd:
                  srcSubDir = srcPath + "\\" + entry
                  dstSubDir = dstPath + "\\" + entry
                  print('\tBegin of migrating from the %d th dir %s to %s.' %(index, srcSubDir, dstSubDir))
                  cmd = "robocopy \"" + srcSubDir + "\" \"" + dstSubDir + "\" /e /w:5 /z /mt:32"
                  if ignoreFile.strip():
                      cmd += " /XF \"" + ignoreFile + "\""
                  cmd += " >> robocopy.log"
                  execute_cmd(cmd)
                  print('\tEnd of migrating from %s to %s.\n' %(srcSubDir, dstSubDir))
              index += 1
      print('Finish to migrate from %s to %s for subdir range[%s, %s] at %s.\n' %(srcPath, dstPath, rangeBegin, rangeEnd, datetime.datetime.now().strftime("%Y-%m-%d_%H:%M:%S")))
   
   def migrate_regfiles(srcPath, dstPath, ignoreFile):
      currTimeStr = datetime.datetime.now().strftime("%Y-%m-%d_%H:%M:%S")
      print('Start to migrate from %s to %s for regular files at %s.\n' %(srcPath, dstPath, currTimeStr))
      for entry in os.listdir(srcPath):
          if os.path.isfile(os.path.join(srcPath, entry)):
              print('\tBegin of migrating %s from %s to %s.\n' %(entry, srcPath, dstPath))
              cmd = "attrib -R \"" + dstPath + "\\\\" + entry + "\""
              execute_cmd(cmd)
              cmd = "copy \"" + srcPath + "\\\\" + entry + "\" \"" + dstPath + "\" /Y"
              if ignoreFile.strip():
                  cmd += " /XF \"" + ignoreFile + "\""
              cmd += " >> robocopy.log"
              execute_cmd(cmd)
              print('\tEnd of migrating %s from %s to %s' %(entry, srcPath, dstPath))
      print('Finish to migrate from %s to %s for regular files at %s.\n' %(srcPath, dstPath, datetime.datetime.now().strftime("%Y-%m-%d_%H:%M:%S")))
   
   
   def main(argv):
      srcPath = ''
      dstPath = ''
      range = ''
      ignoreFile = ''
      try:
          opts, args = getopt.getopt(argv,"hs:d:r:i:f",["srcPath=","dstPath=","range=","ignore="])
      except getopt.GetoptError:
          print('migration.py -s <source path> -d <destination path> [-r BeginIndex:EndIndex | -f] [-i ignoredFile]')
          print('example: migration.py -s x:\pic -d z:\pic [-r 0:100]')
          sys.exit(2)
   
      subdironly = False
      fileonly = False
      for opt, arg in opts:
          if opt == '-h':
              print('migration.py -s <source path> -d <destination path> [-r BeginIndex:EndIndex | -f] [-i ignoredFile]')
              sys.exit()
          elif opt in ("-s", "--srcPath"):
              srcPath = arg
          elif opt in ("-d", "--dstPath"):
              dstPath = arg
          elif opt in ("-r", "--range"):
              range = arg
              subdironly = True
          elif opt in ("-f", "--file"):
              fileonly = True
          elif opt in ("-i", "--ignore"):
              ignoreFile = arg
   
      if not srcPath.strip() or not dstPath.strip():
           print('migration.py -s <source path> -d <destination path> [-r BeginIndex:EndIndex | -f] [-i ignoredFile]')
           sys.exit()
   
      if subdironly and fileonly:
           print('migration.py -s <source path> -d <destination path> [-r BeginIndex:EndIndex | -f] [-i ignoredFile]')
   
           sys.exit()
   
      if not fileonly:
          if not range.strip():
               rangeBegin = 0
               rangeEnd = sys.maxsize-1
          else:
               rangeBegin, rangeEnd = (int(x) for x in range.split(":"))
               if rangeBegin < 0 or rangeEnd >= sys.maxsize or rangeEnd < rangeBegin:
                   print('migration.py -s <source path> -d <destination path> [-r BeginIndex:EndIndex | -f] [-i ignoredFile]')
                   print('example: migration.py -s x:\pic -d z:\pic -r 0:99')
                   sys.exit()
          migrate_subdirs(srcPath, dstPath, rangeBegin, rangeEnd, ignoreFile)
   
      if not subdironly:
           migrate_regfiles(srcPath, dstPath, ignoreFile)
   
   if __name__ == "__main__":
   
     main(sys.argv[1:])

4. Migrate data. Command syntax: python ./migration.py -s <source path> -d <destination path> [-r BeginIndex:EndIndex | -f] [-i ignoredFile].

   The following table describes the parameters.

   Parameter	Description
   -s <source path>	The directory of the mount target of the source file system. For example, -s Z:\ indicates that the root directory of the Z drive is specified for the mount target of the source file system.
   -d <destination path>	The directory of the mount target of the destination file system. For example, -d Y:\ indicates that the root directory of the Y drive is specified for the mount target of the destination file system.
   -r BeginIndex:EndIndex	The range of files to be migrated, from BeginIndex to EndIndex. For example, -r 1:100 indicates that the files from 1 to 100 are migrated.
   -f	Specifies that only ordinary files at the first level in the directory of the source file system are migrated. If you want to migrate all files in the source file system, ignore the -r and -f parameters.
   -i ignoredFile	Specifies the files that do not need to be migrated in the directory of the source file system. For example, i ignored.txt indicates that the file named ignored.txt is not migrated.

   Note

   By default, you do not need to specify the -r or -f parameter. The script automatically copies all subdirectories and then copies ordinary files at the first level. If you want to use the cpu/network directory of multiple clients to concurrently accelerate the migration of different subdirectories, and all clients can access the same source disk, you can specify the range of the subdirectories, for example, -r 0:9999 used by the first client and -r 10000:19999 used by the second client.

   For example, you can migrate data from the Z:\ directory of the source file system to the Y:\ directory of the destination file system. Sample command:

   python ./migration.py -s Z:\ -d Y:\

References

• Release a NAS file system

• Migrate data from a NAS file system to an on-premises storage system

• Manage lifecycle policies

• Recycle bin

• Delete a network instance connection