File Storage NAS:Migrate data to File Storage NAS

Migrate data using an SFTP client

1. Install an SFTP client on your local operating system.

   Various SFTP clients are available. This topic uses FileZilla as an example. Choose a version that is compatible with your local operating system and download and install the SFTP client.

2. Connect the SFTP client to the transit ECS instance.

   1. Open the FileZilla client and configure the parameters as described in the following table. Click Quickconnect.

      After you connect to the ECS server, the left pane displays your local file system, and the right pane displays the server's file system.

      Parameter	Description
      Host	The public IP address of the transit ECS instance. Example: 192.0.2.1.
      Username	The username for the transit ECS instance. This user must have read and write permissions on the NAS file system directory. Example: root. Note The default username for a Linux ECS instance is root or ecs-user. The default username for a Windows ECS instance is administrator.
      Password	The logon password for the transit ECS instance. For example, the logon password for the root user. Note If you forget the password of the transit ECS instance, you can reset it and set a new password. For more information, see Reset the logon password of an instance.
      Port	The SFTP port number. The default is 22.

   2. In the right pane, set the path in the Remote site field to the mount path of the NAS file system, such as /mnt. Press Enter to view the files in the NAS file system.

3. Upload data.

   Drag files or directories from the left pane to the right pane to upload them.

Migrate data using rsync

1. Install rsync.

   • Linux operating systems

     Operating system	Command
     CentOS	Use yum to install rsync. sudo yum install rsync
     Redhat
     Ubuntu	Use apt to install rsync. sudo apt-get install rsync
     Debian

     Note

     If you are using another Linux distribution, use the corresponding package manager to install rsync.

   • macOS

     Download and install the homebrew package manager, and then run the following command to install rsync.

     brew install rsync

   • Windows operating systems

     Download and install the Cygwin environment. You can search for and install rsync during the installation process, or you can manually download, compile, and install rsync.

     Note

     You must open the SSH port (TCP 22) in the VPC security group.

2. Upload data.

   Run the following command to incrementally synchronize and upload a local directory to File Storage NAS.

   rsync -avP DirToSync/ root@192.0.2.0:/mnt/DirToSync/

   Replace the example values in the command with your actual values. The following table describes the parameters.

   Parameter	Description
   DirToSync	The name of the local directory to upload.
   root	The username for the transit ECS instance.
   192.0.2.0	The public IP address of the transit ECS instance.
   /mnt	The mount path of the NAS file system on the transit ECS instance.

   Note

   • The source path in the rsync command must end with a forward slash (/). Otherwise, the data paths will be incorrect after synchronization.

   • To copy data concurrently by using rsync, run the following command:

     threads=<number_of_threads>; 
     src=<source_path/>; 
     dest=<destination_path/>; 
     rsync -av -f"+ */" -f"- *" $src $dest && (cd $src && find . -type f | xargs -n1 -P$threads -I% rsync -av % $dest/% )

     For example, to set the number of threads to 10, the source path to /abc, and the destination path to /mnt1, run the following command:

     threads=10; 
     src=/abc/; 
     dest=/mnt1/; 
     rsync -av -f"+ */" -f"- *" $src $dest && (cd $src && find . -type f | xargs -n1 -P$threads -I% rsync -av % $dest/% )

3. Optional: Configure a scheduled backup task.

   On a local Linux or macOS system, you can use the rsync command with crontab to configure scheduled upload and backup tasks.

   • Linux operating systems

     1. Create a passwordless SSH connection from your local system to the ECS instance. For more information, see Connect to a Linux instance by using a key pair.

        Run the following command to confirm the connection:

        ssh -i ~/.ssh/ecs.pem root@1.2.3.4

        Note

        ~/.ssh/ecs.pem is the path where the key file is stored locally.

     2. Configure crontab.

        Run the crontab -e command to open the editor and configure a scheduled upload task. The following is an example configuration:

        0 23 * * * rsync -av -e "ssh -i ~/.ssh/ecs.pem" ~/Documents/ root@192.0.2.0:/mnt/Documents/

        This crontab configuration automatically backs up the Documents directory of the current user to File Storage NAS at 23:00 every day. Modify the parameters as needed.

   • macOS

     1. Grant disk access permissions to the /usr/sbin/cron directory.

        Go to System Settings, select , click Unlock, and then click +. Select the Macintosh HD directory, press Cmd+Shift+. to show hidden directories, and then select /usr/sbin/cron.

     2. Create a passwordless SSH connection from your local system to the ECS instance. For more information, see Connect to a Linux instance by using a key pair.

        Run the following command to confirm the connection:

        ssh -i ~/.ssh/ecs.pem root@1.2.3.4

        Note

        ~/.ssh/ecs.pem is the path where the key file is stored locally.

     3. Configure crontab.

        Run the crontab -e command to open the editor and configure a scheduled upload task. The following is an example configuration:

        0 23 * * * rsync -av -e "ssh -i ~/.ssh/ecs.pem" ~/Documents/ root@1.2.3.4:/mnt/Documents/

        This crontab configuration automatically backs up the Documents directory of the current user to File Storage NAS at 23:00 every day. Modify the parameters as needed.

   Note

   If you experience poor performance when using rsync to migrate data, use the fpsync tool for a multi-threaded migration. For more information, see Appendix: Migrate data by using the fpsync command-line tool.

Migrate data using Robocopy

Robocopy is a directory replication command-line tool built into Windows. It creates a complete mirror of two directories without copying duplicate files, and preserves all file metadata including date and time stamps.

1. Log on to the transit ECS instance.

2. Migrate data.

   Run the following command to migrate data from the source file system (drive Z) to the destination file system (drive Y):

   robocopy Z:\ Y:\ /e /w:5 /z /mt:32

   Note

   This command migrates only the data within the specified directory, not the directory itself.

   The following table describes important parameters. Replace them with your actual values.

   Parameter	Description
   /mt	The number of concurrent threads. The default is 8. Valid values: 1 to 128. This example uses 32 threads for multi-threaded copying.
   /w	Specifies the wait time in seconds between retries.
   /z	Enables resumable transfer.
   /e	Copies all subdirectories, including empty ones.
   /copyall	Copies all file information. This includes: Data Attributes Timestamps Access Control List (ACL) Owner information Auditing information

   Note

   To accelerate the migration of massive amounts of data, such as more than 100 million small files that exceed 10 TB in total size, you can install the latest Python program on the Windows ECS instance to perform the migration. For more information, see How to accelerate data migration to an SMB file system of Apsara File Storage NAS.

3. Verify the migration result.

   After the migration is complete, run the following Robocopy command to verify that the data in the destination is consistent with the source:

   ROBOCOPY Z:\ Y:\ /e /l /ns /njs /njh /ndl /fp /log:reconcile.txt

   The following table describes important parameters. Replace them with your actual values.

   Parameter	Description
   /e	Lists only directories, including empty ones.
   /l	Reports differences without modifying or copying files.
   /fp	Includes the full paths of files in the log. This is necessary only when /ndl is omitted.
   /ns	Specifies not to include file sizes in the log.
   /ndl	Specifies not to include folders in the log.
   /njs	Specifies not to include the job summary.
   /njh	Specifies not to include the job header.
   /log:reconcile.txt	Writes the migration result to the reconcile.txt log file. If the file already exists, it is overwritten.

Migrate data using IIS FTP

To configure the IIS FTP service on an ECS instance and an FTP client on your local system, see Configure Windows IIS.

Appendix: Migrate data using fpsync

1. Download and install fpsync.

   wget -N https://github.com/martymac/fpart/archive/fpart-1.1.0.tar.gz -P /tmp
   tar -C /tmp/ -xvf /tmp/fpart-1.1.0.tar.gz
   cd /tmp/fpart-fpart-1.1.0
   sudo yum install -y automake libtool
   autoreconf -i
   ./configure
   make
   sudo make install
   sudo yum install parallel -y
   printf "will cite" | parallel --bibtex
   sudo yum install -y rsync

2. Copy the entire file directory.

   fpsync -n 10 -f 10000 /data/src/ /data/dst/

   Note

   For more information about fpsync, see the fpsync tool documentation.

Related documents

If you frequently read and write large amounts of data on NAS from your data center, use a leased line to mount and access the NAS file system directly from the data center. For more information, see Access a file system from an on-premises data center.

If the public network bandwidth of your ECS instance is insufficient for uploading large amounts of data to NAS, consider uploading the data to Object Storage Service (OSS) first and then migrating it to File Storage NAS. For more information, see Upload objects and Implementation of migration.