You can use DTS to migrate data from an external Redis instance to an ApsaraDB for Redis instance.

Migrate data from an external Redis instance to an ApsaraDB for Redis instance

You can use DTS to migrate data between two Redis instances. If the source instance is an external Redis instance that supports incremental data synchronization, then data migration does not affect the write operation on your local applications.

This section describes the procedure of using DTS to migrate data from an external Redis instance to an ApsaraDB for Redis instance. You can follow the same procedure to migrate data from an ApsaraDB for Redis instance to another one.

Migration modes

If the source instance is an external Redis instance, both standard migration and incremental data migration are supported. If the source instance is an ApsaraDB for Redis instance, only standard migration is supported. Standard migration and incremental data migration have the following features and restrictions:

  • Standard migration

    DTS migrates all of the existing keys on an external Redis instance to an ApsaraDB for Redis instance.

  • Incremental data migration

    During incremental data migration, updated keys on an external Redis instance are synchronized to an ApsaraDB for Redis instance. After the synchronization process, the ApsaraDB for Redis instance starts to replicate data dynamically from the external Redis instance . By using incremental data migration, an external Redis instance can smoothly migrate data to an ApsaraDB for Redis instance while providing services normally .

Migration features

Redis incremental data migration supports the following commands:

  • APPEND
  • BITOP, BLPOP, BRPOP, BRPOPLPUSH,
  • DECR, DECRBY, DEL,
  • EVAL, EVALSHA, EXEC, EXPIRE, EXPIREAT,
  • FLUSHALL, FLUSHDB,
  • EOADD, GETSET,
  • HDEL, HINCRBY, HINCRBYFLOAT, HMSET, HSET, HSETNX,
  • INCR, INCRBY, INCRBYFLOAT,
  • LINSERT, LPOP, LPUSH, LPUSHX, LREM, LSET, LTRIM,
  • MOVE, MSET, MSETNX, MULTI,
  • PERSIST, PEXPIRE, PEXPIREAT, PFADD, PFMERGE, PSETEX,PUBLISH,
  • RENAME, RENAMENX, RESTORE,RPOP, RPOPLPUSH, RPUSH, RPUSHX,
  • SADD, SDIFFSTORE, SELECT, SET, SETBIT, SETEX, SETNX, SETRANGE, SINTERSTORE, SMOVE, SPOP, SREM, SUNIONSTORE,
  • ZADD, ZINCRBY, ZINTERSTORE, ZREM, ZREMRANGEBYLEX, ZUNIONSTORE, ZREMRANGEBYRANK, ZREMRANGEBYSCORE

Prerequisites

If the source Redis instance is an external Redis instance connected to an Alibaba Cloud VPC through a leased line, or an ApsaraDB for Redis instance in a VPC, then a proxy is required for data forwarding.

To allow DTS to connect to VPC-connected Redis instances or external Redis instances connected to Alibaba Cloud through a leased line, you must select an ECS instance with an EIP assigned, and deploy Nginx on the ECS instance to facilitate Nginx proxy .

The Redis instance and the ECS instance must be connected to the same VPC. For external Redis instances that are connected to Alibaba Cloud through a leased line, the VPC where the ECS instance resides must be an Alibaba Cloud VPC.

The following section describes how to use Nginx to configure forwarding settings, in order to allow the DTS server to connect to a VPC-connected Redis instance.

  1. Deploy Nginx

    Run the following command on the ECS instance to deploy Nginx .

    yum install nginx
  2. Configure Nginx forwarding settings

    After you have installed Nginx, modify the configuration file /etc/nginx/nginx.conf and set the backend to listen to the Redis instance. Comment out HTTP configuration in the configuration file and add TCP configuration. The HTTP configuration to be commented out are as follows:



    Add the following TCP configuration to the configuration file:

    stream{
        upstream backend{
            hash $remote_addr consistent;
             #Set the connection string and port for connecting the backend server to the Redis instance. The connection timeout period is 10 seconds and the maximum number of attempts is three.
            server  r-bp1b294374634044.redis.rds.aliyuncs.com:6379 max_fails=3 fail_timeout=10s;
        }
        server{
            # Nginx connection port
            Listen 3333;
            #Set the timeout period to 20 seconds for connecting Nginx to the backend server.       
            proxy_connect_timeout 20s;
             #If five minutes have passed since the last successful access (connection or read/write) to the backend server, the connection is timed out and disconnected.
            proxy_timeout 5m;
             #Redirect the TCP connection, data reception, and data forwarding to backend server “backend”.
            proxy_pass backend;
        }
    }

    For example, the connection string of the Redis instance that you want to connect to is r-bp1b294374634044.redis.rds.aliyuncs.com:6379, and the Nginx connection port is 3333, set the TCP configuration as follows:



  3. Connect to the Redis instance through the Nginx connection port

    After the preceding configuration has been completed, run Nginx to start the Nginx proxy service.

    If the EIP of ECS instance where Nginx is deployed is 114.55.89.152, you can then directly use redis_cli to connect to the connection port and test whether the proxy service is running normally.



    As shown in the preceding figure, you can connect to the Redis instance through the Nginx connection port.

When you configure a migration task in DTS, if the source instance is a VPC-connected Redis instance or an external Redis instances connected to Alibaba Cloud through a leased line, you can directly specify the Nginx proxy connection address.

After DTS starts supporting VPC, no proxy service will be required for VPC-connected Redis instances or external Redis instances connected to Alibaba Cloud through a leased line. For the schedule of enabling DTS to support VPC, see the relevant notification on the Alibaba Cloud website.

Configure a migration task

After all of the prerequisites are met, you can start to configure a migration task. This example describes how to migrate data from an external Redis instance connected to Alibaba Cloud through a leased line to an ApsaraDB for Redis instance in the Classic network .

  1. Log on to the DTS console and click Create Migration Task in the upper-right corner to configure a migration task.
  2. Enter the instance connection information

    In this step, you must enter the task name, the information for connecting to the external Redis instance, and the information for connecting to the ApsaraDB for Redis instance. The connection information includes the following parameters :

    • Task name

      DTS automatically generates a task name for each task. Task names are not required to be unique. You can modify the task name as required. We recommend that you choose an appropriate task name for your business.

    • Source instance information

      Instance type: Select an external Redis instance with an EIP assigned.

      Instance region: For an external Redis instance, select a region that is geographically closest to the instance. The closer the region to the Redis instance, the higher the migration performance.

      Database type: Select Redis.

      Instance mode: The mode is set to standalone by default. Support for Redis clusters will soon be available.

      Host name or IP address: The connection string of the external Redis instance. If Nginx proxy is configured, set this parameter to the address that is used to connect to Nginx.

      Port: The listening port of the external Redis instance. If Nginx proxy is configured, set this parameter to the Nginx port.

      Database password: The password used to connect to the external Redis instance. This parameter is not required. If the external Redis instance does not have a password, this parameter can be left empty.

    • Target instance information

      Instance type: Redis instance.

      Instance region: The region where the ApsaraDB for Redis instance has been created.

      Redis instance ID: The ID of the target ApsaraDB for Redis instance to which you want to migrate data.

      Database password: The password used to connect to the ApsaraDB for Redis instance.

  3. After you have configured the connection information, click Authorize Whitelist and Enter into Next Step in the lower-right corner to select migration objects.
  4. Select migration objects and migration types

    In this step, you must configure migration objects and migration types.

    • Migration types

    For Redis instances, DTS supports standard migration and incremental data migration.

    If you want to migrate all of the data, select Standard Migration as the migration type.

    If you need to migrate data without affecting the write operation on the source instance, you can select Standard Migration + Incremental Migration as the migration type.

    • Migration objects

    In this step, you must select a database to migrate. Currently, Redis only supports whole-database migration. You must select a database instead of selecting keys.

  5. Pre-check

    Pre-check is required before migration tasks start. A migration task can be started only after the pre-check has been passed. For more information about the pre-check, see Pre-check.

    If the pre-check fails, check the details of the failure by clicking the button to the right of the specific check items in the check list, troubleshoot the issues, and perform a pre-check again.

  6. Start a migration task

    After the pre-check has been passed, you can start the migration task. You can view the migration status and progress in the task list after the task is successfully started.

    After the preceding operations are completed, you have successfully configured a migration task to migrate data from an external Redis instance to an ApsaraDB for Redis instance .

Pre-check

Before DTS starts the migration, it will perform a pre-check. This section describes the pre-check items.

Check item Description Remarks
Source database connectivity Checks the connectivity between the DTS server and the external Redis instance.
  1. Checks whether the information you have entered is correct. If the information is incorrect, modify the information and run the pre-check again.
  2. Checks whether the port allows connections initiated from other servers.
Target database connectivity Checks the connectivity between the DTS server and the target ApsaraDB for RDS instance. Checks whether the information you have entered is correct. If it is incorrect, modify the information and run the pre-check again.
One-to-one check of databases Checks whether data is migrated from multiple databases to one database. Currently, DTS does not support migrating data from multiple databases to one database. If this occurs, modify the task configuration and run the pre-check again.
Checks whether the target database is empty. Checks whether the database that you want to migrate data to on the target Redis instance is empty. If this check fails, delete all keys in the target database and run the pre-check again.
Checks for topological conflicts of incremental migration tasks Checks whether other incremental migration tasks are running on the target Redis instance. If this check fails, terminate other incremental migration tasks and run the pre-check again.