All Products
Search
Document Center

Migrate from on-premises Redis to ApsaraDB for Redis

Last Updated: Jun 20, 2019

DTS allows you to migrate data from on-premises Redis to ApsaraDB for Redis. With the data change replication function, you can perform the migration without stopping services.

Note: Redis migrations require that the source database support the psync or sync commands. Therefore, currently the source database of Redis migrations must be on-premises, not cloud-based.

Migration types

For data migrations from on-premises Redis to ApsaraDB for Redis, DTS supports the following migration types: migrate existing data and replicate data changes. The features and restrictions of these migration types are as follows:

  • Migrate existing data

    This migrates all existing keys in the source Redis instance to the target Redis instance.

  • Replicate data changes

    If you also choose to replicate data changes, the source instance keys updated during the migration will be synchronized to the target instance, so that eventually the source instance is dynamically synchronous with the target instance. This allows you to perform the migration while on-premises Redis remains operational.

Migration functions

Redis data migrations support the following commands:

  • APPEND
  • BITOP, BLPOP, BRPOP, BRPOPLPUSH
  • DECR, DECRBY, DEL
  • EVAL, EVALSHA, EXEC, EXPIRE, EXPIREAT
  • FLUSHALL, FLUSHDB
  • GEOADD, GETSET
  • HDEL, HINCRBY, HINCRBYFLOAT, HMSET, HSET, HSETNX
  • INCR, INCRBY, INCRBYFLOAT
  • LINSERT, LPOP, LPUSH, LPUSHX, LREM, LSET, LTRIM
  • MOVE, MSET, MSETNX, MULTI
  • PERSIST, PEXPIRE, PEXPIREAT, PFADD, PFMERGE, PSETEX, PUBLISH
  • RENAME, RENAMENX, RESTORE, RPOP, RPOPLPUSH, RPUSH, RPUSHX
  • SADD, SDIFFSTORE, SELECT, SET, SETBIT, SETEX, SETNX, SETRANGE, SINTERSTORE, SMOVE, SPOP, SREM, SUNIONSTORE
  • ZADD, ZINCRBY, ZINTERSTORE, ZREM, ZREMRANGEBYLEX, ZUNIONSTORE, ZREMRANGEBYRANK, ZREMRANGEBYSCORE

Prerequisites

Prerequisites for the source instance

  • DTS can run psync or sync commands on the source instance.
  • The source instance is one of the following: on-premises Redis instance with a public IP address, ECS-based Redis instance, or on-premises Redis instance connected to Alibaba Cloud VPC through dedicated lines (high-speed links) or VPN.

Note: To migrate data from a cloud-based Redis instance, contact Alibaba Cloud technical support.

Prerequisites for the target instance

The target instance is one of the following: ApsaraDB for Redis instance in classic and VPC networks, on-premises Redis instance with a public IP address, ECS-based Redis instance, or on-premises Redis instance connected to Alibaba Cloud VPC through dedicated lines (high-speed links).

Instance access modes

  • If the source Redis instance can directly provide the connection string (format: IP address:Port), DTS can directly access the Redis instance.
  • If the source Redis instance is an on-premises Redis instance connected to Alibaba Cloud VPC through dedicated lines (high-speed links), configure DTS to access the instance through dedicated lines. For more information, see Migrate on-premises databases to Alibaba Cloud through dedicated lines.
  • If the source Redis instance is an on-premises Redis instance connected to Alibaba Cloud VPC through VPN, set up a proxy that forwards data and provides the connection string IP:Port. For more information, see Set up a nginx proxy.

Migration restrictions

  • When calling a Lua script by using EVAL or EVALSHA commands, DTS cannot guarantee the successful script execution because the target does not explicitly return the result of executing the script during incremental data migration.
  • For lists in Redis, since DTS does not flush existing data on the target when retransmission is performed by calling sync or psync, duplicate data may occur.

Set up a nginx proxy

To enable DTS to access an-premises Redis instance connected to Alibaba Cloud VPC through VPN, deploy nginx on an ECS with a public network EIP forward requests.

Note: The VPC where the ECS resides must be the one connected to the on-premises Redis instance.

The following is an example that describes how to use nginx to configure forwarding.

(1) Deploying the nginx proxyOn the ECS server, deploy the nginx proxy using the following command:

  1. yum install nginx

(2) Configuring nginx forwarding

Modify the nginx configuration file /etc/nginx/nginx.conf to set the backend to monitor Redis as follows:

  1. Comment out the following HTTP configurations in the configuration file:

    HTTP Configurations

  2. Add TCP configurations to the configuration file:

    1. stream{
    2. upstream backend{
    3. hash $remote_addr consistent;
    4. # Set the connection string and port number of the back-end Redis. The time-out period is 10 seconds and the maximum number of attempts is 3.
    5. server r-bp1b294374634044.redis.rds.aliyuncs.com:6379 max_fails=3 fail_timeout=10s;
    6. }
    7. server{
    8. # nginx access port
    9. listen 3333;
    10. # Set the time-out period for nginx to connect to the back-end server.
    11. proxy_connect_timeout 20s;
    12. # Disconnect if the previous successful access (connect or read/write) to the back-end server was more than 5 minutes ago.
    13. Proxy_timeout 5m;
    14. # Transfer the TCP connections and data to the backend server called "backend".
    15. proxy_pass backend;
    16. }
    17. }

    For example, if the Redis connection address is r-bp1b294374634044.redis.rds.aliyuncs.com:6379, and the nginx proxy forwarding port is 3333, the TCP configurations are as follows:

    TCP Configuration

(3) Accessing the Redis instance through the forwarding port

Run nginx to start the nginx proxy service.

You can use redis_cli to access the nginx forwarding port and test whether the proxy forwarding is normal. The following example assumes that the EIP of the ECS where nginx resides is 114.55.89.152.

nginx test

As shown in the preceding figure, the Redis instance can be accessed by accessing the nginx proxy forwarding port.

When you create the subsequent DTS migration task for the on-premises Redis instance connected to Alibaba Cloud VPC through VPN, enter the connection address of the nginx proxy.

Create a migration task

When all the preceding prequisites are met, you can begin migrating data. This section describes how to migrate from an on-premises Redis instance with a public IP to an ApsaraDB for Redis instance in a classic network.

  1. Go to the DTS Console and click Create migration task at the upper right corner.

  2. Configure instance connection information.

  • Task name

    DTS generates a task name automatically. The task name is not required to be unique. You can modify it as needed.

  • Source database

    Instance type: Select On-premises databases.
    Instance region: Select the region closest to the source instance.
    Database engine: Select Redis.
    Instance mode: The default value is Single. Cluster instances will be supported later.
    Host name or IP address: Enter the address of the on-premises Redis instance. If nginx proxy forwarding is configured, enter the connection address of the forwarding.
    Port: Enter the listening port number of the on-premises Redis instance. If nginx proxy forwarding is configured, enter the forwarding port number.
    Database password: Enter the password of the on-premises Redis instance. Leave this field blank if the instance does not have a password.

  • Target database
    Instance type: Select Redis Instance.
    Instance region: Select the region where the target instance is located.
    Redis Instance ID: Select the instance ID of the target instance.
    Database password: Enter the password of the target instance.

    connection

After configuring the connection information, click Authorize whitelist and enter into next step at the lower right corner. This adds the DTS server IP address to the IP address whitelist of the target Redis instance; otherwise, the migration will fail because the DTS server cannot access the target Redis instance.

  1. Configure the migration types and objects to be migrated.

    • Migration types

      To migrate only the existing data, select only migrate existing data.

      To perform a migration without stopping services, select both migration types.

    • Objects to be migrated

      Select the databases to be migrated because currently DTS only supports entire database migration for Redis instances.

  2. Start the pre-check.

    DTS performs a pre-check before the migration starts. For more information, see the Pre-check items and rectification methods section.

    If the pre-check fails, click Failed next to the failure items, solve the problem, and perform the pre-check again.

  3. Start the migration task.

    If the pre-check succeeds, start the migration task. You can view the migration status and progress in the task list.

Pre-check items and rectification methods

The following applies to the migrations from on-premises Redis to ApsaraDB for Redis.

Check item Description Rectification method
Source database connectivity check Check the connectivity between the DTS server and the source Redis instance. (1) Correct any incorrect connection information.
(2) Check whether the port allows access from other servers.
Target database connectivity check Check the connectivity between the DTS server and the target MongoDB instance. Correct any incorrect connection information.
Database mapping check Check whether the migration is from multiple databases to one database. Currently DTS does not support migrations from multiple databases to one database. Modify the task configuration.
Empty target database check Check whether the database to be migrated is empty in the target instance. Delete the corresponding database key in the target Redis instance.
Data change replication conflict check Check whether the Redis instance has other ongoing data change replication tasks. End or delete other data change replication tasks.