You can use the sync mode of redis-shake to migrate data from an ApsaraDB for Redis instance to a user-created Redis database in real time. This enables you to migrate the data of a user-created Redis database to the cloud.

Prerequisites

  • The version of the user-created Redis database is Redis 2.8, 3.0, 3.2, 4.0, or 5.0.
    Note If you want to migrate data between different versions of Redis databases, for example, from Redis 2.8 to Redis 4.0, we recommend that you create a pay-as-you-go ApsaraDB for Redis instance to verify the compatibility. If the compatibility verification is passed, you can convert the billing method of the instance to subscription. For more information about how to convert the billing method, see Public bandwidth.
  • The ApsaraDB for Redis instance must meet the following requirements:
    • The instance is of Community Edition (Redis 4.0 or 5.0) or Enhanced Edition (Redis 5.0).
    • The instance is of a standard or cluster architecture.
    Note If you have not created any instances, see Step 1: Create an ApsaraDB for Redis instance.

Redis-shake introduction

Redis-shake is an open source Linux-based tool developed by Alibaba Cloud. You can use this flexible and efficient tool to parse (decode mode), restore (restore mode), back up (dump mode), and synchronize (sync or rump mode) data on ApsaraDB for Redis instances. The sync mode of redis-shake supports full data migration and incremental data migration. The detailed migration process is shown in the following figure:
Figure 1. Redis-shake data migration diagram
Redis-shake data migration diagram
Note The data synchronization feature of Data Transmission Service (DTS) supports more data synchronization use cases. For more information, see Overview.

Procedure

  1. Perform the following operations based on redis-shake installation location:
    Note We recommend that you install redis-shake on the host where the user-created database resides. This reduces network latency and prevents connection failures caused by the firewall settings of the user-created database.
    Redis-shake installation location Operations
    ECS instances
    1. Make sure that the ECS instance and the ApsaraDB for Redis instance are deployed in the same VPC. The ECS instance and ApsaraDB for Redis instance must have the same VPC ID.
      Note If they are deployed in different VPCs, you must change the VPC to which the ECS instance belongs. For more information, see Change the VPC of an ECS instance.
    2. Obtain the internal IP address of an ECS instance. For more information, see How do I query IP addresses of ECS instances?
    3. Add the internal IP address of the ECS instance to the whitelist of the ApsaraDB for Redis instance. For more information, see Set IP address whitelists.
    On-premises host
    1. By default, an ApsaraDB for Redis instance provides only an internal endpoint. You must apply for a public IP address when you want to connect to an ApsaraDB for Redis instance over the Internet. For more information, see Apply for a public endpoint for an ApsaraDB for Redis instance.
    2. Run the curl ipinfo.io |grep ip command on the on-premises host to obtain the public IP address. The returned result is shown in the following figure:View the public IP address
    3. Add the public IP address of the on-premises host to the whitelist of the ApsaraDB for Redis instance. For more information, see Set IP address whitelists.
  2. Install redis-shake.
    1. Log on to the host where redis-shake is to be installed, such as an ECS instance or an on-premises host.
    2. Run the following command to download the redis-shake file.
      wget 'http://docs-aliyun.cn-hangzhou.oss.aliyun-inc.com/assets/attach/120287/cn_zh/1608173646665/redis-shake-v2.0.3.tar.gz'
      Note This topic uses version 2.0.3 as an example to demonstrate the operation process. You can also install other redis-shake versions. For more information, see RedisShake.
    3. Run the following command to decompress the redis-shake file:
      tar xzf redis-shake-v2.0.3.tar.gz
  3. Migrate data on the host where redis-shake is installed.
    1. Run the following command to access the decompressed directory and modify the configuration file:
      cd redis-shake-v2.0.3/ && vim redis-shake.conf
      Note After the command is executed, the system will open an editing interface. Enter a to enter the editing mode.
      Table 1. Parameters
      Parameter Description Example
      parallel (optional) The number of concurrent threads for redis-shake to perform the migration. You can modify this value to improve synchronization performance.
      Note The default value is 32. The minimum value is 1. The maximum value depends on the performance of the server where redis-shake resides.
      32
      source.type (required) Select based on the architecture of the source database (user-created database). Valid values:
      • standalone: master/replica architecture.
      • cluster: cluster architecture.
      standalone
      source.address (required) The connection address and port number of the source database. Separate the connection address and port number with a colon (:).
      Note
      • If the source database and redis-shake are deployed on the same host, enter 127.0.0.1:6379.
      • If the source database is a cluster instance, you must add the prefix master@ to the connection address, for example, master@127.0.0.1:6379.
      127.0.0.1:6379
      source.password_raw (optional) The password of the source database. If no password is set for the destination database, you do not need to specify a password. Rp829dlwa
      target.type (required) Select based on the architecture of the destination ApsaraDB for Redis instance. Valid values: standalone
      target.address (required) The connection address and port number of the destination ApsaraDB for Redis instance. Separate the connection address and port number with a colon (:). For more information about how to obtain the connection address and port number of the ApsaraDB for Redis instance, see View endpoints.
      • The ECS instance is connected through a virtual private cloud (VPC): You must obtain the private endpoint of the ApsaraDB for Redis instance.
      • The on-premises host is connected to the ApsaraDB for Redis instance over the Internet: you must obtain the public endpoint of the ApsaraDB for Redis instance.
      Note If the destination instance is of cluster architecture, you must connect to the instance deployed in a VPC by using a private endpoint. You must add the prefix master@ to the connection endpoint, for example, master@r-bp1mfnrflszg75w****.redis.rds.aliyuncs.com:6379. For more information about how to request a private endpoint, see View or apply for an endpoint.
      r-bp1wcw2rlw76acc5k****.redis.rds.aliyuncs.com:6379
      target.password_raw (required) The account with read and write permissions and password of the destination ApsaraDB for Redis instance. Separate the account and password with a colon (:). For more information about how to create an account, see Create accounts and databases for an ApsaraDB RDS for MySQL instance. testaccount:Rp829dlwa
      target.db (optional) Migrates data from all databases in the source database to the specified databases of the destination database. Valid values: 0 to 15.
      Note The default value -1 indicates that the feature is disabled.
      -1
      key_exists (optional) The data writing policy that is adopted when the keys in the source database are the same as those in the destination database. Valid values:
      • rewrite: specifies whether to overwrite the existing keys in the destination database that are the same as those in the source database.
      • none: This is the default value. Redis-shake stops running and a message appears that indicates conflicting keys.
      • ignore: skips the current key, retains the data in the destination database, and continues the data migration.
      rewrite
      filter.db.whitelist (optional) The names of the databases to be migrated. Separate multiple database names with semicolons (;). The default value null indicates that all databases are to be migrated. 0;1
      filter.db.blacklist (optional) The names of databases that do not need to be migrated. This is a blacklist. Separate multiple database names with semicolons (;). By default, this parameter is not specified and no blacklist is configured. 1;2
      Note You do not need to configure other parameters unless otherwise required. For more information, see the comments on each parameter in the redis-shake.conf file.
    2. Press the ESC key to exit the editing mode and enter :wq to save and close the file.
    3. Run the following command to start redis-shake and migrate data:
      ./redis-shake.linux -type=sync -conf=redis-shake.conf
      Redis-shake outputs the running log on the screen.
      Note For more information about the causes and solutions for related errors, seeFAQ.
  4. Optional:Stop the data migration task as required.
    Note If you need to run redis-shake for a long time to migrate data in real time, skip this step.
    1. Check the logs and wait until the migration task enters the incremental data migration state.
      Task stage Logs
      Full data migration
      2020/12/16 21:02:36 [INFO] DbSyncer[0] total = 4.00MB -       2.18MB [ 54%]  entry=52199
      2020/12/16 21:02:36 [INFO] DbSyncer[0] total = 4.00MB -       4.00MB [100%]  entry=97531
      2020/12/16 21:02:36 [INFO] DbSyncer[0] sync rdb done
      Note When sync rdb done appears in the logs, full synchronization is completed and incremental synchronization starts.
      Incremental data migration
      2020/12/16 21:03:07 [INFO] DbSyncer[0] sync:  +forwardCommands=5      +filterCommands=0      +writeBytes=5095
      2020/12/16 21:03:08 [INFO] DbSyncer[0] sync:  +forwardCommands=7      +filterCommands=0      +writeBytes=7133
      2020/12/16 21:03:09 [INFO] DbSyncer[0] sync:  +forwardCommands=0      +filterCommands=0      +writeBytes=0
      2020/12/16 21:03:10 [INFO] DbSyncer[0] sync:  +forwardCommands=645    +filterCommands=0      +writeBytes=657255
      2020/12/16 21:03:11 [INFO] DbSyncer[0] sync:  +forwardCommands=28     +filterCommands=0      +writeBytes=28532
      2020/12/16 21:03:12 [INFO] DbSyncer[0] sync:  +forwardCommands=0      +filterCommands=0      +writeBytes=0
      The following parameters in logs are described.
      • forwardCommands: the number of commands that are sent from the source database.
      • filterCommands: the number of commands that are filtered. For example, you set to filter specific databases in the configuration file of redis-shake.
      • writeBytes: the number of bytes that are sent from the source database.
    2. Pause writing data to the source database and wait until the value of the writeBytes parameter in the returned log remains 0 for multiple times in succession, and use Ctrl+C to stop running redis-shake.
      Figure 2. Stop running example
      Stop running example
      Note In this case, the data in the source databases and destination databases is consistent. You can switch the database service from the on-premises Redis database to the ApsaraDB for Redis instance.