The sync mode of redis-shake supports full data migration and incremental data migration. The detailed migration process is shown in the following figure.

Prerequisites

The source and destination ApsaraDB for Redis instances must meet the following requirements:
ApsaraDB for Redis instance Requirement
Source ApsaraDB for Redis instance
  • The instance is of Community Edition (Redis 4.0 or 5.0).
    Note If the instance is of Enhanced Edition, you can use the data synchronization feature of Data Transmission Service (DTS) to migrate data. For more information, see Overview.
  • The instance is of a standard architecture or a cluster architecture.
Destination ApsaraDB for Redis instance
Note
  • If you have not created a destination ApsaraDB for Redis instance, see Step 1: Create an ApsaraDB for Redis instance.
  • If you want to migrate data between different versions of Redis databases, for example, from Redis 2.8 to Redis 4.0, we recommend that you create a pay-as-you-go ApsaraDB for Redis instance to verify the compatibility. If the compatibility verification is successful, you can change the billing method of the instance to subscription. For more information about how to change the billing method, see Change the billing method to subscription.

Redis-shake introduction

Redis-shake is an open source tool developed by Alibaba Cloud for Redis data transmission. You can use this flexible and efficient tool to parse (decode mode), restore (restore mode), back up (dump mode), and synchronize (sync or rump mode) data on ApsaraDB for Redis instances. The sync mode of redis-shake supports full data migration and incremental data migration. The detailed migration process is shown in the following figure.
Figure 1. Redis-shake data migration diagram
Redis-shake data migration diagram
Note You can also use the data synchronization feature of Data Transmission Service (DTS) to migrate data. DTS supports more scenarios and features. For more information, see Overview.

Scenarios

  • Migrate data across Alibaba Cloud accounts
  • Migrate data between instances of the same Alibaba Cloud account

Procedure

  1. Perform the following operations based on the redis-shake installation location:
    Note
    • The ApsaraDB for Redis instances mentioned in the following table refer to both the source and destination ApsaraDB for Redis instances. Perform the following operations on both the source and destination ApsaraDB for Redis instances.
    • We recommend that you install redis-shake on Elastic Compute Service (ECS) instances. You can connect to the source and destination ApsaraDB for Redis instances through a virtual private cloud (VPC) to achieve lower network latency and higher security performance.
    • To perform migration tasks across Alibaba Cloud account, we recommend that you deploy redis-shake on an ECS instance within the same Alibaba Cloud account as the source ApsaraDB for Redis instance. Thus, you can connect to the source ApsaraDB for Redis instance through a VPC. You can connect to the destination ApsaraDB for Redis instance over the Internet.
    Redis-shake installation location Operation
    ECS instance (recommended)
    1. Make sure that the ECS instance and the ApsaraDB for Redis instance belong to the same VPC. They share the same VPC ID in the Basic Information section.
      Note
    2. Obtain the internal IP address of the ECS instance. For more information, see How do I query IP addresses of ECS instances.
    3. Add the internal IP address of the ECS instance to the whitelist of the ApsaraDB for Redis instance. For more information, see Configure an IP whitelist.
    On-premises machine
    1. By default, an ApsaraDB for Redis instance provides only an internal endpoint. You must apply for a public endpoint when you want to connect to an ApsaraDB for Redis instance over the Internet. For more information, see Apply for a public endpoint for an ApsaraDB for Redis instance.
    2. Run the curl ipinfo.io |grep ip command on the on-premises machine to obtain the public IP address. The returned result is shown in the following figure.View public IP address results
    3. Add the public IP address of the on-premises machine to the whitelist of the ApsaraDB for Redis instance. For more information, see Configure an IP whitelist.
  2. Install redis-shake.
    1. Log on to the host where redis-shake is to be installed, such as an ECS instance or an on-premises machine.
    2. Run the following command to download the redis-shake file:
      wget 'http://docs-aliyun.cn-hangzhou.oss.aliyun-inc.com/assets/attach/120287/cn_zh/1608173646665/redis-shake-v2.0.3.tar.gz'
      Note This topic uses version 2.0.3 as an example to demonstrate the operation process. You can also install redis-shake of other versions. For more information, see RedisShake.
    3. Run the following command to decompress the redis-shake file:
      tar xzf redis-shake-v2.0.3.tar.gz
  3. Perform data migration tasks on the host where redis-shake is installed.
    1. Run the following command to go to the directory where the redis-shake file is decompressed and modify the configuration file:
      cd redis-shake-v2.0.3/ && vim redis-shake.conf
      Note After the command is run, the system will open an editing interface. Enter a to enter the editing mode.
      Table 1. Parameters
      Parameter Required Description Example
      source.type Yes Select a value based on the architecture of the source ApsaraDB for Redis instance. Valid values: standalone
      source.address Yes The connection address and port number of the source ApsaraDB for Redis instance. Separate the connection address and port number with a colon (:). For more information about how to obtain the connection address and port number of the ApsaraDB for Redis instance, see View endpoints.
      • If you connect an ECS instance to the ApsaraDB for Redis instance through a VPC, you must obtain the internal endpoint of the ApsaraDB for Redis instance in the VPC.
      • If you connect an on-premises machine to the ApsaraDB for Redis instance over the Internet, you must obtain the public endpoint of the ApsaraDB for Redis instance.
      Note If the source ApsaraDB for Redis instance is of cluster architecture, you must connect to the instance deployed in a VPC by using a private endpoint. You must add the master@ prefix to the connection address, for example, master@r-bp1mfnrflszg75w****.redis.rds.aliyuncs.com:6379. For more information about how to request a private endpoint, see Enable a direct connection.
      r-bp1mfnrflszg75w****.redis.rds.aliyuncs.com:6379
      source.password_raw Yes The name of the account with Copy permissions and password of the source ApsaraDB for Redis instance. Separate the account name and password with a colon (:). For more information about how to create an account, see Create and manage database accounts.
      Note Instances of the cluster architecture do not support creating accounts with Copy permissions.To enable this feature, submit a ticket.
      testaccount:Rp829dlwa
      target.type Yes Select a value based on the architecture of the destination ApsaraDB for Redis instance. Valid values: cluster
      target.address Yes The connection address and port number of the destination ApsaraDB for Redis instance. Separate the connection address and port number with a colon (:). For more information about how to obtain the connection address and port number of the ApsaraDB for Redis instance, see View endpoints.
      • If you connect an ECS instance to the ApsaraDB for Redis instance through a VPC, you must obtain the internal endpoint of the ApsaraDB for Redis instance in the VPC.
      • If you connect an on-premises machine to the ApsaraDB for Redis instance over the Internet, you must obtain the public endpoint of the ApsaraDB for Redis instance.
      Note If the destination instance is of cluster architecture, you must connect to the instance deployed in a VPC by using a private endpoint. You must add the master@ prefix to the connection address, for example, master@r-bp1mfnrflszg75w****.redis.rds.aliyuncs.com:6379. For more information about how to request a private endpoint, see Enable a direct connection.
      master@r-bp1mfnrflszg75w****.redis.rds.aliyuncs.com:6379
      target.password_raw Yes The name of the account with read and write permissions and password of the destination ApsaraDB for Redis instance. Separate the account name and password with a colon (:). For more information about how to create an account, see Create and manage database accounts. testaccount:Rp829dlwa
      target.db No Specifies whether to migrate data from all databases in the source ApsaraDB for Redis instance to the specified databases in the destination ApsaraDB for Redis instance. Valid values:
      • -1 (default): disables this feature.
      • 0 to 255: enables this feature. The value specifies the destination database in the destination ApsaraDB for Redis instance. For example, a value of 0 indicates that the data of all databases in the source ApsaraDB for Redis instance is migrated to database 0 in the destination ApsaraDB for Redis instance.
      Note If the source ApsaraDB for Redis instance is of the standard architecture and the destination ApsaraDB for Redis instance is of the cluster architecture, only database 0 is synchronized in this scenario. Data from other databases is not migrated. Set this parameter to 0 to migrate all the databases from the source ApsaraDB for Redis instance to database 0 in the destination ApsaraDB for Redis instance.
      -1
      key_exists No The writing policy that is applied when the keys in the source database are the same as those in the destination database. Valid values:
      • rewrite: overwrites the existing keys in the destination database that are the same as those in the source database.
      • none: This is the default value. Redis-shake stops running and a message appears that indicates conflicting keys.
      • ignore: skips the current key, retains the data in the destination database, and continues migrating data.
      rewrite
      filter.db.whitelist No The names of the databases to be migrated. Separate multiple database names with semicolons (;). The default value is empty, which indicates that all databases are to be migrated. 0;1
      filter.db.blacklist No The names of databases that do not need to be migrated. This is a blacklist. Separate multiple database names with semicolons (;). By default, this parameter is not specified and no blacklist is configured. 1;2
      parallel No The number of concurrent threads for redis-shake to perform the migration tasks. You can specify a large value to improve synchronization performance.
      Note The default value is 32. The minimum value is 1. The maximum value depends on the performance of the server where redis-shake is deployed.
      32
      Note You do not need to configure other parameters unless otherwise required. For more information, see the comments on each parameter in the redis-shake.conf file.
    2. Press the ESC key to exit the editing mode and enter :wq to save and close the file.
    3. Run the following command to start redis-shake and migrate data:
      ./redis-shake.linux -type=sync -conf=redis-shake.conf
      Redis-shake returns the running log on the screen.
      Note For more information about the causes and fixes for related errors, see FAQ.
  4. Optional:Stop the data migration task as required.
    Note If you need to run redis-shake for a long time to migrate data in real time, skip this step.
    1. Check the logs and wait until the migration task enters the incremental data migration state.
      Task stage Logs
      Full data migration
      2020/12/16 21:02:36 [INFO] DbSyncer[0] total = 4.00MB -       2.18MB [ 54%]  entry=52199
      2020/12/16 21:02:36 [INFO] DbSyncer[0] total = 4.00MB -       4.00MB [100%]  entry=97531
      2020/12/16 21:02:36 [INFO] DbSyncer[0] sync rdb done
      Note When sync rdb done appears in the logs, full synchronization is completed and incremental synchronization starts.
      Incremental data migration
      2020/12/16 21:03:07 [INFO] DbSyncer[0] sync:  +forwardCommands=5      +filterCommands=0      +writeBytes=5095
      2020/12/16 21:03:08 [INFO] DbSyncer[0] sync:  +forwardCommands=7      +filterCommands=0      +writeBytes=7133
      2020/12/16 21:03:09 [INFO] DbSyncer[0] sync:  +forwardCommands=0      +filterCommands=0      +writeBytes=0
      2020/12/16 21:03:10 [INFO] DbSyncer[0] sync:  +forwardCommands=645    +filterCommands=0      +writeBytes=657255
      2020/12/16 21:03:11 [INFO] DbSyncer[0] sync:  +forwardCommands=28     +filterCommands=0      +writeBytes=28532
      2020/12/16 21:03:12 [INFO] DbSyncer[0] sync:  +forwardCommands=0      +filterCommands=0      +writeBytes=0
      The logs contain the following parameters:
      • forwardCommands: the number of commands that are sent from the source ApsaraDB for Redis instance.
      • filterCommands: the number of commands that are filtered. For example, you set to filter specific databases in the configuration file of redis-shake.
      • writeBytes: the number of bytes that are sent from the source ApsaraDB for Redis instance.
    2. Stop writing data to the source ApsaraDB for Redis instance and wait until the value of the writeBytes parameter in the returned log remains 0 for multiple times in succession, and then use Ctrl+C to stop the running of redis-shake.
      Figure 2. Stop running example
      Stop running example
      Note The data in the source ApsaraDB for Redis instance and the destination ApsaraDB for Redis instance is now consistent. You can switch the database service from the source ApsaraDB for Redis instance to the destination ApsaraDB for Redis instance.