You can use redis-shake in sync mode to migrate data from a self-managed Redis database to a Tair or ApsaraDB for Redis instance. This method allows you to smoothly migrate data from self-managed Redis databases to Alibaba Cloud.

Prerequisites

  • The self-managed Redis database runs Redis 2.8, 3.0, 4.0, 5.0, 6.0, or 7.0.
  • A Tair instance or an ApsaraDB for Redis instance is created. For more information about how to create an instance, see Step 1: Create an ApsaraDB for Redis instance.
    Note When you create an instance, we recommend that you specify a Redis version that is the same as or later than that of the self-managed Redis database. For example, you can migrate data from a self-managed Redis database that runs Redis 2.8 to an instance that runs Redis 6.0.

    When you migrate data across Redis versions, we recommend that you first create a pay-as-you-go instance to test the compatibility. After you confirm that no compatibility issue exists, you can change the instance into a subscription one. For more information about how to change the billing method of an instance, see Change the billing method to subscription.

redis-shake

redis-shake is an open source Redis tool developed by Alibaba Cloud to transfer and filter data. This tool is flexible, efficient, and easy to deploy. redis-shake is available in three modes: sync, restore, and scan. In sync mode, redis-shake supports full and incremental data migration. In restore mode, redis-shake supports the restoration of local Redis Database (RDB) files to the destination instance. In scan mode, redis-shake runs the SCAN command to obtain the source Redis keys, runs the DUMP command to obtain the key values, and then runs the RESTORE command to restore the keys and their values to the destination instance. The following figure shows the synchronization processes in different modes.
Figure 1. redis-shake modes
redis-shake data migration processes

Precautions

  • If the maxmemory-policy parameter of the destination instance is not set to noeviction, data may be inconsistent between the source database and the destination instance. For more information about data eviction policies, see How does ApsaraDB for Redis evict data by default?
  • If an expiration policy is used on specific keys in the source database, these keys may not be deleted at the earliest opportunity after they expire. Therefore, the number of keys in the destination instance may be less than that in the source database. You can run the info command to view the number of keys in the destination instance.

Procedure

  1. Configure a whitelist based on the device on which you want to install redis-shake.
    Note We recommend that you install redis-shake on the device in which the self-managed Redis database resides to reduce network latency and prevent connection failures caused by the firewall settings of the self-managed Redis database.
    Device on which you want to install redis-shake Operation
    Elastic Compute Service (ECS) instance
    1. Make sure that the ECS instance and the ApsaraDB for Redis instance are deployed in the same VPC. The ECS instance and ApsaraDB for Redis instance must have the same VPC ID.
      Note If they are deployed in different VPCs, you must change the VPC to which the ECS instance belongs. For more information, see Change the VPC of an ECS instance.
    2. Obtain the internal IP address of an ECS instance. For more information, see How do I query IP addresses of ECS instances?
    3. Add the internal IP address of the ECS instance to the whitelist of the ApsaraDB for Redis instance. For more information, see Configure whitelists.
    On-premises device
    1. By default, an ApsaraDB for Redis instance provides only an internal endpoint. You must apply for a public IP address when you want to connect to an ApsaraDB for Redis instance over the Internet. For more information, see Apply for a public endpoint for an ApsaraDB for Redis instance.
    2. Run the curl ipinfo.io |grep ip command on the on-premises host to obtain the public IP address. The returned result is shown in the following figure:View the public IP address
    3. Add the public IP address of the on-premises host to the whitelist of the ApsaraDB for Redis instance. For more information, see Configure whitelists.
  2. Install redis-shake. If it is already installed, skip this step.
    1. Log on to the ECS instance or on-premises device on which you want to install redis-shake.
    2. Run the following command to download the redis-shake package:
      wget 'https://github.com/alibaba/RedisShake/releases/download/v3.1.7/redis-shake-linux-amd64.tar.gz'
      Note In this example, redis-shake v3.1.7 is used. We recommend that you use the latest version of redis-shake. For more information, visit redis-shake-v3.1.7.
    3. Run the following command to decompress the redis-shake package:
      tar xzf redis-shake-linux-amd64.tar.gz
  3. Perform data migration on the device on which redis-shake is installed.
    1. Run the following command in the redis-shake directory to go to the page for modifying the configuration file:
      vim sync.toml
    2. Enter a to enter the edit mode.
    3. Modify the configuration file.
      Table 1. Parameters
      Section Parameter Description Example
      None type

      The redis-shake mode. Set the value to sync.

      sync
      source address The connection address and port number of the source database. Separate the connection address and port number with a colon (:).
      Note
      • If the source database and redis-shake are deployed in the same device, enter 127.0.0.1:6379.
      • If the source database uses the cluster architecture, you must start redis-shake for N nodes. In this case, the address parameter of source is set to the connection address of each node and the address parameter of target is set to the endpoint of the destination instance. If the destination instance uses the standard architecture, cluster architecture, or read/write splitting architecture, the address parameter of target is set to the endpoint of the instance, the endpoint of an instance node, or the endpoint of the master node.
      127.0.0.1:6379
      username

      The account of the source database. If the source database does not use access control lists (ACLs), you do not need to set this parameter.

      ausername
      password

      The account password of the source database. If no password is set for the source database, you do not need to set this parameter.

      Rp829dlwa
      target type The architecture of the destination instance. Valid values: standalone
      address
      The endpoint and port number of the destination instance. Separate the endpoint and port number with a colon (:). For more information about how to obtain the endpoint and port number of an instance, see View endpoints.
      • If the ECS instance and the destination instance reside in the same virtual private cloud (VPC), data can be migrated over the VPC. In this case, you must obtain the VPC endpoint of the destination instance.

        If the ECS instance and the destination instance reside in different regions or VPCs, data can be migrated only over the Internet.

      • If you use an on-premises device, data can be migrated over the Internet. In this case, you must obtain the public endpoint of the destination instance.
      r-bp1wcw2rlw76acc5k****.redis.rds.aliyuncs.com:6379
      password The account and password used to log on to a database of the destination instance. The account must have read and write permissions.
      • If you use the default account whose username is the same as the instance ID, enter the password.
      • If you use a custom account, enter the password in the user:password format. For example, if the username of the custom account is testaccount and the password is Rp829dlwa, enter testaccount:Rp829dlwa as the password.
      For information about how to create a database account, see Create and manage database accounts.
      testaccount:Rp829dlwa
      Note Other parameters do not need to be set in most cases. For more information, see the parameter comments in the sync.toml file.
    4. Press the Esc key to exit the edit mode. Then, enter :wq and press the Enter key to save the configuration file and exit the editor.
    5. Run the following command to start redis-shake. Then, redis-shake starts data migration.
      ./redis-shake sync.toml
      redis-shake displays operational logs on the screen.
      Note For information about the causes and troubleshooting methods of errors, see Common errors and solutions.
  4. Optional: Stop the data migration task.
    Note If you want to run redis-shake for a long time to migrate data in real time, skip this step.
    1. View the displayed logs and wait until the system starts to migrate incremental data.
      Task stage Displayed log
      Full data migration
      2022-10-19 11:02:28 INF RDB AUX fields. key=[repl-offset], value=[0]
      2022-10-19 11:02:28 INF RDB AUX fields. key=[aof-preamble], value=[0]
      2022-10-19 11:02:28 INF RDB resize db. db_size=[9], expire_size=[0]
      2022-10-19 11:02:28 INF start save AOF. address=[127.0.0.1:6379]
      2022-10-19 11:02:28 INF AOFWriter open file. filename=[0.aof]
      2022-10-19 11:02:28 INF send RDB finished. address=[127.0.0.1:6379], repl-stream-db=[0]
      2022-10-19 11:02:29 INF AOFReader open file. aof_filename=[0.aof]
      Note When send RDB finished is displayed in the logs, full data migration is complete and incremental data migration begins.
      Incremental data migration
      2022-10-19 11:02:32 INF syncing aof. allowOps=[2.00], disallowOps=[0.00], entryId=[9], InQueueEntriesCount=[0], unansweredBytesCount=[0]bytes, diff=[0], aofReceivedOffset=[14], aofAppliedOffset=[14]
      2022-10-19 11:02:37 INF syncing aof. allowOps=[0.00], disallowOps=[0.00], entryId=[9], InQueueEntriesCount=[0], unansweredBytesCount=[0]bytes, diff=[0], aofReceivedOffset=[14], aofAppliedOffset=[14]
      2022-10-19 11:02:42 INF syncing aof. allowOps=[0.20], disallowOps=[0.00], entryId=[10], InQueueEntriesCount=[0], unansweredBytesCount=[0]bytes, diff=[0], aofReceivedOffset=[28], aofAppliedOffset=[28]
      2022-10-19 11:02:47 INF syncing aof. allowOps=[0.00], disallowOps=[0.00], entryId=[10], InQueueEntriesCount=[0], unansweredBytesCount=[0]bytes, diff=[0], aofReceivedOffset=[28], aofAppliedOffset=[28]
      The following items describe parameters in the logs:
      • allowOps: the number of commands sent to the destination instance per second.
        Note In most cases, if allowOps is set to 0, data migration is complete and you can stop redis-shake. However, the source database sends PING commands on a regular basis. As such, when data migration is complete, allowOps is not set to 0 on occasion.
      • disallowOps: the number of commands that are filtered out per second.
      • entryId: the total number of commands processed by redis-shake. The value starts from 1.
      • InQueueEntriesCount: the number of commands waiting to be sent.
      Note For information about other parameters, visit redis-shake operational logs.
    2. Stop writing data to the source database, wait until the allowOps parameter in the displayed logs is set to 0 for consecutive times, and then press Ctrl+C to stop redis-shake.
      In this case, the data in the source database is the same as that in the destination instance. You can switch workloads from the self-managed Redis database to the destination instance.