Redis-shake is an open source tool developed by Alibaba Cloud to transmit data between ApsaraDB for Redis instances. You can use redis-shake in sync mode to perform one-way synchronization between ApsaraDB for Redis instances that belong to the same or different Alibaba Cloud accounts.

Prerequisites

ApsaraDB for Redis instance Requirements
Source ApsaraDB for Redis instance
  • An ApsaraDB for Redis instance of Community Edition (Redis 4.0 or 5.0) is created.
    Note If you use an instance of Enterprise Edition, you can use the data synchronization feature of Data Transmission Service (DTS) to synchronize data. For more information, see Overview of data synchronization.
  • The instance uses the standard architecture or the cluster architecture.
Destination ApsaraDB for Redis instance
Note
  • For more information about how to create an ApsaraDB for Redis instance, see Step 1: Create an ApsaraDB for Redis instance.
  • You can synchronize data between ApsaraDB for Redis instances of different versions, for example, between Redis 4.0 and Redis 5.0. We recommend that you create a pay-as-you-go ApsaraDB for Redis instance to verify the compatibility. If no compatibility issue is found, you can then change the billing method of the instance to subscription. For more information about how to change the billing method, see Change the billing method to subscription.

Redis-shake introduction

Redis-shake is an open source tool developed by Alibaba Cloud to transmit data between ApsaraDB for Redis instances. You can use this tool to parse (decode mode), restore (restore mode), back up (dump mode), and synchronize (sync or rump mode) data on ApsaraDB for Redis instances. The redis-shake tool in sync mode supports full data and incremental data synchronization. The following figure shows the detailed synchronization process.
Figure 1. Redis-shake synchronization diagram
Redis-shake synchronization diagram
Note Compared with redis-shake, DTS is applicable to more scenarios and supports more features, such as two-way synchronization. For more information, see Overview of data synchronization.

Precautions

  • If the data eviction policy (maxmemory-policy) of the destination database is not set to noeviction, data may become inconsistent between the source and destination databases. For more information about the data eviction policy, see How does ApsaraDB for Redis evict data by default?
  • After you run the info command to query the keys in the destination database, you may find that the destination database contains fewer keys than the source database. This is caused by the key expiration mechanism of Redis. This situation occurs if the source database contains keys that are not deleted upon expiration.
    Note The numbers of keys that do not have a validity period in the source and destination databases are the same.

Scenarios

  • Synchronize data between ApsaraDB for Redis instances that belong to different Alibaba Cloud accounts
  • Synchronize data between ApsaraDB for Redis instances that belong to the same Alibaba Cloud account

Procedure

  1. Perform the following operations based on the location where redis-shake is installed:
    Note
    • In the table in this section, ApsaraDB for Redis instances refer to the source and destination ApsaraDB for Redis instances. Perform the following operations on the source and destination ApsaraDB for Redis instances.
    • We recommend that you install redis-shake on Elastic Compute Service (ECS) instances. You can connect to the source and destination ApsaraDB for Redis instances over a virtual private cloud (VPC) to achieve lower network latency and higher security.
    • To perform synchronization tasks across Alibaba Cloud accounts, we recommend that you deploy redis-shake on the ECS instance that belongs to your Alibaba Cloud account. Then, you can connect to the source ApsaraDB for Redis instance over a VPC. You can connect to the destination ApsaraDB for Redis instance over the Internet.
    Location where redis-shake is installed Operation
    ECS instance (recommended)
    1. Make sure that the ECS instance and the ApsaraDB for Redis instance belong to the same VPC. They share the same VPC ID in the Basic Information section.
      Note
    2. Obtain the internal IP address of the ECS instance. For more information, see How do I query IP addresses of ECS instances.
    3. Add the internal IP address of the ECS instance to the whitelist of the ApsaraDB for Redis instance. For more information, see Configure an IP whitelist.
    On-premises machine
    1. By default, an ApsaraDB for Redis instance provides only an internal endpoint. You must apply for a public endpoint when you want to connect to an ApsaraDB for Redis instance over the Internet. For more information, see Apply for a public endpoint.
    2. Run the curl ipinfo.io |grep ip command on the on-premises machine to obtain the public IP address. The returned result is shown in the following figure.View public IP address results
    3. Add the public IP address of the on-premises machine to the whitelist of the ApsaraDB for Redis instance. For more information, see Configure an IP whitelist.
  2. Install redis-shake.
    1. Log on to the host where you want to install redis-shake. The host may be an ECS instance or an on-premises machine.
    2. Run the following command to download the redis-shake file.
      wget 'http://docs-aliyun.cn-hangzhou.oss.aliyun-inc.com/assets/attach/120287/cn_zh/1608173646665/redis-shake-v2.0.3.tar.gz'
      Note This example shows how to install redis-shake 2.0.3. You can also install redis-shake of other versions. For more information, see RedisShake.
    3. Run the following command to decompress the redis-shake file:
      tar xzf redis-shake-v2.0.3.tar.gz
  3. Synchronize data from the host where redis-shake is installed.
    1. Run the following command to access the directory of the decompressed redis-shake file and modify the configuration file:
      cd redis-shake-v2.0.3/ && vim redis-shake.conf
      Note After the command is run, the system opens an editing interface. Enter a to use editing mode.
      Table 1. Parameters
      Parameter Required Description Example
      source.type Yes Select a value based on the architecture of the source ApsaraDB for Redis instance. Valid values: standalone
      source.address Yes The endpoint and port number of the source ApsaraDB for Redis instance. Separate the endpoint and port number with a colon (:). For more information about how to obtain the endpoint and port number of the ApsaraDB for Redis instance, see View endpoints.
      • If your ECS instance is connected to the ApsaraDB for Redis instance over a VPC, you must obtain the internal endpoint of the ApsaraDB for Redis instance.
      • If an on-premises machine is connected to the ApsaraDB for Redis instance over the Internet, you must obtain the public endpoint of the ApsaraDB for Redis instance.
      Note If the source instance uses the cluster architecture, you must connect to the instance that is deployed in a VPC by using a private endpoint. You must add the master@ prefix to the endpoint. Example: master@r-bp1mfnrflszg75w****.redis.rds.aliyuncs.com:6379. For more information about how to request a private endpoint, see Enable the direct connect mode.
      r-bp1mfnrflszg75w****.redis.rds.aliyuncs.com:6379
      source.password_raw Yes The account that has Copy permissions and the password of the source ApsaraDB for Redis instance. Separate the account and password with a colon (:). For more information about how to create an account, see Create and manage database accounts.
      Note You cannot create accounts that have Copy permissions for instances that use the cluster architecture. To enable this feature, submit a ticket.
      testaccount:Rp829dlwa
      target.type Yes Select a value based on the architecture of the destination ApsaraDB for Redis instance. Valid values: cluster
      target.address Yes The endpoint and port number of the destination ApsaraDB for Redis instance. Separate the endpoint and port number with a colon (:). For more information about how to obtain the endpoint and port number of the ApsaraDB for Redis instance, see .
      • If your ECS instance is connected to the ApsaraDB for Redis instance over a VPC, you must obtain the internal endpoint of the ApsaraDB for Redis instance.
      • If an on-premises machine is connected to the ApsaraDB for Redis instance over the Internet, you must obtain the public endpoint of the ApsaraDB for Redis instance.
      Note If the destination instance uses the cluster architecture, you must connect to the instance that is deployed in a VPC by using a private endpoint. You must add the master@ prefix to the endpoint. Example: master@r-bp1mfnrflszg75w****.redis.rds.aliyuncs.com:6379. For more information about how to request a private endpoint, see Enable the direct connect mode.
      master@r-bp1mfnrflszg75w****.redis.rds.aliyuncs.com:6379
      target.password_raw Yes The account that has read and write permissions and the password of the destination ApsaraDB for Redis instance. Separate the account and password with a colon (:). For more information about how to create an account, see Create and manage database accounts. testaccount:Rp829dlwa
      target.db No Specifies whether to synchronize data from all databases in the source ApsaraDB for Redis instance to the specified databases in the destination ApsaraDB for Redis instance. Valid values:
      • -1: disables the feature. This is the default value.
      • 0 to 255: enables the feature. The value specifies the database of the destination ApsaraDB for Redis instance. For example, a value of 0 specifies that the aggregated data of all databases in the source ApsaraDB for Redis instance is synchronized to database 0 in the destination ApsaraDB for Redis instance.
      Note If the source ApsaraDB for Redis instance uses the standard architecture and the destination ApsaraDB for Redis instance uses the cluster architecture, only database 0 can be synchronized. Data from other databases cannot be synchronized. Set this parameter to 0 to synchronize all the databases from the source ApsaraDB for Redis instance to database 0 in the destination ApsaraDB for Redis instance.
      -1
      key_exists No The writing policy that is applied when the keys in the source database are the same as those in the destination database. Valid values:
      • rewrite: overwrites the existing keys in the destination database that are the same as those in the source database.
      • none: This is the default value. Redis-shake stops running and the system prompts conflicting keys.
      • ignore: skips the current key, retains the data in the destination database, and continues to synchronize data.
      rewrite
      filter.db.whitelist No The names of the databases that you want to synchronize. Separate multiple database names with semicolons (;). By default, this parameter is empty. This specifies that all databases are synchronized. 0;1
      filter.db.blacklist No The names of databases that you want to exclude from the synchronization task. This parameter is equivalent to a blacklist. Separate multiple database names with semicolons (;). By default, this parameter is empty. This specifies that no database is added to the blacklist. 1;2
      parallel No The number of parallel threads that is required for redis-shake to perform synchronization tasks. You can increase this value to improve synchronization performance.
      Note The default value is 32. The minimum value is 1. The maximum value is based on the performance of the server where redis-shake is deployed.
      32
      Note You do not need to configure other parameters unless required. For more information, see the comments on each parameter in the redis-shake.conf file.
    2. Press the Esc key to exit the editing mode and enter :wq to save and close the file.
    3. Run the following command to start redis-shake and synchronize data:
      ./redis-shake.linux -type=sync -conf=redis-shake.conf
      Redis-shake prints the operational log on the screen.
      Note For more information about the causes of related errors and how to fix the errors, see FAQ.
  4. Optional:Stop the data synchronization task as required.
    Note If you want to run redis-shake to synchronize incremental data for long-term services, skip this step.
    1. Check the log data and wait until the synchronization task enters the incremental data synchronization state.
      Synchronization stage Returned log
      Full data synchronization
      2020/12/16 21:02:36 [INFO] DbSyncer[0] total = 4.00MB -       2.18MB [ 54%]  entry=52199
      2020/12/16 21:02:36 [INFO] DbSyncer[0] total = 4.00MB -       4.00MB [100%]  entry=97531
      2020/12/16 21:02:36 [INFO] DbSyncer[0] sync rdb done
      Note If full data synchronization is completed, sync rdb done appears in the log data and incremental data synchronization starts.
      Incremental data synchronization
      2020/12/16 21:03:07 [INFO] DbSyncer[0] sync:  +forwardCommands=5      +filterCommands=0      +writeBytes=5095
      2020/12/16 21:03:08 [INFO] DbSyncer[0] sync:  +forwardCommands=7      +filterCommands=0      +writeBytes=7133
      2020/12/16 21:03:09 [INFO] DbSyncer[0] sync:  +forwardCommands=0      +filterCommands=0      +writeBytes=0
      2020/12/16 21:03:10 [INFO] DbSyncer[0] sync:  +forwardCommands=645    +filterCommands=0      +writeBytes=657255
      2020/12/16 21:03:11 [INFO] DbSyncer[0] sync:  +forwardCommands=28     +filterCommands=0      +writeBytes=28532
      2020/12/16 21:03:12 [INFO] DbSyncer[0] sync:  +forwardCommands=0      +filterCommands=0      +writeBytes=0
      The log contains the following fields:
      • forwardCommands: the number of commands that are sent from the source ApsaraDB for Redis instance.
      • filterCommands: the number of commands that are filtered out. For example, you can configure the configuration file of redis-shake to filter out specific databases.
      • writeBytes: the number of bytes that are sent from the source ApsaraDB for Redis instance.
    2. Stop writing data to the source database. Wait until the value of the writeBytes parameter in the log returns a value of 0 multiple times. Then, press Ctrl+C to stop redis-shake.
      Figure 2. Stop running example
      Stop running example
      Note In this scenario, the data in the source ApsaraDB for Redis instance and the destination ApsaraDB for Redis instance is consistent. You can switch the workloads from the source ApsaraDB for Redis instance to the destination ApsaraDB for Redis instance.