Use redis-shake to migrate data

Redis-shake introduction

Redis-shake is an open source Linux-based tool developed by Alibaba Cloud. You can use this flexible and efficient tool to parse (decode mode), restore (restore mode), back up (dump mode), and synchronize (sync or rump mode) data on ApsaraDB for Redis instances. The sync mode of redis-shake supports full data migration and incremental data migration. The detailed migration process is shown in the following figure:

Procedure

1. Perform the following operations based on redis-shake installation location:
   Note We recommend that you install redis-shake on the host where the user-created database resides. This reduces network latency and prevents connection failures caused by the firewall settings of the user-created database.

   Redis-shake installation location	Operations
   ECS instances	Make sure that the ECS instance and the ApsaraDB for Redis instance are deployed in the same VPC. The ECS instance and ApsaraDB for Redis instance must have the same VPC ID. Note If they are deployed in different VPCs, you must change the VPC to which the ECS instance belongs. For more information, see Change the VPC of an ECS instance. Obtain the internal IP address of an ECS instance. For more information, see How do I query IP addresses of ECS instances? Add the internal IP address of the ECS instance to the whitelist of the ApsaraDB for Redis instance. For more information, see Set IP address whitelists.
   On-premises host	By default, an ApsaraDB for Redis instance provides only an internal endpoint. You must apply for a public IP address when you want to connect to an ApsaraDB for Redis instance over the Internet. For more information, see Apply for a public endpoint for an ApsaraDB for Redis instance. Run the curl ipinfo.io |grep ip command on the on-premises host to obtain the public IP address. The returned result is shown in the following figure: Add the public IP address of the on-premises host to the whitelist of the ApsaraDB for Redis instance. For more information, see Set IP address whitelists.

2. Install redis-shake.
   1. Log on to the host where redis-shake is to be installed, such as an ECS instance or an on-premises host.
   2. Run the following command to download the redis-shake file.

      wget 'http://docs-aliyun.cn-hangzhou.oss.aliyun-inc.com/assets/attach/120287/cn_zh/1608173646665/redis-shake-v2.0.3.tar.gz'

      Note This topic uses version 2.0.3 as an example to demonstrate the operation process. You can also install other redis-shake versions. For more information, see RedisShake.
   3. Run the following command to decompress the redis-shake file:

      tar xzf redis-shake-v2.0.3.tar.gz

3. Migrate data on the host where redis-shake is installed.
   1. Run the following command to access the decompressed directory and modify the configuration file:

      cd redis-shake-v2.0.3/ && vim redis-shake.conf

      Note After the command is executed, the system will open an editing interface. Enter a to enter the editing mode.

      Table 1. Parameters

      Parameter	Description	Example
      parallel (optional)	The number of concurrent threads for redis-shake to perform the migration. You can modify this value to improve synchronization performance. Note The default value is 32. The minimum value is 1. The maximum value depends on the performance of the server where redis-shake resides.	32
      source.type (required)	Select based on the architecture of the source database (user-created database). Valid values: standalone: master/replica architecture. cluster: cluster architecture.	standalone
      source.address (required)	The connection address and port number of the source database. Separate the connection address and port number with a colon (:). Note If the source database and redis-shake are deployed on the same host, enter 127.0.0.1:6379. If the source database is a cluster instance, you must add the prefix master@ to the connection address, for example, master@127.0.0.1:6379.	127.0.0.1:6379
      source.password_raw (optional)	The password of the source database. If no password is set for the destination database, you do not need to specify a password.	Rp829dlwa
      target.type (required)	Select based on the architecture of the destination ApsaraDB for Redis instance. Valid values: standalone: standard (master/replica) architecture. cluster: cluster architecture.	standalone
      target.address (required)	The connection address and port number of the destination ApsaraDB for Redis instance. Separate the connection address and port number with a colon (:). For more information about how to obtain the connection address and port number of the ApsaraDB for Redis instance, see View endpoints. The ECS instance is connected through a virtual private cloud (VPC): You must obtain the private endpoint of the ApsaraDB for Redis instance. The on-premises host is connected to the ApsaraDB for Redis instance over the Internet: you must obtain the public endpoint of the ApsaraDB for Redis instance. Note If the destination instance is of cluster architecture, you must connect to the instance deployed in a VPC by using a private endpoint. You must add the prefix master@ to the connection endpoint, for example, master@r-bp1mfnrflszg75w****.redis.rds.aliyuncs.com:6379. For more information about how to request a private endpoint, see View or apply for an endpoint.	r-bp1wcw2rlw76acc5k****.redis.rds.aliyuncs.com:6379
      target.password_raw (required)	The account with read and write permissions and password of the destination ApsaraDB for Redis instance. Separate the account and password with a colon (:). For more information about how to create an account, see Create accounts and databases for an ApsaraDB RDS for MySQL instance.	testaccount:Rp829dlwa
      target.db (optional)	Migrates data from all databases in the source database to the specified databases of the destination database. Valid values: 0 to 15. Note The default value -1 indicates that the feature is disabled.	-1
      key_exists (optional)	The data writing policy that is adopted when the keys in the source database are the same as those in the destination database. Valid values: rewrite: specifies whether to overwrite the existing keys in the destination database that are the same as those in the source database. none: This is the default value. Redis-shake stops running and a message appears that indicates conflicting keys. ignore: skips the current key, retains the data in the destination database, and continues the data migration.	rewrite
      filter.db.whitelist (optional)	The names of the databases to be migrated. Separate multiple database names with semicolons (;). The default value null indicates that all databases are to be migrated.	0;1
      filter.db.blacklist (optional)	The names of databases that do not need to be migrated. This is a blacklist. Separate multiple database names with semicolons (;). By default, this parameter is not specified and no blacklist is configured.	1;2

      Note You do not need to configure other parameters unless otherwise required. For more information, see the comments on each parameter in the redis-shake.conf file.
   2. Press the ESC key to exit the editing mode and enter :wq to save and close the file.
   3. Run the following command to start redis-shake and migrate data:

      ./redis-shake.linux -type=sync -conf=redis-shake.conf

      Redis-shake outputs the running log on the screen.

      Note For more information about the causes and solutions for related errors, seeFAQ.
4. Optional:Stop the data migration task as required.
   Note If you need to run redis-shake for a long time to migrate data in real time, skip this step.
   1. Check the logs and wait until the migration task enters the incremental data migration state.

      Task stage	Logs
      Full data migration	2020/12/16 21:02:36 [INFO] DbSyncer[0] total = 4.00MB - 2.18MB [ 54%] entry=52199 2020/12/16 21:02:36 [INFO] DbSyncer[0] total = 4.00MB - 4.00MB [100%] entry=97531 2020/12/16 21:02:36 [INFO] DbSyncer[0] sync rdb done Note When sync rdb done appears in the logs, full synchronization is completed and incremental synchronization starts.
      Incremental data migration	2020/12/16 21:03:07 [INFO] DbSyncer[0] sync: +forwardCommands=5 +filterCommands=0 +writeBytes=5095 2020/12/16 21:03:08 [INFO] DbSyncer[0] sync: +forwardCommands=7 +filterCommands=0 +writeBytes=7133 2020/12/16 21:03:09 [INFO] DbSyncer[0] sync: +forwardCommands=0 +filterCommands=0 +writeBytes=0 2020/12/16 21:03:10 [INFO] DbSyncer[0] sync: +forwardCommands=645 +filterCommands=0 +writeBytes=657255 2020/12/16 21:03:11 [INFO] DbSyncer[0] sync: +forwardCommands=28 +filterCommands=0 +writeBytes=28532 2020/12/16 21:03:12 [INFO] DbSyncer[0] sync: +forwardCommands=0 +filterCommands=0 +writeBytes=0 The following parameters in logs are described. forwardCommands: the number of commands that are sent from the source database. filterCommands: the number of commands that are filtered. For example, you set to filter specific databases in the configuration file of redis-shake. writeBytes: the number of bytes that are sent from the source database.

   2. Pause writing data to the source database and wait until the value of the writeBytes parameter in the returned log remains 0 for multiple times in succession, and use Ctrl+C to stop running redis-shake.

      

      Note In this case, the data in the source databases and destination databases is consistent. You can switch the database service from the on-premises Redis database to the ApsaraDB for Redis instance.