Use redis-shak to migrate data from a self-managed Redis database to Alibaba Cloud
Last Updated: Apr 16, 2021
You can set redis-shake to sync mode and then use it to migrate data from a self-managed
Redis database to an ApsaraDB for Redis instance in real time. The migration does
not cause service interruption.
Prerequisites
The version of the self-managed Redis database is Redis 2.8, 3.0, 3.2, 4.0, or 5.0.
Note If you want to migrate data between different versions of Redis databases, for example,
from Redis 2.8 to Redis 4.0, we recommend that you create a pay-as-you-go ApsaraDB
for Redis instance to verify the compatibility. If no compatibility issue is found,
you can change the billing method of the instance to subscription. For more information
about how to change the billing method, see Change the billing method to subscription.
The destination ApsaraDB for Redis instance must meet the following requirements:
The destination ApsaraDB for Redis instance is of Community Edition (Redis 4.0 or
5.0) or Enhanced Edition (Redis 5.0).
The destination ApsaraDB for Redis instance use a standard or cluster architecture.
Redis-shake is an open source tool developed by Alibaba Cloud for Redis data transmission. You
can use this flexible and efficient tool to parse (decode mode), restore (restore
mode), back up (dump mode), and synchronize (sync or rump mode) data on ApsaraDB for
Redis instances. When redis-shake works in sync mode, it supports both full data migration
and incremental data migration. The detailed migration process is shown in the following
figure.
Figure 1. Redis-shake data migration diagram
Note You can also use the data synchronization feature of Data Transmission Service (DTS)
to migrate data. DTS is applicable to more scenarios and supports more features. For
more information, see Overview of data synchronization.
Precautions
If the data eviction policy (maxmemory-policy) of the destination database is not set to noeviction, data may become inconsistent between the source and destination databases. For more
information about the data eviction policy, see How does ApsaraDB for Redis evict data by default?
After you run the info command to query the keys in the destination database, you
may find that the destination database contains fewer keys than the source database.
This is caused by the key expiration mechanism of Redis. This situation occurs if
the source database contains keys that are not deleted upon expiration.
Note The numbers of keys that do not have a validity period in the source and destination
databases are the same.
Procedure
Perform the following operations based on the location where redis-shake is installed:
Note We recommend that you install redis-shake on the host where the self-managed database
is deployed. This reduces network latency and prevents connection failures caused
by the firewall settings of the self-managed database.
Make sure that the ECS instance and the ApsaraDB for Redis instance belong to the
same VPC. They share the same VPC ID in the Basic Information section.
Note
If they are deployed in different VPCs, you can change the VPC to which the ECS instance
belongs. For more information, see Change the VPC of an ECS instance.
The network types are distinct. For example, the ECS instance belongs to a classic
network and the ApsaraDB for Redis instance belongs to a VPC. For more information
about how to connect to an ApsaraDB for Redis instance from an ECS instance when they
are deployed in different types of networks, see Connect an ECS instance to an ApsaraDB for Redis instance in different types of networks.
Add the internal IP address of the ECS instance to the whitelist of the ApsaraDB for
Redis instance. For more information, see Configure an IP whitelist.
On-premises machine
By default, an ApsaraDB for Redis instance provides only an internal endpoint. You
must apply for a public endpoint when you want to connect to an ApsaraDB for Redis
instance over the Internet. For more information, see Apply for a public endpoint.
Run the curl ipinfo.io |grep ip command on the on-premises machine to obtain the public IP address. The returned
result is shown in the following figure.
Add the public IP address of the on-premises machine to the whitelist of the ApsaraDB
for Redis instance. For more information, see Configure an IP whitelist.
Install redis-shake.
Log on to the host where you want to install redis-shake. The host may be an ECS instance or an on-premises machine.
Run the following command to download the redis-shake file.
The connection address and port number of the destination ApsaraDB for Redis instance.
Separate the connection address and port number with a colon (:). For more information
about how to obtain the connection address and port number of the ApsaraDB for Redis
instance, see .
The ECS instance is connected through a virtual private cloud (VPC): You must obtain
the internal endpoint of the ApsaraDB for Redis instance.
If you connect an on-premises machine to the ApsaraDB for Redis instance over the
Internet, you must obtain the public IP address of the ApsaraDB for Redis instance.
Note If the destination instance uses the cluster architecture, you must connect to the
instance deployed in a VPC by using a private endpoint. You must add the prefix master@ to the connection address, for example, master@r-bp1mfnrflszg75w****.redis.rds.aliyuncs.com:6379. For more information about how to request a private endpoint, see Enable the direct connect mode.
The account with read and write permissions and the password of the destination ApsaraDB for Redis instance. Separate
the account and password with a colon (:). For more information about how to create
an account, see Create and manage database accounts.
testaccount:Rp829dlwa
target.db
No
Migrates data from all databases in the source database to the specified databases
of the destination database. Valid values: 0 to 15.
Note The default value -1 indicates that the feature is disabled.
-1
key_exists
No
The data writing policy that is applied when the keys in the source database are the
same as those in the destination database. Valid values:
rewrite: overwrites the existing keys in the destination database that are the same as those
in the source database.
none: This is the default value. Redis-shake stops running and a message that indicates
conflicting keys appears.
ignore: skips the current key, retains the data in the destination database, and continues
to migrate data.
rewrite
filter.db.whitelist
No
The names of the databases to be migrated. Separate multiple database names with semicolons
(;). By default, this parameter is left empty. This indicates that all databases are
migrated.
0;1
filter.db.blacklist
No
The names of databases that you do not want to migrate. This parameter is equivalent
to a blacklist. Separate multiple database names with semicolons (;). By default,
this parameter is left empty. This indicates that no database is added to the blacklist.
1;2
parallel
No
The number of concurrent threads for redis-shake to perform the migration tasks. You
can increase this value to improve synchronization performance.
Note The default value is 32. The minimum value is 1. The maximum value depends on the
performance of the server where redis-shake is deployed.
32
Note You do not need to configure other parameters unless otherwise required. For more
information, see the comments on each parameter in the redis-shake.conf file.
Press the Esc key to exit the editing mode and enter :wq to save and close the file.
Run the following command to start redis-shake and migrate data:
forwardCommands: the number of commands that are sent from the source database.
filterCommands: the number of commands that are filtered out. For example, you can set the configuration
file of redis-shake to filter out specific databases.
writeBytes: the number of bytes that are sent from the source database.
Stop writing data to the source database and wait until the value of the writeBytes parameter in the returned log remains 0 for multiple times in succession, and then
press Ctrl+C to stop the running of redis-shake.
Figure 2. Stop running example
Note The preceding figure indicates that the data in the source databases and destination
databases is consistent. You can switch workloads from the self-managed Redis database
to the ApsaraDB for Redis instance.