Migration Hub:Migration solution for self-hosted Redis to self-hosted Redis on Alibaba Cloud

Migrate a self-hosted Redis database from another cloud, data center, or environment to a self-hosted Redis instance on an Alibaba Cloud Elastic Compute Service (ECS) instance. This solution uses Redis-shake to perform full data migration followed by continuous incremental synchronization, enabling zero or minimal downtime.

Use cases

This solution covers two migration types:

• Full data migration: Migrates all historical data from the source Redis database to the target in a single operation.

• Incremental data migration: After full migration completes, continuously synchronizes real-time data changes from source to target — keeping both in sync until you cut over traffic.

Migration tools

Redis-shake

Redis-shake is an open-source tool from Alibaba for migrating and processing Redis data. It is the core tool for this solution.

Key features:

• Supports both full data migration and continuous incremental synchronization

• Compatible with Redis versions 2.8 to 7.2 across standalone, primary-secondary, Sentinel, and cluster deployments

• Works with major cloud Redis services, including ApsaraDB for Redis and Tair

• Provides multiple data export modes: PSync, RDB, and SCAN

• Supports key filtering and transformation through custom scripts

Redis-cli and custom scripts

Redis-cli is the official Redis command-line client. During data validation, use it to run commands such as INFO keyspace to retrieve and compare keyspace information between source and target. For large datasets or many key-value pairs, write custom scripts to automate the comparison.

Migration solution

Migration flow

1. Prepare environments: Verify network connectivity, resource capacity, version compatibility, and authentication. Deploy Redis-shake.

2. Create and configure the target instance: Set up the self-hosted Redis instance on an Alibaba Cloud ECS instance. Confirm version compatibility with the source and ensure sufficient memory, CPU, and network bandwidth.

3. Migrate data: Use Redis-shake to run full data migration, then switch to continuous incremental synchronization.

4. Validate data: After incremental sync stabilizes, run quantity and content checks to confirm consistency between source and target.

5. Switch over service traffic: Redirect application connections from the source to the target and validate core business flows.

6. Clean up and maintain a rollback plan: After confirming stable operation, decommission the old environment. Keep a tested rollback plan ready throughout.

Data migration solution

Migration modes

Redis-shake supports two migration modes. Use PSync mode unless your source instance does not support the PSync protocol.

SCAN mode limitations: The DUMP command is CPU-intensive and puts significant load on the source. KSN-based incremental sync is not aware of FLUSHALL or FLUSHDB commands — if these run on the source during migration, data inconsistency may result.

Prerequisites

1. Network connectivity: The Redis-shake host must have stable network access to both source and target Redis instances. Confirm that firewall rules allow traffic on the relevant ports (default: 6379).

2. Source Redis instance:

   • Redis version 2.8 to 7.2 (supported by Redis-shake). The target version must be equal to or later than the source version.

   • If the source uses password authentication or an Access Control List (ACL) user, have the credentials ready for Redis-shake.

   • sync_reader mode: Set client-output-buffer-limit replica on the source Master to a large enough value to prevent disconnection from buffer overflow.

   • scan_reader + KSN mode: Enable notify-keyspace-events in the source configuration and set its value to include AE.

3. Target Redis instance:

   • Version equal to or later than the source to avoid command and encoding incompatibilities.

   • Enough memory, CPU, and network bandwidth to handle all migrated data and post-migration traffic.

   • If the target uses password authentication or an ACL user, have the credentials ready for Redis-shake.

4. Redis-shake host (ECS instance):

   • Enough CPU, memory, and disk space for staging RDB files, especially for large datasets.

   • The operating system must support Go runtime (if compiling from source) or be able to run the provided binary directly.

Risks and notes

1. Version downgrade: Never migrate from a later Redis version to an earlier one. Redis-shake does not support this because newer versions may use commands or encoding methods that older versions cannot handle. Always confirm the target version is equal to or later than the source.

2. CPU load from SCAN mode: In scan_reader mode, DUMP is CPU-intensive and can noticeably affect source database availability. Run SCAN migration during off-peak hours and tune the count parameter to control the load.

3. Disk space for temporary files: Redis-shake creates temporary RDB files at runtime. Make sure the host has enough disk space. Never run two Redis-shake processes in the same folder — temporary files will overwrite each other, causing unpredictable behavior.

4. Network stability: Interruptions between the source, Redis-shake, and the target can cause migration failure or data inconsistency. Use a stable, low-latency network connection throughout.

5. Expired key count discrepancy: Redis applies expiration lazily and randomly. After migration, the number of keys with expiration times may differ slightly between source and target. This is expected behavior. Focus validation on the count of non-expiring keys (keys - expires), which must match exactly.

6. Sentinel and PSync mode: When using sync_reader to connect to a Master managed by Sentinel, Sentinel may treat Redis-shake as a secondary node, which can cause unexpected behavior. Connect to a secondary node as the source instead.

7. client-output-buffer-limit parameter: In PSync mode, if Redis-shake disconnects from the source, the likely cause is that client-output-buffer-limit replica on the source is too small — increase it. In KSN mode, if a disconnection occurs, increase client-output-buffer-limit pubsub.

Data validation solution

Run all three validation types after Redis-shake completes full sync and enters incremental sync.

1. Quantity validation: Run INFO keyspace on both source and target and compare keys and expires per database. The count of non-expiring keys (keys - expires) must match exactly. A small discrepancy in expires is acceptable due to Redis's random expiration policy.

2. Content validation (sampling): Use SCAN to retrieve a random sample of keys from the source, then compare type, TTL, and value on both source and target using TYPE, TTL, GET, LRANGE, SMEMBERS, HGETALL, and ZRANGE. Automate this with a Python or Shell script for larger datasets.

3. Business-side validation: Before the final cutover, connect a test application to the target and run core business flows. Monitor error logs and performance metrics to confirm behavior is correct.

Migration procedure

Migrate data using Redis-shake

Before starting, confirm you have completed these preparations: created the Alibaba Cloud ECS instance, deployed and configured the target Redis database, and downloaded Redis-shake.

Step 1: Deploy Redis-shake

1. Download the latest Redis-shake binary from the GitHub Releases page: https://github.com/alibaba/RedisShake/releases. Select the package that matches your ECS operating system, for example, redis-shake-linux-amd64.tar.gz.

2. Upload the package to your ECS instance and extract it into a dedicated folder, for example, /opt/redis-shake. Use a separate folder for each Redis-shake instance — running two processes in the same folder overwrites shared temporary files and causes errors.

Step 2: Create the Redis-shake configuration file

In /opt/redis-shake, create a file named shake.toml with the following content. This example uses sync_reader (PSync) mode.

# shake.toml

[sync_reader]
# Source Redis configuration
cluster = false            # Set to true if the source Redis is in cluster mode. Set to false for standalone or primary-secondary mode.
address = "Source_Redis_IP_address:port" # Example: "192.168.1.100:6379". For cluster mode, enter the address of any node.
username = ""              # If the source Redis uses an ACL account, enter the username. Otherwise, leave it blank.
password = "Source_Redis_password" # If the source Redis has a password, enter it. Otherwise, leave it blank.
tls = false                # Set to true if the source Redis has TLS/SSL enabled. Otherwise, set to false.
sync_rdb = true            # Enable RDB full synchronization.
sync_aof = true            # Enable AOF incremental synchronization.

# [sync_reader.sentinel] # If the source Redis is monitored by Sentinel and you want to get the Master address through Sentinel
# master_name = "mymaster"
# address = "127.0.0.1:26380" # Sentinel address
# username = ""
# password = ""
# tls = false

[redis_writer]
# Target Redis configuration (self-hosted Redis on Alibaba Cloud ECS)
cluster = false            # Set to true if the target Redis is in cluster mode. Set to false for standalone or primary-secondary mode.
address = "Target_Redis_IP_address:port" # Example: "10.0.0.10:6379". For cluster mode, enter the address of any node.
username = ""              # If the target Redis uses an ACL account, enter the username. Otherwise, leave it blank.
password = "Target_Redis_password" # If the target Redis has a password, enter it. Otherwise, leave it blank.
tls = false                # Set to true if the target Redis has TLS/SSL enabled. Otherwise, set to false.

[filter]
# Example: Filter out keys that start with "test:"
# type = "lua"
# args = """
# function filter(cmd)
#     if string.sub(cmd[2], 1, 5) == "test:" then
#         return false
#     end
#     return true
# end
# """

[advanced]
# max_commands_per_second = 0 # Limits the number of write commands per second. 0 means no limit.
# rps = 0 # Limits the number of commands synchronized per second.
# source_db = 0 # Source Redis database index. Default is 0.
# target_db = 0 # Target Redis database index. Default is 0.
# network_timeout = 5 # Network timeout in seconds.
# temp_file_compaction = true # Compresses temporary files to save disk space.

Key configuration parameters:

• Replace Source_Redis_IP_address:port and Source_Redis_password with the actual connection details for your source Redis database.

• Replace Target_Redis_IP_address:port and Target_Redis_password with the actual connection details for your target Redis database on the Alibaba Cloud ECS instance.

• Adjust cluster, username, and tls to match your environment.

• To discover the source Master address through Sentinel, uncomment and configure the [sync_reader.sentinel] section. Note that when sync_reader connects to a Master managed by Sentinel, Sentinel may treat Redis-shake as a secondary node. Connect to a secondary node as the source to avoid this.

• The filter and advanced sections provide optional settings for key filtering and rate limiting.

Step 3: Start Redis-shake

From the folder containing shake.toml, run:

./redis-shake shake.toml

Redis-shake starts and begins migrating data. Monitor progress using the following methods:

• Console output: Redis-shake prints real-time logs showing connection status, RDB synchronization progress, and AOF stream traffic. When you see RDB dump finished, the full migration is complete and incremental sync has started.

• Log file: Logs are also written to shake.log in the current folder.

• Process status: Run ps -ef | grep redis-shake to confirm the process is running.

• Target Redis metrics: Watch memory usage, key count, and queries per second (QPS) on the target to confirm data is being written.

Once RDB dump finished appears in the logs, start the data validation procedure.

Data validation procedure

Run all validation steps after Redis-shake enters incremental sync (after RDB dump finished appears in the logs).

Quantity validation

Connect to both source and target databases using redis-cli and compare keyspace information.

1. Get source keyspace information:

redis-cli -h Source_Redis_IP_address -p Source_Redis_port -a Source_Redis_password INFO keyspace

Sample output:

# Keyspace
db0:keys=4463175,expires=2,avg_ttl=333486
db1:keys=1000,expires=0,avg_ttl=0

Record the keys and expires values for each database.

1. Get target keyspace information:

redis-cli -h Target_Redis_IP_address -p Target_Redis_port -a Target_Redis_password INFO keyspace

Record the keys and expires values for each database.

1. Compare the results:

   • Non-expiring keys (keys - expires): This value must be identical on source and target for each database.

   • Expiring keys (expires): A small difference is acceptable due to Redis's random expiration policy. A large discrepancy requires investigation.

Content validation (sampling)

Sample a subset of keys and compare their content on both source and target.

1. Sample keys from the source: Use SCAN to traverse the source and randomly select keys. For example, pick 10 keys from every 100 returned by SCAN.

# Example: Get keys from db0
redis-cli -h Source_Redis_IP -p Source_Redis_port -a Source_Redis_password SCAN 0 COUNT 100

Repeat SCAN until you have enough sample keys.

1. Compare each sampled key:

   • Type and TTL: Confirm key types match and TTLs are similar. A millisecond-level difference is expected.

   • Value: Use the appropriate command for the key type on both source and target:

     • String: GET <key_name>

     • Hash: HGETALL <key_name>

     • List: LRANGE <key_name> 0 -1

     • Set: SMEMBERS <key_name>

     • ZSet: ZRANGE <key_name> 0 -1 WITHSCORES

Write a Python or Shell script to automate sampling, retrieval, and comparison for large datasets.

Business-side validation

After confirming data consistency, run business-side validation before the final traffic cutover.

1. Switch application connections: Redirect some or all business application connections from the source to the target. Start with a small traffic slice or a test environment to confirm that business functions work correctly.

2. Verify core business flows: Run core read and write operations — such as user login, data queries, data writing, and order creation — and monitor error logs and performance metrics (response time, throughput) against baseline.

3. Verify data write-back: Confirm that data written to the target after the switch is stored and read back correctly. Run a sampling check on newly written data to verify integrity.

Migration notes

• Version compatibility: The target Redis version must be equal to or later than the source. Migrating to an older version causes data encoding and command incompatibilities.

• Resource planning:

  • Redis-shake host: Reserve enough CPU, memory, and disk space for RDB staging, especially for large datasets.

  • Target Redis instance: Provision enough memory to hold all migrated data with headroom for growth, plus enough CPU and network bandwidth to handle peak traffic during and after migration.

• Network bandwidth and latency: Bandwidth between source, Redis-shake, and target directly affects migration speed. Low latency reduces synchronization delay and improves real-time performance.

• Source database load:

  • sync_reader (PSync) has minimal impact on the source, but still monitor CPU, memory, and network I/O.

  • scan_reader (SCAN) is CPU-intensive. Run full synchronization during off-peak hours and tune the count parameter to limit load.

• Data consistency:

  • sync_reader provides the strongest data consistency.

  • A small discrepancy in expiring key counts is normal and generally acceptable.

• Business downtime: PSync mode supports zero-downtime or minimal-downtime migration, keeping the traffic switchover window to seconds. Coordinate with your team to define an acceptable interruption window and schedule the cutover during off-peak hours.

• Monitoring and alerts: Monitor the source, Redis-shake process, target, and business applications throughout the migration. Set alerts on CPU utilization, memory usage, network traffic, Redis connections, QPS, and error logs to detect and respond to issues quickly.

• Rollback plan: Prepare and test a rollback plan before starting. The plan must cover how to redirect traffic back to the source and how to handle data written to the target during the rollback window (reverse sync or re-sync may be required). Verify the rollback path before the migration begins.

• Configuration file security: Store shake.toml securely to prevent credential leaks.

• Log analysis: Monitor Redis-shake logs closely during migration. Logs provide detailed status, error messages, and the primary data for troubleshooting.