Data Transmission Service:Self-managed Redis cluster to Tair (Redis OSS-Compatible) synchronization

Prerequisites

• The self-managed Redis database is version 2.8, 3.0, 3.2, 4.0, or 5.0.

  Note

  The destination Tair (Redis OSS-Compatible) instance must be version 4.0 or 5.0. Cross-version synchronization is supported only from earlier to later versions. Verify compatibility by creating a pay-as-you-go Redis cluster instance for testing, then release or convert it.

• The destination Tair (Redis OSS-Compatible) instance must have more storage than the source Redis database uses.

• Each node in the source Redis cluster must support the psync command, and all nodes must share the same password.

• The repl-timeout parameter controls the replication timeout between replica and primary nodes (default: 60 seconds). Set it to 600 seconds by running config set repl-timeout 600. For large databases, increase repl-timeout further as needed.

Limits

• Initial full data synchronization consumes resources from both the source and destination databases, increasing server load. If your database experiences high traffic or runs on low-specification servers, this may increase the pressure on your database and can even make it unavailable. We recommend that you evaluate the performance impact before you start and run the data synchronization task during off-peak hours.

• Increase repl-backlog-size in redis.conf to maintain synchronization link stability.

• DTS inserts a DTS_REDIS_TIMESTAMP_HEARTBEAT key into the source Redis database to track update timestamps.

• Do not run FLUSHDB or FLUSHALL on the source cluster. This causes data inconsistency between source and destination.

• If the destination database runs out of memory and triggers data eviction, data may become inconsistent between the source and destination. This occurs because the default data eviction policy (maxmemory-policy) for Tair (Redis OSS-Compatible) is volatile-lru. However, this does not affect normal task operation.

  To prevent this issue, we recommend setting the data eviction policy for the destination database to noeviction. With this policy, if the destination runs out of memory, write operations fail, the task stops, and no data is lost to eviction.

  Note

  For more information about data eviction policies, see Data eviction policies in Redis.

• If the source uses key expiration, the destination may report fewer keys (via info) because expired keys are not always deleted immediately from the source.

  Note

  The number of keys that do not have an expiration policy or have not expired is the same in both the source and destination databases.

• During synchronization, if you scale out or scale in the self-managed Redis database, such as by adding or removing shards, or change its specifications, such as by increasing memory, you must reconfigure the task. To ensure data consistency, we recommend clearing the data synchronized to the destination Redis instance before you reconfigure the task.

• During synchronization, if the connection endpoint of the self-managed Redis database changes, you must reconfigure the task.

• When synchronizing from a standalone to a cluster Redis instance, a limitation arises because Redis clusters allow a single command to operate on only a single slot. If you perform a multi-key operation on the source where the keys do not belong to the same slot, the following error is returned:

  CROSSSLOT Keys in request don't hash to the same slot

  To prevent task interruptions, perform only single-key operations during DTS synchronization.

• If a destination cluster shard reaches its memory limit or the instance runs out of storage, the DTS task fails with an Out of Memory error.

• DTS cannot synchronize data to a destination instance with transparent data encryption (TDE) enabled.

Billing

Supported synchronization topologies

• One-to-one, one-way synchronization

• One-to-many, one-way synchronization

• Cascading one-way synchronization

For more information about synchronization topologies and their usage notes, see Data synchronization topologies.

Supported synchronization commands

• APPEND

• BITOP, BLPOP, BRPOP, BRPOPLPUSH

• DECR, DECRBY, DEL

• EVAL, EVALSHA, EXEC, EXPIRE, EXPIREAT

• GEOADD, GETSET

• HDEL, HINCRBY, HINCRBYFLOAT, HMSET, HSET, HSETNX

• INCR, INCRBY, INCRBYFLOAT

• LINSERT, LPOP, LPUSH, LPUSHX, LREM, LSET, LTRIM

• MOVE, MSET, MSETNX, MULTI

• PERSIST, PEXPIRE, PEXPIREAT, PFADD, PFMERGE, PSETEX

• RENAME, RENAMENX, RESTORE, RPOP, RPOPLPUSH, RPUSH, RPUSHX

• SADD, SDIFFSTORE, SELECT, SET, SETBIT, SETEX, SETNX, SETRANGE, SINTERSTORE, SMOVE, SPOP, SREM, SUNIONSTORE

• ZADD, ZINCRBY, ZINTERSTORE, ZREM, ZREMRANGEBYLEX, ZUNIONSTORE, ZREMRANGEBYRANK, ZREMRANGEBYSCORE

• SWAPDB, UNLINK (supported only if the source Redis instance is version 4.0 or later)

• XADD, XCLAIM, XDEL, XAUTOCLAIM, XGROUP CREATECONSUMER, XTRIM

Procedure

1. Purchase a data synchronization task.

   Note

   During purchase, set both source and destination instance types to Redis.

2. Log on to the Data Transmission Service console.

   Note

   If you are redirected to the Data Management (DMS) console, click the icon in the lower-right corner and then click the icon to return to the previous version of the DTS console.

3. In the left-side navigation pane, click Data Synchronization.

4. At the top of the Synchronization Tasks page, select the region where the destination instance resides.

5. Find the purchased synchronization instance and click Configure Task.

6. Configure the source and destination instances.

   Category	Setting	Description
   N/A	Synchronization task name	DTS auto-generates a task name. Specify a descriptive name for easy identification. Uniqueness is not required.
   Source instance details	Instance type	Select Self-managed database on ECS. Depending on your deployment, select Self-managed database on ECS or Self-managed database connected via Express Connect/VPN Gateway/Smart Access Gateway. This example uses Self-managed database on ECS. The process is similar for other self-managed Redis types.
   	Instance region	The region you selected when purchasing the synchronization task. This setting cannot be changed.
   	ECS instance ID	Select the ECS instance that hosts a master node of the self-managed Redis cluster.
   	Database type	This is fixed at Redis.
   	Instance mode	Select Cluster.
   	Port	Enter the service port of a master node. Example: 7000.
   	Database password	Enter the password for your self-managed Redis database. Note Optional. Leave blank if no password is set.
   Destination instance details	Instance type	Select Redis instance.
   	Instance region	The region you selected when purchasing the synchronization task. This setting cannot be changed.
   	Instance ID	Select the ID of the destination Tair (Redis OSS-Compatible) instance.
   	Database password	Enter the password for the destination Tair (Redis OSS-Compatible) instance. Note The password must be in the <user>:<password> format. For example, if the username is admin and the password is Rp829dlwa, enter admin:Rp829dlwa.

7. Click Authorize whitelist and go to next step.

   Note

   • For Alibaba Cloud database instances such as ApsaraDB RDS for MySQL or ApsaraDB for MongoDB, or self-managed databases on ECS, DTS automatically adds the CIDR blocks of DTS servers for the corresponding region to the whitelist or security group rules.

   • Remove the DTS server CIDR blocks after the task is complete or released.

8. Configure the handling mode for existing objects in the destination and the synchronization objects.

   Setting	Description
   Handling mode for existing objects	Precheck and Report Errors: Checks if the destination database is empty. If it is empty, the precheck passes. If not, the precheck fails, and the data synchronization task does not start. Ignore Errors and Proceed: Skips the check for an empty destination database. Warning If you select Ignore Errors and Proceed, data from the source database will overwrite data with the same keys in the destination database during synchronization. Use this option with caution.
   Synchronization objects	In the Source Objects box, click the database that you want to synchronize and click the icon to move it to the Selected Objects box. You can select databases as synchronization objects. You cannot select individual keys.
   Edit mapped name	You cannot rename objects in this scenario.
   Replicate temporary tables during online DDL with DMS	If you use Data Management (DMS) to perform online DDL changes on the source database, you can choose whether to synchronize the temporary tables generated by the DDL changes. Yes: Synchronizes the temporary tables generated by online DDL changes. Note If a large amount of temporary table data is generated by online DDL changes, the data synchronization task may be delayed. No: Does not synchronize the temporary tables generated by online DDL changes. Only the original DDL operations from the source database are synchronized. Note This option causes tables in the destination database to be locked.
   Connection retry duration	If DTS cannot connect to the source or destination instance, it retries for 720 minutes (12 hours) by default. You can also specify a custom retry duration. If DTS reconnects to the source or destination instance within the specified duration, the synchronization task automatically resumes. Otherwise, the task fails. Note You are billed for task run time during connection retries. Customize the retry duration based on your business needs, or release the DTS instance as soon as the source and destination instances are released.

9. After you complete the preceding configurations, click Next.

10. The initial synchronization method is fixed as Full Data + Incremental Data.

    Note

    • DTS synchronizes the existing data from the source Redis instance to the destination Redis instance, and then synchronizes incremental data.

    • If a version-related error message appears, upgrade the source Redis instance to the specified version as prompted. For more information, see Upgrade the major version and Upgrade a minor version and a proxy version.

11. After you finish the configuration, click Precheck and Start in the lower-right corner of the page.

    Note

    • Before the synchronization task starts, DTS performs a precheck. The task can start only after passing the precheck.

    • If the precheck fails, click the icon next to a check item to view the failure details.

      • You can fix the issues based on the details and run the precheck again.

      • If you do not need to fix the warning items, you can click Ignore or Ignore and Rerun Precheck to proceed.

12. After Precheck Passed is displayed in the Precheck dialog box, close the Precheck dialog box. The synchronization job will then start.

13. Wait for the task to initialize. Its status changes to Synchronizing.

    Note

    You can view the task status on the Data Synchronization page.