Data Transmission Service:Synchronize a self-managed Redis to a Redis instance

Prerequisites

• The source Redis database is version 2.8, 3.0, 3.2, 4.0, or 5.0.

  Note

  If you synchronize data across versions (supported only from earlier to later versions), you must first confirm compatibility. For example, you can create a pay-as-you-go Redis instance for testing. After the test is complete, you can release the instance or convert its billing method to subscription.

• The destination Redis Instance must have more available storage space than the source database uses.

• If the source Redis database uses a cluster architecture, each node in the cluster must be able to run the psync command and have the same connection password.

• The replication timeout parameter repl-timeout between the replica and master nodes of the source Redis instance is 60 seconds by default. Run the config set repl-timeout 600 command to set the value to 600 seconds. If the source database contains a large amount of data, you can increase the value of the repl-timeout parameter.

Notes

• Initial full data synchronization consumes resources from both the source and destination databases, increasing server load. If your database experiences high traffic or runs on low-specification servers, this may increase the pressure on your database and can even make it unavailable. We recommend that you evaluate the performance impact before you start and run the data synchronization task during off-peak hours.

• If some keys in the source database are set to expire, they might not be deleted immediately upon expiration. As a result, the key count in the destination database, which you can check using the info command, might be lower than in the source database.

  Note

  The number of keys that do not have an expiration policy or have not expired is the same in the source and destination databases.

• If the bind parameter is configured in the redis.conf file of the source database, set this parameter to the internal IP address of the ECS instance. This ensures that DTS can connect to the source database.

• To ensure task stability, we recommend increasing the value of the repl-backlog-size parameter in the redis.conf file of the source Redis database.

• To ensure synchronization quality, DTS inserts a key prefixed with DTS_REDIS_TIMESTAMP_HEARTBEAT into the source Redis database to record the update timestamp.

• If you configure data synchronization between Redis clusters, do not run the FLUSHDB or FLUSHALL command on the source cluster. Otherwise, data between the source and destination will become inconsistent.

• If the destination database runs out of memory and triggers data eviction, data may become inconsistent between the source and destination. This occurs because the default data eviction policy (maxmemory-policy) for Tair (Redis OSS-Compatible) is volatile-lru. However, this does not affect normal task operation.

  To prevent this issue, we recommend setting the data eviction policy for the destination database to noeviction. With this policy, if the destination runs out of memory, write operations fail, the task stops, and no data is lost to eviction.

  Note

  For more information about data eviction policies, see Data eviction policies in Redis.

• During synchronization, if you scale out or scale in the self-managed Redis database, such as by adding or removing shards, or change its specifications, such as by increasing memory, you must reconfigure the task. To ensure data consistency, we recommend clearing the data synchronized to the destination Redis instance before you reconfigure the task.

• During synchronization, if the connection endpoint of the self-managed Redis database changes, you must reconfigure the task.

• The following limitations apply when you synchronize data from a standalone Redis instance to a cluster-mode Redis instance. A Redis cluster only allows a command to operate on keys that are in the same slot. If you perform a multi-key operation on the source database where the keys span multiple slots, the following error occurs:

  CROSSSLOT Keys in request don't hash to the same slot

  We recommend that you perform only single-key operations during data synchronization to prevent task interruptions.

• If the destination instance uses a cluster architecture and one of its shards reaches the memory limit, or if the destination instance has insufficient storage space, the DTS task fails due to an out-of-memory (OOM) error.

• To ensure synchronization quality, Data Transmission Service (DTS) inserts a key with the prefix DTS_REDIS_TIMESTAMP_HEARTBEAT into the source database to record update timestamps. If the source database uses a cluster architecture, DTS inserts this key into each shard. This key is filtered out during synchronization and expires after the task is complete.

• If the source database is a read-only instance or the account used by DTS does not have write (SETEX) permissions, the reported latency may be inaccurate.

• If transparent data encryption (TDE) is enabled for the destination Redis instance, you cannot use DTS to synchronize data to it.

• Certain conditions can trigger a full data re-synchronization to the destination, which may lead to data inconsistency. These conditions include the following:

  • A transient connection interruption on the source or destination Redis causes resumable transmission to fail.

  • A primary/secondary switchover or failover occurs on the source or destination Redis.

  • The connection address of the source or destination Redis changes.

  • You modify the synchronization objects of the DTS instance.

• If a Tair (Redis OSS-Compatible) instance has Transport Layer Security (TLS) encryption enabled, you must connect to DTS using an SSL-encrypted. TLSv1.3 is not supported. You cannot connect to an SSL-enabled Tair (Redis OSS-Compatible) instance if you set the Alibaba Cloud Instance to Alibaba Cloud Instance.

• If a task fails, DTS support staff will attempt to restore it within eight hours. During restoration, they may restart the task or adjust its parameters.

  Note

  Only DTS task parameters are modified—not database parameters. Parameters that may be adjusted include those listed in Modify instance parameters.

Billing

Supported synchronization topologies

• One-to-one, one-way synchronization

• One-to-many, one-way synchronization

• Cascading one-way synchronization

For more information about synchronization topologies and their usage notes, see Data synchronization topologies.

Supported synchronization commands

• APPEND

• BITOP, BLPOP, BRPOP, BRPOPLPUSH

• DECR, DECRBY, DEL

• EVAL, EVALSHA, EXEC, EXPIRE, EXPIREAT

• GEOADD, GETSET

• HDEL, HINCRBY, HINCRBYFLOAT, HMSET, HSET, HSETNX

• INCR, INCRBY, INCRBYFLOAT

• LINSERT, LPOP, LPUSH, LPUSHX, LREM, LSET, LTRIM

• MOVE, MSET, MSETNX, MULTI

• PERSIST, PEXPIRE, PEXPIREAT, PFADD, PFMERGE, PSETEX

• RENAME, RENAMENX, RESTORE, RPOP, RPOPLPUSH, RPUSH, RPUSHX

• SADD, SDIFFSTORE, SELECT, SET, SETBIT, SETEX, SETNX, SETRANGE, SINTERSTORE, SMOVE, SPOP, SREM, SUNIONSTORE

• ZADD, ZINCRBY, ZINTERSTORE, ZREM, ZREMRANGEBYLEX, ZUNIONSTORE, ZREMRANGEBYRANK, ZREMRANGEBYSCORE

• SWAPDB, UNLINK (supported only if the source Redis instance is version 4.0 or later)

• XADD, XCLAIM, XDEL, XAUTOCLAIM, XGROUP CREATECONSUMER, XTRIM

Procedure

1. Purchase a data synchronization instance. For more information, see Purchase a data synchronization task.

   Note

   On the purchase page, set both the source and destination instance types to Redis.

2. Log on to the Data Transmission Service console.

   Note

   If you are redirected to the Data Management (DMS) console, click the icon in the lower-right corner and then click the icon to return to the previous version of the DTS console.

3. In the left-side navigation pane, click Data Synchronization.

4. At the top of the Synchronization Tasks page, select the region where the destination instance is located.

5. Find the purchased data synchronization instance and click Configure Task in the Actions column.

6. Configure the source and destination instances.

   Section	Parameter	Description
   N/A	Synchronization task name	DTS automatically generates a task name. We recommend specifying a descriptive name for easy identification.
   Source instance details	Instance type	Select User-Created Database in ECS Instance.
   	Instance region	The source region that you selected when you purchased the data synchronization instance. This parameter cannot be changed.
   	ECS instance ID	Select the ID of the ECS instance that serves as the data source. Note If the source Redis database uses a cluster architecture, select the ID of the ECS instance where a master node resides.
   	Database type	Fixed to Redis.
   	Instance mode	Select Standalone or Cluster based on the architecture of the source Redis database.
   	Port	Enter the service port of the source Redis database. The default value is 6379. In this example, 7000 is entered. Note If the source Redis database uses a cluster architecture, enter the service port of a master node.
   	Database password	Enter the password to connect to the source Redis database. Note This parameter is optional. If no password is set, you can leave this parameter empty.
   Destination instance details	Instance type	Select Redis Instance.
   	Instance region	The destination region that you selected when you purchased the data synchronization instance. This parameter cannot be changed.
   	Instance ID	Select the ID of the destination Redis instance.
   	Database password	Enter the password to connect to the destination Redis instance. Note The password must be in the <user>:<password> format. For example, if the username is admin and the password is Rp829dlwa, enter admin:Rp829dlwa.

7. In the lower-right corner of the page, click Set Whitelist and Next.

   Note

   • If the source or destination database is an Alibaba Cloud database instance, such as ApsaraDB RDS for MySQL or ApsaraDB for MongoDB, or a self-managed database on an ECS instance, DTS automatically adds the IP CIDR blocks of the DTS servers in the corresponding region to the whitelist of the database instance or the security group of the ECS instance. You do not need to add them manually. For more information, see IP addresses of DTS servers.

   • After the DTS task is complete or released, we recommend manually removing the added IP CIDR blocks of the DTS servers.

8. Configure the handling mode for existing objects in the destination and the synchronization objects.

   Setting	Description
   Handling mode for existing objects	Precheck and Report Errors: Checks if the destination database is empty. If it is empty, the precheck passes. If not, the precheck fails, and the data synchronization task does not start. Ignore Errors and Proceed: Skips the check for an empty destination database. Warning If you select Ignore Errors and Proceed, data from the source database will overwrite data with the same keys in the destination database during synchronization. Use this option with caution.
   Synchronization objects	In the Source Objects box, click the database that you want to synchronize and click the icon to move it to the Selected Objects box. You can select databases as synchronization objects. You cannot select individual keys.
   Edit mapped name	You cannot rename objects in this scenario.
   Replicate temporary tables during online DDL with DMS	If you use Data Management Service (DMS) to perform online DDL changes on the source database, you can specify whether to synchronize the temporary tables generated by these changes. Yes: Synchronize data from temporary tables generated by online DDL changes. Note If online DDL changes generate a large volume of data in temporary tables, data synchronization may be delayed. No: Do not synchronize data from temporary tables. Only the original DDL data from the source database is synchronized. Note This option causes table locking in the destination database.
   Connection retry duration	By default, if DTS fails to connect to the source or destination database, it retries for 720 minutes (12 hours). You can also specify a custom retry duration. The synchronization task automatically resumes if DTS reconnects within the specified duration; otherwise, the task fails. Note Because DTS instances are charged during the connection retry period, set a custom retry duration based on your business requirements, or release the DTS instance promptly after the source and destination instances are released.

9. After you complete the preceding configurations, click Next.

10. The initial synchronization method is fixed as Full Data + Incremental Data.

    Note

    • DTS synchronizes the existing data from the source Redis instance to the destination Redis instance, and then synchronizes incremental data.

    • If a version-related error message appears, upgrade the source Redis instance to the specified version as prompted. For more information, see Upgrade the major version and Upgrade a minor version and a proxy version.

11. After completing the configuration, click Precheck and Start in the lower-right corner of the page.

    Note

    • DTS performs a precheck before a synchronization task starts. The task can start only after it passes the precheck.

    • If the precheck fails, click the icon next to the failed item to view details.

      • Fix the issues based on the provided details, and then rerun the precheck.

      • If a warning item does not require a fix, you can select Ignore or Ignore Warnings and Precheck Again to skip the item and rerun the precheck.

12. After Precheck Passed is displayed in the Precheck dialog box, close the Precheck dialog box. The synchronization job will then start.

13. Wait for the task to initialize. Its status changes to Synchronizing.

    Note

    You can view the task status on the Data Synchronization page.