Data Transmission Service:Synchronize self-managed Redis to ECS via private line

Prerequisites

• The source Redis database must be version 2.8, 3.0, 3.2, 4.0, or 5.0.
  Note The destination Redis database supports versions 2.8, 3.0, 3.2, 4.0, and 5.0. The destination version must be the same as or later than the source version. If you are synchronizing data between different versions, verify their compatibility beforehand.
• The destination Redis database must have more available storage space than the storage used by the source database.
• If the source Redis database uses a cluster architecture, each node in the cluster must support the psync command and all nodes must use the same password.
• repl-timeoutThe repl-timeout parameter, which specifies the replication timeout period between the replica and primary nodes of the source Redis instance, is 60 seconds by default. We recommend that you run the config set repl-timeout 600 command to set the value to 600 seconds. If the source database contains a large amount of data, you can increase the value of the repl-timeout parameter as needed.
• A data synchronization account has been created in the destination database and configured with a password.

Usage notes

• Initial full data synchronization increases the load on both the source and destination. If your databases handle high traffic or run on low-spec servers, synchronization may cause service interruptions. Evaluate the performance impact and run the task during off-peak hours.

• If the source database has keys with an expiration policy, the destination database may report fewer keys when checked with a command like info. This is because expired keys may not be immediately deleted.

  Note

  The number of keys that do not have an expiration policy or have not expired is the same in both the source and destination databases.

• If you set the bind parameter in the redis.conf configuration file of the source database, set its value to the private IP address of the ECS instance to ensure that DTS can connect to the source database.
• To ensure link stability, we recommend that you increase the value of the repl-backlog-size parameter in the redis.conf file of the source Redis database.
• To ensure synchronization quality, DTS inserts a key with the prefix DTS_REDIS_TIMESTAMP_HEARTBEAT into the source Redis database to record the update timestamp.
• If you configure data synchronization between Redis clusters, do not run the FLUSHDB or FLUSHALL command on the source cluster. Otherwise, data inconsistency will occur between the source and destination databases.

• If the destination runs out of memory, data eviction may cause inconsistency between the source and destination. The default maxmemory-policy for Tair (Redis OSS-Compatible) is volatile-lru, which does not affect task operation but may evict data.

  Set the destination eviction policy to noeviction to prevent data loss. With this policy, write operations fail and the task stops when memory is exhausted, but no data is evicted.

  Note

  Data eviction policies in Redis.

• During synchronization, if the self-managed Redis database undergoes scaling (such as adding or removing shards) or a specification change (such as increasing memory), you must reconfigure the task. To ensure data consistency, we recommend that you clear the data already synchronized to the destination Redis database before you reconfigure the task.
• During synchronization, if the connection address of the self-managed Redis database changes, you must reconfigure the task.

• When synchronizing from a standalone to a cluster Redis instance, Redis clusters allow a single command to operate on only one slot. Multi-key operations on keys in different slots return the following error:

  CROSSSLOT Keys in request don't hash to the same slot

  To prevent task interruptions, perform only single-key operations during DTS synchronization.

• If the target instance is a cluster instance and a shard reaches the memory limit, or if the target instance has insufficient storage space, the DTS task fails due to an Out of Memory error.

• DTS inserts a key prefixed with DTS_REDIS_TIMESTAMP_HEARTBEAT into the source database to record update timestamps. For clusters, this key is inserted into each shard. The key is filtered during synchronization and expires after the task completes.

• If the source database is a read-only instance or the DTS account does not have write (SETEX) permissions, the reported latency may be inaccurate.

Billing

Supported synchronization topologies

• One-to-one, one-way synchronization

• One-to-many, one-way synchronization

• Cascading one-way synchronization

Topology details and usage notes are covered in Data synchronization topologies.

Supported synchronization commands

• APPEND

• BITOP, BLPOP, BRPOP, BRPOPLPUSH

• DECR, DECRBY, DEL

• EVAL, EVALSHA, EXEC, EXPIRE, EXPIREAT

• GEOADD, GETSET

• HDEL, HINCRBY, HINCRBYFLOAT, HMSET, HSET, HSETNX

• INCR, INCRBY, INCRBYFLOAT

• LINSERT, LPOP, LPUSH, LPUSHX, LREM, LSET, LTRIM

• MOVE, MSET, MSETNX, MULTI

• PERSIST, PEXPIRE, PEXPIREAT, PFADD, PFMERGE, PSETEX

• RENAME, RENAMENX, RESTORE, RPOP, RPOPLPUSH, RPUSH, RPUSHX

• SADD, SDIFFSTORE, SELECT, SET, SETBIT, SETEX, SETNX, SETRANGE, SINTERSTORE, SMOVE, SPOP, SREM, SUNIONSTORE

• ZADD, ZINCRBY, ZINTERSTORE, ZREM, ZREMRANGEBYLEX, ZUNIONSTORE, ZREMRANGEBYRANK, ZREMRANGEBYSCORE

• SWAPDB, UNLINK (supported only if the source Redis instance is version 4.0 or later)

• XADD, XCLAIM, XDEL, XAUTOCLAIM, XGROUP CREATECONSUMER, XTRIM

Procedure

1. Purchase a data synchronization instance. For more information, see Purchase a data synchronization instance.
   Note When purchasing, set both the source and destination instance types to Redis.

2. Log on to the Data Transmission Service console.

   Note

   If you are redirected to the Data Management (DMS) console, click the icon in the lower-right corner and then click the icon to return to the previous version of the Data Transmission Service (DTS) console.

3. In the left-side navigation pane, click Data Synchronization.

4. At the top of the Synchronization Tasks page, select the region where the destination instance is located.

5. Find the data synchronization instance and click Configure Task in the Actions column.
6. Configure the source and destination instances.

   Section	Parameter	Description
   N/A	Synchronization task name	DTS automatically generates a task name. We recommend that you specify a descriptive name for easy identification. The name does not need to be unique.
   Source instance details	Instance type	Select User-Created Database Connected Over Express Connect, VPN Gateway, or Smart Access Gateway.
   	Instance region	The source region that you selected when you purchased the data synchronization instance. This parameter cannot be changed.
   	Peer VPC	Select the ID of the VPC connected to the self-managed database.
   	Database type	This is fixed to Redis.
   	Instance mode	Select Standalone or Cluster based on the architecture of the source Redis database.
   	IP address	Enter the IP address of the source Redis database. Note If the source Redis database uses a cluster architecture, enter the IP address of the server on which any primary node resides.
   	Port number	Enter the service port number of the source Redis database. The default value is 6379. Note If the source Redis database uses a cluster architecture, enter the service port number of any primary node.
   	Database password	Enter the password to connect to the source Redis database. Note This parameter is optional. If no password is set, you can leave it blank.
   Destination instance details	Instance type	Select User-Created Database in ECS Instance.
   	Instance region	The destination region that you selected when you purchased the data synchronization instance. This parameter cannot be changed.
   	ECS instance ID	Select the ID of the ECS instance that serves as the synchronization destination. Note If the destination Redis database uses a cluster architecture, select the ID of the ECS instance where any primary node is located.
   	Instance mode	Select Standalone or Cluster based on the architecture of the destination Redis database.
   	Port number	Enter the service port number of the destination Redis database. The default value is 6379. Note If the destination Redis database uses a cluster architecture, enter the service port number of any primary node.
   	Database password	Enter the password to connect to the destination Redis database. Note This parameter is required. The precheck will fail if you leave it blank.

7. In the lower-right corner of the page, click Set Whitelist and Next.
   Note

   • If the source or destination database is an Alibaba Cloud database instance (for example, RDS for MySQL or ApsaraDB for MongoDB) or a self-managed database on an ECS instance, DTS automatically adds the IP addresses of DTS servers in the related region to the whitelist of the Alibaba Cloud database instance or to the ECS security group rule. You do not need to add them manually. For more information, see IP addresses of DTS servers.
   • After the DTS task is complete or released, we recommend that you manually remove the added DTS server IP address ranges.

8. Configure the handling mode for existing destination objects and select synchronization objects.

   Setting	Description
   Handling mode for existing objects	Precheck and Report Errors: Checks whether the destination database is empty. If not empty, the precheck fails and the task does not start. Ignore Errors and Proceed: Skips the check for an empty destination database. Warning If you select Ignore Errors and Proceed, source data overwrites destination data with the same keys during synchronization. Use with caution.
   Synchronization objects	In the Source Objects box, click the database that you want to synchronize and click the You can select databases as synchronization objects. You cannot select individual keys.
   Edit mapped name	You cannot rename objects in this scenario.
   Replicate temporary tables during online DDL with DMS	If you use Data Management (DMS) to perform online DDL changes on the source database, you can choose whether to synchronize the temporary tables generated by the DDL changes. Yes: Synchronizes the temporary tables generated by online DDL changes. Note If a large amount of temporary table data is generated by online DDL changes, the data synchronization task may be delayed. No: Does not synchronize the temporary tables generated by online DDL changes. Only the original DDL operations from the source database are synchronized. Note This option causes tables in the destination database to be locked.
   Connection retry duration	If DTS cannot connect to the source or destination instance, it retries for 720 minutes (12 hours) by default. You can also specify a custom retry duration. If DTS reconnects to the source or destination instance within the specified duration, the synchronization task automatically resumes. Otherwise, the task fails. Note You are billed for task run time during connection retries. Customize the retry duration based on your business needs, or release the DTS instance as soon as the source and destination instances are released.

9. Click Next.

10. The initial synchronization method is fixed as Full Data + Incremental Data.

    Note

    • DTS synchronizes existing data from the source to the destination Redis instance, then synchronizes incremental data.

    • If a version-related error appears, upgrade the source Redis instance as prompted. Upgrade the major version. Upgrade a minor version and a proxy version.

11. After you finish the configuration, click Precheck and Start in the lower-right corner of the page.

    Note

    • Before the synchronization task starts, DTS performs a precheck. The task can start only after passing the precheck.

    • If the precheck fails, click the icon next to a check item to view the failure details.

      • You can fix the issues based on the details and run the precheck again.

      • If you do not need to fix the warning items, you can click Ignore or Ignore and Rerun Precheck to proceed.

12. After Precheck Passed is displayed in the Precheck dialog box, close the Precheck dialog box. The synchronization job will then start.

13. Wait for the task to initialize. Its status changes to Synchronizing.

    Note

    You can view the task status on the Data Synchronization page.