You can use Data Transmission Service (DTS) to perform one-way data synchronization between ApsaraDB for Redis instances that belong to different Alibaba Cloud accounts. This feature is suitable for scenarios such as resource migration or resource merging across Alibaba Cloud accounts and business architecture adjustment.

Prerequisites

  • The engine version of the source ApsaraDB for Redis instance is 4.0 or 5.0 if the instance is a Community Edition instance, and is 5.0 if the instance is an Enhanced Edition instance.
    Note The engine version of the destination ApsaraDB for Redis instance can be 4.0 or 5.0. The engine version of the destination ApsaraDB for Redis instance cannot be earlier than that of the source ApsaraDB for Redis instance. Before you synchronize data between two ApsaraDB for Redis instances that use different engine versions, check the compatibility of the two versions. For example, you can create a destination ApsaraDB for Redis instance that adopts the pay-as-you-go billing method to verify the compatibility between the source and destination instances. Then, you can release this instance or change the billing method of the instance to subscription.
  • Cluster, standard, or read/write splitting instances with local disks are configured as the source and destination instances.
  • For instances with cloud disks, standard instances are configured.
  • The source instance is deployed in a virtual private cloud (VPC). If the source ApsaraDB for Redis instance is deployed in the classic network, you can change the network type to VPC. For more information, see Change the network type from classic network to VPC.
  • Secure Sockets Layer (SSL) encryption is disabled for the source instance. For more information, see Configure SSL encryption.
  • The available storage space of the destination instance is larger than the total size of the data in the source instance.
  • To ensure compatibility between the source and destination instances, if an ApsaraDB for Redis Enhanced Edition (Tair) instance is configured as the source instance, the destination instance is an ApsaraDB for Redis Enhanced Edition (Tair) instance. ApsaraDB for Redis Enhanced Edition (Tair) is integrated with more Redis modules than ApsaraDB for Redis Community Edition.
  • The appendonly parameter is set to yes if a persistent memory-optimized instance is configured as the source instance.
  • No storage-optimized instance of the ApsaraDB for Redis instance of Enhanced Edition (Tair) is configured as the source instance.

Background information

Assume that two ApsaraDB for Redis instances are created by different Alibaba Cloud accounts. You want to synchronize data from the instance of Account A to the instance of Account B. The following figure shows the architecture of the synchronization solution. Data synchronization between ApsaraDB for Redis instances across different accounts

The following table describes how to configure a one-way data synchronization task for this scenario.

Step Description
1. Use Account A to log on to the Resource Access Management (RAM) console and grant the required permissions to a RAM role. For more information, see the "Preparations" section of this topic. When you configure the RAM role, you can set Account B as the trusted account, and authorize the RAM role to access the resources of Account A.
2. Use Account B to log on to the DTS Console and configure the data synchronization task. For more information, see the "Procedure" section of this topic. DTS can read information about the source ApsaraDB for Redis instance across Alibaba Cloud accounts.

Supported source and destination databases

The following table describes the supported types of databases when you use DTS to synchronize data between Redis databases across Alibaba Cloud accounts.
Source database Destination database
  • ApsaraDB for Redis instance
  • Self-managed database hosted on Elastic Compute Service (ECS)
  • Self-managed database connected over Express Connect, VPN Gateway, or Smart Access Gateway
  • Self-managed database connected over CEN
  • ApsaraDB for Redis instance
  • Self-managed database hosted on ECS
  • Self-managed database connected over Express Connect, VPN Gateway, or Smart Access Gateway
  • Self-managed database connected over CEN

Limits

Category Description
Limits on the source database
  • The collections to synchronize must have PRIMARY KEY or UNIQUE constraints, and all fields must be unique. Otherwise, the destination database may contain duplicate data records.
  • To ensure the synchronization quality, Data Transmission Service (DTS) adds the following key to the source database: DTS_REDIS_TIMESTAMP_HEARTBEAT. This key is used to record the time when data is synchronized to the destination database.
  • To ensure the stability of data synchronization, we recommend that you increase the value of the repl-backlog-size parameter in the redis.conf file.
  • The following requirements for append-only file (AOF) logging must be met:
    • The AOF logging feature must be enabled.
    • If you perform only incremental data synchronization, the AOF logs of the source database must be stored for more than 24 hours. If you perform both full data synchronization and incremental data synchronization, the AOF logs of the source database must be stored for at least seven days. After the full data synchronization is complete, you can set the retention period to more than 24 hours. Otherwise, DTS may fail to obtain the AOF logs and your data synchronization task may fail. In exceptional circumstances, data inconsistency or loss may occur. Make sure that you set the retention period of AOF logs in accordance with the preceding requirements. Otherwise, the Service Level Agreement (SLA) of DTS does not ensure service reliability or performance.

    Note This limit is not applied to ApsaraDB for Redis instances.
  • If an expiration policy is enabled for some keys in the source database, these keys may not be deleted in a timely manner after they expired. Therefore, the number of keys in the destination database may be less than that in the source database. You can run the info command to view the number of keys in the destination database.
  • If the data eviction policy (maxmemory-policy) of the destination database is not set to noeviction, data inconsistency may occur between the source and destination databases. For more information about data eviction policies, see How does ApsaraDB for Redis evict data by default?
  • Limits on synchronizing data from a standalone ApsaraDB for Redis instance to an ApsaraDB for Redis cluster instance: Each command can be run only on a single slot in an ApsaraDB for Redis cluster instance. If you perform operations on multiple keys in the source database and the keys belong to different slots, the following error occurs:
    CROSSSLOT Keys in request don't hash to the same slot
    We recommend that you perform operations on only one key during data synchronization. Otherwise, the data synchronization task will be interrupted.
Other limits
  • During data synchronization, if the number of shards in the source ApsaraDB for Redis instance is increased or decreased, or if the database specifications are changed (for example, the memory capacity is scaled up), you must reconfigure the task. To ensure data consistency, we recommend that you clear the data that has been synchronized to the destination Redis database before you reconfigure the task.
  • During data synchronization, if the endpoint of the source self-managed Redis database is changed, you must submit a ticket to update the change. Otherwise, the AOF files of the self-managed Redis database may be reset. In this case, you must reconfigure the task.
  • To ensure compatibility, the version of the destination database must be the same as or later than that of the source database. If the version of the destination database is earlier than that of the source database, database compatibility issues may occur.
  • Before you synchronize data, evaluate the impact of data synchronization on the performance of the source and destination databases. We recommend that you synchronize data during off-peak hours. During full data synchronization, DTS uses read and write resources of the source and destination databases. This may increase the loads of the database servers.
  • During data synchronization, we recommend that you use only DTS to write data to the destination database. This prevents data inconsistency between the source and destination databases.

Billing

Synchronization type Task configuration fee
Schema synchronization and full data synchronization Free of charge.
Incremental data synchronization Charged. For more information, see Pricing.

SQL operations that can be synchronized

Edition SQL operation
ApsaraDB for Redis Community Edition instance
  • APPEND
  • BITOP, BLPOP, BRPOP, and BRPOPLPUSH
  • DECR, DECRBY, and DEL
  • EVAL, EVALSHA, EXEC, EXPIRE, and EXPIREAT
  • GEOADD and GETSET
  • HDEL, HINCRBY, HINCRBYFLOAT, HMSET, HSET, and HSETNX
  • INCR, INCRBY, and INCRBYFLOAT
  • LINSERT, LPOP, LPUSH, LPUSHX, LREM, LSET, and LTRIM
  • MOVE, MSET, MSETNX, and MULTI
  • PERSIST, PEXPIRE, PEXPIREAT, PFADD, PFMERGE, and PSETEX
  • RENAME, RENAMENX, RESTORE, RPOP, RPOPLPUSH, RPUSH, and RPUSHX
  • SADD, SDIFFSTORE, SELECT, SET, SETBIT, SETEX, SETNX, SETRANGE, SINTERSTORE, SMOVE, SPOP, SREM, and SUNIONSTORE
  • ZADD, ZINCRBY, ZINTERSTORE, ZREM, ZREMRANGEBYLEX, ZUNIONSTORE, ZREMRANGEBYRANK, and ZREMRANGEBYSCORE
  • SWAPDB and UNLINK. These two SQL operations can be synchronized only if the engine version of the source instance is Redis 4.0.
ApsaraDB for Redis Enhanced Edition (Tair) instance
  • APPEND
  • BITOP, BLPOP, BRPOP, and BRPOPLPUSH
  • DECR, DECRBY, and DEL
  • EVAL, EVALSHA, EXEC, EXPIRE, and EXPIREAT
  • GEOADD and GETSET
  • HDEL, HINCRBY, HINCRBYFLOAT, HMSET, HSET, and HSETNX
  • INCR, INCRBY, and INCRBYFLOAT
  • LINSERT, LPOP, LPUSH, LPUSHX, LREM, LSET, and LTRIM
  • MOVE, MSET, MSETNX, and MULTI
  • PERSIST, PEXPIRE, PEXPIREAT, PFADD, PFMERGE, and PSETEX
  • RENAME, RENAMENX, RPOP, RPOPLPUSH, RPUSH, and RPUSHX
  • SADD, SDIFFSTORE, SELECT, SET, SETBIT, SETEX, SETNX, SETRANGE, SINTERSTORE, SMOVE, SPOP, SREM, and SUNIONSTORE
  • UNLINK, ZADD, ZINCRBY, ZINTERSTORE, ZREM, ZREMRANGEBYLEX, ZUNIONSTORE, ZREMRANGEBYRANK, and ZREMRANGEBYSCORE
Note
  • PUBLISH operations cannot be synchronized.
  • If you run the EVAL or EVALSHA command to call Lua scripts, DTS cannot identify whether these Lua scripts are executed on the destination instance. This is because the destination instance does not explicitly return the execution results of Lua scripts during incremental data synchronization.
  • When DTS runs the SYNC or PSYNC command to transfer data of the LIST type, DTS does not clear the existing data in the destination instance. As a result, the destination instance may contain duplicate data records.

Preparations

Use the Alibaba Cloud account that owns the source ApsaraDB for Redis instance to log on to the RAM console. Then, create a RAM role and authorize the RAM role to access the resources of the Alibaba Cloud account. For more information, see Configure RAM authorization for cross-account data migration and synchronization.

Procedure

  1. Use the Alibaba Cloud account that owns the destination ApsaraDB for Redis instance to log on to the new DTS console and go to the Data Synchronization page.
    Note You can also log on to the Data Management (DMS) console. In the top navigation bar, choose DTS > Data Synchronization.
  2. In the top navigation bar, select the region where the data synchronization instance resides.
    Region
  3. Click Create Task. On the page that appears, configure the source and destination databases.
    Warning After you select the source and destination instances, we recommend that you read the limits displayed at the top of the page. This helps you create and run the data synchronization task.
    Section Parameter Description
    N/A Task Name

    DTS automatically generates a task name. We recommend that you specify an informative name to identify the task. You do not need to use a unique task name.

    Source Database Select Instance
    Select whether to use an existing instance.
    • If you use an existing instance, DTS automatically applies the parameter settings of the instance.
    • If you do not use an existing instance, you must configure the following parameters.
    Database Type Select ApsaraDB for Redis Enhanced Edition (Tair).
    Access Method Select Alibaba Cloud Instance.
    Instance Region The region where the source ApsaraDB for Redis instance resides.
    Replicate Data Across Alibaba Cloud Accounts Select Yes.
    Note The two accounts can be an Alibaba Cloud account on the China site (aliyun.com) and an Alibaba Cloud account on the International site (alibabacloud.com).
    Alibaba Cloud Account The ID of the Alibaba Cloud account that owns the source ApsaraDB for Redis instance.
    Note To obtain the ID of the Alibaba Cloud account, you must log on to the Account Management console by using this account. The account ID is displayed on the Basic Information page.
    Obtain an Alibaba Cloud account ID
    RAM Role Name The name of the RAM role that you created in Preparations.
    RDS Instance ID The ID of the source ApsaraDB for Redis instance.
    Database Password The database password of the source ApsaraDB for Redis instance. The database account that corresponds to the password must have read permissions. If you forget the password, you can reset the password. For more information, see Change or reset the password.
    • If you use the default account whose username is the same as the instance ID, enter the password.
    • If you create a custom account, specify the password in the <Custom account username>:<Password> format. Example: testaccount:Test1234.
    Note This parameter is optional and can be left empty if no database password is set.
    Destination Database Select Instance
    Select whether to use an existing instance.
    • If you use an existing instance, DTS automatically applies the parameter settings of the instance.
    • If you do not use an existing instance, you must configure the following parameters.
    Database Type Select ApsaraDB for Redis Enhanced Edition (Tair).
    Access Method Select Alibaba Cloud Instance.
    Instance Region The region where the destination ApsaraDB for Redis instance resides.
    Instance ID The ID of the destination ApsaraDB for Redis instance.
    Database Password The database password of the destination ApsaraDB for Redis instance. The database account that corresponds to the password must have read permissions. If you forget the password, you can reset the password. For more information, see Change or reset the password.
    • If you use the default account whose username is the same as the instance ID, enter the password.
    • If you create a custom account, specify the password in the <Custom account username>:<Password> format. Example: testaccount:Test1234.
  4. In the lower part of the page, click Test Connectivity and Proceed.
    Note
    • You do not need to modify the security settings for ApsaraDB instances (such as ApsaraDB RDS for MySQL and ApsaraDB for MongoDB) and ECS-hosted databases. DTS automatically adds the CIDR blocks of DTS servers to the whitelists of ApsaraDB instances or the security group rules of Elastic Compute Service (ECS) instances. For more information, see Add the CIDR blocks of DTS servers to the security settings of on-premises databases.
    • After data synchronization is completed, we recommend that you remove the CIDR blocks of DTS servers from the whitelists or security groups.
  5. Select objects for the task and configure advanced settings.
    • Basic Settings
      Parameter Description
      Task Stages By default, Incremental Data Synchronization is selected. You must also select Full Data Synchronization. After the precheck, DTS synchronizes the historical data of the selected objects from the source instance to the destination instance. The historical data is the basis for subsequent incremental synchronization.
      Note Full data synchronization is supported only for ApsaraDB for Redis Enhanced Edition (Tair) instances.
      Processing Mode of Conflicting Tables
      • Precheck and Report Errors: checks whether the destination database contains tables that have the same names as tables in the source database. If the source and destination databases do not contain identical table names, the precheck is passed. Otherwise, an error is returned during the precheck and the data synchronization task cannot be started.

      • Ignore Errors and Proceed: skips the precheck for identical table names in the source and destination databases.
        Warning If you select Ignore Errors and Proceed, data consistency is not guaranteed and your business may be exposed to potential risks.
        • If the source and destination databases have the same schema, and a data record has the same primary key as an existing data record in the destination database:
          • During full data synchronization, DTS does not synchronize the data record to the destination database. The existing data record in the destination database is retained.
          • During incremental data synchronization, DTS synchronizes the data record to the destination database. The existing data record in the destination database is overwritten.
        • If the source and destination databases have different schemas, data may fail to be initialized. In this case, only some columns are synchronized or the data synchronization task fails.
      Synchronization Topology Select One-way Synchronization.
      Select Objects

      Select one or more objects from the Source Objects section and click the Rightwards arrow icon to add the objects to the Selected Objects section.

      Note You can select databases as the objects to be synchronized. Keys cannot be selected as the objects to be synchronized.
      Rename Databases and Tables In this scenario, you cannot rename objects.
      Select the SQL operations to be synchronized In the Selected Objects section, right-click an object. In the dialog box that appears, select the SQL operations that you want to synchronize. For more information, see the "SQL operations that can be synchronized" section of this topic.
    • Advanced Settings
      Parameter Description
      Set Alerts
      Specify whether to set alerts for the data synchronization task. If the task fails or the synchronization latency exceeds the threshold, the alert contacts will receive notifications.
      • Select No if you do not want to set alerts.
      • Select Yes to set alerts. In this case, you must also set the alert threshold and alert contacts.
      Retry Time for Failed Connection
      Specify the retry time range for failed connections. If a data synchronization task is disconnected, DTS immediately retries a connection within the specified time range. Valid values: 10 to 1440. Unit: minutes. Default value: 120. We recommend that you set the retry time range to more than 30 minutes. If DTS reconnects to the source and destination databases within the specified time range, DTS resumes the data synchronization task. Otherwise, the data synchronization task fails.
      Note
      • If multiple DTS instances have the same source or destination database, the lowest value of this parameter takes effect. For example, the retry time range is set to 30 minutes for Instance A and 60 minutes for Instance B, DTS retries failed connections within 30 minutes.
      • When DTS retries a connection, you are charged for the DTS instance. We recommend that you specify the retry time range based on your business requirements. You can also release the DTS instance at your earliest opportunity after the source and destination instances are released.
      Configure ETL
      Specify whether you want to enable the ETL feature. For more information, see What is ETL?.
      • If you select Yes, the ETL feature is enabled. You must specify a policy in the text box.
      • If you select No, the ETL feature is disabled.
  6. In the lower part of the page, click Next: Save Task Settings and Precheck.
    Note
    • Before you can start the data synchronization task, DTS performs a precheck. You can start the data synchronization task only after the task passes the precheck.
    • If the task fails to pass the precheck, you can click the Info icon icon next to each failed item to view details.
      • You can troubleshoot the issues based on the causes and run a precheck again.
      • If you do not need to troubleshoot the issues, you can ignore failed items and run a precheck again.
  7. Wait until the Success Rate becomes 100%. Then, click Next: Purchase Instance.
  8. On the Purchase Instance page, specify the Billing Method and Instance Class parameters for the data synchronization instance. The following table describes the parameters.
    Section Parameter Description
    Parameters Billing Method
    • Subscription: You pay for your subscription when you create an instance. We recommend that you select the subscription billing method for long-term use because it is more cost-effective than the pay-as-you-go billing method. You can save more costs with longer subscription periods.
    • Pay-as-you-go: A pay-as-you-go instance is billed on an hourly basis. We recommend that you select the pay-as-you-go billing method for short-term use. If you no longer require a pay-as-you-go instance, you can release the instance to reduce costs.
    Instance Class DTS provides several instance classes that have different performance in synchronization speed. You can select an instance class based on your business scenario. For more information, see Specifications of data synchronization instances.
    Subscription Length If you select the subscription billing method, set the subscription length and the number of instances that you want to create. The subscription length can be one to nine months or one to three years.
    Note This parameter is available only if you select the subscription billing method.
  9. Read and select Data Transmission Service (Pay-as-you-go) Service Terms.
  10. Click Buy and Start to start the data synchronization task. You can view the progress of the task in the task list.