All Products
Search

This Product

    Data Transmission Service:Migrate from SLS to a ClickHouse cluster

    Document Center

    Data Transmission Service:Migrate from SLS to a ClickHouse cluster

    Last Updated:Jun 22, 2026

    You can use Data Transmission Service (DTS) to migrate data from Simple Log Service (SLS) to an ApsaraDB for ClickHouse cluster.

    Important

    This feature is in preview. To enable this feature, submit a ticket.

    Prerequisites

    • You have collected the data to be migrated into an SLS Logstore. For more information, see Data collection.

    • You have created a destination ApsaraDB for ClickHouse cluster. For more information, see Create a cluster.

    • You have created a database and a table in the destination ApsaraDB for ClickHouse cluster to receive the data. For more information, see Create a database, Create a table, and Object naming limits.

      Note

      • A Logstore in SLS corresponds to a database in the ApsaraDB for ClickHouse cluster, and a topic corresponds to a table.

      • In the ApsaraDB for ClickHouse cluster, do not create columns for index fields in the Logstore that start with __ and end with __.

    Usage notes

    Type

    Description

    Source database limits

    • Ensure that data scheduled for migration is retained in the Logstore for at least 7 days.

    • Disable automatic sharding for the source Logstore. For more information, see Modify LogStore configuration.

    • Do not split or merge shards while the DTS instance is running. Otherwise, data loss may occur.

    Other limits

    • This feature is in canary release and is available only to specific users.

    • Schema migration is not supported. You must manually create tables and add the necessary fields in the destination database. For more information about table and field requirements, see Table and field information.

    • You cannot migrate an index field that starts with __ and ends with __.

    • If you want to migrate data from all topics in a Logstore to the same table in an ApsaraDB for ClickHouse cluster, enter .* in the Table Name text box when you configure migration objects. Otherwise, some topics may be filtered out, resulting in data loss.

    • Restarting a data migration task may result in a small amount of duplicate data.

    • A task includes both full and incremental data migration but appears only as incremental data migration in the task list.

    • The number of databases to be migrated cannot exceed the ApsaraDB for ClickHouse limit of 256.

    • If you purchase an instance before configuring it, you must select the correct value for Number of distributed database nodes during the purchase.

      Note

      The value for Number of distributed database nodes must match the number of shards in the source Logstore. For more information about how to view the number of shards in a Logstore, see Manage Shards.

    • During full data migration, DTS consumes read and write resources from the source and destination databases. Assess database performance before starting the migration and run the task during off-peak hours, for example, when CPU utilization of both databases is below 30%.

    • To prevent data inconsistency, do not write data to the destination database from sources other than DTS during the data migration.

    • If a task fails, DTS support staff will attempt to restore it within eight hours. During restoration, they may restart the task or adjust its parameters.

      Note

      Only DTS task parameters are modified—not database parameters. Parameters that may be adjusted include those listed in Modify instance parameters.

    Billing

    Migration type

    Task configuration fee

    Internet traffic fee

    Full data migration

    Free.

    Free in this example.

    Incremental data migration

    Charged. For details, see the billing overview.

    Database account permissions

    Database

    Permissions

    References

    Destination ApsaraDB for ClickHouse cluster

    • Versions 22.8 and later require read and write permissions on the destination database. A privileged account has these permissions.

    • Version 21.8 requires Read, Write, and Set Permissions and Enable DDL.

    Manage database accounts for Community-compatible Edition

    Prerequisites

    Before you create a DTS instance, create a RAM user and grant it the required permissions. DTS uses the RAM user's AccessKey pair to read data from SLS.

    1. Create a RAM user.

      1. Go to the Users page on the RAM console.

      2. In the upper-left corner of the page, click Create User.

      3. Enter a Logon Name. For Access Mode, select to generate a permanent AccessKey.

      4. Click OK.

        Important

        After the RAM user is created, save the AccessKey ID and AccessKey Secret.

    2. Return to the Users page and find the RAM user that you created.

    3. For the target RAM user, click Add Permissions in the Actions column.

    4. In the search box under permission policy, search for and select the AliyunLogReadOnlyAccess policy.

      Note

      The AliyunLogReadOnlyAccess permission policy grants read-only permissions on SLS. A RAM user with this policy can access SLS. You can also create a custom policy to meet your specific needs. For more information, see Create a custom policy.

    5. Click Grant Permissions.

    Procedure

    1. Navigate to the migration task list page for the destination region using one of the following methods.

      From the DTS console

      1. Log on to the Data Transmission Service (DTS) console.

      2. In the navigation pane on the left, click Data Migration.

      3. In the upper-left corner of the page, select the region where the migration instance is located.

      From the DMS console

      Note

      The actual operations may vary based on the mode and layout of the DMS console. For more information, see Simple mode console and Customize the layout and style of the DMS console.

      1. Log on to the Data Management (DMS) console.

      2. In the top menu bar, choose Data + AI > Data Transmission (DTS) > Data Migration.

      3. To the right of Data Migration Tasks, select the region where the migration instance is located.

    2. Click Create Task to navigate to the task configuration page.

    3. Optional: In the upper-right corner of the page, click New Configuration Page.

      Note

      • If you are already on the new configuration page (the button in the upper-right corner is Back to Previous Version), you can skip this step.

      • The parameters on the new and old configuration pages are different. We recommend that you use the new configuration page.

    4. Configure the source and destination databases.

      Category

      Parameter

      Description

      N/A

      Task Name

      DTS automatically generates a task name. We recommend that you specify a descriptive name for easy identification. The name does not need to be unique.

      Source Database

      Select Existing Connection

      • To use a database instance that has been added to the system (created or saved), select the desired database instance from the drop-down list. The database information below will be automatically configured.

        Note

        In the DMS console, this parameter is named Select a DMS database instance..

      • If you have not registered the database instance with the system, or do not need to use a registered instance, manually configure the database information below.

      Database Type

      Select SLS.

      Access Method

      Select Alibaba Cloud Instance.

      Instance Region

      Select the region of the source SLS Project.

      Project

      Select the source SLS Project.

      Logstore

      Select the source SLS Logstore.

      AccessKey ID

      Enter the AccessKey ID and AccessKey Secret of the RAM user that you created in the prerequisites.

      AccessKey Secret

      Destination Database

      Select Existing Connection

      • To use a database instance that has been added to the system (created or saved), select the desired database instance from the drop-down list. The database information below will be automatically configured.

        Note

        In the DMS console, this parameter is named Select a DMS database instance..

      • If you have not registered the database instance with the system, or do not need to use a registered instance, manually configure the database information below.

      Database Type

      Select ClickHouse.

      Access Method

      Select Alibaba Cloud Instance.

      Instance Region

      Select the region of the destination ApsaraDB for ClickHouse cluster.

      Replicate Data Across Alibaba Cloud Accounts

      In this example, a database instance under the current Alibaba Cloud account is used. Select No.

      Cluster Type

      Select the type of the ApsaraDB for ClickHouse cluster based on your business requirements.

      Cluster ID

      Select the ID of the destination ApsaraDB for ClickHouse cluster.

      Database Account

      Enter the database account for the destination ApsaraDB for ClickHouse cluster. For information about the required permissions, see Permissions required for database accounts.

      Database Password

      Enter the password for the database account.

    5. After you complete the configuration, click Test Connectivity and Proceed at the bottom of the page.

      Note

      Ensure that the DTS service IP address segments are automatically or manually added to the security settings of the source and destination databases to allow access from DTS servers. For more information, see Add DTS server IP addresses to a whitelist.

    6. Configure the task objects.

      1. On the Configure Objects page, configure the objects that you want to migrate.

        Parameter

        Description

        Migration Types

        Only Full Data Migration + Incremental Data Migration is supported.

        Processing Mode of Conflicting Tables

        • Precheck and Report Errors: Checks whether tables with the same names exist in the destination database. If no tables with the same names exist, the precheck is passed. If tables with the same names exist, an error is reported during the precheck, and the data migration task does not start.

          Note

          If a table in the destination database has the same name but cannot be easily deleted or renamed, you can change the name of the table in the destination database. For more information, see Object name mapping.

        • Ignore Errors and Proceed: Skips the check for tables with the same names.

          Warning

          Selecting Ignore Errors and Proceed may cause data inconsistency and business risks. For example:

          • If the table schemas are consistent and a record in the destination database has the same primary key value as a record in the source database:

            • During full migration, DTS keeps the record in the destination database. The record from the source database is not migrated.

            • During incremental migration, DTS does not keep the record in the destination database. The record from the source database overwrites the record in the destination database.

          • If the table schemas are inconsistent, only some columns of data may be migrated, or the migration may fail. Proceed with caution.

        Capitalization of Object Names in Destination Instance

        You can configure the case sensitivity policy for the names of migrated objects, such as databases, tables, and columns, in the destination instance. By default, DTS default policy is selected. You can also choose to keep the case sensitivity consistent with the default policy of the source or destination database. For more information, see Case sensitivity of object names in the destination database.

        Selected Objects

        1. Specify the database to receive data.

          1. In the Selected Objects section, hover over the source Logstore and click the Edit icon that appears.

          2. In the Edit Schema dialog box, change the Schema Name to the name of the database that will receive the data in the destination ApsaraDB for ClickHouse cluster.

            Note

            By default, DTS maps the Logstore name to the database name of the destination ClickHouse cluster. If the Logstore name contains the - character, the character is replaced with _.

          3. Click OK.

        2. Specify the table to receive data.

          1. Click Add Table next to Table.

          2. In the Add Table dialog box, configure the migration objects and the destination table.

            You can specify the granularity of migration objects as a Logstore or a topic:

            • Logstore granularity: In the Table Name text box, enter .*.

              Important

              .* indicates all topics in the Logstore. Data from all topics in a Logstore can only be written to a single table.

            • Topic granularity: In the Table Name text box, enter the name of the source topic in the Logstore.

          3. In the Destination Table Name text box, enter the name of the table that will receive data in the destination ApsaraDB for ClickHouse cluster.

            Note

            • If the migration object is a topic, click Add Table and map each topic to a table in the ApsaraDB for ClickHouse cluster.

            • If the migration object is a Logstore, do not click Add Table.

          4. Click OK.

        3. Specify the columns to receive data.

          Note

          If the migration objects are topics, you must repeat the following steps to map each topic to a table in the ApsaraDB for ClickHouse cluster.

          1. In the Selected Objects section, hover over the newly created table and click the Edit icon that appears.

          2. In the Edit Table dialog box, determine the index fields (columns) to migrate.

            Note

            To exclude a field from migration, click the image icon in the Actions column for that field.

          3. In the Destination Column Name text box, edit the name to match the corresponding column in the destination table.

          4. Optional: In the Target Column Type column, select the data type for the corresponding column.

            Note

            For information about the default data type mappings in DTS, see Migrate data from SLS to an ApsaraDB for ClickHouse cluster.

          5. Optional: Configure non-indexed fields to migrate.

            Note

            You can repeat the following steps to configure the non-indexed fields you need to migrate.

            1. Click the image icon to add a new row and configure it.

            2. In the Column Name column, enter the name of the non-indexed field to migrate.

            3. In the Destination Column Name column, enter the name of the column that will receive the data in the ApsaraDB for ClickHouse cluster.

            4. Optional: In the Target Column Type column, select the data type for the corresponding column.

              Note

              For information about the default data type mappings in DTS, see Migrate data from SLS to an ApsaraDB for ClickHouse cluster.

          6. After configuring each field to be migrated, click Confirm.

      2. Click Next: Advanced Settings to configure advanced parameters.

        Parameter

        Description

        Dedicated Cluster for Task Scheduling

        By default, DTS schedules tasks on a shared cluster. You do not need to select one. If you want more stable tasks, you can purchase a dedicated cluster to run DTS migration tasks.

        Retry Time for Failed Connections

        After the migration task starts, if the connection to the source or destination database fails, DTS reports an error and immediately begins to retry the connection. The default retry duration is 720 minutes. You can customize the retry time to a value from 10 to 1440 minutes. We recommend that you set the duration to more than 30 minutes. If DTS reconnects to the source and destination databases within the specified duration, the migration task automatically resumes. Otherwise, the task fails.

        Note

        • For multiple DTS instances that share the same source or destination, the network retry time is determined by the setting of the last created task.

        • Because you are charged for the task during the connection retry period, we recommend that you customize the retry time based on your business needs, or release the DTS instance as soon as possible after the source and destination database instances are released.

        Retry Time for Other Issues

        After the migration task starts, if a non-connectivity issue, such as a DDL or DML execution exception, occurs in the source or destination database, DTS reports an error and immediately begins to retry the operation. The default retry duration is 10 minutes. You can customize the retry time to a value from 1 to 1440 minutes. We recommend that you set the duration to more than 10 minutes. If the related operations succeed within the specified retry duration, the migration task automatically resumes. Otherwise, the task fails.

        Important

        The value of Retry Time for Other Issues must be less than the value of Retry Time for Failed Connections.

        Start Incremental Synchronization from Selected Offset

        You can choose whether to start migrating data from a specific checkpoint based on your business requirements. In this example, the default setting No is used.

        Note

        If you select Yes, you must also select the checkpoint from the drop-down list.

        Enable Throttling for Incremental Data Migration

        If required, you can also choose to set speed limits for the incremental migration task. You can set RPS of Incremental Data Migration and Data migration speed for incremental migration (MB/s) to reduce the load on the destination database.

        Note

        • This configuration item is available only if you select Incremental Data Migration for Migration Types.

        • You can also adjust the incremental migration speed after the migration instance is running.

        Environment Tag

        Optional. Select an environment tag to identify the instance.

        Configure ETL

        Choose whether to enable the extract, transform, and load (ETL) feature. For more information, see What is ETL? Valid values:

        Monitoring and Alerting

        Select whether to set alerts and receive alert notifications based on your business needs.

        • No: Does not set an alert.

        • Yes: Configure alerts by setting an alert threshold and an alert notifications. If a migration fails or the latency exceeds the threshold, the system sends an alert notification.

    7. Save the task and run a precheck.

      • To view the parameters for configuring this instance when you call the API operation, move the pointer over the Next: Save Task Settings and Precheck button and click Preview OpenAPI parameters in the bubble that appears.

      • If you do not need to view or have finished viewing the API parameters, click Next: Save Task Settings and Precheck at the bottom of the page.

      Note

      • Before the migration task starts, DTS performs a precheck. The task starts only after it passes the precheck.

      • If the precheck fails, click View Details next to the failed check item, fix the issue based on the prompt, and then run the precheck again.

      • If a warning is reported during the precheck:

        • For check items that cannot be ignored, click View Details next to the failed item, fix the issue based on the prompt, and then run the precheck again.

        • For check items that can be ignored, you can click Confirm Alert Details, Ignore, OK, and Precheck Again to skip the alert item and run the precheck again. If you choose to ignore a warning, it may cause issues such as data inconsistency and pose risks to your business.

    8. Purchase the instance.

      1. When the Success Rate is 100%, click Next: Purchase Instance.

      2. On the Purchase page, select the link specification for the data migration instance. For more information, see the following table.

        Category

        Parameter

        Description

        New Instance Class

        Resource Group Settings

        Select the resource group to which the instance belongs. The default value is default resource group. For more information, see What is Resource Management?

        Instance Class

        DTS provides migration specifications with different performance levels. The link specification affects the migration speed. You can select a specification based on your business scenario. For more information, see Data migration link specifications.

      3. After the configuration is complete, read and select Data Transmission Service (Pay-as-you-go) Service Terms.

      4. Click Buy and Start. In the OK dialog box that appears, click OK.

        You can view the progress of the migration task on the Data Migration Tasks list page.

        Note

        • If the migration task does not include incremental migration, it stops automatically after the full migration is complete. After the task stops, its Status changes to Completed.

        • If the migration task includes incremental migration, it does not stop automatically. The incremental migration task continues to run. While the incremental migration task is running, the Status of the task is Running.

    Appendix

    Data type mapping

    SLS

    ClickHouse cluster

    TEXT

    STRING

    JSON

    STRING

    DOUBLE

    FLOAT64

    LONG

    INT64

    Table and field information

    Table information

    When you manually create a destination table, ensure that it meets the following requirements.

    Important

    The ENGINE of the destination table must be ENGINE = ReplicatedReplacingMergeTree(_version, _is_deleted) to prevent data inconsistency.

    • For a ClickHouse Community Edition instance, you must create a local table and a distributed table. The name of the distributed table must match the table name specified for the synchronization objects. The local table must be named <distributed_table_name>+_local.

    • For a ClickHouse Enterprise Edition instance, you must create a table with the same name specified for the synchronization objects.

    Field information

    In addition to the source fields, the destination table must contain the following fields. Run the SELECT * FROM table_name final WHERE _sign>0; statement to query data. The WHERE clause filters out deleted data, and the final keyword deduplicates rows based on the sorting key.

    Version

    Parameter

    Type

    Default

    Description

    ClickHouse Community Edition earlier than 23.8

    _sign

    Int8

    1

    The DML operation type.

    • INSERT: The value is 1.

    • UPDATE: The value is 1.

    • DELETE: The value is -1.

    _version

    UInt64

    1

    The timestamp when the data is written to the ClickHouse cluster.

    ClickHouse Enterprise Edition and ClickHouse Community Edition 23.8 and later

    _sign

    Int8

    1

    The DML operation type.

    • INSERT: The value is 1.

    • UPDATE: The value is 1.

    • DELETE: The value is -1.

    _is_deleted

    UInt8

    0

    Indicates whether the record is deleted.

    • INSERT: The value is 0.

    • DELETE: The value is 1.

    • UPDATE: The value is 0.

    _version

    UInt64

    1

    The timestamp when the data is written to the ClickHouse cluster.