All Products
Search

This Product

    DataWorks:Real-time synchronization from LogHub (SLS) to MaxCompute

    Document Center

    DataWorks:Real-time synchronization from LogHub (SLS) to MaxCompute

    Last Updated:Jul 17, 2026

    DataWorks Data Integration provides real-time, single-table synchronization tasks that capture data changes (insertions, deletions, and updates) at the source and apply them to the destination with low latency and high throughput. This topic describes how to configure a real-time synchronization task from LogHub (SLS) to MaxCompute.

    Prerequisites

    Step 1: Create a synchronization task

    1. Log on to the DataWorks console. In the target region, click Data Integration > Data Integration in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Integration.

    2. In the left-side navigation pane, click Synchronization Task. At the top of the page, click Create Synchronization Task and configure the task parameters. This example writes real-time data from LogHub (SLS) to MaxCompute.

      • Source Type: LogHub.

      • Destination Type: MaxCompute.

      • Specific Type: Single-table real-time.

      • Synchronization Mode:

        • Schema Migration: Automatically creates database objects such as tables, fields, and data types at the destination to match the source schema. This step does not include data.

        • Incremental Sync: Continuously captures data changes such as new, modified, and deleted data at the source and synchronizes the changes to the destination.

    Note

    For more information about supported data sources and synchronization solutions, see Supported data sources and synchronization solutions.

    Step 2: Configure data sources and runtime resources

    1. For Source Information, select the LogHub data source that you added. For Destination, select the MaxCompute data source that you added.

    2. In the Running Resources section, select a Resource Group for the synchronization task and allocate Resource Group CUs to the task. To precisely control resource usage and prevent waste, you can set CUs separately for full and incremental synchronization. If your synchronization task fails due to an out-of-memory (OOM) error, increase the CU allocation for the resource group.

      Note

      A CU is a billing unit for a resource group, where one CU is approximately 1 vCPU and 4 GB of memory. We recommend that you estimate the required throughput for the initial configuration and adjust the value based on monitoring data.

    3. Make sure that both the source and destination data sources pass the Connectivity Check.

    Step 3: Configure the synchronization solution

    1. Configure the data source

    In the Source section, configure the LogHub (SLS) data source.

    1. In the Source Information section, select the logstore that you want to synchronize.

      • Logstore: Select an existing logstore from the drop-down list.

    2. Click Data Sampling to sample and preview the LogHub data.

      In the dialog box that appears, set Start From and Sampled Data Records, and then click Start Collection. The sampling result can be used as the preview input for subsequent data processing nodes.

    3. In the Configure Output Field section, the system automatically loads the list of fields from the logstore. You can change the Data Type, Delete fields, or Add Output Field.

    Note

    If a configured field does not exist in SLS, its value is written as NULL to the destination.

    2. Data processing

    Turn on the Data Processing switch. Five data processing methods are provided: Data Masking, String Replace, Data Filtering, JSON Parsing, and Field Editing and Assignment. You can arrange these methods in the desired order. The task processes data in the specified order.

    After you configure a data processing node, you can click the Preview Data Output button in the upper-right corner.

    1. In the table below the input data, you can view the results of Data Sampling from the previous step. You can click Re-obtain Output of Ancestor Node to refresh the results.

    2. If no output is generated from the upstream node, you can also Manually Construct Data to simulate the upstream output.

    3. Click Preview to view the processed output from the upstream node.

    4. This preview shows the final output.

      On the data processing preview page, the Input Data section displays the raw logs, which include fields such as _source_, _time_, _topic_, and check_rows. The Preview Result section displays the processed data with newly parsed fields such as client_ip and db. The page indicates that 0 dirty data records are found.

    Note

    The data output preview feature relies on the Data Sampling feature of the LogHub (SLS) source. Before you preview the data output, you must first sample data from the LogHub (SLS) source.

    3. Configure the data destination

    1. In the Destination section, select a Tunnel resource group. By default, Public transport resources (the free quota for MaxCompute) is selected.

    2. Select whether to write data to a New table or an Existing table.

      1. If you want to write data to a new table, select Create from the drop-down list. By default, this creates a table with the same schema as the source. You can manually modify the name and schema of the destination table.

      2. If you want to use an existing table, select the destination table from the drop-down list.

    3. (Optional) Edit the table schema.

      Click the edit icon next to the table name to edit the table schema. To automatically generate a schema based on the upstream node's output, click Re-generate Table Schema Based on Output Column of Ancestor Node. You can select a column in the automatically generated schema as the primary key.

    4. Configure field mapping

    1. The system automatically maps source columns to destination table columns based on the Map Fields with the Same Name rule. You can adjust the mappings as required. You can map a single source column to multiple destination table columns, but you cannot map multiple source columns to a single destination table column. If a source column is not mapped, its data is not written to the destination table.

    2. (Optional) Configure partitioning.

      1. Automatic Time-based Partitioning: partitions data based on a business time field, which is _timestamp in this example. The first-level partition is year, the second-level partition is month, and so on.

      2. Dynamic Partitioning By Field Value: Maps a source field to a partition field in the destination MaxCompute table. This ensures rows are written to the correct partition in the MaxCompute table based on the value of the source field.

    Step 4: Advanced settings

    Synchronization tasks provide advanced parameters for fine-grained control. The default values work for most use cases. To customize them:

    1. In the upper-right corner of the configuration tab, click Advanced Settings to go to the Advanced Parameters configuration page.

      Note

      In Data Development, advanced settings are on a tab on the right side of the task configuration page.

    2. You can set parameters separately for the read and write sides of the synchronization task. Set Auto-configure runtime settings to false to customize the Runtime configuration.

    3. Modify the parameter values as prompted. For more information about the parameters, see the description that follows each parameter name. For information about recommended settings of some parameters, see Advanced parameters for real-time synchronization.

    Important

    Modify these parameters only if you fully understand their purpose and potential consequences to avoid unexpected errors or data quality issues.

    Step 5: Run a simulation

    Note

    For single-table real-time synchronization tasks in Data Development, the simulated run feature is available in the toolbar.

    After you configure the task, click Perform Simulated Running in the lower-left corner to debug it. The simulation runs the task on a small data sample and displays the output without writing to the destination, giving you immediate feedback on configuration errors, runtime exceptions, or dirty data.

    1. In the dialog box that appears, set the sampling parameters, which are Start time and Sampled Data Records.

    2. Click Start Collection to obtain the sample data.

    3. Click the Preview Result button to simulate the task run and view the output.

    The output of a simulated run is for preview only and is not written to the destination.

    Step 6: Publish and run the task

    1. After you complete the configuration, click Save at the bottom of the page.

    2. Data Integration tasks must be published to the production environment to run. Therefore, you must Deploy any new or edited task for it to take effect. When you publish the task, if you select the Start immediately after deployment option, the task starts automatically. Otherwise, after the task is published, go to the Data Integration > Synchronization Task page and manually start the task in the Actions column of the target task.

    3. In the Tasks, click the Name/ID of the task to view the detailed execution information.

    Step 7: Configure alert rules

    After a task is published and running, configure alert rules to receive immediate notifications when issues occur. In the Data Integration task list, find the target task and choose More > Configure Alert Rule in the Actions column.

    1. Add alert rule

    In the Add Alert dialog box, select Use Custom Rule or Use Public Alert Rule. Specify Alert Name and Description. Supported notification methods include Mail, SMS, Phone, DingTalk, webhook, and Lark. You can select recipients from an On-call Schedule or specify Others.

    (1) Click Create Rule to configure an alert rule.

    By setting Alert Reason, you can monitor metrics such as Business delay, Failover, Task status, DDL Notification, and Task Resource Utilization. You can set CRITICAL or WARNING alerts based on specified thresholds.

    • After you set the alert method, you can use Configure Advanced Parameters to control the notification interval and prevent excessive alerts.

    • If you select Business delay, Task status, or Task Resource Utilization as the alert reason, you can also enable recovery notifications to inform recipients when the task returns to a normal state.

    (2) Manage alert rules.

    For created alert rules, you can use the switch to enable or disable them. You can also send alerts to different personnel based on the alert level.

    2. View alerts

    Find the desired task in the task list and choose More > Configure Alert Rule. Then, go to the Alert Events page to view historical alert information.

    Next steps

    After the task starts, you can:

    • In the synchronization task list, view the running status and execution summary of the task.

    • Click the task name to go to the task details page and view the details of Schema Migration and Real-time Data Synchronization.

    • In the Actions column, click Start or Stop. You can also choose More to perform other operations such as Edit and View.

    FAQ