DataWorks:Real-time synchronization: Entire MySQL database to LogHub (SLS)

Step 1: Select the synchronization task type

1. Log on to the DataWorks console. In the target region, click Data Integration > Data Integration in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Integration.

2. In the left-side navigation pane, click Synchronization Task. At the top of the page, click Create Synchronization Task. In the dialog box, configure the following basic settings:

   • Source Type: MySQL.

   • Destination Type: LogHub.

   • Specific Type: real-time synchronization of entire database.

   • Synchronization Mode: incremental synchronization. This method continuously synchronizes binlog data from your business database to LogHub.

Step 2: Configure network and resources

1. In the Network and Resource Configuration section, select the Resource Group for the synchronization task. You can allocate compute units (CUs) for the Task Resource Usage.

2. For Source Information, select the added MySQL data source. For Destination, select the added LogHub data source. Then, click Test Connectivity.

3. After both connectivity tests pass, click Next.

Step 3: Select databases and tables

In this step, in the Source Table area, select the source tables to synchronize and click the icon to move them to the Selected Tables area on the right.

In the Source Table area, you can also select tables by pasting a list. You can select up to 5,000 tables.

• Select Specific Databases/Tables:

  • In the Source Table area, use the Database Filter and Table Filter fields to search for the databases and tables you want to synchronize. Select the items and click the icon to move them to the Selected Databases/Tables area.

  • In the Selected Databases/Tables area, use the Database Filter and Table Filter fields to find any tables you want to exclude. Select them and click the icon to move them back to the Source Databases/Tables area.

• Use regex for table selection (supports adding/removing tables by regex during runtime).:

  Enter regular expressions in the Database Filter and Table Filter fields. Click Confirm to select the matching tables.

  Note

  For example, to synchronize tables from databases with names that start with a and tables with names that start with order, enter a.* in the Database Filter box and order.* in the Table Filter box.

Step 4: Map destination tables

After you select the tables to synchronize, they are automatically displayed on this page. However, the destination tables are not yet mapped. You must define the mappings between the source and destination tables, which specify how data is read and written. After you define the mappings, click Refresh Mapping to proceed to the next step. You can refresh the mappings now or after you customize the rules for destination tables.

Step 5: Configure alerts

To prevent data synchronization delays caused by task errors, you can configure alert policies for synchronization tasks.

1. Click Alert Settings in the upper-right corner of the page to go to the Alert Rule Configurations for Real-time Synchronization Subnode settings page.

2. Click Add Alert Rule to configure an alert rule.

   Note

   The alert rules defined here apply to the real-time synchronization subtasks generated by this task. After you finish configuring the task, you can go to the Run and manage real-time synchronization tasks page to view and modify the monitoring alert rules for the real-time synchronization subtask.

3. Manage alert rules.

   For existing alert rules, you can use the alert toggle to enable or disable an alert rule. You can also send alerts to different contacts based on the alert severity level.

Step 6: Configure advanced parameters

To fine-tune the task, click Configuration in the Custom Advanced Parameters column to modify advanced parameters.

1. In the upper-right corner of the page, click Advanced Parameters to go to the advanced parameter configuration page.

2. The following table describes key parameters.

   Parameter	Description
   Expand business data	Specifies whether to flatten the business data fields from the source table to the top level of the log entry. true: Expands the data. For the expansion format, you can select Partial expansion or Full expansion in the Data Expansion Format section. false (default): Does not expand the data. The data is encapsulated within a single JSON field.
   Pass through DDL information	Controls whether to capture Data Definition Language (DDL) operations from the source table, such as CREATE TABLE and ALTER TABLE, and synchronize them to SLS as a log entry. true (default): Passes through the DDL information. false: Does not pass through the DDL information.
   Source NULL value handling	Specifies the value to write to the destination field when the source field is NULL. By default, this is empty, which preserves the NULL value.
   Data Expansion Format	Defines the structure of the business data (the data field) when it is written to SLS. This option is visible only when the Expand business data option is set to true. The format you select directly affects how downstream systems consume the logs. For format examples, see Appendix: Data expansion format examples. Partial expansion (default): Compatible with the Logtail collection format for MySQL binlog. Full expansion: Flattens all business data fields, including both before and after images of the change, as independent top-level key-value pairs in the data.

Step 7: Configure the resource group

Click Configure Resource Group in the upper-right corner of the page to view and switch the resource group used by the current task.

Step 8: Deploy and run the synchronization task

1. After you complete all configurations, click Save to save the task.

2. On the Data Integration > Sync Tasks page, find the synchronization task that you created, and click Deploy in the Operation column. If you select Start immediately after deployment during deployment, the task runs immediately after you confirm. Otherwise, you must manually start the task.

   Note

   Data Integration tasks must be deployed to the production environment before they can run. Therefore, you must perform the Publish operation after you create or edit a task for the changes to take effect.

3. Click the Name/ID of the corresponding task in Tasks to view the detailed execution process of the task.

View task status

After creating the synchronization task, you can view the list of created tasks and their basic information on the synchronization task page. After a task is successfully created, it appears in the task list with the status Running. In the execution overview, the real-time synchronization status is Pending Start, and the resource group is Serverless_Group.

• In the Operations column, you can Start or Stop the synchronization task. Under More, you can find other actions such as Edit and View.

• For a running task, you can check the Execution Overview to see its operational status, or click an area in the overview to view details. The Basic Information section on the task details page displays the task ID, data sources, resource group, synchronization solution, and current CU usage. You can switch between the Execution Status, Run Log, Failover, and Resource Usage tabs to monitor the task.

  The real-time synchronization task from MySQL to LogHub (SLS) displays real-time statistics, including progress, DDL records, DML records, and alert information.

Rerun a task