All Products
Search

This Product

    DataWorks:Synchronize an entire MySQL database to an OSS data lake

    Document Center

    DataWorks:Synchronize an entire MySQL database to an OSS data lake

    Last Updated:Jun 20, 2026

    Data Integration allows you to synchronize an entire database from sources such as MySQL and PolarDB to OSS in real time. This topic demonstrates how to use Data Integration to ingest data from MySQL into an OSS data lake in real time.

    Prerequisites

    Limitations

    The primary key column in the source cannot contain null or empty string values. Otherwise, the sync task fails.

    Real-time database ingestion

    • You can migrate the schema of a source MySQL database to an OSS data lake. This involves a one-time full synchronization of historical data from the entire database or specific tables, followed by real-time synchronization of incremental data.

    • The process supports source DDL operations, such as creating tables and adding columns. The synchronization automatically adapts to changes in the number of tables and columns in the source database.

      Important

      Other source DDL operations, such as deleting or renaming tables, are not supported and cause the sync task to fail.

    • If you have Data Lake Formation (DLF) activated in the same region, the task can automatically create the corresponding metadatabase and metatables in DLF during data ingestion.

      Note

      Cross-region metadata creation is not supported.

    Procedure

    Step 1: Select the sync task type

    1. Log on to the DataWorks console. In the target region, click Data Integration > Data Integration in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Integration.

    2. In the left-side navigation pane, click Synchronization Task. At the top of the page, click Create Synchronization Task to go to the task creation page. Configure the following basic information:

      • Source and Destination: MySQL → OSS

      • New Node Name: Enter a custom name for the sync task.

      • Synchronization Method: real-time database synchronization.

      • Synchronization Mode: Select both Full Sync and Incremental Sync.

    Step 2: Configure network and resources

    1. In the Network and Resource Configuration section, select the Resource Group for the sync task. You can also specify the number of CUs for Task Resource Usage.

    2. For Source Information, select your existing MySQL data source. For Destination, select your existing OSS data source. Then, click Test Connectivity.

    3. After both the source and destination data sources pass the connectivity test, click Next.

    Step 3: Configure destination settings

    • Write Format: Select one of the supported formats: Hudi, Paimon, or Iceberg.

    • Select Storage Path: Select the OSS path where the ingested data will be stored.

    • Location for Creating Metadatabase: Choose whether to automatically create a metadatabase in DLF.

      Note

      Metadatabases can be created only in a DLF instance located in the same region.

    Step 4: Select databases and tables

    Select the tables you want to synchronize from the source data source in the source tables area. Then, click the image icon to move them to the selected tables area on the right.

    • Select Specific Databases/Tables:

      • In the Database Filter and Table Filter fields of the Source Tables area, enter keywords to find the databases and tables to synchronize. Select them and click the image icon to move them to the Selected Databases/Tables area.

      • In the Selected Databases/Tables area, you can use the Database Filter and Table Filter fields to find tables you want to exclude. Select them and click the image icon to move them back to the Source Databases/Tables area.

    • Use regex for table selection (supports adding/removing tables by regex during runtime).:

      Use regular expressions in the Database Filter and Table Filter fields to select tables. Then, click Confirm Selection.

      Note

      For example, to select tables from databases with names starting with a and table names starting with order, enter a.* in the Database Filter field and order.* in the Table Filter field.

    Step 5: Configure destination table mapping

    After you select the tables to synchronize, they are automatically displayed on this page. However, the destination tables are not yet mapped. You must define the mappings between the source and destination tables, which specify how data is read and written. After you define the mappings, click Refresh Mapping to proceed to the next step. You can refresh the mappings now or after you customize the rules for destination tables.

    Note

    • You can select the tables to be synchronized and click Batch Refresh Mapping. If no mapping rule is configured, the default table name rule is ${tableName}. If a matching table does not exist in the destination, it is created automatically.

    • In the Modify Mapping Rule for Destination Database Name column, click Configure to define a naming rule.

      You can combine and edit built-in variables with custom strings to generate the destination database name. For example, you can create a rule that appends a suffix to the source database name to form the destination database name.

    • In the Customize Mapping Rules for Destination Table Names column, click Edit to customize the naming rules for destination tables.

      You can concatenate built-in variables and manually entered strings to create the final destination table name. You can also edit the built-in variables. For example, you can create a new rule that creates the destination table name by appending a suffix to the source table name.

    1. Edit column type mappings

    A synchronization task provides default mappings between source and destination column types. To customize these mappings, click Edit Field Type Mapping in the upper-right corner of the table. When you finish, click Apply and Refresh Mapping.

    2. Edit destination table structure and assign column values

    When the destination table is in the To Be Created state, you can add new columns to its original table structure. To do so, follow these steps:

    1. Add columns to the destination table.

      • To add a column to a single table, click the image.png icon in the Destination Table Name column.

      • To add columns to multiple tables in batch, select all the tables to synchronize, and then choose Batch Modify > Target Table Structure - Batch Modify and Add Fields at the bottom of the table.

    2. Assign values to the newly added columns. You can use the following methods to assign values.

      • To assign values for a single table, click Configure in the Value assignment column.

      • To assign values for multiple tables in batch, choose Batch Modify > Value assignment at the bottom of the list to assign the same values to identical columns across destination tables.

      Note

      You can assign constants or variables as values. Click the image icon to switch between assignment modes.

    3. Configure DML rules

    Data Integration provides default DML processing rules. You can also define custom processing rules for DML commands that are written to the destination tables based on your business requirements.

    • To define rules for a single table, click Configure in the DML Rule Configuration column.

    • To define rules for multiple tables in batch, select all the tables to synchronize, and then choose Batch Modify > DML Rule Configuration at the bottom of the list.

    Step 6: Configure alerts

    To prevent data synchronization delays caused by task errors, you can configure alert policies for synchronization tasks.

    1. Click Alert Settings in the upper-right corner of the page to go to the Alert Rule Configurations for Real-time Synchronization Subnode settings page.

    2. Click Add Alert Rule to configure an alert rule.

      Note

      The alert rules defined here apply to the real-time synchronization subtasks generated by this task. After you finish configuring the task, you can go to the Run and manage real-time synchronization tasks page to view and modify the monitoring alert rules for the real-time synchronization subtask.

    3. Manage alert rules.

      For existing alert rules, you can use the alert toggle to enable or disable an alert rule. You can also send alerts to different contacts based on the alert severity level.

    Step 7: Configure advanced parameters

    The synchronization task provides configurable parameters that you can modify as needed. For example, you can set a maximum number of connections to prevent the synchronization task from overloading the database and affecting production workloads.

    Note

    Modify parameters only after you fully understand their meanings to avoid unexpected errors or data quality issues.

    1. Click Advanced Parameter Configuration in the upper-right corner of the page to go to the advanced parameter configuration page.

    2. On the Advanced Parameter Configuration page, modify the relevant parameter values.

    Step 8: Configure DDL capabilities

    The source data source may contain various DDL operations. Based on your business requirements, you can click DDL Capability Configuration in the upper-right corner of the page to go to the DDL capability configuration page and define processing policies for different DDL messages to be synchronized to the destination.

    Note

    For more information about different DDL message processing policies, see DDL message processing rules.

    Step 9: Configure resource group

    Click Configure Resource Group in the upper-right corner of the page to view and switch the resource group used by the current task.

    Step 10: Run the sync task

    Step 9: Run the synchronization task

    1. After you complete all configurations, click Save to save the task.

    2. On the Data Integration > Sync Tasks page, find the synchronization task that you created, and click Deploy in the Operation column. If you select Start immediately after deployment during deployment, the task runs immediately after you confirm. Otherwise, you must manually start the task.

      Note

      Data Integration tasks must be deployed to the production environment before they can run. Therefore, you must perform the Publish operation after you create or edit a task for the changes to take effect.

    3. Click the Name/ID of the corresponding task in Tasks to view the detailed execution process of the task.

    Monitor and manage the sync task

    View task status

    After creating a sync task, you can view a list of all your tasks and their basic information on the Sync Task page.

    • The Start column provides options to Start or Stop the task. The Edit menu provides other actions, such as Edit and View.

    • For a running task, view its status in the Execution Overview section and click an overview area to see execution details. The Basic information section shows the task ID, data source, synchronization solution, resource group, status, and current CU usage. Tabs such as Execution status, Run logs, Failover, and Resource utilization provide detailed operational information about the sync task.

      A real-time synchronization task from MySQL to OSS consists of three phases:

      • structure migration: Displays how destination tables are created, either from existing tables or by automatic creation. If tables are created automatically, the DDL statements are displayed.

      • full initialization: Displays information about the offline synchronization, including the tables, progress, and the number of rows written.

      • real-time data synchronization: Displays real-time synchronization statistics, including progress, DDL records, DML records, and alert information.

    Rerun a task

    In certain scenarios, such as when you need to add or remove tables, or modify destination table schema or table name information, you can click Rerun in the Operations column of the synchronization task. The system synchronizes only the newly added or modified tables. Previously synchronized or unmodified tables are not synchronized again.

    • Click Rerun to rerun the full initialization and real-time synchronization.

    • Edit the task to add or remove tables, save the task, and then deploy it. After the deployment, the Apply Updates button appears in the Operations column. Click Apply Updates to trigger a rerun of the modified task. Only the newly added or modified tables are synchronized. Previously synchronized tables are not synchronized again.