All Products
Search

This Product

    :Real-time synchronization from a full ApsaraDB for OceanBase database to MaxCompute

    Document Center

    :Real-time synchronization from a full ApsaraDB for OceanBase database to MaxCompute

    Last Updated:Feb 26, 2026

    Data Integration supports the real-time synchronization of entire databases from sources such as ApsaraDB for OceanBase, MySQL, Oracle, PolarDB, and PolarDB-X 2.0 to MaxCompute. This topic uses ApsaraDB for OceanBase as the source and MaxCompute as the destination to demonstrate how to synchronize full and incremental data from an entire ApsaraDB for OceanBase database to a MaxCompute Delta Table.

    Background information

    A Data Integration task for the real-time synchronization of an entire ApsaraDB for OceanBase database to MaxCompute first synchronizes the full data from the source database to a MaxCompute Delta Table. The Delta Table can be a partitioned or non-partitioned table. Then, the task synchronizes incremental data in real time. The synchronized data is visible in near real-time. Currently, incremental data synchronized to a Delta Table is visible in as little as five minutes.

    For more information about MaxCompute Delta Tables, see Near-real-time data warehouse overview.

    Prerequisites

    Limits

    Synchronizing source data to MaxCompute external tables is not supported.

    Procedure

    1. Select the sync task type

    1. Go to the Data Integration page.

      Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose Data Integration > Data Integration. On the page that appears, select the desired workspace from the drop-down list and click Go to Data Integration.

    2. In the navigation pane on the left, click Synchronization Task. Then, click Create Synchronization Task at the top of the page and configure the following basic information.

      • Source Type: ApsaraDB for OceanBase

      • Destination Type: MaxCompute

      • Task Name: Enter a custom name for the sync task.

      • Task Type: Real-time migration of entire database.

      • Sync Procedure:select Full initialization and Incremental synchronization.

    2. Configure network and resources

    1. In the Network and Resource Configuration area, select the Resource Group for the sync task. You can allocate CUs to the task in the Task Resource Usage area.

    2. For Source, select the ApsaraDB for OceanBase data source. For Destination, select the MaxCompute data source. Then, click Test Connectivity.

      image

    3. After you verify that the source and destination data sources are connected, click Next.

    3. Select the databases and tables to synchronize

    In this step, select the tables to synchronize from the source data source in the Source Table section. Click the image icon to move them to the Selected Tables section on the right.

    image

    • Select Specified Database and Table:

      • In the Database Filtering and Search for Tables fields, you can filter the source databases and tables to sync by entering keywords from their names. Select the databases and tables that you want to sync, and click the image icon to move them to the Selected Tables area.

      • In the Selected Databases And Tables area, enter identifying information in the Database Filter and Table Filter fields to find the databases and tables that you do not want to sync. Select them, click the image icon, and move them to the Source Databases And Tables area.

    • Use Regular Expression (table addition and removal during running based on a regular expression supported):

      You can filter the database and table information using the regular expressions in Database Filtering and Table Filtering. Then, click OK to synchronize the selected databases and tables.

      Note

      For example, to filter databases with the prefix a and tables with the prefix order, enter a.* in the Database Filtering box and order.* in the Table Filtering box.

    4. Configure destination table mapping

    After you select the tables from which you want to synchronize data, the selected tables are automatically displayed in the Mapping Rules for Destination Tables section. The properties of the destination tables are waiting to be mapped. You must manually define mappings between the source tables and destination tables to determine the data reading and writing relationships. Then, you can click Refresh in the Actions column. You can directly refresh mappings between source tables and destination tables. You can also refresh mappings between source tables and destination tables after you configure settings related to destination tables.

    Note

    • You can select the tables that you want to synchronize and click Batch Refresh Mapping Results. If no mapping rules are configured, the default table naming rule is ${source_database_name}_${table_name}. If a table with the same name does not exist in the destination database, a new table is automatically created.

    • In the Customize Mapping Rules column, you can click Edit to customize the target table naming rules.

      You can combine built-in variables and manually entered strings to create the destination table name. You can also edit the built-in variables. For example, you can create a new naming rule that adds a suffix to the source table name.

    • When synchronizing data to a MaxCompute Delta Table, you must specify a primary key for the destination table. By default, the primary key of the source table is used. If the source table has no primary key, you can specify custom primary key columns. The sync task cannot be saved if no primary key is specified.

    • By default, an automatically created Delta table has 16 buckets. You can change this number in the Table Bucket Num field in the target table mapping.

      You cannot change the number of buckets for an existing table. The number of buckets determines how the table data is partitioned, which allows operations such as queries, writes, and data merges to be executed concurrently at the bucket level. However, using too many buckets can increase the number of small files. Therefore, you must configure this setting based on your specific needs. For more information, see Table Operations and Data Storage Bucketing.

    • By default, the query period for historical data in an automatically created Delta Table is 0 hours. You can specify this period in the Acid Data Retain Hours setting of the target table mapping.

      You cannot change the historical data query period for an existing table. This period determines the time range for which you can query historical versions of the data. Historical data that is older than the specified period cannot be queried. A longer period means that more historical data is retained, which increases storage costs. Therefore, you should set this period based on your needs. For more information, see Table Operations and Time Travel.

    a. Modify data type mappings for fields

    Default mappings exist between data types of source fields and data types of destination fields. You can click Edit Mapping of Field Data Types in the upper-right corner of the Mapping Rules for Destination Tables section to configure data type mappings between source fields and destination fields based on your business requirements. After the configuration is complete, click Apply and Refresh Mapping.

    b. Modify the schema of a destination table to add fields to the table and assign values to the fields

    If a destination table is in the To Be Created state, you can perform the following steps to add fields to the table and assign values to the fields:

    1. Add fields to one or more destination tables.

      • Add fields to a single destination table: Find the destination table to which you want to add fields and click the image.png icon in the Destination Table Name column. In the dialog box that appears, add fields.

      • Add fields to multiple destination tables at a time: Select the destination tables to which you want to add fields at a time, click Batch Modify in the lower part of the page, and then click Destination Table Schema - Batch Modify and Add Field.

    2. Assign values to the fields. You can perform one of the following operations to assign values to the fields:

      • Assign values to the fields that are added to a single destination table: Find the destination table in which you want to assign values to newly added fields and click Configure in the Value assignment column. In the Additional Field dialog box, assign values to the fields.

      • Assign values to the fields that are added to multiple destination tables at a time: Select the destination tables in which you want to assign values to newly added fields, click Batch Modify in the lower part of the page, and then click Value assignment to assign values to the same fields in the selected destination tables at a time.

      Note

      You can click the image icon to switch the value assignment method and assign constants and variables to the fields that are added to a destination table.

    c. Configure DML processing rules

    Data Integration provides default DML processing rules. You can also configure DML processing rules for destination tables based on your business requirements.

    • Configure DML processing rules for a single destination table: Find the destination table for which you want to configure DML processing rules and click Configure in the Configure DML Rule column to configure DML processing rules for the table.

    • Configure DML processing rules for multiple destination tables at a time: Select the destination tables for which you want to configure DML processing rules, click Batch Modify in the lower part of the page, and then click Configure DML Rule.

    4. Configure full synchronization

    If you chose Full initialization in the Sync Procedure when you selected the sync task type, you can also disable full sync for specific tables here.

    5. Configure alert rules

    To prevent the failure of the synchronization task from causing latency on business data synchronization, you can configure different alert rules for the synchronization task.

    1. In the upper-right corner of the page, click Configure Alert Rule to go to the Alert Rule Configurations for Real-time Synchronization Subnode panel.

    2. In the Configure Alert Rule panel, click Add Alert Rule. In the Add Alert Rule dialog box, configure the parameters to configure an alert rule.

      Note

      The alert rules that you configure in this step take effect for the real-time synchronization subtask that will be generated by the synchronization task. After the configuration of the synchronization task is complete, you can refer to Run and manage real-time synchronization tasks to go to the Real-time Synchronization Task page and modify alert rules configured for the real-time synchronization subtask.

    3. Manage alert rules.

      You can enable or disable alert rules that are created. You can also specify different alert recipients based on the severity levels of alerts.

    6. Configure advanced parameters

    You can change the values of specific parameters configured for the synchronization task based on your business requirements. For example, you can specify an appropriate value for the Maximum read connections parameter to prevent the current synchronization task from imposing excessive pressure on the source database and data production from being affected.

    Note

    To prevent unexpected errors or data quality issues, we recommend that you understand the meanings of the parameters before you change the values of the parameters.

    1. In the upper-right corner of the configuration page, click Configure Advanced Parameters.

    2. In the Configure Advanced Parameters panel, change the values of the desired parameters.

    7. Configure DDL processing rules

    DDL operations may be performed on the source. You can click Configure DDL Capability in the upper-right corner of the page to configure rules to process DDL messages from the source based on your business requirements.

    Note

    For more information, see Configure rules to process DDL messages.

    8. View and change resource groups

    You can click Configure Resource Group in the upper-right corner of the page to view and change the resource groups that are used to run the current synchronization task.

    9. Run the synchronization task

    1. After the configuration of the synchronization task is complete, click Complete in the lower part of the page.

    2. In the Synchronization Task section of the Data Integration page, find the created synchronization task and click Start in the Operation column.

    3. Click the Name/ID of the synchronization task in the Tasks section and view the detailed running process of the synchronization task.

    O&M for sync tasks

    View the task status

    After you create a sync task, you can view the list of created tasks and their basic information on the Sync Task page.

    image

    • In the Operation column, you can Start or Stop a sync task. The More menu provides additional operations, such as Edit and View.

    • For a running task, you can view its basic status in the Execution Overview. You can also click the task's overview area to view its execution details.

      image

      A real-time sync task for an entire ApsaraDB for OceanBase database to MaxCompute consists of three steps:

      • Schema migration: Includes the creation method for the destination table (existing table or automatic creation). If the table is created automatically, the Data Definition Language (DDL) statement is displayed.

      • Full initialization: Includes information about the tables synchronized offline, the synchronization progress, and the number of written rows.

      • Real-time data synchronization: Includes real-time synchronization statistics, such as real-time progress, DDL records, DML records, and alert information.

    Rerun the synchronization task

    In some special cases, if you add tables to or remove tables from the source, or change the schema or name of a destination table, you can click More in the Operation column of the synchronization task and then click Rerun to rerun the task after the change. During the rerun process, the synchronization task synchronizes data only from the newly added tables to the destination or only from the mapped source table to the destination table whose schema or name is changed.

    • If you want to rerun the synchronization task without modifying the configuration of the task, click More in the Actions column and then click Rerun to rerun the task to perform full synchronization and incremental synchronization again.

    • If you want to rerun the synchronization task after you add tables to or remove tables from the task, click Complete after the change. In this case, Apply Updates is displayed in the Actions column of the synchronization task. Click Apply Updates to trigger the system to rerun the synchronization task. During the rerun process, the synchronization task synchronizes data from the newly added tables to the destination. Data in the original tables is not synchronized again.