:Synchronize incremental data to MaxCompute at a time

Configure a sync solution

1. Go to the Create Data Synchronization Solution wizard. Select the source and the destination for data synchronization from the drop-down lists. In this scenario, select MaxCompute as the destination. After that, select One-click batch synchronization to MaxCompute (Once Increment) from the available sync solutions.
   For more information, see Select a data synchronization solution.
2. Configure network connection for data synchronization.
   Select the data source, exclusive resource group for Data Integration, and destination data source as prompted, and then test the network connectivity. After that, click Next Step. You must prepare the exclusive resource groups and the network connection solution that you want to use. In addition, you must create connections to data sources in DataWorks and configure network connectivity as required, such as a whitelist. This avoids failures in connectivity tests. For more information, see Plan and configure resources.
3. Configure the source and rules for data synchronization.
   1. Configure the basic information.
      In the Basic Configuration section, set the parameters that are described in the following table.

      Parameter	Description
      Solution Name	The name of the sync solution. The name can be a maximum of 50 characters in length.
      Description	The description of the sync solution. The description can be a maximum of 50 characters in length.
      Location	If you select Automatic Workflow Creation, DataWorks automatically creates a workflow named in the format of clone_database_Source name+to+Destination name. All sync nodes generated by the sync solution are placed in the Data Integration folder of this workflow. If you clear Automatic Workflow Creation, you must select a directory from the Select Location drop-down list. All sync nodes generated by the sync solution are placed in the specified directory.

   2. Check the data source information.
      The information about the data source selected in the preceding step is displayed in the Data Source section, and the encoding type is specified by default. You must check the information and determine whether to change the encoding type.
   3. Select the source tables for synchronization.
      Select the tables whose data you want to synchronize to MaxCompute as prompted. After you select the source tables, data is synchronized from the selected tables to MaxCompute based on the configuration of this sync solution.

      Notice If a selected table does not have a primary key, the table data cannot be synchronized in real time.
   4. Configure mapping rules for the names of the source and destination tables.
      Click Add rule, select a rule type, and then configure the mapping rules. Supported rule types are Conversion Rule for Table Name and Rule for Destination Table name.

      • Conversion Rule for Table Name: the rule used to convert the names of source tables to those of destination tables.
      • Rule for Destination Table name: the rule used to add a prefix or a suffix to the converted names of destination tables.
   5. Click Next Step.
4. Configure the destination.
   1. The destination data source selected in the preceding step is displayed on the page. Check whether the displayed information is valid.
   2. Click the icon next to Time automatic partition setting. In the Edit dialog box, modify the partition settings for the destination tables. You can configure daily partitions.
   3. Click Refresh source table and MaxCompute Table mapping to create the mappings between the source tables and destination MaxCompute tables.
   4. View the mapping progress, source tables, and mapped destination tables.
      

      No.	Description
      1	The progress of mapping the source tables to the destination tables. Note The mapping may require a long period of time if you want to synchronize data from a large number of tables.
      2	The source of the destination table. Valid values: Create Table and Use Existing Table.
      3	The name of the destination table. The table name that appears varies based on the value that you selected from the drop-down list in the Table creation method column. If you set the Table creation method parameter to Create Table, the name of the destination table that is automatically created appears. You can click the table name to view and modify the table creation statements. If you set the Table creation method parameter to Use Existing Table, you must select a table name from the drop-down list in the MaxComputeTable name column.
      4	If a source table does not have a primary key, an error message appears to remind you that the current source table does not have a primary key and cannot be synchronized. The synchronization can be performed if one of the selected source tables has a primary key. Source tables without primary keys are ignored during the synchronization.

   5. Click Next Step.
5. Configure synchronization rules.
   1. Configure rules for incremental data synchronization.
      You can use the SQL WHERE clause to extract incremental data from the source tables. You need only to enter the WHERE clause in the Incremental Condition field without the need to enter the WHERE keyword. In addition, you can use the built-in system variables to write the WHERE clause. For example, the ${bizdate} variable specifies the data timestamp, and the ${cyctime} variable specifies the scheduling time. For more information, see Built-in system variables.
   2. Click Next Step.
6. Configure the resources required by the sync solution.
   In the Set Resources for Solution Running step, check the name of the sync node to be generated by the sync solution, the resource group for Data Integration, and the resource group for scheduling. Then, set the Maximum number of connections supported by source read parameter.

   Note The Maximum number of connections supported by source read parameter specifies the maximum number of Java Database Connectivity (JDBC) connections that are allowed for the source. You must set this parameter based on the capabilities of the source. If the specified number of connections is beyond the capabilities of the source, the data may not be read from the source.
7. Click Complete Configuration. The sync solution is configured.

Run the sync solution

On the Tasks page, find the configured sync solution and click Submit and Run in the Operation column to run the sync solution.

View the running status and result of the data synchronization nodes

• On the Task list page, find the solution that is run and choose More > Execution details in the Operation column. Then, you can view the running details of all nodes.
• Find a node whose running details you want to view and click Execution details in the Status column. In the dialog box that appears, click the provided link to go to the DataStudio page.