Real-time synchronization of a single Oracle table to Tablestore - DataWorks

This topic describes how to use Data Integration to synchronize data from an Oracle table to Tablestore in real time using a single Oracle table to Tablestore synchronization as an example.

Prerequisites

A serverless resource group or an exclusive resource group for Data Integration is purchased.
Oracle and Tablestore data sources are created. For more information, see Create a data source for Data Integration.
Network connections between the resource group and data sources are established. For more information, see Network connectivity solutions.

Procedure

Step 1: Select a synchronization task type

Go to the Data Integration page.
Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose Data Integration > Data Integration. On the page that appears, select the desired workspace from the drop-down list and click Go to Data Integration.
In the left-side navigation pane, click Synchronization Task, and then click Create Synchronization Task at the top of the page to go to the synchronization task creation page. Configure the following basic information:
- Source And Destination: Oracle→Tablestore
- New Task Name: Customize the synchronization task name.
- Synchronization Type: Single Table Realtime.

Step 2: Configure network and resources

In the Network And Resource Configuration section, select the Resource Group for the synchronization task. You can allocate the number of CUs for Task Resource Usage.
For Source Data Source, select the added Oracle data source. For Destination Data Source, select the added Tablestore data source, then click Test Connectivity.
After you ensure that both the source and destination data sources are connected successfully, click Next.

Step 3: Configure the synchronization link

1. Configure the Oracle source

Click the Oracle data source at the top of the page to edit the Oracle Source Information.

In the Oracle Source Information section, select the schema where the Oracle table you want to read is located, and the source table.
Click Data Sampling in the upper-right corner.
In the Preview Data Output dialog box, specify the Sampled Data Records and click Start Collection to sample data from the specified Oracle table and preview the data in the Oracle table, which provides input for subsequent visual configuration.

2. Configure data processing nodes

You can click the icon to add data processing methods. The following data processing methods are supported: Data Masking, Replace String, Data filtering, JSON Parsing, and Edit Field and Assign Value. You can arrange the data processing methods based on your business requirements. When the synchronization task is run, data is processed based on the processing order that you specify.

After you complete the configuration of a data processing node, you can click the Preview Data Output button in the upper-right corner. In the dialog box that appears, click Re-obtain Output Of Ancestor Node to simulate the result after the Oracle table sample data is processed by the current data processing node.

Note

The data output preview strongly depends on the Data Sampling of the Oracle source. Before executing the data output preview, you need to complete data sampling in the Oracle source form.

3. Configure the Tablestore destination information

Click the Tablestore data destination at the top of the page to edit the Tablestore Destination Information.

In the Tablestore Destination Information section, select the Tablestore table to which you want to write data.
Note
The Tablestore table must be created in advance.
Select the Data Update Mode. The following modes are supported:
- replace: Row change. Directly overwrite the original row with the updated data.
- update: Column change. Update only the columns whose data has changed.
Configure field mappings.
In the Field Mappings section on the right, you can click Batch Add All Source Fields As Attribute Fields, or manually click Add Attribute Field in the Output Field section, and then click Same-name Mapping, Same-row Mapping, or Clear Mapping to configure the mappings.

4. Configure alert rules

To prevent the failure of the synchronization task from causing latency on business data synchronization, you can configure different alert rules for the synchronization task.

In the upper-right corner of the page, click Configure Alert Rule to go to the Alert Rule Configurations for Real-time Synchronization Subnode panel.
In the Configure Alert Rule panel, click Add Alert Rule. In the Add Alert Rule dialog box, configure the parameters to configure an alert rule.
Note
The alert rules that you configure in this step take effect for the real-time synchronization subtask that will be generated by the synchronization task. After the configuration of the synchronization task is complete, you can refer to Run and manage real-time synchronization tasks to go to the Real-time Synchronization Task page and modify alert rules configured for the real-time synchronization subtask.
Manage alert rules.
You can enable or disable alert rules that are created. You can also specify different alert recipients based on the severity levels of alerts.

5. Configure advanced parameters

DataWorks allows you to modify the configurations of specific parameters. You can change the values of these parameters based on your business requirements.

Note

To prevent unexpected errors or data quality issues, we recommend that you understand the meanings of the parameters before you change the values of the parameters.

In the upper-right corner of the configuration page, click Configure Advanced Parameters.
In the Configure Advanced Parameters panel, change the values of the desired parameters.

Step 6: Configure DDL capabilities

DDL operations may be performed on the source. You can click Configure DDL Capability in the upper-right corner of the page to configure rules to process DDL messages from the source based on your business requirements.

Note

For more information, see Configure rules to process DDL messages.

Step 7: Configure resource groups

You can click Configure Resource Group in the upper-right corner of the page to view and change the resource groups that are used to run the current synchronization task.

Step 8: Perform a test run

After the preceding configuration is complete, you can click Perform Simulated Running in the upper-right corner of the configuration page to enable the synchronization task to synchronize the sampled data to the destination table. You can view the synchronization result in the destination table. If specific configurations of the synchronization task are invalid, an exception occurs during the test run, or dirty data is generated, the system reports an error in real time. This can help you check the configurations of the synchronization task and determine whether expected results can be obtained at the earliest opportunity.

In the dialog box that appears, configure the parameters for data sampling from the specified table, including the Start At and Sampled Data Records parameters.
Click Start Collection to enable the synchronization task to sample data from the source.
Click Preview to enable the synchronization task to synchronize the sampled data to the destination.

Step 9: Execute the synchronization task

After the configuration of the synchronization task is complete, click Complete in the lower part of the page.
In the page of Data Integration > Synchronization Task page, find the created synchronization task and click Start in the Operation column.
Click the Name or ID of the synchronization task in the Tasks section and view the detailed running process of the synchronization task.

Perform O&M operations on the synchronization task

View the status of the synchronization task

After the synchronization task is created, you can go to the Synchronization Task page to view all synchronization tasks that are created in the workspace and the basic information of each synchronization task.

You can Start or Stop the synchronization task in the Actions column. In the More menu, you can perform operations such as Edit and View on the synchronization task.
For tasks that have been started, you can see the basic status of the task in Running Details, or click the corresponding details area to view the execution details.

The real-time synchronization task from Oracle to Tablestore includes Real-time Data Synchronization. You can view running information, DDL records, and alert information.

Rerun the synchronization task

In some special cases, if you want to modify the fields to synchronize, the fields in a destination table, or table name information, you can also click Rerun in the Operation column of the desired synchronization task. This way, the system synchronizes the changes that are made to the destination. Data in the tables that are already synchronized and are not modified will not be synchronized again.

Directly click Rerun without modifying the configurations of the synchronization task to enable the system to rerun the synchronization task.
Modify the configurations of the synchronization task and then click Complete. Click Apply Updates that is displayed in the Operation column of the synchronization task to rerun the synchronization task for the latest configurations to take effect.