DataWorks:Build an enterprise data warehouse with AnalyticDB for MySQL

Configure an AnalyticDB for MySQL 3.0 data source

1. 1. Log on to the DataWorks console. In the target region, click More > Management Center in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Management Center.

   2. On the Workspace Management page, click Data Sources in the left-side navigation pane to open the data source page.

2. In the Add Connection dialog box, select AnalyticDB for MySQL (V3.0) as the data source type.

3. Configure the parameters for the AnalyticDB for MySQL 3.0 data source. For more information, see Configure an AnalyticDB for MySQL 3.0 data source.

   Note

   • To run AnalyticDB for MySQL node tasks that access an AnalyticDB for MySQL instance in a VPC, use a serverless resource group (recommended) or an exclusive resource group for scheduling. Other types of resource groups cannot access the instance and cause connection timeouts. For more information about serverless resource groups, see Use serverless resource groups.

   • AnalyticDB for MySQL V2.0 uses your account's AccessKey pair for authentication.

   • AnalyticDB for MySQL V3.0 uses a database username and password for authentication. After you activate a V3.0 database, you must first create a username and password in the console.

4. Click Test Connectivity.

5. After the connectivity test passes, click Complete.

Configure the whitelist for AnalyticDB for MySQL 3.0

AnalyticDB for MySQL V3.0 uses username and password authentication. You must configure a client whitelist to allow database connections.

Create a workflow

1. Log on to the DataWorks console. Select a region, and then in the left-side navigation pane, choose Data Development and O&M > DataStudio. From the drop-down list, select your workspace and click Data Analytics.

2. On the Data Development page, click Create > Create Workflow in the top navigation bar. The Create Workflow page opens. In the dialog box, enter the Workflow Name and Description.

3. Click Create.

Create a batch synchronization task

1. In the workflow, right-click Data Integration and choose Create > Batch Sync.

2. In the Create Node dialog box, enter a Name and click Submission.

3. Configure the Data Source and Destination.

4. Configure field mappings.

   The source fields on the left and the destination fields on the right are mapped one-to-one. Click Add Row to add a field. To remove a field, hover over the field and click the Delete icon.

5. Configure channel control.

   Configure the maximum job rate and the rules for handling dirty data.

   Parameter	Description
   Expected Maximum Concurrency	The maximum number of concurrent threads for reading data from the source or writing data to the destination. In wizard mode, specify the number of concurrent threads on the UI to control the task's degree of parallelism.
   Bandwidth Throttling	Set a synchronization rate to prevent the source database from being overloaded by excessive data extraction speeds. Set a reasonable data extraction rate based on the configuration of the source database.
   Policy for Dirty Data Records	The maximum number of dirty data records allowed.
   Distributed Execution	Select a resource group to run the task. If you have a large number of tasks and resource contention occurs on the default resource group, consider purchasing a serverless resource group. For more information, see Use serverless resource groups.

6. In the right-side panel, click Scheduling Settings to configure the scheduling properties for the node.

7. After completing the configuration, click the icon to Save the node and then click the icon to Submission the task.

Create a data development task

1. In the workflow, right-click Custom and choose Create > AnalyticDB for MySQL.

2. In the Create Node dialog box, enter a Node Name for the node and click Submission.

3. Select the appropriate data source and write SQL statements based on the syntax supported by AnalyticDB for MySQL. Data Manipulation Language (DML) and Data Definition Language (DDL) statements are supported.

4. In the right-side panel, click Scheduling Settings to configure the scheduling properties for the node. In the node editor, click the Scheduling Dependencies section to configure dependencies. Click Parse Dependencies from Code to automatically resolve upstream dependencies. You can also manually search for and add upstream node outputs in the Upstream Dependencies section. In the Cross-cycle Dependencies section, select Current Node, Level-1 Child Nodes, or Other Nodes as needed. In the Current Node Output Name section, confirm or add the output information for the current node.

5. After completing the configuration, click the Save icon to save the node. Then, click the Run icon to run the SQL statements immediately.

Operations and maintenance

After you commit and deploy a task, it runs periodically based on its scheduling properties. You can click O&M in the upper-right corner of the node editing page to view the scheduling and execution status of the periodic task. For more information, see Manage periodic tasks.

Metadata management

Navigate to the DataWorks console and choose Data Governance > DataMap to manage metadata. For more information, see Overview of Data Map.