DataWorks:Synchronize data

Create a MySQL data source

The basic user information for this tutorial is stored in a MySQL database. Create a MySQL data source named user_behavior_analysis_mysql to pull the ods_user_info_d table into MaxCompute.

1. Log on to the DataWorks console. In the top navigation bar, select your region. In the left-side navigation pane, choose More > Management Center. Select your workspace from the drop-down list and click Go to Management Center.

2. In the left-side navigation pane, click Data Sources.

3. Click Add Data Source, then search for and select MySQL.

4. On the Add MySQL Data Source page, configure the parameters below. Use these values for both the development and production environments. Keep the default values for any parameters not listed.

   Parameter	Value
   Data Source Name	user_behavior_analysis_mysql
   Data Source Description	This data source is for DataWorks tutorials. Read data from this data source when you configure an offline sync task to access the test data provided by the platform. This data source can only be read in Data Integration scenarios. Other modules cannot use it.
   Configuration Mode	Connection String Mode
   Connection Address	Host IP: rm-bp1z69dodhh85z9qa.mysql.rds.aliyuncs.com, Port: 3306
   Database Name	workshop
   Username	workshop
   Password	workshop#2017
   Authentication Method	No authentication

5. In the Connection Configuration section, click Test Network Connectivity for both environments. Confirm that the status shows Connected.

   Important

   The resource group must be attached to the workspace with public network access enabled. Otherwise, data synchronization fails.

6. Click Complete Creation.

Create an HttpFile data source

The user website access logs for this tutorial are stored in OSS (Object Storage Service). Create an HttpFile data source named user_behavior_analysis_httpfile to pull user_log.txt into MaxCompute.

1. In the left-side navigation pane, click Data Sources.

2. Click Add Data Source, then search for and select HttpFile.

3. On the Add HttpFile Data Source page, configure the parameters below. Use these values for both the development and production environments. Keep the default values for any parameters not listed.

   Parameter	Value
   Data Source Name	user_behavior_analysis_httpfile
   Data Source Description	This data source is for DataWorks tutorials. Read data from this data source when you configure an offline sync task to access the test data provided by the platform. This data source can only be read in Data Integration scenarios. Other modules cannot use it.
   URL	https://dataworks-workshop-2024.oss-cn-shanghai.aliyuncs.com

4. In the Connection Configuration section, click Test Network Connectivity for both environments. Confirm that the status shows Connected.

   Important

   The resource group must be attached to the workspace with public network access enabled. Otherwise, data synchronization fails.

5. Click Complete Creation.

Configure the initial node

1. On the workflow canvas, hover over workshop_start and click Open Node.

2. In the right-side pane, click Scheduling and set the following parameters. Keep the default values for any parameters not listed.

   Scheduling parameter	Value
   Scheduling Type	dry-run scheduling
   Resource Group	Select the Serverless resource group created in Prepare the environment.
   Scheduling Dependencies	Click Use Workspace Root Node to trigger execution from the workspace root node (workspace_name_root).

Configure the user data sync task (ods_user_info_d)

1. On the workflow canvas, hover over ods_user_info_d and click Open Node.

2. Configure the source and destination:

   Parameter	Value
   Source — Data source type	MySQL
   Source — Data source name	user_behavior_analysis_mysql
   Resource Group	Select the Serverless resource group from Prepare the environment.
   Destination — Data destination type	MaxCompute(ODPS)
   Destination — Data source name	Select the MaxCompute resource attached in Prepare the environment, for example MaxCompute_Source.

3. Click Next to configure the sync task details.

   1. Configure source and destination. Set the following parameters. Keep the default values for any parameters not listed.

      Module	Configuration Item	Value
      Data source	Table	ods_user_info_d
      Data source	Split key	uid (the primary key of the ods_user_info_d table in MySQL)
      Destination	Tunnel Resource Group	Common transmission resources (default). If you have an exclusive Tunnel Quota, you can select it from the drop-down list. For more information, see Purchase and use an exclusive resource group for Data Transmission Service.
      Destination	Table	Click Generate Destination Table Schema. In the Statement for Creating Table box, paste the following DDL and click Create Table:

      Module	Configuration Item	Value
      Destination	Partition information	${bizdate}

         CREATE TABLE IF NOT EXISTS ods_user_info_d (
             uid STRING COMMENT 'User ID',
             gender STRING COMMENT 'Gender',
             age_range STRING COMMENT 'Age range',
             zodiac STRING COMMENT 'Zodiac sign'
         )
         PARTITIONED BY (
             dt STRING
         )
         LIFECYCLE 7;

      Using ${bizdate} lets you pass a constant during testing and a dynamic value during scheduled runs. For supported variable formats, see Scheduling Parameters.

   2. Confirm field mapping and channel control. DataWorks maps source fields to destination fields based on the field mapping configuration. Set Policy for Dirty Data Records to Disallow Dirty Data Records. Keep the default values for all other settings. For details, see Configure a sync task using the codeless UI.

4. Configure debug parameters. In the right-side pane, click Debugging Configurations and set the following:

   Configuration item	Value
   Resource Group	Select the Serverless resource group from Prepare the environment.
   Script Parameters	Leave blank. When you run the workflow in Step 4, set Value Used in This Run to a constant such as 20250223. The job replaces ${bizdate} with this value.

5. (Optional) Configure scheduling properties. In the right-side pane, click Scheduling. The workflow-level scheduling parameters already apply to this node; no separate configuration is required. For details, see Node scheduling.

6. In the node toolbar, click Save.

Configure the user log sync task (ods_raw_log_d)

1. On the workflow canvas, hover over ods_raw_log_d and click Open Node.

2. Configure the source and destination:

   Parameter	Value
   Source — Data source type	HttpFile
   Source — Data source name	user_behavior_analysis_httpfile
   Resource Group	Select the Serverless resource group from Prepare the environment.
   Destination — Data destination type	MaxCompute(ODPS)
   Destination — Data source name	Select the MaxCompute resource from Prepare the environment, for example MaxCompute_Source.

3. Click Next to configure the sync task details.

   1. Configure source and destination. Set the following parameters. Keep the default values for any parameters not listed.

      Important

      After you configure the data source settings, click Confirm Data Structure to verify that DataWorks can read the log file correctly before proceeding.

      Module	Configuration Item	Value
      Data source	File path	/user_log.txt
      Data source	Text type	text
      Data source	Column Delimiter	|
      Data source	Advanced configuration > Skip Header	No
      Destination	Tunnel Resource Group	Common transmission resources (default). If you have an exclusive Tunnel Quota, you can select it from the drop-down list. For more information, see Purchase and use an exclusive resource group for Data Transmission Service.
      Destination	Table	Click Generate Destination Table Schema. In the Statement for Creating Table box, paste the following DDL and click Create Table:

      Module	Configuration Item	Value
      Destination	Partition information	${bizdate}

         CREATE TABLE IF NOT EXISTS ods_raw_log_d
         (
             col STRING
         )
         PARTITIONED BY
         (
             dt STRING
         )
         LIFECYCLE 7;

   2. Confirm field mapping and channel control. Set Policy for Dirty Data Records to Disallow Dirty Data Records. Keep the default values for all other settings. For details, see Configure a sync task using the codeless UI.

4. Configure debug parameters. In the right-side pane, click Debugging Configurations and set the following:

   Configuration item	Value
   Resource Group	Select the Serverless resource group from Prepare the environment.
   Script Parameters	Leave blank. When you run the workflow in Step 4, set Value Used in This Run to a constant such as 20250223.

5. (Optional) Configure scheduling properties. In the right-side pane, click Scheduling. The workflow-level parameters apply to this node automatically. For details, see Node scheduling.

6. In the node toolbar, click Save.