DataWorks:Synchronize data

Follow these steps to synchronize the user information from the MySQL data source to the OSS bucket you created.

1. Open the batch synchronization node.

   On the workflow orchestration page, hover over the batch synchronization node ods_user_info_d_2oss_emr and click Open Node that appears above it to go to the task configuration page.

2. Configure the data source and runtime resource.

   Configure the data source and resource group based on the following parameters.

   Parameter	Description
   Source	Data source: MySQL. Data Source Name: user_behavior_analysis_mysql.
   Destination	Data going: OSS. Data Source Name: Select the OSS data source you created earlier. In this example, select test_g.
   Running Resources	For Resource Group, select the serverless resource group you purchased.

3. Configure the source and destination.

   1. The following table describes the key parameters. You can keep the default values for other parameters.

      Parameter	Description
      Data source	Table: Select ods_user_info_d from the data source. Shard Key: We recommend that you use a primary key or an indexed column as the split key. Only integer columns are supported. Set the split key to uid.
      Data going	File Type: Select text. Object Name (Path Included): Enter the path based on your OSS directory. In this example, enter ods_user_info_d/user_${bizdate}/user_${bizdate}.txt. In this path, ods_user_info_d is the OSS directory name, and $bizdate retrieves the date of the previous day. Field Delimiter: Enter | as the column delimiter.

   2. Verify the Field Mapping and Advanced Configuration.

      DataWorks maps source columns to destination columns so that data from specified source columns is written to specified destination columns. You can also configure the concurrency and dirty data handling policy in the advanced settings. In this tutorial, set Policy for Dirty Data Records to Disallow Dirty Data Records and keep the default values for other settings. For more information, see Configure a batch synchronization task.

4. Configure debug parameters.

   On the configuration page, click Run Configuration on the right side. Configure the following parameters to test the run in Step 4.

   Configuration item	Configuration
   Resource Group	Select the serverless resource group you purchased in Prepare an environment.
   Script Parameters	No configuration is required. The sample code in this tutorial uses ${bizdate} uniformly to represent the business date. When you debug the workflow in Step 4, set This operation value to a specific constant (for example, 20250223). The task run will use this constant to replace the variables defined in the task.

5. Save the batch synchronization task.

   After you complete the configuration, click the icon on the toolbar to save the node.

Follow these steps to synchronize the log data from the HttpFile data source to the OSS bucket you created.

1. Open the batch synchronization node.

   On the workflow orchestration page, hover over the batch synchronization node ods_raw_log_d_2oss_emr and click Open Node that appears above it to go to the task configuration page.

2. Configure the data source and runtime resource.

   Configure the data source and resource group based on the following parameters.

   Parameter	Description
   Data source	Data source: HttpFile. Data Source Name: user_behavior_analysis_httpfile.
   Data going	Data going: OSS. Data Source Name: Select the OSS data source you created earlier. In this example, select test_g.
   Running Resources	For Resource Group, select the serverless resource group you purchased.

3. Configure the source and destination.

   1. The following table describes the key parameters. You can keep the default values for other parameters.

      Parameter	Description
      Source	File Path: /user_log.txt. File Type: Select text. Field Delimiter: Enter | as the column delimiter. Compression Format: Four types are available: None, Gzip, Bzip2, and Zip. Select None. Skip Header: Select No.
      Destination	File Type: Select text. Object Name (Path Included): Enter the path based on your OSS directory. In this example, enter ods_raw_log_d/log_${bizdate}/log_${bizdate}.txt. In this path, ods_raw_log_d is the OSS directory name, and $bizdate retrieves the date of the previous day. Field Delimiter: Enter | as the column delimiter.

   2. Configure debug parameters.

      On the configuration page, click Run Configuration on the right side. Configure the following parameters to test the run in Step 4.

      Configuration item	Configuration
      Resource Group	Select the serverless resource group you purchased in Prepare an environment.
      Script Parameters	No configuration is required. The sample code in this tutorial uses ${bizdate} uniformly to represent the business date. When you debug the workflow in Step 4, set This operation value to a specific constant (for example, 20250223). The task run will use this constant to replace the variables defined in the task.

4. Save the batch synchronization task.

   After you complete the configuration, click the icon on the toolbar to save the node.

1. Develop code.

   On the workflow orchestration page, hover over the EMR Hive node ods_user_info_d_emr and click Open Node that appears above it to go to the EMR Hive node editing page. Enter the following sample SQL statements.

   CREATE EXTERNAL TABLE IF NOT EXISTS ods_user_info_d_emr
   (
       `uid` STRING COMMENT 'User ID',
       `gender` STRING COMMENT 'Gender',
       `age_range` STRING COMMENT 'Age range',
       `zodiac` STRING COMMENT 'Zodiac sign'
   ) PARTITIONED BY (
     dt STRING
   )
   ROW FORMAT  delimited fields terminated by '|'
   LOCATION 'oss://dw-emr-demo/ods_user_info_d/';
   
   ALTER TABLE ods_user_info_d_emr ADD IF NOT EXISTS PARTITION (dt='${bizdate}')
   LOCATION 'oss://dw-emr-demo/ods_user_info_d/user_${bizdate}/';

   Note

   The LOCATION in the preceding code is a sample path, which is the same as the destination path you configured for the batch synchronization task. Enter the file path you created. In this path, dw-emr-demo is the OSS bucket name you created during environment setup.

2. Configure debug parameters.

   On the configuration page, click Debug Configuration on the right side. Configure the following parameters to test the run during debugging.

   Configuration item	Configuration
   Computing Resources	Select the EMR compute resource associated during environment setup.
   Resource Group	Select the serverless resource group you purchased in Prepare an environment.
   Script Parameters	No configuration is required. The sample code in this tutorial uses ${bizdate} uniformly to represent the business date. When you debug a task, set This operation value to a specific constant (for example, 20250223). The task run will use this constant to replace the variables defined in the task.

3. After you complete the configuration, click Save to save the node.

1. Develop code.

   On the workflow orchestration page, hover over the EMR Hive node ods_raw_log_d_emr and click Open Node that appears above it to go to the EMR Hive node editing page. Enter the following sample SQL statements.

   --Create target table for OSS logs
   CREATE EXTERNAL TABLE IF NOT EXISTS ods_raw_log_d_emr
   (
     `col` STRING
   ) PARTITIONED BY (
     dt STRING
   );
   ALTER TABLE ods_raw_log_d_emr ADD IF NOT EXISTS PARTITION (dt='${bizdate}')
   LOCATION 'oss://dw-emr-demo/ods_raw_log_d/log_${bizdate}/';

   Note

   The LOCATION in the preceding code is a sample path, which is the same as the destination path you configured for the batch synchronization task. Enter the file path you created. In this path, dw-emr-demo is the OSS bucket name you created during environment setup.

2. Configure debug parameters.

   On the configuration page, click Debug Configuration on the right side. Configure the following parameters to test the run during debugging.

   Configuration item	Configuration
   Computing Resources	Select the EMR compute resource associated during environment setup.
   Resource Group	Select the serverless resource group you purchased in Prepare an environment.
   Script Parameters	No configuration is required. The sample code in this tutorial uses ${bizdate} uniformly to represent the business date. When you debug a task, set This operation value to a specific constant (for example, 20250223). The task run will use this constant to replace the variables defined in the task.

3. Save and run the node.

   After you complete the configuration, click Save and then Run to run the node.