DataWorks:Data development: Developers

Go to the DataStudio page

1. Log on to the DataWorks console.

2. In the left-side navigation pane, click Workspaces.

3. In the top navigation bar, select the region in which the desired workspace resides. On the Workspaces page, find the workspace and click DataStudio in the Actions column.

Procedure

1. Step 1: Create a workflow

   Code is developed based on a workflow in DataStudio. Before you perform development operations, you must create a workflow.

2. Step 2: Create tables

   DataWorks allows you to create tables in a visualized manner and displays tables in a directory structure. Before data development, you must create a table in the MaxCompute compute engine instance to store the data processing results.

3. Step 3: Create a node

   Data development in DataWorks is based on nodes, and tasks of different types of compute engines are encapsulated into different types of nodes in DataWorks. You can select a suitable node type for node development based on your business requirements.

4. Step 4: Configure the node

   You can write code for the node on the node configuration tab based on the syntax that is supported by the related database.

5. Step 5: Configure scheduling properties for the node

   You can configure scheduling attributes for the node to schedule and run the node at regular intervals.

6. Step 6: Debug the code of the node

   You can use the quick run feature for code snippets, or the Run feature or Run with Parameters feature to debug and check the logic of the code of the node.

7. Step 7: Save and commit the node

   After the node is debugged, you must save and commit the node.

8. Step 8: Perform smoke testing

   To ensure efficient running of a node in the production environment and prevent wastage of computing resources, you can commit the node to the development environment and perform smoke testing in the development environment before you deploy the node. This helps ensure the correctness of the code of the node.

9. Step 9: Deploy the node

   DataWorks can schedule only nodes that are deployed to the production environment. After the node passes the smoke testing, you must deploy the node to the production environment to enable DataWorks to schedule the node at regular intervals.

Step 1: Create a workflow

DataWorks organizes data development processes by using workflows. DataWorks provides dashboards for different types of nodes in each workflow and allows you to use tools and optimize and manage nodes on the dashboards. This facilitates data development and management. You can place nodes of the same type in one workflow based on your business requirements.

1. Go to the DataStudio page.

2. Create a workflow.

   You can use one of the following methods to create a workflow:

   • Method 1: Move the pointer over the icon and click Create Workflow.

   • Method 2: Right-click Business Flow in the Scheduled Workflow pane and select Create Workflow.

3. Configure the Workflow Name parameter and the Description parameter for the workflow, and click Create.

   In this example, the Workflow Name parameter is set to Create the first auto triggered node. You can configure the Workflow Name parameter based on your business requirements in actual data development scenarios.

   Note

   For more information about how to use workflows, see Create and manage workflows.

Step 2: Create tables

A data development node of DataWorks cleanses and processes source data. Before data development, you must create a table in the required compute engine instance to store the data cleansing results and define the table schema.

1. Create tables.

   1. Click Business Flow in the Scheduled Workflow pane. Find the workflow that is created in Step 1, click the workflow name, right-click MaxCompute, and then select Create Table.

   2. Configure the Engine Type, Path, and Name parameters.

   In this example, the following tables are created.

   Table name	Description
   bank_data	Used to store raw business data.
   result_table	Used to store the data cleansing results.

   Note

   • For information about table creation statements, see Table creation statements.

   • For information about how to create tables in different compute engine instances in a visualized manner, such as creating a MaxCompute table or an EMR table, see Create tables.

2. Generate table schemas.

   Go to the editing pages of the tables, switch to the DDL mode, and use DDL statements to generate schemas for the tables. After the table schemas are generated, configure the Display Name parameter in the General section, click Commit to Development Environment, and then click Commit to Production Environment in the top toolbar. For information about how to view the compute engines that are associated with workspaces in different environments, see Manage MaxCompute projects.

   Note

   • Operations such as table creation and table update can take effect in the related compute engine instances only after they are committed to the required environment.

   • You can also follow the on-screen instructions that are displayed in the DataWorks console to configure the table schemas in a visualized manner based on your business requirements. For more information about how to create a table in a visualized manner, see Create and manage MaxCompute tables.

   In this example, the following statement is used to generate the schema of the bank_data table:

   CREATE TABLE IF NOT EXISTS bank_data
   (
    age             BIGINT COMMENT 'Age',
    job             STRING COMMENT 'Job type',
    marital         STRING COMMENT 'Marital status',
    education       STRING COMMENT 'Education level',
    default         STRING COMMENT 'Credit card',
    housing         STRING COMMENT 'Mortgage',
    loan            STRING COMMENT 'Loan',
    contact         STRING COMMENT 'Contact information',
    month           STRING COMMENT 'Month',
    day_of_week     STRING COMMENT 'Day of the week',
    duration        STRING COMMENT 'Duration',
    campaign        BIGINT COMMENT 'Number of contacts during the campaign',
    pdays           DOUBLE COMMENT 'Interval from the last contact',
    previous        DOUBLE COMMENT 'Number of contacts with the customer',
    poutcome        STRING COMMENT 'Result of the previous marketing campaign',
    emp_var_rate    DOUBLE COMMENT 'Employment change rate',
    cons_price_idx  DOUBLE COMMENT 'Consumer price index',
    cons_conf_idx   DOUBLE COMMENT 'Consumer confidence index',
    euribor3m       DOUBLE COMMENT 'Euro deposit rate',
    nr_employed     DOUBLE COMMENT 'Number of employees',
    y               BIGINT COMMENT 'Time deposit available or not'
   );

   In this example, the following statement is used to generate the schema of the result_table table:

   CREATE TABLE IF NOT EXISTS result_table
   (
   education STRING COMMENT 'Education level',
   num BIGINT COMMENT 'Number of persons'
   )
   PARTITIONED BY
   (
   day STRING,
   hour STRING
   );

3. Upload data.

   Upload raw business data to a table in DataWorks. In this example, data is uploaded to the bank_data table. In this example, a file named banking.txt is uploaded from an on-premises machine to DataWorks. The following figure shows the procedure. For more information about how to upload data, see Upload a file from your on-premises machine to the bank_data table.

Step 3: Create a node

1. Create a node.

   You can use one of the following methods to create a node:

   • Method 1: Create a node in the Scheduled Workflow pane

     1. In the Scheduled Workflow pane of the DataStudio page, click Business Flow, find the workflow that you created, and then click the name of the workflow.

     2. Right-click the compute engine that you want to use, and select a suitable node type after you move the pointer over Create Node.

   • Method 2: Create a node on the configuration tab of the workflow

     1. In the Scheduled Workflow pane of the DataStudio page, click Business Flow, find the workflow that you created, and then click the name of the workflow.

     2. Double-click the name of the workflow to go to the configuration tab of the workflow.

     3. In the left-side section of the configuration tab, click the required node type or drag the required node type to the canvas on the right side.

2. Configure the Engine Instance, Node Type, Path, and Name parameters for the node.

   In this example, an ODPS SQL node named result_table is created. The name of the node is the same as the name of the result table that is created in Step 2.

   Note

   When you use DataWorks for data development, you need to use a compute node to cleanse the data and then store the cleansing results in a result table. We recommend that you use the name of the result table as the name of the node to quickly locate the table data that is generated by the node.

   

Step 4: Configure the node

Find the node that you created in Step 3, and double-click the name of the node to go to the node configuration tab. On the node configuration tab, write the code of the node based on the syntax that is supported by the related database.

INSERT OVERWRITE TABLE result_table partition (day='${day}', hour='${hour}')
SELECT education
, COUNT(marital) AS num
FROM bank_data
GROUP BY education;

Step 5: Configure scheduling properties for the node

You can configure scheduling properties for a node to enable periodical scheduling for the node. In the right-side navigation pane of the node configuration tab, click the Properties tab. You can configure scheduling properties in different sections of the tab for the node based on your business requirements.

Tab	Description
General	In this section, the node name, node ID, node type, and owner of the node are automatically displayed. You do not need to configure additional settings. Note By default, the owner of the node is the current user. You can modify the owner of the node based on your business requirements. You can select only a member in the current workspace as the owner of the node. An ID is automatically generated after the node is committed.
Parameters	In this section, you can configure the scheduling parameters that are used to define how the node is scheduled. DataWorks provides scheduling parameters that can be classified into custom parameters and built-in variables based on their value assignment methods. Scheduling parameters support dynamic parameter settings for node scheduling. If a variable is defined during the modification of the code of the node in Step 4, you can assign a value to the variable in the Parameters section. In this example, the following variables are defined in Step 4, and values are assigned to the variables to write the data that is generated in the 24 hours of the previous day in the bank_data table to the partition in the result_table table. Assign ${yyyymmdd} to day as the value. Assign $[hh24] to hour as the value.
Schedule	In this section, you can configure time properties for the node, such as the instance generation mode, the scheduling cycle, the point in time when you want to schedule the node to start, the rerun settings, and the timeout period. Note You can commit the node only after you configure the rerun settings. The scheduling time that you specify for a node takes effect only on the node. The point in time when the node starts to run is related to the scheduling time of the ancestor node of the node. The node can start to run only if the scheduling time of the ancestor node arrives and the ancestor node is successfully run, even if the scheduling time of the node is earlier than the scheduling time of the ancestor node. In this example, the result_table node is scheduled to run at an interval of 1 hour from 00:00. The data that is generated each hour in the bank_data table in the 24 hours of the previous day is written to the related hourly partition in the result_table table each hour.
Resource Group	In this section, you can select the resource group for scheduling that you want to use to deploy the node to the production environment. When you activate DataWorks, the service provides the shared resource group for scheduling. In this example, the shared resource group for scheduling is used. Note If a large number of nodes need to run in parallel, exclusive computing resources are required to ensure that the nodes can be run as scheduled. In this case, we recommend that you use an exclusive resource group for scheduling. For more information about exclusive resource groups for scheduling, see Exclusive resource groups for scheduling.
Dependencies	In this section, you can configure scheduling dependencies for the node. We recommend that you configure scheduling dependencies for the node based on the lineage of the node. If the ancestor node of the current node is successfully run, the table data that the current node needs to use is generated. This way, the current node can obtain the table data. Note If a SELECT statement is specified in the code of the current node to query the table data that is not generated by an auto triggered node, you can disable the automatic parsing feature and use the root node of the workspace to schedule the current node. If a SELECT statement is specified in the code of the current node to query the table data that is generated by other nodes, you can use the following methods to configure the ancestor node that is used by the current node as a dependency. The ancestor node generates the table data that is queried by the current node, and is used to schedule the current node. If the ancestor node does not belong to the current workflow or workspace, specify the value of the output name parameter of the ancestor node in the Parent Nodes table. If the ancestor node belongs to the current workflow, configure scheduling dependencies for the current node by drawing lines in the canvas of the workflow. In this example, if the result_table node queries the data in the bank_data table that is not generated by a node in the current workflow, configure the root node of the workspace as the ancestor node of the result_table node, and use the root node to schedule the result_table node.
(Optional) Parameters	In this section, you can configure input parameters and output parameters for the node. The configurations in this section are optional. A node can obtain the values of parameters that are configured for its ancestor node by using specific parameters. Note In most cases, this process requires assignment nodes or scheduling parameters. For more information about assignment nodes, see Configure an assignment node. For information about scheduling parameters, see Supported formats of scheduling parameters.

Step 6: Debug the code of the node

Step 7: Save and commit the node

1. Click the icon in the top toolbar to save the node.

2. Click the icon in the top toolbar to commit the node to the development environment.

Step 8: Perform smoke testing

In this example, smoke testing is performed to check whether the scheduling parameters that are configured meet user requirements. The result_table node is scheduled to run at an interval of 1 hour from 00:00 to 23:59. When the smoke testing is performed on the node, two instances are generated. The scheduling time of the instances are 00:00 and 01:00.

Note

• Auto triggered node instances are snapshots that are generated for an auto triggered node when the node is scheduled to run based on the specified scheduling cycle.

• The result_table node is scheduled by hour. You must specify the data timestamp of the node for the smoke testing. You must also select the start time and end time of the test.

• For more information about how to perform smoke testing in the development environment, see Perform smoke testing.

Step 9: Deploy the node

In a workspace in standard mode, the operations that are committed on the DataStudio page, including addition, update, and deletion of data development nodes, resources, and functions, are in the pending state on the Create Deploy Task page. You can click Deploy to go to the Create Deploy Task page, and deploy the related operations to the production environment. The operations take effect only after they are deployed to the production environment. For more information, see Deploy nodes.