DataWorks:Data development process guidance

Personal directory development (for personal testing, temporary queries, and cross-project code synchronization)

Files in the personal directory are visible only to the current account, cannot be scheduled or published to production, and can be used for personal testing or temporary queries. Files in the personal directory are visible across different workspaces and can be used for cross-workspace synchronization. You can submit files from your personal directory to project directories in different workspaces, and then configure scheduling and publishing in the corresponding workspace's project directory. For more information, see Personal directory.

1. In the left navigation bar of Data Studio, click to access the data development directory.

2. In the Personal Directory section, click to create a directory, and then create files in the directory as needed.

3. If you need to submit files from the personal directory to the workspace project directory, you can click Submit To Project Directory at the top of the editing window. For subsequent steps, see Project directory development (for production environment).

Project directory development (for production environment)

Files in the project directory can be developed collaboratively by teams, support creating different types of nodes, and orchestrating upstream and downstream node relationships. For more information, see Project directory.

1. In the left navigation bar of Data Studio, click to access the data development directory.

2. Create project directories, nodes, and workflows.

   In the Project Directory section, click to create directories, nodes, or workflows as needed.

   • Directory: You can use directories to manage nodes and workflows.

   • Node: Data Studio supports various node types, such as Data Integration, Notebook, and MaxCompute SQL. For the functions and differences of different nodes, see Node development.

   • Workflow: A workflow is an automated management tool for data processing processes. It provides a visual canvas that supports integrating various types of task nodes through visual drag-and-drop, easily establishing dependencies between tasks, accelerating the construction of data processing processes, and effectively improving task development efficiency. For more information, see Recurring workflow.

3. Node orchestration.

   • Node: For individually created nodes, you need to configure upstream and downstream relationships through node scheduling dependencies.

     Click Scheduling Configuration on the right side of the node editing page, and configure Node Scheduling parameters to define the upstream and downstream dependencies of the node. Through dependency relationships, upstream and downstream nodes run in order, meaning that downstream nodes will only start running after upstream nodes run successfully, ensuring that the current node obtains data in a timely and correct manner.

   • Workflow: Workflows support visually dragging and dropping to orchestrate upstream and downstream relationships between nodes on the canvas, which you can plan yourself.

4. Node development.

   Data Studio supports various node types with different configurable content. You can refer to Node development to complete node configuration.

   Note

   You can define variables using the ${variable name} format during node development, then assign constant values to variables during testing, and dynamically assign values to variables during scheduling configuration.

Manual directory development (for manually triggered node instances)

You can create manually triggered tasks or manually triggered workflows in the manual directory for one-time data processing scenarios that do not require recurring scheduling.

1. In the left navigation bar of Data Studio, click to access the manual directory.

2. Create development directories and nodes under Manually Triggered Task or Manually Triggered Workflow as needed. For more information, see Manually triggered task and Manually triggered workflow.

Scheduling configuration

After node debugging is complete, if the node needs to be published to the production environment for recurring automatic scheduling, click Scheduling Configuration on the right side of the node editing page to configure scheduling properties for the node.

• Scheduling Parameters: Used to define parameters used during node scheduling. DataWorks provides multiple assignment formats. If you have defined variables using the ${variable name} format during node development, scheduling parameters can implement dynamic assignment of variables in scheduling scenarios.

• Scheduling Policy: Used to define scheduling attributes other than execution frequency and specific execution time points for nodes in the scheduling environment.

• Scheduling Time: Used to define the execution frequency and specific execution time points for nodes in the scheduling environment.

• Scheduling Dependencies: Used to define the upstream and downstream dependency relationships of the task. Through dependency relationships, upstream and downstream nodes run in order, meaning that downstream nodes will only start running after upstream nodes run successfully, ensuring that the current node obtains data in a timely and correct manner.

Node publishing

After node scheduling configuration is complete, you need to click the Publish button at the top of the node editing page to publish the node to the production environment through the publishing process. After that, the node will be scheduled periodically. For more information, see Node/workflow publishing.

Click the Publish button in the toolbar to enter the publishing process, then click Start Publishing To Production, and the task will be published according to the publishing check process.