DataWorks:Data development process guide

Personal directory (for personal testing, temporary queries, and cross-project code synchronization)

Files in the Personal Folder are visible only to the current account. You can use these files for personal testing or temporary queries, but you cannot configure scheduling for them or publish them to the production environment. The Personal Folder is visible across all your workspaces and supports cross-workspace synchronization. To schedule and publish a file, you must first submit it from the Personal Folder to a Project Folder. You can then configure scheduling and publish the file from that Project Folder. For more information, see Personal Folder.

1. In the navigation pane on the left, click to go to the Data Development folder.

2. In the Personal Directory section, you can click to create a folder, and then create files in it as needed.

3. To submit a file from the Personal Folder to the Project Folder in the workspace, click Commit To Workspace Directory at the top of the editing window. For the next steps, see Workspace directories (for production environments).

Workspace directories (for production environments)

Files in the Project Folder support collaborative team development. You can create different types of nodes and orchestrate their upstream and downstream dependencies. For more information, see Project Folder.

1. In the navigation pane on the left, click to go to the Data Development folder.

2. Create a project directory, nodes, and workflows.

   In the Workspace Directories section, you can click to create a folder, node, or workflow.

   • Directory: You can use directories to manage nodes and workflows.

   • Node: Data Studio supports a wide range of node types, such as Data Integration, Notebook, and MaxCompute SQL. For more information about the functions and differences of various nodes, see Node development.

   • Workflow: A workflow is a tool for automating the management of data processing. It provides a visual canvas that lets you integrate various types of subtask nodes by dragging and dropping them. This makes it easy to establish dependencies between tasks, accelerate the creation of data processing flows, and improve development efficiency. For more information, see Auto triggered workflows.

3. Node orchestration.

   • Node: For a standalone node, you must configure its upstream and downstream dependencies in the scheduling dependency settings.

     On the node editing page, click Scheduling in the right pane. Configure the Node Scheduling parameters to define the upstream and downstream dependencies for the node. Dependencies ensure that nodes run in the correct order. A descendant node runs only after its ancestor nodes run successfully. This ensures that the current node can retrieve the correct data at the right time.

   • Workflow: A workflow lets you visually orchestrate the upstream and downstream dependencies of nodes on a canvas. You can plan the orchestration as required.

4. Node development.

   Data Studio supports a wide range of node types. The configurable content varies by node type. For more information, see Node development to complete the node configuration.

   Note

   During node development, you can define variables using the ${Variable_Name} format. You can then assign constant values to the variables during the testing phase and dynamically assign values during scheduling configuration.

Manually triggered workflow/task directory

You can create manually triggered workflows or tasks for scenarios that do not require recurring scheduling.

1. In the navigation pane on the left, click to go to the Manual Folder.

2. Create development folders and nodes under Manually Triggered Workflows or Manually Triggered Tasks as needed. For more information, see Manually triggered task and Manually Triggered Workflow.

Scheduling configuration

After you debug the node, if it needs to be published to the production environment for recurring automatic scheduling, click Scheduling on the right side of the node editing page to configure its scheduling properties.

• Scheduling Parameters: Define the parameters used for node scheduling. DataWorks provides multiple assignment formats. If you defined variables using the ${Variable_Name} format during node development, you can use scheduling parameters to dynamically assign values to variables in scheduling scenarios.

• Scheduling Policies: Define scheduling properties for the node in the scheduling environment, other than the execution frequency and specific execution time.

• Scheduling Time: Define the execution frequency and specific execution time for the node in the scheduling environment.

• Scheduling Dependencies: Define the upstream and downstream dependencies for the task. Dependencies ensure that nodes run in the correct order. A descendant node starts only after its ancestor nodes run successfully. This process ensures that the current node retrieves data correctly and at the right time.

Node publishing

After you configure the scheduling properties of the node, click the Publish button at the top of the node editing page. The node is then published to the production environment and scheduled to run periodically. For more information, see Publish a node or workflow.

Click the Publish button in the toolbar and then click Start Deployment to Production Environment. This action publishes the task according to the publishing check process.