In DataWorks, you create data development tasks as nodes, each encapsulating a task for a specific compute engine. DataStudio also lets you develop complex tasks by using resources, functions, and various logical processing nodes. This topic walks you through the common data development workflow.
Prerequisites
-
The required data sources must be bound. For more information, see Prepare for data development: Bind a compute resource or cluster.
-
You must have the Development role. For more information about how to grant permissions, see Add workspace members and manage member role permissions.
Go to DataStudio
Log on to the DataWorks console. In the target region, click in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Development.
The following procedure describes how to create and manage tasks in DataStudio.
Development workflow
The following figure shows the common data development workflow.
|
Step |
Description |
Related documentation |
|
Step 1: Create a workflow |
In DataWorks, you organize all development tasks and code into workflows. |
|
|
Step 2: Create a table |
DataWorks provides a visual interface for creating and managing tables, displayed in a directory structure. Before you start data development, create tables in the engine to store raw data and cleansed results. The type of table you create depends on your use case. |
Create and use tables: View and manage tables: |
|
Step 3: Create and upload a resource (Optional) |
If your development process requires resources such as text files or JAR packages, upload and manage them in DataWorks for use with a specific compute engine. Note
The supported compute engines and resource types are displayed in the DataWorks UI. |
|
|
Step 4: Create a scheduling node |
In DataWorks, tasks are encapsulated as nodes, with each node type corresponding to a specific compute engine. Select the node type that matches your engine and business needs. DataWorks also provides node management operations such as cloning nodes in batches with node groups and restoring deleted nodes from the recycle bin. |
DataWorks supports multiple engines, including: Different node types are available for different tasks in each engine. For a detailed list of supported node types, see Supported node types. For more information about node management, see the following topics: |
|
Step 5: Reference a resource in a node (Optional) |
To use a resource in a node, load it into the node's runtime environment first. |
|
|
Step 6: Register a function (Optional) |
If your development process requires a function, register it through the DataWorks visual interface. Before registering a function, upload the required resources to DataWorks. Note
The compute engines that support function registration are displayed in the DataWorks UI. |
|
|
Step 7: Edit node code |
On the node editor page, write business code using the syntax of the corresponding engine and database. Syntax may vary by node type. Note
After you finish editing the code, save it promptly ( |
For a detailed list of supported node types, see Supported node types. Usage notes for common engines: |
) to prevent data loss.Next steps
After you develop a task, perform the following operations as needed:
-
Debug the code: Run a single task or an entire workflow to debug it. For more information, see Task debugging process.
-
Configure scheduling: Configure scheduling properties for the node so that it runs periodically. For more information, see Configure scheduling for a node.
-
Commit and deploy the node: After development is complete, commit the node to the target environment for scheduled execution. If you use a workspace in standard mode, after a successful commit, you must also deploy the node by clicking Deploy in the upper-right corner. For more information, see Deploy a node.
-
Perform O&M operations: After a node is deployed, it appears in Operation Center for the production environment by default. Go to Operation Center to view node run status and perform O&M tasks. For more information, see Operation Center overview.