DataWorks provides various MaxCompute node types to support diverse development requirements. It also offers comprehensive scheduling configurations for flexible task execution. This topic describes how to create and manage MaxCompute nodes.
Scope
To develop MaxCompute tasks, your account must be added to the workspace and assigned the Development or Workspace Manager role. The Workspace Manager role grants extensive permissions. Assign this role with caution. For more information, see Add members to a workspace.
Create a MaxCompute node
Go to the DataStudio page.
Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose . On the page that appears, select the desired workspace from the drop-down list and click Go to Data Development.
Associate the MaxCocmpute computing source with the workspace and create a workflow.
In Data Development (DataStudio), workflows organize tasks for specific compute engines. You must create a workflow before adding nodes.
The following steps demonstrate how to create an ODPS SQL node.
Right-click a workflow and select . Alternatively, click the New button in the top toolbar and follow the prompts.
ImportantIf the path is not visible, click Compute Engines in the navigation pane on the left verify that a MaxCompute engine is associated with the workspace. You must associate a MaxCompute engine and refresh the page before creating nodes.
In the dialog box that appears, enter a name for the node and click Confirm. After the node is created, you can develop and configure the MaxCompute task in the node editor.
Develop a MaxCompute task
DataWorks supports multiple node types for developing MaxCompute tasks.
When you run a MaxCompute task, the system displays an estimated cost. This is for reference only. Actual charges depend on your final bill. For details, see Billable items and billing methods.
If a cost estimation error occurs (for example, if a table does not exist or permissions are missing), you can ignore the warning and troubleshoot based on the actual error message generated during execution.
Node type | Use scenario | Task development guide |
ODPS SQL | Develop MaxCompute SQL tasks. | |
SQL Snippet | Develop reusable SQL processes. Use this node when multiple SQL processes share similar logic and compatible table structures but use different table names. You can abstract the SQL process into a template where input and output tables are defined as variables (request and response parameters), enabling code reuse. | |
PyODPS 3 | Develop PyODPS tasks using Python 3. | |
PyODPS 2 | Develop PyODPS tasks using Python 2. | |
ODPS Spark | Develop MaxCompute Spark tasks. | |
ODPS Script | Develop script-mode SQL tasks (for complex script logic). | |
ODPS MR | Develop MaxCompute MapReduce tasks. |
Create tables, resources, and functions
In addition to task development, DataWorks provides tools to manage the tables, resources, and functions required for MaxCompute jobs.
MaxCompute tables: Use the DataWorks UI to create, view, and manage tables. For more information, see Create and manage MaxCompute tables and Manage tables.
MaxCompute functions and resources:
You can directly call MaxCompute built-in functions within your tasks. For a complete list, see Use built-in functions.
You can upload custom code as resources in DataWorks and register them as MaxCompute functions. These functions can then be called directly during task development. For details, see Create and use MaxCompute resources and Create and use user-defined functions.
You can create resources directly in DataWorks or upload locally developed packages.
DataWorks supports various file types, including text files, Python scripts, and archives (such as .zip, .tgz, .tar.gz, .tar, and .jar). These resources are accessible when running UDFs and MapReduce tasks. For details on uploading and using resources, see Create and use MaxCompute resources.
What to do next
After developing the task logic, proceed with the following operations:
Configure scheduling properties: Define periodic scheduling rules for the node. If a task requires recurring execution, you must configure properties such as run frequency, rerun policies, and scheduling dependencies. For more information, see Overview of task scheduling properties.
Debug the task: Test the node code to verify that the logic performs as expected. For more information, see Task debugging process.
Publish the task: Deploy the node to the production environment. Once published, the task executes automatically based on its scheduling configuration. For more information, see Publish tasks.