DataWorks provides PAI Designer nodes, which load Machine Learning Designer pipelines and run them on a periodic schedule. Use this node type to integrate existing PAI pipelines into your DataWorks scheduling workflows.
Prerequisites
Before you begin, make sure you have:
-
DataWorks authorized to access PAI. Complete the authorization with a few clicks on the authorization page. This creates the AliyunServiceRoleForDataworksEngine service-linked role. For details, see Role 1: AliyunServiceRoleForDataworksEngine. Only an Alibaba Cloud account or a RAM user with the AliyunDataWorksFullAccess policy can perform one-click authorization.
-
A PAI Designer node created in DataWorks.
Step 1: Develop the task
On the configuration tab of the PAI Designer node, set up your pipeline and configure scheduling.
Load or create a pipeline
To load an existing pipeline, search for it by name on the configuration tab. Create the pipeline in PAI before starting — it must exist before you can find it in search results.
To create a pipeline directly from the configuration tab, choose one of the following methods:
| Method | Description |
|---|---|
| Blank pipeline | Add components and use drag-and-drop to build a model from scratch. See Create a blank pipeline. |
| Preset template | Start from a built-in template, then modify components or their configurations. See Create a pipeline from a preset template. |
| Custom template | Use a pipeline previously saved as a custom template by a workspace member. See Create a pipeline from a custom template. |
Configure scheduling parameters
Scheduling parameters let you pass dynamic values into node code at runtime. DataWorks replaces each variable reference in your node code with the value you define in the Properties tab — this substitution happens automatically every time the node runs.
To configure scheduling parameters:
-
Click Properties in the right-side navigation pane.
-
Define a variable in the node code using the
${Variable}format:--command='echo '\''${Variable}'\'';' \ --You can assign a specific scheduling parameter to the variable. -
In the Scheduling Parameters section of the Properties tab, assign a value to the variable.
-
Click Save.
For supported variable formats, see Supported formats of scheduling parameters.
Step 2: Deploy and monitor the node
-
Deploy the node. See Deploy nodes.
-
After deployment, click Perform O&M below Prod Online to open the node in Operation Center and view its run status. See Getting started with Operation Center.
Run a PAI Designer node manually
PAI Designer nodes have no run entry point in DataStudio. To trigger a run manually, go to Operation Center in the development environment and choose Auto Triggered Node O&M > Auto Triggered Nodes in the Node O&M section. Find the node, then click Test or Backfill Data in the Actions column.