You can use the resource group orchestration feature to change resource groups for scheduling for multiple nodes in a workflow at the same time. If multiple resource groups for scheduling exist in your workspace, you can reallocate resource groups for scheduling for nodes in the workspace based on your business requirements. This helps you optimize resource allocation and utilization. This topic describes how to use the resource group orchestration feature of DataWorks.
Prerequisites
- A workflow is created. For more information about how to create a workflow, see Create an auto triggered workflow.
- One or more resource groups for scheduling are created for your workspace. This way, you can use the resource group orchestration feature to allocate resource groups for scheduling in the current workspace.
After you activate DataWorks, the system provides the shared resource group for scheduling for you to run nodes. If the shared resource group for scheduling cannot meet your requirements, you can purchase exclusive resource groups for scheduling based on your business requirements. For more information about exclusive resource groups for scheduling, see Exclusive resource groups for scheduling.
Background information
The resource group orchestration feature is used to change the resource groups for scheduling for multiple nodes in a specific workflow at the same time during data development. The maximum number of nodes that can be run in parallel on a resource group for scheduling varies based on the specifications of the resource group. You can run nodes on resource groups for scheduling with different specifications based on your business requirements. For more information about the specifications of exclusive resource groups for scheduling and the maximum number of parallel nodes that are supported by the exclusive resource groups for scheduling with different specifications, see Billing of exclusive resource groups for scheduling (subscription).Limits
- DataWorks allows you to change the resource groups for scheduling only for auto triggered nodes.
- Zero load nodes do not occupy scheduling resources. Therefore, you cannot change the resource groups for scheduling for zero load nodes.
Procedure
Go to the DataStudio page.
Log on to the DataWorks console.
In the left-side navigation pane, click Workspaces.
In the top navigation bar, select the region where the workspace resides. On the Workspaces page, find the workspace in which you want to create tables, and click in the Actions column.
- Go to the Resource Group Orchestration tab. On the DataStudio page, move the pointer over the desired workflow in the Business Flow section and click the icon to the right of the workflow name to go to the Resource Group Orchestration tab.Note Before you go to the Resource Group Orchestration tab, you must find the workflow in which you want to change the resource groups for scheduling for nodes.
- Change the resource groups for scheduling for multiple nodes. Note
- You can specify the same resource group for scheduling for multiple nodes in one batch.
- If you want to specify different resource groups for scheduling for multiple nodes, you must perform the operations in batches.
- Optional:Search for the desired nodes by using specific filter conditions, such as the node name, node type, and engine type.
- Check the nodes in the search results and select the nodes for which you want to change the resource groups for scheduling.
- Click Switch Resource Groups and select a resource group for scheduling from the list.
- Click OK.
You can also perform the following operations on the resource groups for scheduling:- Modify the configurations of a resource group: You can modify the configurations of only exclusive resource groups for scheduling. After you click Settings to the right of the name of an exclusive resource group for scheduling, the Exclusive Resource Groups tab appears. You can view the details of the exclusive resource group for scheduling and modify the configurations of the resource group for scheduling based on your business requirements.
- Purchase resource groups for scheduling: If no resource groups for scheduling are available in the current workspace, you can click Purchase Resource Group in the upper-right corner of the Resource Group Orchestration tab to purchase resource groups for scheduling.
- Change the resource group for scheduling for a single node: You can find the desired node and click Switch in the Actions column to change the resource group for scheduling for the node.
- Click the icon to commit the changes to the resource groups.
- Deploy the nodes whose resource groups for scheduling are changed. After the resource groups for scheduling of nodes are changed, you must go to the Deploy page to redeploy the nodes to the production environment. After you redeploy the nodes, the nodes are run on the new resource group for scheduling in the production environment. For more information about how to deploy a node, see Deploy nodes.