Purchase an exclusive resource group for scheduling - DataWorks

DataWorks no longer recommends exclusive resource groups for scheduling

DataWorks no longer recommends exclusive resource groups for scheduling. We recommend that you use serverless resource groups. Serverless resource groups consolidate the core capabilities of legacy resource groups (exclusive resource groups for scheduling, exclusive resource groups for Data Integration, exclusive resource groups for Data Service, and shared resource groups) into a single resource group, with which you can synchronize data, schedule and run tasks, and call and manage API services.

Prerequisites

Review the specifications, concurrency limits, and pricing in Billing of exclusive resource groups for scheduling. Plan your resource group specifications and subscription duration based on your business requirements.
Review the Use cases for exclusive resource groups for scheduling.

Notes

(Optional) You must understand network connectivity and allowlist configurations if the exclusive resource group for scheduling needs to interact with data sources or other network environments, such as accessing a self-managed database or an internal address by using a Shell script, or if you use engines like EMR or CDH that require an exclusive resource group for scheduling. Network connectivity and allowlist configurations.

Note

If you only need to mitigate task delays from resource contention in shared scheduling resource groups and do not need database connectivity, skip the network configuration steps. You can purchase an exclusive resource group in any availability zone without configuring network settings.
Exclusive resource groups can access the internet by default, but over shared bandwidth with no performance guarantee. If your tasks rely heavily on internet access, use a Serverless resource group instead.

Limitations

Only an Alibaba Cloud account or a RAM user with the AliyunBSSOrderAccess and AliyunDataWorksFullAccess permissions can purchase resource groups.
Only a workspace administrator can associate a resource group with a workspace or modify the association.
Detailed permissions are listed in Resource group permission control policies.
You can also create custom policies to fine-tune access. Create a custom policy and grant permissions.
An exclusive resource group for scheduling of the 4c8g type supports binding up to 2 VPCs. Exclusive resource groups for scheduling of other specifications support binding up to 3 VPCs.
Exclusive resource groups will be deprecated in the future. If you have never activated DataWorks in any region, you can only purchase and use serverless resource groups after activation, and cannot purchase or use legacy resource groups. Serverless resource group release notes.
Tasks that run on an exclusive resource group for scheduling can create up to 100 files at runtime, and each file cannot exceed 150MB in size.

Step 1: Purchase a resource group

Exclusive resource groups for scheduling use subscription billing and must be purchased separately.

Note

Only users with the AliyunBSSOrderAccess and AliyunDataWorksFullAccess permissions can purchase resource groups.

Log on to the DataWorks console.

In the left-side navigation pane, click Resource Group to go to the Resource Groups page. On the Exclusive Resource Group tab, click Create Resource Group for Scheduling of Old Version. On the purchase page, configure the following key parameters based on your business needs.

Parameter	Description
Region	Select the region where you want to use this exclusive resource group. Note Exclusive resource groups cannot be used across regions. For example, a resource group in China (Shanghai) can only be used by workspaces in China (Shanghai).
Exclusive Resource Group Type	Set this parameter to Exclusive Resources for Scheduling.
Exclusive Resources for Scheduling	Select the resource group specifications. Different specifications have different costs and concurrency limits. Billing of exclusive resource groups for scheduling.
Resources	The number of machines in the resource group. Select at least two for production high availability.
Billing Cycle	Subscription billing. Select Auto Renewal to avoid service interruption. You can manage auto-renewal later on the Alibaba Cloud Renewals page. Renew an exclusive resource group.
Resource Name	Set a unique name for the resource group within the tenant. Duplicate names cause an error during confirmation. Note A tenant is an Alibaba Cloud account, which can have multiple RAM users.

Click Buy Now and follow the on-screen instructions to complete the payment and purchase the exclusive resource group for scheduling.

After purchase, DataWorks initializes the exclusive resource group. When the status changes to running, the resource group appears in the console.

Note
The initialization of an exclusive resource group takes about 20 minutes. Wait until the status changes to Running.

After the resource group is ready, associate it with a workspace before using it in task configurations.

Step 2: Associate with a workspace

Note

Only a workspace administrator can associate a resource group with a workspace or modify the association.

An exclusive resource group for scheduling must be associated with a workspace before use. A resource group can be associated with multiple workspaces, but only within the same region. For example, a resource group in China (Shanghai) can only be associated with workspaces in China (Shanghai).

Log on to the DataWorks console.
On the Resource Groups page, click the Exclusive Resource Group tab, and then click Associate Workspace in the row of the target resource group.
On the Associate Workspace page, click Bind in the row of the target workspace to associate the exclusive resource group with the workspace.

Step 3: Configure network settings

Bind a VPC

You must understand network connectivity and allowlist configurations if the exclusive resource group for scheduling needs to interact with data sources or other network environments, such as accessing a self-managed database or an internal address by using a Shell script, or if you use engines like EMR or CDH that require an exclusive resource group for scheduling, you must also configure VPC binding and allowlist settings as described below.

An exclusive resource group is deployed in a DataWorks-managed VPC. Bind your own VPC to the resource group to enable connectivity to your data sources.

Important

An exclusive resource group for scheduling of the 4c8g type supports binding up to 2 VPCs. Other specifications support up to 3 VPCs.

Log on to the DataWorks console.
On the Resource Groups page, click the Exclusive Resource Group tab, and then click Network Settings in the row of the target resource group to go to the binding page.

Before binding, use your Alibaba Cloud account (only the Alibaba Cloud account has the required permissions) to complete RAM authorization for DataWorks to access your cloud resources. On the Cloud Resource Access Authorization page, select the AliyunDataWorksAccessingENIRole role, and click Confirm Authorization Policy.

Bind a VPC.

On the VPC Binding page, click Add Binding in the upper-left corner. In the Add VPC Binding dialog box, configure the parameters based on your network environment.

Note

For use cases that involve Alibaba Cloud instances or self-managed ECS instances, the network connectivity solution and configuration depend on whether the DataWorks workspace and the data source belong to the same primary account.

Parameter	Same account and region	Different accounts or regions
VPC	If your data source and exclusive resource group are in the same Alibaba Cloud account, we recommend that you select the VPC where the data source resides. If they are not in the same Alibaba Cloud account, follow the instructions for the scenario where they are in different regions.	If your data source and exclusive resource group are in different regions or under different primary accounts, select a VPC in the current account that is connected to the data source network. For example, if the data source is not in an Alibaba Cloud VPC, you can click Create VPC to create a VPC for the exclusive resource group. After the VPC is created, select the new VPC or a VPC that is already connected to the target database network. Note If the DataWorks workspace and the data source are in different regions or under different primary accounts, you must also use a service such as VPN Gateway or Express Connect to connect the VPC that is bound to the exclusive resource group to the VPC where the data source is located. You must also manually add a route that points to the target database IP address to ensure connectivity between the networks. For more information, see Network connectivity solutions.
Zone	Select the zone where the database is located.	Select a zone that is already connected to the target database network.
vSwitch	If you select the VPC where the data source resides, we recommend that you select the vSwitch that is associated with the data source. Note After you bind any vSwitch in the data source's VPC, a route is automatically added to the entire VPC CIDR block to ensure network connectivity for the exclusive resource group for Data Integration within that VPC.	Select a vSwitch that is already connected to the target database network. If no vSwitch is available, click Create VSwitch to create one for the exclusive resource group. After the vSwitch is created, select it here.

Click Determine to complete the VPC binding.

Note

If the data source and the exclusive resource group are in different regions or under different primary accounts, you must bind a VPC and then add a route rule that points to the target database IP address.

Optional: Configure a hostname-to-IP mapping.

If your data source is accessed using a hostname instead of an IP address, you must configure a hostname-to-IP mapping. Otherwise, connectivity tests that use the hostname fail when you add the data source.

Click Hostname-to-IP Mapping. On the page, click Add in the upper-left corner. In the Create Hostname-to-IP Mapping dialog box, configure the following parameters.

Parameter	Description
IP Address	Enter the actual IP address of the data source.
Hostname	Enter the host domain that the data source uses for external access. If there are multiple host domains, enter each one on a new line.

To add multiple mappings, click Add again.
Note
- The IP address and host domains in a new mapping must not duplicate any IP address or host domain in existing mappings.
- A single mapping supports a one-to-many relationship between an IP address and host domains. An IP address can be mapped to multiple host domains, but a host domain can be mapped to only one IP address.

Add an allowlist

If the resource group and data source are in the same availability zone but share the same VPC and vSwitch and still cannot connect, the data source may have allowlist restrictions. Add the following entries to the data source allowlist.

Internal network: Add the vSwitch CIDR block used when binding the resource group to a VPC to the data source allowlist.

After binding the VPC, go to the Resource Groups > Exclusive Resource Group page. Click Network Settings for the target resource group, and then click the VPC Binding tab to view the vSwitch CIDR Block. Find the CIDR block address in the vSwitch CIDR Block column.
Internet: Add the EIP Address of the resource group to the data source allowlist. Find the EIP Address on the Basic Information page of the resource group.

Step 4: Test network connectivity

After completing the network configuration, test the connectivity between the resource group and the data source.

1. Log on to the DataWorks console. In the target region, click More > Management Center in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Management Center.
2. On the Workspace Management page, click Data Sources in the left-side navigation pane to open the data source page.
Click Edit in the Operation column of the target data source.
On the data source editing page, click Test Connectivity next to the target scheduling resource group. If the connectivity status is Connectable, the connection is successful.

In the Connection Settings section, click the Data Scheduling tab to view the resource group list and the connectivity status for both development and production environments. Click Finish Editing to save.
Note
- The configuration interface varies by data source type.
- If the data source has separate development and production environments, test connectivity in each environment separately.
- Different network environments may require specific configurations. Network connectivity solutions.

More operations

View resource group usage and monitoring

Check resource group usage and queuing status in the DataWorks console, or use Operation Center intelligent monitoring to track resource consumption and queued instances. If a task takes too long to execute, check whether resources have been fully consumed. View resource group usage. Resource group monitoring.

Use O&M Assistant to run commands on a resource group

Use O&M Assistant to run commands on an exclusive resource group during development, such as installing third-party PyODPS packages.

Change the availability zone of a resource group

To change the availability zone of a resource group, perform the following steps:

Log on to the DataWorks console.
In the left-side navigation pane, click Resource Group. On the Resource Groups page, click the Exclusive Resource Group tab, and find the resource group whose Purpose is Data Scheduling.
In the Operation column of the target resource group, click the icon, and select Change Zone to open the Change Zone for Resource Group dialog.
In the Change Zone for Resource Group dialog, select the Current Zone and Machines of the resource group to change, and then select the target New Zone and Number of Machines to Use.
Click OK to complete the availability zone change for the resource group.

Changing the availability zone may cause network changes:

Resource group CIDR block: Each availability zone has an independent CIDR block. If the availability zone changes, the CIDR block changes accordingly.
Resource group primary NIC IP: The primary NIC IP always changes. A new IP is assigned within the CIDR block of the target availability zone.
Elastic network interface bound to the resource group: If the vSwitch CIDR block is in the allowlist, no update is needed. However, if the allowlist contains the ENI IP directly, update the allowlist after the zone change.

Appendix: Switch the scheduling resource group

You can switch the resource group used by tasks in the following ways.

Environment	Supported switching operations	Interface entry
Switch the production environment resource group	Batch switching	In the left-side navigation pane of Operation Center, click Auto Triggered Task O&M > Auto Triggered Task. Select the tasks for which you want to change the resource group, and then click Change Resource Group for Scheduling in the bottom toolbar to change the resource group in batches. Important Virtual nodes, workflows, and Machine Learning Platform for AI (PAI) tasks do not support resource group changes. Do not select these task types.
Switch the development environment resource group	Single task switching Batch switching	Go to the Datastudio page and configure as follows: Single task switching Go to the editing page of the target node. You can change the resource group in Scheduling Settings > Resource Group. Batch switching Click the icon, select the tasks for which you want to change the resource group, and then click More > Change Resource Group for Scheduling in the bottom toolbar to change the resource group in batches.
Switch the Data Studio debug resource group	Single task switching	Go to the code editing page of a single node in Datastudio, and click the button in the toolbar to switch the scheduling resource group used for code debugging.