DataWorks:Create and use an exclusive resource group for Data Integration

Procedure

Precautions

• Only an Alibaba Cloud account or a RAM user to which the AliyunBSSOrderAccess and AliyunDataWorksFullAccess policies are attached can create a resource group.
• Only a workspace administrator can associate a resource group with a workspace and change the workspace with which a resource group is associated.
• For information about the permissions that are required to use the features and perform operations on the Resource Groups page of the DataWorks console, see Policies that can be used to manage permissions on resource groups.
• For information about how to create a custom policy and attach the custom policy to a RAM user, see (Optional) Create a custom policy.
• You can associate an exclusive resource group for Data Integration that uses the specifications of 4 vCPUs and 8 GiB of memory with a maximum of two VPCs. You can associate an exclusive resource group for Data Integration that uses the other specifications with a maximum of three VPCs.
• You can purchase a maximum of 20 Elastic Compute Service (ECS) instances for each exclusive resource group for Data Integration, and the ECS instances must be of the same specifications.

Create an exclusive resource group for Data Integration

1. Log on to the DataWorks console.
2. In the left-side navigation pane, click Resource Groups. On the Exclusive Resource Groups tab of the Resource Groups page, click Create Resource Group for Data Integration. Then, configure the parameters in the Create a dedicated resource group panel and on the DataWorks Exclusive Resources page. The following table describes the parameters that need to be configured.

   Parameter	Description
   Region	The region in which you want to use the exclusive resource group. Note An exclusive resource group cannot be shared across regions. For example, exclusive resource groups in the China (Shanghai) region can be used only by the workspaces in the China (Shanghai) region.
   Resource Group Type	The type of the exclusive resource group. Select Exclusive Resource Group for Data Integration for this parameter.
   Resource Group Name	The name of the exclusive resource group for Data Integration. The name must be unique within a tenant. Otherwise, an error is reported when you click OK. Note A tenant refers to an Alibaba Cloud account. Each tenant can have multiple RAM users.
   Resource Group Description	The description of the exclusive resource group for Data Integration.
   Duration	Exclusive resource groups for Data Integration are charged based on the subscription billing method. To ensure service continuity, we recommend that you select Auto Renewal. You can also go to the Renewal Management page to enable or disable auto renewal after the resource group is created. For more information, see General reference: Disable auto-renewal for subscription resources.

   You can configure other parameters based on your business requirements.

   Note You can purchase a maximum of 20 Elastic Compute Service (ECS) instances for each exclusive resource group for Data Integration, and the ECS instances must be of the same specifications.
3. After you configure the parameters in the Create a dedicated resource group panel, click OK.
   Then, DataWorks starts to initialize the resource group. When the resource group enters the Running state, the resource group is created in the DataWorks console.

   Note DataWorks requires approximately 20 minutes to initialize the exclusive resource group for Data Integration. Wait until the status of the resource group changes to Running.

Associate the exclusive resource group for Data Integration with a workspace

You must associate the exclusive resource group for Data Integration with a workspace before you can select the resource group in the workspace. An exclusive resource group for Data Integration can be shared among multiple workspaces but cannot be used across regions. For example, you can associate an exclusive resource group for Data Integration in the China (Shanghai) region only with workspaces in the China (Shanghai) region. To associate an exclusive resource group for Data Integration with a workspace, perform the following steps:

1. Log on to the DataWorks console.
2. On the Exclusive Resource Groups tab of the Resource Groups page, find the created resource group and click Change Workspace in the Actions column.
3. In the Modify home workspace dialog box, find the workspace with which you want to associate the resource group and click Bind in the Actions column.

Associate the exclusive resource group with a VPC

1. Log on to the DataWorks console.
2. In the left-side navigation pane, click Resource Groups. On the Exclusive Resource Groups tab of the Resource Groups page, find the created resource group and click Network Settings in the Actions column. On the page that appears, you can associate the resource group with a VPC.
   Before you associate the exclusive resource group with a VPC, you must log on to the RAM console with your Alibaba Cloud account and authorize DataWorks to access your cloud resources. You can go to the Cloud Resource Access Authorization page to authorize DataWorks to access your cloud resources. You can also authorize DataWorks to access your cloud resources by clicking the related button in the dialog box that is displayed the first time you log on to the DataWorks console with your Alibaba Cloud account.
3. Associate the exclusive resource group with a VPC.
   1. On the VPC Binding tab, click Add Binding. In the Add VPC Binding panel, configure the parameters.
      Note If you want to use the resource group to access a data source, such as an Alibaba Cloud data source or a self-managed data source hosted on an ECS instance, you can select a network connectivity solution and configure network settings based on whether the resource group and data source belong to the same Alibaba Cloud account.

      Parameter	Description (same region and Alibaba Cloud account)	Description (different regions or Alibaba Cloud accounts)
      VPC	If your data source and the exclusive resource group belong to the same Alibaba Cloud account, we recommend that you select the VPC in which your data source resides. If your data source and the exclusive resource group belong to different Alibaba Cloud accounts, configure this parameter based on the description for the scenario where your data source and the exclusive resource group reside in different regions.	If your data source and the exclusive resource group belong to different Alibaba Cloud accounts or reside in different regions, you must select a VPC that connects to the data source. For example, if your data source does not reside in a VPC, you can click Create VPC to create a VPC for the exclusive resource group. After the VPC is created, you can select it from the VPC drop-down list. You can also select a VPC that connects to your data source. Note If your data source and the exclusive resource group reside in different regions or belong to different Alibaba Cloud accounts, you must use VPN Gateway or Express Connect to establish a connection between the VPC with which the exclusive resource group is associated and the VPC in which the data source resides and add a route that points to the IP address of the data source for the exclusive resource group. For more information, see Establish a network connection between a resource group and a data source.
      Zone	Select the zone in which your data source resides.	Select a zone from which a network connection to your data source is established.
      VSwitch	If you set the VPC parameter to the VPC in which your data source resides, we recommend that you select the vSwitch to which the data source is connected. Note After you associate the exclusive resource group with the VPC in which the data source resides and a vSwitch that resides in the VPC, a route that points to the CIDR block of the VPC is automatically added. This ensures that the exclusive resource group can access the data sources in this VPC.	Select the vSwitch to which the data source is connected. If no vSwitch is available, you can click Create VSwitch to create a vSwitch for the exclusive resource group. After a vSwitch is created, select the vSwitch.
      Security Groups	Security groups allow or deny access to the exclusive resource group over the Internet or an internal network. You can select an existing security group based on your business requirements, or click Create Security Group to create a security group for the resources in the exclusive resource group. For more information about how to create a security group, see Add a security group rule.

   2. Click OK.
   Note If your data source and the exclusive resource group reside in different regions or belong to different Alibaba Cloud accounts, you must add a route that points to the IP address of your data source after you associate the exclusive resource group with a VPC.
4. Optional:Add host configurations.
   You may fail to access your data source by using IP addresses. For example, you can access your data source only by using hostnames. In this case, you must perform the following steps to add host configurations. Otherwise, the connectivity test fails when you add the data source by using its hostnames.
   1. Click the Hostname-to-IP Mapping tab. Then, click Add. In the Create Hostname-to-IP Mapping dialog box, configure the parameters. The following table describes the parameters.

      Parameter	Description
      IP Address	The actual IP address of the data source.
      Hostname	The hostname that is used to access the data source. If you want to specify multiple hostnames, place each hostname on a separate line.

   2. If the data source has multiple IP addresses, click Add to add more host configurations.
      Note

      • The IP address or hostnames that are added in a host configuration must be different from the IP addresses or hostnames in existing host configurations.
      • You can map one IP address to multiple hostnames in a host configuration. However, one hostname can point to only one IP address.

Configure the IP address whitelist of a data source

Test the network connectivity of the exclusive resource group for Data Integration

After you complete the preceding network configuration, you need to test the network connectivity between the resource group and your data source by performing the following operations:

1. On the Workspaces page, find the desired workspace, move the pointer over the icon in the Actions column, and then select Workspace Settings. In the Workspace Settings panel, click More. The Workspace Management page appears.
2. In the left-side navigation pane, click Data Source to go to the Data Source page.
3. Find the desired data source and click Edit in the Operation column.
4. In the dialog box that appears, select Data Integration for Resource Group connectivity. Find the exclusive resource group for Data Integration and click Test connectivity in the Actions column. If the connectivity status is Connected, a network connection is established between the resource group and data source.
   Note If a network connection cannot be established between the resource group and data source, click Troubleshoot in the Connectivity status column to choose a diagnostic tool to diagnose network connection exceptions. For more information about the solutions for network connectivity between an exclusive resource group and data sources that reside in various network environments, see Establish a network connection between a resource group and a data source.
5. Click Complete.

Use the exclusive resource group for Data Integration to run nodes

Environment for the operation	Supported change operation	Entry point
Production environment	Change the resource groups for Data Integration for multiple nodes in the production environment at the same time	Go to the Operation Center page and choose Cycle Task Maintenance > Cycle Task in the left-side navigation pane. Select the nodes for which you want to change the resource groups, click More in the lower part of the Cycle Task page, and then select Modify Data Integration Resource Group.
Development environment	Change the resource group for Data Integration for a single node in the development environment Change the resource groups for Data Integration for multiple nodes in the development environment at the same time	Go to the DataStudio page. Change the resource group for a single node Go to the configuration tab of the node for which you want to change the resource group, click More in the Establish Network Connections step, select the type of the resource group that you want to use, and then select the resource group from the drop-down list. Change the resource groups for multiple nodes at the same time Click the icon. On the Batch Operation-Data Development tab, select the nodes for which you want to change the resource groups, click More in the lower part of the tab, and then select Change Resource Group for Data Integration. Note If you cannot find the entry point of changing the resource groups for nodes, you can select Offline synchronization from the Node Type drop-down list in the filter condition section to search for all batch synchronization nodes.

View the resource usage of the exclusive resource group for Data Integration and monitor the resource group

You can view the resource usage of the exclusive resource group for scheduling and the number of instances that are waiting for resources in the resource group in the DataWorks console. You can also use the intelligent monitoring feature provided in Operation Center to monitor the resource usage of the resource group and the number of instances that are waiting for resources in the resource group. For more information about how to view the resource usage of a resource group, see View the resource usage of an exclusive resource group. For more information about how to monitor a resource group, see Create a custom alert rule.