FAQs and solutions for resource group operations and network connectivity - DataWorks

Data synchronization network connectivity issues
Resource group configuration issues

What should I know about DataWorks and its network capabilities before running a batch synchronization task?

Before you synchronize data using Data Integration, make sure that you are familiar with the following information:

The VPC, vSwitch, and region of the database from which you want to synchronize data, and the region of the DataWorks workspace.
Whether special conditions exist, such as the database and the DataWorks workspace being under different Alibaba Cloud accounts or in different regions.

If you encounter issues during task synchronization, you can first refer to Troubleshoot data synchronization issues for self-service troubleshooting.

If you encounter issues when testing data source connectivity, you can first refer to Troubleshoot data source connectivity issues to select the appropriate solution for troubleshooting.

When you use an exclusive resource group for Data Integration, follow these steps: purchase the resource group, associate it with the VPC where the database is connected, evaluate whether to add routes, configure database allowlists, and associate the exclusive resource group with the workspace. For more information, see Configure an exclusive resource group for Data Integration.

How do I ensure network connectivity between DataWorks and a self-managed database on an ECS instance?

If you want to use an exclusive resource group for Data Integration to synchronize data from a self-managed database on an ECS instance over the internal network, you must configure network settings for the exclusive resource group for Data Integration. For more information, see Configure an exclusive resource group for Data Integration. The key points of the connectivity solution are as follows:

Associate the exclusive resource group for Data Integration with the VPC where the ECS instance resides. A route pointing to the VPC CIDR block is automatically added. We recommend that you do not manually delete this route, as doing so may cause access failures to other databases and task errors.
Add the vSwitch CIDR block associated with the exclusive resource group for Data Integration to the database allowlist. For more information, see Add allowlist entries.

How do I ensure network connectivity when my database and DataWorks workspace are in different regions?

Before you begin, refer to Troubleshoot data source connectivity issues to select the appropriate network connectivity solution. The key points of the connectivity solution are as follows:

If you use the Internet to synchronize data from a database in a different region, you must add the EIP of the exclusive resource group to the database allowlist. For more information, see Add allowlist entries.

Note

Internet-based data synchronization incurs Internet traffic fees. For more information, see Pricing.

How do I ensure network connectivity when my database and DataWorks workspace are under different Alibaba Cloud accounts?

Before you begin, refer to Troubleshoot data source connectivity issues to select the appropriate solution for troubleshooting.

If you use the Internet to synchronize data from a database under a different account, you must add the EIP of the exclusive resource group to the database allowlist. For more information, see Add allowlist entries.

Note
Internet-based data synchronization incurs Internet traffic fees. For more information, see Pricing.
If you want to synchronize data from a database under a different account over the internal network, perform the following steps:
1. Connect the networks of the two Alibaba Cloud accounts by using a networking service such as VPN Gateway or Express Connect.
2. Associate the exclusive resource group for Data Integration with the VPC that is already connected to the network of the other Alibaba Cloud account.
3. After you associate the VPC, add a custom route, select "On-premises IDC", and add the IP address of the destination database.
4. Add the vSwitch CIDR block associated with the exclusive resource group to the database allowlist. Note that the vSwitch CIDR block is the one associated with the resource group. For more information, see Add allowlist entries.

What should I do if a connectivity test fails for a database in a VPC?

If you added the data source by using a VPC endpoint:
1. Make sure that you have associated the exclusive resource group for Data Integration with the VPC where the database resides.
2. Make sure that the vSwitch CIDR block associated with the exclusive resource group for Data Integration has been added to the database allowlist. For more information, see Add allowlist entries.
If you added the data source by using a public endpoint and the connectivity test fails with the exclusive resource group for Data Integration, check whether the EIP of the exclusive resource group for Data Integration has been added to the database allowlist. For more information, see Add allowlist entries.

Note
Internet-based data synchronization incurs Internet traffic fees. For more information, see Pricing.

What should I do if data source connectivity is intermittent?

Check whether you are using a shared resource group. The network of shared resource groups is unstable. We recommend that you use an exclusive resource group for Data Integration to ensure connection stability.

How do I ensure network connectivity between a serverless resource group and Hologres when syncing data with Data Integration?

The serverless resource group must be associated with the VPC where the Hologres instance resides. Perform the following steps:

Log on to the DataWorks console and find the target serverless resource group in the resource group list.
Click Network Settings. In the Data Scheduling and Data Integration section, click Add VPC Association.
Select the VPC, zone, and vSwitch where the Hologres instance resides, and then click OK.
After the association is complete, create a Hologres data source in the data source management section of Data Integration, and click Test Connectivity to verify that the connection is successful.

If the CIDR block of the VPC already associated with the resource group conflicts with the Hologres VPC CIDR block, disassociate the conflicting VPC first and then add the Hologres VPC. Associating a VPC creates elastic network interfaces (ENIs) in the VPC that consume quota. Do not delete these ENIs.

For detailed operations on switching Hologres networks, see Switch Hologres network configurations.

What should I do if I purchased an exclusive resource group but cannot find it when testing data source connectivity or running a task?

Make sure that the exclusive resource group is associated with the DataWorks workspace. For more information, see Associate a resource group with a workspace.

How do I check logs to determine the type of resource group a task is running on?

If a task is running on the default resource group, the following information appears in the logs: running in Pipeline[basecommon_ group_xxxxxxxxx].
If a task is running on a custom resource group for Data Integration, the following information appears in the logs: running in Pipeline[basecommon_xxxxxxxxx].
If a task is running on an exclusive resource group for Data Integration, the following information appears in the logs: running in Pipeline[basecommon_S_res_group_xxx].

How do I switch the resource group used by a task in the scheduling system?

Switch the scheduling resource group and Data Integration resource group for task execution in Operation Center: On the Operation Center > Scheduled Task O&M > Scheduled Tasks page, select the tasks for which you want to switch the resource group, and click Change Scheduling Resource Group or Change Data Integration Resource Group at the bottom of the page to make batch changes.
Go through the deployment process in Data Studio to change the resource group used by tasks in the production environment.

Note
When you change the resource group by using the following method, remember to deploy the changes. In standard mode, submitting changes only takes effect in the development environment. Changes take effect for automatic scheduling of scheduled tasks in the production environment only after deployment. After deployment, you can check whether the resource group change has taken effect in the scheduled tasks section of Operation Center.

1. Change the resource group for task scheduling: Open the Schedule Settings panel of the task node. In the Resource Properties section, click the Scheduling Resource Group drop-down list and select the target resource group (for example, Shared Scheduling Resource Group).

2. Change the Data Integration resource group for task execution: Click the Data Integration Resource Group Configuration tab on the right sidebar. In the Exclusive Resource Group for Data Integration drop-down list, select the target resource group (for example, xiangcui_vpc).

How do I troubleshoot a custom resource group that is waiting for a gateway?

Log on to the DataWorks console. In the left-side navigation pane, click Resource Group List and go to the Custom Resource Groups tab. Click Server Management next to the scheduling resource group used by the task, and check whether the server is stopped or occupied by other tasks.

If the preceding troubleshooting does not resolve the issue, run the following command to restart the service.

su - admin /home/admin/alisatasknode/target/alisatasknode/bin/serverctl restart

How do I find the public IP address of a resource group for an IP address allowlist?

If you use an exclusive resource group for Data Integration to synchronize data over the Internet, add the EIP of the exclusive resource group for Data Integration to the database allowlist. You can obtain the EIP as follows:

On the Resource Groups page of the DataWorks console, click the Exclusive Resource Groups tab. Click Details next to the Data Integration resource group, and copy the EIP from the basic information section to the database allowlist.

Why do I get an "insufficient resources" error when the resource group has available resources?

Check the resource group details. In most cases, the remaining resources of the resource group are insufficient for new tasks. For example, the resource group may have queued tasks at this time.

How many VPCs can be associated with a serverless resource group, and what should I do if I reach the quota?

Each serverless resource group supports a maximum of two VPC associations.

Log on to the DataWorks console. On the Resource Groups page, find the target serverless resource group and click Details. In the basic information section of the resource group details page, the VPCs That Can Be Associated field displays the current quota usage in the format X associated, Y remaining.

When the quota of two VPCs is fully used (2 associated, 0 remaining), you cannot add more VPC associations. You can resolve this issue by using the following methods:

Evaluate whether you can reuse an already-associated VPC. Deploy the data source that requires connectivity in an already-associated VPC, or connect the VPC where the data source resides to the already-associated VPC by using Cloud Enterprise Network (CEN).
Use Cloud Enterprise Network (CEN) for cross-VPC connectivity. Add the already-associated VPC and the target VPC to the same CEN instance to establish network connectivity. This way, you can access data sources in the target VPC without consuming additional VPC association quota.

VPC associations in the network settings of a resource group are divided into two sections: Data Service and Data Scheduling. The quota of two VPCs is a resource-group-level total quota shared by the two sections.