Network connectivity issues for Offline Synchronization
Resource group configuration issues
DataWorks network capabilities for offline synchronization
Before you perform data synchronization, understand the following:
The Virtual Private Cloud (VPC), vSwitch, and Region of your database, and the Region of your DataWorks workspace.
Whether your database and DataWorks workspace are in different Alibaba Cloud Accounts or Regions.
If you encounter issues during data synchronization, see Supported data source types, Reader plug-ins, and Writer plug-ins for troubleshooting.
If a data source connectivity test fails, see the solutions in Network connectivity solutions.
When you use an exclusive resource group for Data Integration, follow these steps: purchase a resource group, associate the resource group with the VPC to establish network connectivity, evaluate whether to add a route, configure the database IP address whitelist, and associate the exclusive resource group with your workspace. For more information, see Use exclusive resource groups for Data Integration.
Network connectivity for self-managed ECS databases
To use an exclusive resource group for Data Integration to synchronize data from a self-managed database on an Elastic Compute Service (ECS) instance over an internal network, you must configure the resource group's network settings. For more information, see Use exclusive resource groups for Data Integration. The key steps for establishing connectivity are:
Associate the exclusive resource group for Data Integration with the VPC where the ECS instance resides. A route to the VPC CIDR block is automatically added. We recommend that you do not manually delete this route. Deleting it may cause access failures for other databases and lead to task errors.
Add the vSwitch CIDR block that is associated with the exclusive resource group for Data Integration to the database IP address whitelist. For more information, see Configure an IP address whitelist.
Cross-region network connectivity
Before you begin, select a solution from Network connectivity solutions. The key step for establishing connectivity is as follows:
To synchronize data over the Internet from a database in another Region, add the Elastic IP Address (EIP) of the exclusive resource group to the database IP address whitelist. For more information, see Configure an IP address whitelist.
Data transfer over the Internet incurs fees. For more information, see Internet traffic billing.
Cross-account network connectivity
Before you begin, select a solution from Network connectivity solutions.
To synchronize data over the Internet from a database in a different Alibaba Cloud Account, add the Elastic IP Address (EIP) of the exclusive resource group to the database IP address whitelist. For more information, see Configure an IP address whitelist.
NoteData transfer over the Internet incurs fees. For more information, see Internet traffic billing.
To synchronize data over an internal network from a database in a different Alibaba Cloud Account, perform the following steps:
Establish a connection between the networks of the two Alibaba Cloud Accounts by using a cloud product such as VPN Gateway or Express Connect.
Associate the exclusive resource group for Data Integration with the VPC that is connected to the network of the other Alibaba Cloud Account.
After associating the VPC, add a custom route. Set the next hop to 'Local IDC' and specify the IP address of the target database.
Add the vSwitch CIDR block that is associated with the exclusive resource group to the database IP address whitelist. For more information, see Configure an IP address whitelist.
Troubleshoot failed VPC connectivity tests
If you added the data source by using a VPC endpoint:
Ensure that you have associated the exclusive resource group for Data Integration with the VPC where the database resides.
Ensure that you have added the vSwitch CIDR block that is associated with the exclusive resource group for Data Integration to the database IP address whitelist. For more information, see Configure an IP address whitelist.
If you added the data source by using a public endpoint and the connectivity test fails when using an exclusive resource group for Data Integration, ensure that you have added the Elastic IP Address (EIP) of the resource group to the database IP address whitelist. For more information, see Configure an IP address whitelist.
NoteData transfer over the Internet incurs fees. For more information, see Internet traffic billing.
Troubleshoot intermittent connectivity failures
Check if you are using a shared resource group. Network connectivity for shared resource groups can be unstable. We recommend that you use an exclusive resource group for Data Integration to ensure a stable connection.
Exclusive resource group not found
Ensure that the exclusive resource group is associated with the DataWorks workspace. For more information, see Use exclusive resource groups for Data Integration.
Resource group type in logs
If the task runs on the default resource group, the following log entry appears:
running in Pipeline[basecommon_ group_xxxxxxxxx].If the task runs on a custom resource group for Data Integration, the following log entry appears:
running in Pipeline[basecommon_xxxxxxxxx].If the task runs on an exclusive resource group for Data Integration, the following log entry appears:
running in Pipeline[basecommon_S_res_group_xxx].
Change resource groups for scheduled tasks
In Operation Center, switch the resource group for scheduling and the execution resource group for the Data Integration task:
In DataStudio, deploy the changes to the Production Environment to modify the task's resource group.
NoteIf you use this method, remember to deploy the changes. In standard mode, a commit operation takes effect only in the Development Environment. You must deploy the changes to apply them to scheduled tasks in the Production Environment. After deployment, you can verify the resource group change for the Cycle Task in Operation Center.
1. Modify the resource group for scheduling:
2. Modify the execution resource group for the Data Integration task:
Troubleshoot custom resource groups waiting for a gateway
Log on to the DataWorks console. In the left-side navigation pane, click Resource Groups and go to the Custom Resource Groups tab. Click Server Management for the task's resource group for scheduling. Check if the server is stopped or occupied by other tasks.
If the issue persists, run the following command to restart the service.
su - admin /home/admin/alisatasknode/target/alisatasknode/bin/serverctl restartFind the public IP for whitelisting
To synchronize data over the Internet by using an exclusive resource group for Data Integration, add the Elastic IP Address (EIP) of the resource group to the database IP address whitelist. To obtain the EIP, perform the following steps:
On the Resource Groups page of the DataWorks console, go to the Exclusive Resource Groups tab. Click Details next to the Data Integration resource group. In the Basic Information section, copy the EIPAddress and add it to the database IP address whitelist.
'Insufficient resources' error with available resources
Check the details of the resource group. This issue usually occurs because the remaining resources are insufficient to start a new task. For example, you may see other tasks queued in the resource group.