If a large number of data synchronization nodes in Data Integration must be run in parallel, exclusive computing resources are required to ensure fast and stable data transmission. In this case, we recommend that you use exclusive resource groups for Data Integration. This topic provides an overview of exclusive resource groups for Data Integration.

Benefits

Exclusive resource groups for Data Integration have the following benefits:
  • Support data synchronization in complex network environments. For example, you can use an exclusive resource group for Data Integration to synchronize data across cloud environments (Alibaba Finance Cloud and Alibaba Gov Cloud), across Alibaba Cloud accounts, or from or to data centers.
  • Support data synchronization for various scenarios.
    • You can use an exclusive resource group for Data Integration for batch synchronization. For example, you can use an exclusive resource group for Data Integration to synchronize full and incremental data or synchronize data from a single table in a database or from tables in sharded databases. For more information, see Overview of the batch synchronization feature.
    • You can use an exclusive resource group for Data Integration for real-time synchronization. For example, you can use an exclusive resource group for Data Integration to synchronize data from a single table in real time or synchronize data from all tables in a database in real time. For more information, see Overview of the real-time synchronization feature.
    • You can use exclusive resource groups for Data Integration to run various data synchronization solutions. For example, you can use exclusive resource groups for Data Integration to run the following types of synchronization solutions: batch synchronization of data from all tables in a database and one-time full synchronization and real-time incremental synchronization from a table or database. For more information, see Overview of the solution-based synchronization feature.
Note

An exclusive resource group for Data Integration ensures the number of threads that a data synchronization instance can simultaneously run but not the number of data synchronization instances that can be simultaneously run. To ensure the number of data synchronization instances that can simultaneously run, you can purchase an exclusive resource group for scheduling.

Limits

  • An exclusive resource group for Data Integration is charged based on the subscription billing method. You cannot delete or release an exclusive resource group for Data Integration before the resource group expires. An exclusive resource group for Data Integration is suspended and released at the specified points in time after it expires.
  • An exclusive resource group for Data Integration cannot be used across regions. For example, an exclusive resource group for Data Integration in the China (Shanghai) region can be used only by workspaces in the China (Shanghai) region.
  • An exclusive resource group for Data Integration cannot access data sources that are deployed in the classic network of Alibaba Cloud. If your data source is deployed in the classic network, we recommend that you migrate the data source to the VPC in which the exclusive resource group for Data Integration is deployed.
  • You can purchase a maximum of 20 Elastic Compute Service (ECS) instances for each exclusive resource group, and the ECS instances must be of the same specifications.

Performance metrics

SpecificationsMaximum number of parallel threads supported for a batch synchronization solution or node
4c8g8
8c16g16
12c24g24
16c32g32
24c48g48
ScenarioRecommended specifications for a single real-time synchronization solution or node
Real-time synchronization of data in one MySQL database4c8g
Real-time synchronization of data in two to five MySQL databases8c16g
Real-time synchronization of data in more than six MySQL databases12c24g
Real-time synchronization of data in PolarDB-X 1.012c24g
Real-time synchronization of data in Kafka4c8g
Real-time synchronization of data in a single table of another type of source4c8g

Billing and related operations

If you want to obtain professional pre-sales purchase guide, you can refer to Purchase guide. You can also scan the following QR code to join the DataWorks DingTalk group to contact pre-sales personnel. DingTalk group

(1) Billing

An exclusive resource group for Data Integration is charged based on the subscription billing method. You can purchase an exclusive resource group for Data Integration of appropriate specifications based on your business requirements. For more information, see Billing of exclusive resource groups for Data Integration (subscription).

(2) Scaling

You can purchase an exclusive resource group for scheduling based on your business requirements. When you purchase an exclusive resource group for scheduling, you can specify the specifications and number of ECS instances that you need to use. After the purchase is complete, you can scale out or scale in the resource group. For more information about how to scale out and scale in a resource group, see Scale out or in a resource group.

(3) Specification change (specification upgrade or downgrade)

If the specifications of your exclusive resource group for scheduling no longer meet your business requirements, you can change the specifications of the resource group. After you change the specifications of the resource group, the specifications of all ECS instances in the resource group are changed. For more information about how to change the specifications of a resource group and the related precautions, see Change the specifications of a resource group.

(4) Renewal, suspension, and release of an exclusive resource group for scheduling

You can renew an exclusive resource group for scheduling when the resource group is about to expire. If you do not renew the resource group before it expires, the resource group is suspended upon expiration and automatically released. For more information, see Expiration and renewal.

Use an exclusive resource group for Data Integration

After you have a command of the billing of an exclusive resource group for Data Integration, you can purchase an exclusive resource group for Data Integration based on your business requirements and use the resource group to run data synchronization nodes in Data Integration. To purchase and use an exclusive resource group for Data Integration, perform the following steps:
  1. Create an exclusive resource group for Data Integration.
  2. Associate the exclusive resource group for Data Integration with a workspace.
  3. Associate the exclusive resource group for Data Integration with a VPC.
  4. Add the elastic IP address (EIP) of the exclusive resource group for Data Integration or the CIDR block of the vSwitch with which the resource group is associated to the IP address whitelist of the data source that the resource group needs to access.
  5. Use the created exclusive resource group for Data Integration.
For more information, see Create and use an exclusive resource group for Data Integration.

Network connectivity solutions

Similar to other types of exclusive resource groups, an exclusive resource group for Data Integration is a group of Alibaba Cloud ECS instances. Before you run a data synchronization node in Data Integration on an exclusive resource group for Data Integration, you must make sure that network connections are established between the resource group and the data sources used for the node. You must also make sure that special security settings such as IP address whitelists do not affect the network connections between the resource group and data sources.

After you purchase an exclusive resource group for Data Integration, you must associate the resource group with a VPC. Then, you can select a network connectivity solution based on the network environment in which the data source you want to access is deployed. For more information, see Establish a network connection between a resource group and a data source.

  • Network connectivity solutions
    Network environmentNetwork connectivity solution
    The data source is deployed on the Internet.The exclusive resource group for Data Integration that is deployed in a VPC can directly access the data source.
    The data source is deployed in a VPC and resides in the same region as the exclusive resource group for Data Integration.We recommend that you associate the exclusive resource group for Data Integration with the VPC in which the data source is deployed and with a vSwitch that belongs to the VPC. Then, the system adds a route for the exclusive resource group for Data Integration. This way, the exclusive resource group for Data Integration can access the data source.
    The data source is deployed in a VPC and resides in a different region from the exclusive resource group for Data Integration.Use Express Connect circuits or VPN gateways to connect the data source to the VPC with which the exclusive resource group for Data Integration is associated, and add a route that points to the IP address of the database you want to access to ensure the network connection between the data source and resource group.
    The data source is deployed in a data center.Use Express Connect circuits or VPN gateways to connect the data source to the VPC with which the exclusive resource group for Data Integration is associated, and add a route that points to the IP address of the database you want to access to ensure the network connection between the data source and resource group.
    The data source is deployed in the classic network.Exclusive resource groups are deployed in VPCs of Alibaba Cloud. If your data source is deployed in the classic network of Alibaba Cloud, your exclusive resource group cannot access the data source. In this case, we recommend that you migrate the data source to the VPC in which your exclusive resource group is deployed.
  • Whitelist settings

    If an IP address whitelist is configured for the data source that you want to access, you must add the EIP of your exclusive resource group for Data Integration or the CIDR block of the vSwitch with which the resource group is associated to the whitelist. For more information, see Add the EIP or CIDR block of an exclusive resource group for Data Integration to the whitelist of a data source.

FAQ

For information about FAQ about exclusive resource groups, see Exclusive resource groups.