If you plan to switch your legacy DataWorks resource groups to Serverless resource groups, you must assess the resource consumption of your existing tasks to ensure a smooth migration. Then, switch to a Serverless resource group with sufficient capacity to handle all your tasks. This topic provides examples of how to estimate the required compute units (CUs) for different tasks, describes the potential impacts of the switch, and provides instructions on how to switch from legacy resource groups to Serverless resource groups.
New resource groups
DataWorks supports exclusive resource groups for Data Integration, scheduling, and DataService Studio. However, you must purchase and configure these resource groups separately. To improve resource management across DataWorks features and provide a unified user experience, DataWorks introduced Serverless resource groups. A single Serverless resource group can be used for Data Integration, task scheduling, and DataService Studio. This simplifies resource group management and improves operational consistency.
Billing
Before you switch, review the billing information for legacy resource groups: Billing for legacy resource groups.
After you switch, review the billing information for Serverless resource groups: Billing for Serverless resource groups.
Procedure
Step 1: Query tasks in the resource groups to be switched
Data Integration
Switch data integration tasks on the Data Integration page:
Go to the Data Integration page.
Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose . On the page that appears, select the desired workspace from the drop-down list and click Go to Data Integration.
In the navigation pane on the left, click Sync Tasks. In the Task List section, click Expand. Filter the tasks by setting Resource Group to the data integration resource group that you want to switch.
Switch data integration tasks on the new Data Studio page:
Go to the Workspaces page in the DataWorks console. In the top navigation bar, select a desired region. Find the desired workspace and choose in the Actions column.
The Data Studio page appears by default. Click the
icon for Project Directory to perform batch operations on nodes.Filter the nodes by the resource group that you want to switch, and select the relevant node types (Real-time Sync and Offline Sync).
Switch data integration tasks on the legacy Data Studio page:
Go to the DataStudio page.
Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose . On the page that appears, select the desired workspace from the drop-down list and click Go to Data Development.
In the navigation pane on the left, click Data Development. Find the target business flow, right-click it, and choose Batch Operations.
Set Node Type to Offline Sync and Real-time Sync. Set Data Integration Resource Group to the data integration resource group that you want to switch.
Scheduling tasks
Go to the Operation Center page.
Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose . On the page that appears, select the desired workspace from the drop-down list and click Go to Operation Center.
In the navigation pane on the left, choose . Filter the tasks by setting Scheduling Resource Group to the scheduling resource group that you want to switch.
DataService Studio
Go to the DataService Studio page.
Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose . On the page that appears, select the desired workspace from the drop-down list and click Go to DataService Studio.
In the navigation pane on the left, click Service Development. Then, click the
icon to go to the Batch Operations page. Filter the services by setting Resource Group to the DataService Studio resource group that you want to switch.
Step 2: Assess the specifications for Serverless resource groups before the switch
After you switch to a Serverless resource group, data computing tasks (such as PyODPS2 and EMR Hive) will incur computing fees.
Before you switch, assess the resource consumption of your existing tasks, including sync tasks, scheduling tasks, and DataService Studio tasks. This helps you determine the size of the Serverless resource group to purchase and ensures that the resource group can handle your business workloads.
The following are recommendations for the assessment:
Data synchronization
Batch synchronization tasks
Parallelism configured for a batch synchronization task | Recommended specifications | Required minimum specifications |
<4 | 0.5 CUs | 0.5 CUs |
>=4 |
|
Real-time synchronization tasks
Synchronization task type | Recommended specifications | Required minimum specifications | |
Real-time synchronization from MySQL | One source database | 2.5 CUs | Minimum specifications that are required to run such a real-time synchronization task: 1 CU |
Two to five source databases | 4 CUs | ||
Six or more source databases | 7 CUs | ||
Real-time synchronization from PolarDB for Xscale (PolarDB-X) | 7 CUs | ||
Real-time synchronization from Kafka | 2.5 CUs | ||
Real-time synchronization of data in a single table of another source type | 2.5 CUs | ||
Real-time synchronization of all data in a database | - | Minimum specifications that are required to run such a synchronization task: 2 CUs | |
Task scheduling
A serverless resource group supports a maximum of 200 concurrently running instances. The CU specification of the serverless resource group can be ignored. The default number of concurrently running instances is 50. You can set the maximum number of concurrent scheduling tasks to 200 on the resource group details page.
DataService Studio
Maximum QPS | Required minimum specifications | Service availability (SLA) |
500 | 4 CUs | 99.95% |
1000 | 8 CUs | |
2000 | 16 CUs |
Data computing
DataWorks supports different types of nodes, such as PyODPS and EMR Hive nodes. Some of the nodes are issued to the related compute engines for running, and the data computing fees are charged by the Alibaba Cloud services to which the compute engines belong. The other nodes are directly run on DataWorks resource groups or issued to the related compute engines for running after they are started by using DataWorks resource groups. The running consumes the computing resources of the resource groups and are charged by DataWorks for data computing.
When you use a serverless resource group to run a data computing task, the CUs in the resource group are consumed. For information about the list of computing tasks, and the default and actual CU configurations of computing tasks, see Configuration of CUs for data computing tasks.
Step 3: Purchase a Serverless resource group
Purchase a Serverless resource group based on your assessment. For more information, see Use a Serverless resource group.
Step 4: Switch to the Serverless resource group
Switch the resource group for Data Integration
NoteAfter you switch to the Serverless resource group, DataWorks automatically recommends the number of CUs based on the original task configuration. To manually set the number of CUs for the resource group, refer to the recommendations in Step 2: Assess the specifications for Serverless resource groups before the switch.
Switch the resource group for scheduling tasks
NoteScheduling tasks run in the DataWorks resource group. A portion of the resource group quota is allocated to the CU configuration of compute-optimized tasks. When you change the scheduling resource group, the resource group used for task computing is also changed.
Switch the resource group for DataService Studio
NoteBefore you switch, you must set a quota for DataService Studio. If no quota is set, you cannot select the Serverless resource group during the switch. For more information about how to set a quota for DataService Studio, see Allocate a CU quota to a task.
More operations
After you switch to the Serverless resource group, you can unsubscribe from the legacy resource group if you no longer need it. For more information, see General reference: Unsubscribe from a subscription product.