To ensure a smooth migration from a legacy DataWorks resource group to a serverless resource group, you must first assess the resource consumption of your existing tasks. This allows you to select a serverless resource group with adequate capacity to handle all your tasks. This topic provides examples for estimating the required CUs, explains the potential impacts of the switch, and guides you through the migration process.
New resource groups
DataWorks supports dedicated resource groups for Data Integration, scheduling, and DataService Studio. However, these resource groups must be purchased and configured separately. To streamline resource management and unify the user experience, DataWorks introduces serverless resource groups. A single serverless resource group can be used across Data Integration, scheduling, and DataService Studio. This simplifies resource group interactions and improves operational consistency.
Serverless vs. legacy resource groups
Legacy resource groups have exclusive and public variants, each tied to a single feature.
|
Feature |
Serverless resource group |
Legacy resource groups |
|
Scope |
Works with all DataWorks features |
Requires separate purchases per feature |
|
Billing |
Pay-as-you-go and subscription |
Exclusive: subscription only. Public: pay-as-you-go only |
|
Scaling |
Scale CUs without affecting running tasks |
Scaling (upgrading/downgrading specs or adding/removing nodes) disrupts running tasks |
|
Minimum purchase |
2 CUs; scale in 1-CU increments |
One node with 4 vCPUs and 8 GB memory; scale in full-node increments |
|
Network security |
Bound to your VPC with full network control |
DataWorks-managed public endpoints shared across tenants |
|
Development status |
Actively developed as the standard resource type |
Being phased out; no new features planned |
Billing
-
Before the switch, see Billing for legacy resource groups.
-
After the switch, see Billing for serverless resource groups.
Procedure
Step 1: Find tasks in the resource group
Data Integration
On the Data Integration page:
Log on to the DataWorks console. In the target region, click in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Integration.
-
In the navigation pane on the left, click Synchronization Task. In the Task List area, click Expand and filter by Resource Group to find the Data Integration resource group that you want to switch.
On the Data Studio page:
Go to the Workspaces page in the DataWorks console. In the top navigation bar, select a desired region. Find the desired workspace and choose in the Actions column.
-
Next to Project Directory, click the
icon to open the batch operations page for nodes. -
Use the Filter to find the resource group that you want to switch, and select the related node types: Real-time Synchronization and Offline Synchronization.
On the legacy Data Development page:
Log on to the DataWorks console. In the target region, click in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Development.
-
In the navigation pane on the left, click DataStudio. Find the target business flow, right-click it, and select Batch Operation.
-
Set Node Type to Batch Sync and Real-time Synchronization. For Resource Group for Data Integration, select the Data Integration resource group that you want to switch.
Schedulingtasks
Log on to the DataWorks console. In the target region, click in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Operation Center.
-
In the navigation pane on the left, choose and use the Resource Group for Scheduling filter to find the resource group to switch.
DataService Studio
Log on to the DataWorks console. In the target region, click in the left-side navigation pane. Select a workspace from the drop-down list and click Go to DataService Studio.
-
In the navigation pane on the left, click Service Development, and then click the
icon to go to the Batch Operation page. Filter by Resource Group to find the DataService Studio resource group that you want to switch.
Step 2: Estimate the required size
After you switch to a serverless resource group, you will be charged computing fees for data compute tasks such as PyODPS2 and EMR Hive.
Before you switch, you need to assess the resource consumption of your existing synchronization, scheduling, and DataService Studio tasks. This helps you determine the size of the serverless resource group to purchase and ensures it can handle your workloads.
Data integration
Batch synchronization
|
Task concurrency |
Recommended specifications |
Minimum specifications |
|
<4 |
0.5 CU |
0.5 CU |
|
>=4 |
|
Real-time synchronization
|
Task type |
Recommended specifications |
Minimum specifications |
|
|
MySQL real-time synchronization |
1 database |
2 CU |
Minimum for a real-time task: 1 CU |
|
2 to 5 databases |
2 CU |
||
|
6 or more databases |
2 CU |
||
|
Kafka real-time synchronization |
1 CU |
||
|
Other types of table-level real-time tasks |
1 CU |
||
|
Full-database real-time synchronization |
- |
Minimum for a full-database task: 2 CU |
|
Task scheduling
A serverless resource group supports a maximum of 200 concurrent instances, independent of its CU specifications. The default concurrency is 50. You can set the maximum task scheduling concurrency to 200 on the resource group details page.
Data service
|
Maximum QPS |
Minimum specifications |
Service availability (SLA) |
|
500 |
4 CU |
99.95% |
|
1000 |
8 CU |
|
|
2000 |
16 CU |
Data compute
Each data compute task has a default CU value. For more information, see Task types and CU consumption.
Step 3: Purchase a serverless resource group
Based on the size you estimated in the previous step, purchase a serverless resource group. For instructions, see Use a serverless resource group.
Step 4: Switch to the serverless group
-
Switch the resource group for Data Integration
NoteAfter you switch to a serverless resource group, DataWorks automatically sets the recommended CU count based on your original task configuration. If you need to manually set the CU count for the resource group, see the recommendations in Step 2: Estimate the required size.
-
Switch the resource group for scheduling tasks
NoteA portion of the resource group quota is allocated for CU configuration for compute-optimized tasks depending on whether task computing occurs within a DataWorks resource group. When you change the scheduling resource group, the resource group used for task computing is also updated.
-
Switch the resource group for DataService Studio
NoteBefore you switch, you must set a quota for DataService Studio. If you do not set a quota, you cannot select the serverless resource group during the switch. For instructions on how to set a quota, see Allocate a CU quota to a task.
Next steps
After you switch to the serverless resource group, you can unsubscribe from the legacy resource group if you no longer need it. For instructions, see General reference: Unsubscribe from a subscription product.