Before you migrate from a legacy DataWorks resource group to a serverless resource group, assess the resource consumption of your existing tasks and select a serverless resource group with adequate capacity. This topic covers CU estimation examples, potential impacts, and step-by-step migration instructions.
New resource groups
DataWorks supports dedicated resource groups for Data Integration, scheduling, and DataService Studio, but each must be purchased and configured separately. Serverless resource groups unify all three features into a single resource group, simplifying management and improving operational consistency.
Serverless vs. legacy resource groups
Legacy resource groups have exclusive and public variants, each tied to a single feature.
|
Feature |
Serverless resource group |
Legacy resource groups |
|
Scope |
Works with all DataWorks features |
Requires separate purchases per feature |
|
Billing |
Pay-as-you-go and subscription |
Exclusive: subscription only. Public: pay-as-you-go only |
|
Scaling |
Scale CUs without affecting running tasks |
Scaling (upgrading/downgrading specs or adding/removing nodes) disrupts running tasks |
|
Minimum purchase |
2 CUs; scale in 1-CU increments |
One node with 4 vCPUs and 8 GB memory; scale in full-node increments |
|
Network security |
Bound to your VPC with full network control |
DataWorks-managed public endpoints shared across tenants |
|
Development status |
Actively developed as the standard resource type |
Being phased out; no new features planned |
Billing
-
Before the switch, see Billing for legacy resource groups.
-
After the switch, see Billing for serverless resource groups.
Procedure
Step 1: Find tasks in the resource group
Data Integration
On the Data Integration page:
Log on to the DataWorks console. In the target region, click in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Integration.
-
In the navigation pane on the left, click Synchronization Task. In the Task List area, click Expand and filter by Resource Group to find the Data Integration resource group that you want to switch.
On the Data Studio page:
Go to the Workspaces page in the DataWorks console. In the top navigation bar, select a desired region. Find the desired workspace and choose in the Actions column.
-
Next to Project Directory, click the
icon to open the batch operations page for nodes. -
Use the Filter to find the resource group that you want to switch, and select the related node types: Real-time Synchronization and Offline Synchronization.
On the legacy Data Development page:
Log on to the DataWorks console. In the target region, click in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Development.
-
In the navigation pane on the left, click DataStudio. Find the target business flow, right-click it, and select Batch Operation.
-
Set Node Type to Batch Sync and Real-time Synchronization. For Resource Group for Data Integration, select the Data Integration resource group that you want to switch.
Schedulingtasks
Log on to the DataWorks console. In the target region, click in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Operation Center.
-
In the navigation pane on the left, choose and use the Resource Group for Scheduling filter to find the resource group to switch.
DataService Studio
Log on to the DataWorks console. In the target region, click in the left-side navigation pane. Select a workspace from the drop-down list and click Go to DataService Studio.
-
In the navigation pane on the left, click Service Development, and then click the
icon to go to the Batch Operation page. Filter by Resource Group to find the DataService Studio resource group that you want to switch.
Step 2: Estimate the required size
After you switch to a serverless resource group, you will be charged computing fees for data compute tasks such as PyODPS2 and EMR Hive.
Before you switch, assess the resource consumption of your existing synchronization, scheduling, and DataService Studio tasks to determine the appropriate size of the serverless resource group.
Data Integration
Batch synchronization
|
Batch synchronization task concurrency |
Recommended specifications |
Minimum specifications |
|
<4 |
0.5 CU |
0.5 CU |
|
>=4 |
|
Real-time synchronization
|
Synchronization task type |
Recommended specifications |
Minimum specifications |
|
|
MySQL real-time synchronization |
1 database |
2 CU |
Minimum specifications for running a real-time synchronization task: 1 CU |
|
2 to 5 databases |
2 CU |
||
|
6 or more databases |
2 CU |
||
|
Kafka real-time synchronization |
1 CU |
||
|
Other single-table real-time tasks |
1 CU |
||
|
Full-database real-time synchronization |
- |
Minimum specifications for running a full-database synchronization task: 2 CU |
|
Task scheduling
A serverless resource group supports a maximum of 200 concurrent instances. You do not need to plan for CU specifications. The default concurrency is 50, and you can increase the maximum task scheduling concurrency to 200 on the resource group details page.
Data Service
|
Maximum QPS |
Minimum specifications |
Service availability (SLA) |
|
500 |
4 CU |
99.95% |
|
1000 |
8 CU |
|
|
2000 |
16 CU |
Data compute
Each data compute task has a default CU allocation. For details, see Task types and CU consumption.
Step 3: Purchase a serverless resource group
Based on the size you estimated in the previous step, purchase a serverless resource group. For instructions, see Use a serverless resource group.
Step 4: Switch to the serverless group
-
Switch the resource group for Data Integration
NoteAfter you switch to a serverless resource group, DataWorks automatically sets the recommended CU count based on your original task configuration. If you need to manually set the CU count for the resource group, see the recommendations in Step 2: Estimate the required size.
-
Switch the resource group for scheduling tasks
NoteA portion of the resource group quota is allocated for CU configuration for compute-optimized tasks depending on whether task computing occurs within a DataWorks resource group. When you change the scheduling resource group, the resource group used for task computing is also updated.
-
Switch the resource group for DataService Studio
NoteBefore you switch, you must set a quota for DataService Studio. If you do not set a quota, you cannot select the serverless resource group during the switch. For instructions on how to set a quota, see Allocate a CU quota to a task.
Next steps
After you switch to the serverless resource group, you can unsubscribe from the legacy resource group if you no longer need it. For instructions, see General reference: Unsubscribe from a subscription product.