DataWorks real-time sync tasks support elastic scaling for resources. You can scale resources for integration tasks according to preset cycles to reduce costs. This topic describes how to perform resource elastic scaling.
Function introduction
To improve resource utilization of Data Integration and leverage the characteristics of elastic cloud, DataWorks Data Integration supports dynamic scaling based on Serverless resource groups at the task level. By adjusting resource configurations for different cycles and execution times, you can ensure that integration tasks obtain more compute units (CUs) during peak data periods and run with minimal compute units during low data periods. This prevents resource waste and reduces costs.
Notes
In the same execution plan, the trigger time of a newly added resource configuration must be at least half an hour apart from existing resource configurations to ensure successful completion of existing configurations. Otherwise, the new resource configuration cannot be saved.
For subscription resource groups, if resource elastic scaling exceeds the Data Integration resource quota limit, a rollback operation will be triggered, returning the sync task resources to the resource size of the last successful run.
Scaling operations will cause the job to be temporarily interrupted for no more than 3 minutes.
Access resource optimization
Go to the Data Integration homepage.
Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose . On the page that appears, select the desired workspace from the drop-down list and click Go to Data Integration.
On the Data Integration page, click the
button to enter the Sync Tasks page. You can view the created data integration tasks in the Task List module.Access the Create/modify Execution Plan page.
You can create or modify execution plans for data integration tasks in the following two ways:
Create/modify from resource optimization:
After creating a data integration task, you can create or modify execution plans from the resource optimization side.
On the Task List page, in the Actions column of the integration task for which you want to configure Create/Modify Execution Plan, click to access the Create/modify Execution Plan page.
Create/modify from task basic information:
When a data integration task is running, you can create or modify execution plans from the task basic information side.
On the Task List page, click the Name/id of the integration task for which you want to create or modify an execution plan to access the Task Details page. In , click Resource Optimization to access the Create/modify Execution Plan page.
Create/modify execution plan
On the Create/modify Execution Plan page, you can click New Plan in the plan list to add an execution plan. You can also click the Stop button in the Actions column to stop a running plan, and then click the Edit button in the Actions column of an existing plan to modify it.
Click the Add Resource Configuration button to configure the cycle, time, and CU size for the new task resource scaling. You can also click the Edit button for an existing resource configuration to modify it.
Plan Name: When creating an execution plan, you need to define a custom plan name. The plan name cannot be edited or modified after it is saved.
Add Resource Configuration: When creating or modifying an execution plan, you can add or modify the following configuration items.
Configuration Item
Configuration Description
Cycle Frequency
Configure the cycle frequency based on the resource amount required by the current data integration task during runtime. You can configure One-time Execution, Daily, Weekly, and Monthly execution
Execution Date
When the cycle frequency is configured as one-time execution, weekly, or monthly, you can set the specific execution date.
Execution Time
Configure the specific runtime within the execution plan's running time. You can configure different time zones and times
Configuration Size
Configure based on the number of CUs required for the data integration task to run.
Actions
You can Save, Cancel, or Delete configured items.
You can Edit or Delete saved task resource execution plans.
You can add multiple resource configurations within a task execution plan by clicking Add Resource Configuration.
After completing the task resource optimization configuration, click OK to save the task execution plan.
Apply execution plan
After creating an execution plan, you can click the Apply button in the Actions column of the plan list to apply the execution plan to the current integration task.
A data integration task only supports applying one execution plan. If there are multiple resource configuration time periods in multiple execution plans, you can configure multiple resource configurations within a single plan task.
View optimization execution history
After accessing the Create/modify Execution Plan page, you can view the created and executed plans on the Plan List page. You can also view the Execution Results, Execution Time, and Execution Information of the resource execution plans applied to the data integration task in the Execution History module.
More operations
After creating or modifying an execution plan, you can perform more management operations on existing execution plans in the plan list on the Create/modify Execution Plan page.
Delete Execution Plan: Applicable to existing but unapplied execution plans. If you need to delete an applied execution plan, you must first stop the applied execution plan before deleting it.
View Execution Plan: Applicable to applied execution plans. You can view the resource configuration information of the selected execution plan.
Stop Execution Plan: Applicable to applied execution plans. You can stop the selected execution plan. Stopped execution plans can be deleted and edited.