If you want to reduce the cost of subscription computing resources consumed by jobs, or if a job cannot be completed as expected based on existing resource configurations, you can use the computing cost optimization feature. You can use this feature to generate a resource configuration optimization plan for the computing resources in a level-1 subscription quota based on the actual number of job requests and expected resource configurations. You can also use this feature to generate a resource configuration optimization plan that changes the pay-as-you-go computing resources of a project to subscription computing resources and view the estimated optimization effect. This helps you further reduce computing costs and improve resource utilization. This topic describes the implementation principle of the computing cost optimization feature and how to use this feature.
How it works
The following examples describe the logic for generating a resource configuration optimization plan in MaxCompute. In this topic, you can also learn about the related terms that help you configure the parameters for the computing resource optimization feature based on your business requirements.
Generate an optimization plan for reconfiguring subscription computing resources
Scenario
A company purchased specific subscription computing resources. In the current resource configuration plan, 1,000 reserved compute units (CUs) are configured for jobs. Two batches of jobs consume the reserved CUs every day.
First batch of jobs: All jobs are submitted and started before 01:00 every day. The company expects to complete the jobs before 02:00. The jobs cannot be completed as expected based on the current resource configuration.
Second batch of jobs: All jobs are submitted and started before 05:00 every day. The company expects to complete the jobs before 06:00.
Monthly resource cost (catalog price): The monthly cost of computing resources is USD 22,000, which is calculated by using the following formula: 1,000 CUs × USD 22.0 per CU.
Plan generation logic
The system estimates the number of daily CU requests based on the basic scenario information.
Based on the running results of two batches of jobs in 30 days, the system estimates the CU-hours that are separately consumed by the two batches of jobs. The following content lists the estimated values.
First batch of jobs
1,500 CUs are consumed for the jobs to run for 1 hour. 1,500 CU-hours are used, which is calculated by using the following formula: 1,500 × 1.
Second batch of jobs
750 CUs are consumed for the jobs to run for 1 hour. 750 CU-hours are used, which is calculated by using the following formula: 750 × 1.
The system sets two estimation time points 02:00 and 06:00 based on the expected completion time of the two batches of jobs.
The system estimates resource consumption based on the current resource configuration.
NoteThe data of the following metrics is estimated based on CU-hours. For more information about the calculation logic, see "Terms" in this topic.
For the first batch of jobs, 1,500 CU-hours are required, and 1,000 reserved CUs are configured. The first batch of jobs can be completed in 1.5 hours. In this case, the first batch of jobs can be completed at 02:30. The job latency is 30 minutes.
For the second batch of jobs, 750 CU-hours are required, and 1,000 reserved CUs are configured. The second batch of jobs can be completed in 0.75 hours. In this case, all jobs are completed at about 05:45. Redundant CUs are available and no job latency exists.
The following table describes the current resource configuration plan based on the estimation result.
Estimation metric
Calculation formula
First batch of jobs
Second batch of jobs
Requirement satisfaction
Calculated based on resource consumption at the expected completion time:
Used CU-hours/Actually required CU-hours66.67%
(Resource insufficiency)
100%
(Resource redundancy)
Job latency
Calculated if jobs are not completed before the expected completion time:
Actual completion time - Expected completion time30 minutes
0 minutes
The system generates an optimal resource configuration optimization plan based on the output objective. The output objective refers to the time before which all the jobs involved are expected to be completed.
Optimization plan: 750 reserved CUs and 750 elastically reserved CUs for the period from 01:00 to 02:00.
Optimization results:
For the two batches of jobs, the requirement satisfaction is 100%, and the job latency is 0 minutes.
Monthly resource cost (catalog price): USD 17,598, which is calculated by using the following formula: 750 CUs × USD 22.0 per CU + 750 CUs × USD 0.0488 per CU-hour × 1 hour × 30. Compared with the monthly cost of computing resources before the optimization USD 22,000 (1,000 CUs × USD 22.0 per CU), a total of USD 4,402 is saved.
Generate an optimization plan for purchasing a subscription quota for a pay-as-you-go project
Scenario
A company activates MaxCompute billed in pay-as-you-go mode. In Project A, two batches of jobs run at fixed periods of time every day.
First batch of jobs: All jobs are submitted and started before 01:00 every day. The company expects to complete the jobs before 02:00.
Second batch of jobs: All jobs are submitted and started before 05:00 every day. The company expects to complete the jobs before 06:00.
Monthly resource cost (catalog price): The pay-as-you-go SQL jobs are billed based on the amount of data scanned. The average amount of data scanned by the jobs every day is about 1,200 GB. The monthly cost of computing resources is USD 1,576.8, which is calculated by using the following formula: 1,200 GB × USD 0.0438 per GB × 30.
The company wants to purchase a subscription computing quota to run the computing jobs in Project A.
Plan generation logic
The system estimates the number of daily CU requests based on the basic scenario information.
Based on the running results of two batches of jobs in 30 days, the system estimates the CU-hours that are separately consumed by the two batches of jobs. The following content lists the estimated values.
First batch of jobs
100 CUs are consumed for the jobs to run for 1 hour. 100 CU-hours are used, which is calculated by using the following formula: 100 × 1.
Second batch of jobs
50 CUs are consumed for the jobs to run for 1 hour. 50 CU-hours are used, which is calculated by using the following formula: 50 × 1.
The system sets two estimation time points 02:00 and 06:00 based on the expected completion time of the two batches of jobs. The system also sets the output objective. By default, job latency is not allowed.
The system generates an optimal resource configuration optimization plan based on the output objective. The output objective refers to the time before which all the jobs involved are expected to be completed.
Optimization plan: 50 reserved CUs and 50 elastically reserved CUs for the period from 01:00 to 02:00.
Optimization results:
For the two batches of jobs, the requirement satisfaction is 100%, and the job latency is 0 minutes.
Monthly resource cost (catalog price): USD 1,173.2, which is calculated by using the following formula: 50 CUs × USD 22.0 per CU + 50 CUs × USD 0.0488 per CU-hour × 1 hour × 30. Compared with the monthly cost of computing resources before the optimization USD 1,576.8, a total of USD 403.6 is saved.
Generate an optimization plan for using an existing subscription quota in a pay-as-you-go project
Scenario
A company purchased specific subscription computing resources. In the current resource configuration plan, 1,000 reserved compute units (CUs) are configured for jobs. Two batches of jobs consume the reserved CUs every day.
First batch of jobs: All jobs are submitted and started before 01:00 every day. The company expects to complete the jobs before 02:00.
Second batch of jobs: All jobs are submitted and started before 05:00 every day. The company expects to complete the jobs before 06:00.
Monthly resource cost (catalog price): The monthly cost of computing resources is USD 22,000, which is calculated by using the following formula: 1,000 CUs × USD 22.0 per CU.
The company activates MaxCompute billed in pay-as-you-go mode. In Project A, two batches of jobs run with pay-as-you-go resources at fixed periods of time every day.
First batch of jobs: All jobs are submitted and started before 01:00 every day. The company expects to complete the jobs before 02:00.
Second batch of jobs: All jobs are submitted and started before 05:00 every day. The company expects to complete the jobs before 06:00.
Monthly resource cost (catalog price): The pay-as-you-go SQL jobs are billed based on the amount of data scanned. The average amount of data scanned by the jobs every day is about 4,200 GB. The monthly cost of computing resources is USD 5,518.8, which is calculated by using the following formula: 4,200 GB × USD 0.0438 per GB × 30.
The total monthly cost of computing resources is USD 27,518.8, which is calculated by using the following formula: 22,000 + 5,518.8.
The company wants to run the computing jobs of Project A in the existing subscription quota and ensure that the jobs can be completed at the expected time.
Plan generation logic
The system estimates the number of daily CU requests of the pay-as-you-go project and the subscription quota based on the basic scenario information.
Based on the running results of two batches of jobs that use the subscription quota in 30 days, the system estimates the CU-hours that are separately consumed by the two batches of jobs. The following content lists the estimated values.
First batch of jobs
1,500 CUs are consumed for the jobs to run for 1 hour. 1,500 CU-hours are used, which is calculated by using the following formula: 1,500 × 1.
Second batch of jobs
750 CUs are consumed for the jobs to run for 1 hour. 750 CU-hours are used, which is calculated by using the following formula: 750 × 1.
Based on the running results of two batches of jobs of Project A in 30 days, the system estimates the CU-hours that are separately consumed by the two batches of jobs. The following content lists the estimated values.
First batch of jobs
500 CUs are consumed for the jobs to run for 1 hour. 500 CU-hours are used, which is calculated by using the following formula: 500 × 1.
Second batch of jobs
250 CUs are consumed for the jobs to run for 1 hour. 250 CU-hours are used, which is calculated by using the following formula: 250 × 1.
The system sets two estimation time points 02:00 and 06:00 based on the expected completion time of the two batches of jobs.
The system estimates resource consumption based on the current resource configuration of the subscription quota.
NoteThe data of the following metrics is estimated based on CU-hours. For more information about the calculation logic, see "Terms" in this topic.
After the first batch of jobs of pay-as-you-go Project A are added to the subscription quota, 2,000 CU-hours are required to run all the first batch of jobs in the quota, and 1,000 reserved CUs are configured. The first batch of jobs can be completed in 2 hours. In this case, the first batch of jobs is completed at 03:00. The job latency is 1 hour.
After the second batch of jobs of pay-as-you-go Project A are added to the subscription quota, 1,000 CU-hours are required to run all the second batch of jobs in the quota, and 1,000 reserved CUs are configured. The second batch of jobs can be completed in 1 hour. In this case, all jobs are completed at about 06:00. No job latency exists.
The following table describes the current resource configuration plan based on the estimation result.
Estimation metric
Calculation formula
First batch of jobs
Second batch of jobs
Requirement satisfaction
Calculated based on resource consumption at the expected completion time: Used CU-hours/Actually required CU-hours
50% (Resource insufficiency)
100%
Job latency
Calculated if jobs are not completed before the expected completion time: Actual completion time - Expected completion time
1 hour
0 minutes
The system generates an optimal resource configuration optimization plan based on the output objective. The output objective refers to the time before which all the jobs involved are expected to be completed. The system estimates resource consumption based on the current resource configuration of the subscription quota.
Optimization plan: 1,000 reserved CUs and 1,000 elastically reserved CUs for the period from 01:00 to 02:00.
Optimization results:
For the two batches of jobs, the requirement satisfaction is 100%, and the job latency is 0 minutes.
Monthly resource cost (catalog price): USD 23,464, which is calculated by using the following formula: 1,000 CUs × USD 22.0 per CU + 1,000 CUs × USD 0.0488 per CU-hour × 1 hour × 30. Compared with the monthly cost of computing resources before the optimization USD 27,518.8, a total of USD 4,054.8 is saved.
Terms (Click to expand)
Limits
MaxCompute can analyze computing resources and generate resource configuration optimization plans only for computing resources in a level-1 subscription quota and pay-as-you-go computing resources.
NoteThe billing methods supported by MaxCompute computing resources are pay-as-you-go, subscription, and pay-by-hour. For more information about the billing of different types of resources, see Overview.
You can use the quota management feature to allocate MaxCompute computing resources to different levels of quotas. For more information, see Manage quotas in the new MaxCompute console.
This feature is available in the China (Hangzhou), China (Shanghai), China (Beijing), China (Zhangjiakou), China (Shenzhen), and China (Chengdu) regions.
Procedure
Go to the Resource Optimization page.
Log on to the MaxCompute console. In the top navigation bar, select a region. In the left-side navigation pane, choose Cost Management > Cost Optimization.
Select a quota.
Select a quota from the Quota Name drop-down list based on your business requirements:
If you want to view the optimization plan for reconfiguring subscription computing resources, select a subscription quota.
If you want to view the resource configuration optimization plan for changing the pay-as-you-go computing resources of a project to subscription computing resources and view the estimated optimization effect, select a pay-as-you-go quota. The default name of the pay-as-you-go quota is Default pay-as-you-go quota_p.
Optional. Select a project whose resource configuration you want to change.
We recommend that you change the billing method of MaxCompute resources at the project level. If you select a pay-as-you-go quota in Step 2, you must select a project with which pay-as-you-go computing resources are associated.
NoteThe project list contains only pay-as-you-go projects that meet the requirements for resource reconfiguration. A project whose jobs are initiated every day in the last seven days meets the requirements for resource reconfiguration. Data statistics for the projects in the project list are collected until 00:00:00 on the current day.
To ensure a smooth page experience, you can select up to 20 projects for estimation at a time.
Select a reconfiguration method. By default, MaxCompute selects a reconfiguration method for you based on the quota of the project.
If you select a pay-as-you-go quota in Step 2, MaxCompute automatically selects a reconfiguration method based on whether the project uses a subscription quota.
If the project already uses a subscription quota, MaxCompute estimates resource consumption and generates an optimization plan based on the existing resource requests of your subscription quota.
If the project does not use a subscription quota, MaxCompute recommends purchasing a subscription quota based on the resource requests of the project.
View computing requirements.
On the Cost Optimization page, click Analyze Computing Requirements. On the page that appears, you can view the chart of the estimated daily CU requests that consume resources in a specified level-1 subscription quota group. You can learn about the number of resources that are required per day based on the estimated daily CU requests. If Reconfiguration Method is set to Use Existing Subscription Level-1 Quota, you can view the accumulative estimated computing requests for the project and the existing subscription quota.
NoteIf a small amount of historical job request data is available or the data has no apparent regularity, you cannot use this feature.
The estimated values in the chart are calculated by using the period identification and decomposition algorithm based on the job request data in 30 days.
Set the estimation time point.
In the Set Estimation Time Point section, select one or more estimation time points. You can set the estimation time points to the time points before which jobs are expected to be completed.
NoteMaxCompute estimates whether resources are sufficient and whether job latency exists in different resource configuration plans based on the specified estimation time point and job completion time. For more information about the logic, see How it works in this topic.
You can select 1 to 24 estimation time points. Only time points on the hour or the half hour can be configured as estimation time points.
Configuration examples:
If you want all jobs to be completed before 05:00 every day, you can set the estimation time point to 05:00.
All jobs submitted between 05:00:00 on the previous day and 05:00:00 on the current day are considered as a batch of jobs. You can estimate requirement satisfaction and job latency of this batch of jobs at 05:00:00 on the current day.
If you want the first batch of jobs to be completed before 02:00 and the second batch of jobs to be completed before 10:00 every day, you can set two estimation time points, such as 02:00 and 10:00.
All jobs submitted between 10:00:00 on the previous day and 02:00:00 on the current day are considered as the first batch of jobs. You can estimate requirement satisfaction and job latency of this batch of jobs at 02:00:00 on the current day.
All jobs submitted between 02:00:00 and 10:00:00 on the current day are considered as the second batch of jobs. You can estimate requirement satisfaction and job latency of this batch of jobs at 10:00:00 on the current day.
Optional. Click Current Plan Estimation to view the output of the jobs based on the current resource configuration. If Reconfiguration Method is set to Purchase Subscription Level-1 Quota, estimation on the current plan is not required.
NoteMaxCompute simulates CU consumption to show the output of jobs based on the current resource configuration. For more information about the metrics and estimation logic, see How it works in this topic.
You can view the requirement satisfaction and job latency metrics to obtain the consumption of resources and output of jobs based on the current resource configuration.
Optional. In the Set Optimization Goal section, configure the time point before which the jobs are expected to be completed after the optimization.
By default, the estimation time point is used. You can also configure a time point that is earlier than the estimation time point to reserve sufficient time for job output. Alternatively, you can configure the maximum allowed job latency, which allows for a specific latency for job output to reduce resource costs.
Click Generate Recommended Plan to view the optimization results and plan details based on the recommended resource configuration.
The optimization effect is still obtained based on the simulation of CU consumption (recommended plan). The obtained information is consistent with the estimation chart. You can compare the obtained information with data in the estimation chart.
ImportantThe recommended plan is for reference only and may not fully achieve the optimization objective. We recommend that you gradually change the configuration based on your business requirements and take note of the effect after the change.
In the Recommended Plan section, click Download to download the recommended plan to your on-premises machine.
We recommend that you download the recommended plan as an
.xlsxfile for easy storage and comparison of different recommended plans. Then, click Configure Now to configure plans on the Quotas page based on your business requirements.
What to do next
After an optimization plan is generated, estimate the cost of the computing resources after the optimization based on the computing resource configuration provided by the optimization plan.
Monthly cost of computing resources (catalog price) = Number of purchased subscription CUs × USD 22.0 per month + Number of elastically reserved CUs × Scaling period (hours) × USD 0.0488 per CU-hour × 30For more information about the billing of subscription resources, see Computing pricing. You can obtain the actual price on the buy page.
Estimate the cost and completion time after optimization based on the cost and optimization objective in the optimization plan. Purchase subscription resources based on the optimization plan, gradually change the configuration, and take note of the job running result after the change.
References
Best practices for using the computing resource optimization feature: Use the computing resource optimization feature to achieve cost-effectiveness.
For more information about MaxCompute cost optimization, see the following topics: