All Products
Search
Document Center

MaxCompute:Consumption control

Last Updated:Mar 15, 2024

This topic describes how to configure limits and alert rules on the consumption of the pay-as-you-go resources that are consumed by computing jobs.

Description

Pay-as-you-go resources in MaxCompute are scalable. Computing jobs can use these resources without limits. To prevent unexpected consumption, we recommend that you monitor the resource consumption of computing jobs. MaxCompute allows you to configure limits and alert rules on the resource consumption:

  • Upper limit on the resource consumption of an SQL statement: MaxCompute allows you to configure an upper limit for the resources that are consumed by an SQL statement. Before you execute an SQL statement, you can estimate the resources that can be consumed by this statement. Such SQL statements include SQL statements that consume pay-as-you-go computing resources and SQL statements that consume subscription computing resources. If the estimated resource consumption exceeds the specified upper limit, the SQL statement cannot be executed, and a failure message is returned. You can use this feature to prevent unexpected charges caused by an SQL statement.

  • Monitoring and alerting for real-time resource consumption: You can use CloudMonitor to monitor and generate alerts for the resources consumed by standard SQL, MapReduce, and Spark computing jobs in real time before bills are generated. You can specify a threshold in the CloudMonitor console. CloudMonitor monitors the consumed resources by project on a daily or monthly basis. For more information about the billing, see Computing pricing. If the accumulative resource consumption exceeds the specified threshold, CloudMonitor sends alert notifications by phone calls, text messages, emails, or DingTalk messages.

    Note

    Real-time resource consumption is the consumption of resources before bills are generated. Coupons or discounts are not included.

Configure an upper limit for the resources that are consumed by an SQL statement

You can configure an upper limit for the resources that are consumed by an SQL statement for a project or a session.

  • Configure an upper limit for a project.

    The project owner or a user who is assigned the Super_Administrator role can impose or remove the upper limit on or from the resource consumption of an SQL statement. The configuration takes effect for all subsequent SQL jobs whose resource consumption can be estimated in a project.

    -- Impose an upper limit on the resource consumption of an SQL statement. 
    setproject odps.sql.metering.value.max=<m_value>;
    
    -- Remove an upper limit from the resource consumption of an SQL statement. 
    setproject odps.sql.metering.value.max;
  • Configure an upper limit for a session.

    You can submit the following command with the SQL statement that you want to execute to impose the upper limit on the resources that are consumed by the SQL statement. If the command is not added before the SQL statement, the upper limit is removed. The configuration takes effect only for the current statement.

    set odps.sql.metering.value.max=<m_value>;

m_value specifies an upper limit for the resources that are consumed by an SQL statement. m_value is calculated by using the following formula: m_value = Number of gigabytes read by an SQL statement × Complexity of the SQL statement. m_value does not indicate your consumption fees.

Note

The resources that can be consumed by an SQL statement for a project are limited by m_value. If the estimated consumption value of an SQL statement exceeds the value of m_value, the SQL statement cannot be executed. The resource consumption of some SQL statements, such as SQL statements in which external tables are involved, cannot be estimated. In this case, consumption control does not take effect for the SQL statements.

Take note of the following points:

  • We recommend that you specify m_value for a project based on the historical consumption records and resources that are expected to be consumed by an SQL statement.

    You can download daily consumption information and calculate the consumption of an SQL statement whose data category is ComputationSql in a project by using the following formula: Consumption of an SQL statement = Number of gigabytes read by the SQL statement × Complexity of the SQL statement. After the calculation is complete, you can determine the value of m_value based on the maximum daily consumption and business fluctuation.

    You can also convert computing resource consumption into fees, determine an appropriate amount of fees based on your business requirements, and then convert the fees into a consumption value.

  • We recommend that you specify m_value for a project and then for a session for some SQL statements.

    Note
    • Before you configure an upper limit for an SQL statement at the session level, you must make sure that this upper limit is required for the SQL statement. This prevents abuse of consumption control.

    • If you configure an upper limit for both a project and its session, the configuration for the session takes precedence over the configuration for the project. For example, if you set m_value to 100 for a project and 200 for a session in the project, the SQL statement cannot be executed if the estimated resource consumption value exceeds 200. SQL statements that do not have a session-level upper limit are subject to the project-level upper limit. If the estimated resource consumption value exceeds 100, these SQL statements cannot be executed.

  • If you configure an upper limit for a project or a session, the execution of a job is adversely affected. We recommend that you use the consumption control feature together with the job failure alerting feature, such as the intelligent monitoring feature of DataWorks. This way, failed jobs are immediately detected and processed to minimize the negative impact of the jobs on your business.

Configure alert rules on real-time resource consumption

The accumulated resource consumption amount of each SQL, MapReduce, or Spark job that runs successfully is calculated based on the catalog price that is calculated by using the billing formula for pay-as-you-go resources. MaxCompute calculates the accumulated resource consumption amount on a daily or monthly basis and pushes the amount to CloudMonitor. CloudMonitor helps you monitor resource consumption in real time. To configure an alert rule, perform the following steps:

  1. Log on to the CloudMonitor console.

  2. Create an alert contact. If you have already created an alert contact, skip this step. For more information about how to create an alert contact, see Create an alert contact or alert contact group.

  3. Create an alert rule. For more information, see Create an alert rule. The following table describes the parameters that you must configure when you create an alert rule on real-time resource consumption.

    Section

    Parameter

    Description

    Associated Resources

    Product

    Set this parameter to MaxCompute_PayAsYouGo.

    Resource Range

    The resources to which the alert rule is applied.

    • All Resources: Alert notifications are sent when any project meets the alert rule.

    • Application Groups: The alert rule applies to all resources in the specified application group of the specified cloud service. The alert rule takes effect on the resources that are newly added.

    • Instances: You can select a project that you want to monitor based on your business requirements. For example, you can monitor a project that uses pay-as-you-go resources. We recommend that you set the Resource Range parameter to Instances.

    Associated Resources

    If you set the Resource Range parameter to Instances, select a project from the Associated Resources drop-down list.

    Rule Description

    Metric

    • If you set this parameter to PayAsYouGo jobs' daily consumption in USD, the resource consumption amount is accumulated by project on a daily basis.

    • If you set this parameter to PayAsYouGo jobs' monthly consumption in USD, the resource consumption amount is accumulated by project on a monthly basis.

    Important

    If a metric is measured in the unit of CNY, the metric is invalid.

    Threshold

    Unit: USD. The resource consumption amount is the catalog price that is calculated by using the billing formula for pay-as-you-go resources.