AnalyticDB:Spot instances

Prerequisites

An AnalyticDB for MySQL Enterprise Edition, Basic Edition, or Data Lakehouse Edition cluster is created.

What are spot instances

AnalyticDB for MySQL resource pools may have idle Spark executor resources that are sold at discounted prices. These resources are called spot instances.

The following section describes the differences between spot instance resources and AnalyticDB compute unit (ACU) elastic resources:

• Spot instance resources cost approximately 30% less than ACU elastic resources.

• When you submit a Spark job in a job resource group with the spot instance feature enabled, the AnalyticDB for MySQL cluster preferentially preempts idle and low-cost spot instance resources. If preemption succeeds, the Spark job runs using spot instance resources and you are charged accordingly. If preemption fails, the Spark job runs using ACU elastic resources and you are charged for those resources instead.

• By default, Alibaba Cloud provides a 1-hour protection period for AnalyticDB for MySQL spot instance resources to ensure that the resources are not preempted or released within 1 hour after they are used. After the protection period ends, the resources may be preempted. The probability of preemption increases with the duration of spot instance resource usage.

• Spark jobs that use spot instance resources have lower priority. For example, if Spark job A is running using spot instance resources and another AnalyticDB for MySQL cluster submits Spark job B that does not use spot instance resources, Spark job B may preempt the spot instance resources used by Spark job A if it lacks sufficient resources. This results in the release of spot instance resources and the failure of Spark job A. The probability of Spark job failures due to resource release is 1% to 5%.

Scenarios and suggestions

We recommend that you enable the spot instance feature in the following scenarios:

• Spark job development scenarios.

• Cost-sensitive scenarios.

• Scenarios that have high tolerance for Spark job failures.

• Scenarios in which most Spark jobs run within 1 hour.

For Spark jobs that require more than 1 hour to run and those that run within 1 hour, we recommend the following operations:

• Create two job resource groups. Enable the spot instance feature for the first group and run only Spark jobs with short execution times. Do not enable the spot instance feature for the second group and use it for Spark jobs with longer execution times.

• Create a job resource group and enable the spot instance feature for the resource group. When you run a Spark job, configure the spark.adb.spotInstance.disabled parameter to disable the usage of spot instance resources for the Spark job.

Limits

• The spot instance feature is supported only in the following regions: China (Qingdao), China (Beijing), China (Zhangjiakou), China (Hangzhou), China (Shanghai), China (Shenzhen), China (Guangzhou), China (Hong Kong), Japan (Tokyo), Singapore, US (Silicon Valley), US (Virginia), Germany (Frankfurt), and UK (London).

• You can enable the spot instance feature only for job resource groups. After you enable the feature, Spark jobs in the job resource group attempt to use spot instance resources, but this does not guarantee that the resources will be available.

• Only Spark jobs can use spot instance resources.

• Only the following specifications of Spark executor resources support spot instance resources: large and 2xlarge.

Billing rules

Spot instance resources cost approximately 30% less than ACU elastic resources. For more information, see Pricing for Enterprise Edition and Basic Edition and Pricing for Data Lakehouse Edition.

Enable the spot instance feature

1. Log on to the AnalyticDB for MySQL console. In the upper-left corner of the console, select a region. In the left-side navigation pane, click Clusters. Find the cluster that you want to manage and click the cluster ID.

2. In the navigation pane on the left, choose Cluster Management > Resource Management. On the page that appears, click the Resource Groups tab.

   Enable the spot instance feature for an existing job resource group

   1. Find the job resource group that you want to manage and click Modify in the Actions column.

   2. In the Modify Resource Group panel, turn on Spot Instance.

   3. Click OK.

   4. Check whether the spot instance feature is enabled in the Spot Instance column of the job resource group.

      After you enable the spot instance feature, Spark jobs that run in the job resource group attempt to use the spot instance resources.

   Enable the spot instance feature when you create a job resource group

   1. Click Create Resource Group in the upper-right corner of the resource group list and configure the resource group information.

      Parameter	Description
      Resource Group Name	The name of the resource group. The name must be 2 to 30 characters in length and can contain letters, digits, and underscores (_). The name must start with a letter.
      Job Type	The job type of the resource group. Select Job from the drop-down list. Job resource groups are suitable for scaling computing resources on demand in high-throughput batch processing scenarios.
      Minimum Computing Resources	The minimum amount of computing resources. Set the value to 0 ACUs.
      Maximum Computing Resources	The maximum amount of computing resources. Set this parameter within the range of 8 ACUs to 1,024 ACUs in increments of 8 ACUs. If you want to change the maximum value of the Maximum Computing Resources parameter, submit a ticket.
      Spot Instance	Turn on Spot Instance. After you enable the spot instance feature for the job resource group, Spark jobs that run in the job resource group attempt to use the spot instance resources.

   2. Click OK.

   3. After you create the job resource group, check whether the spot instance feature is enabled in the Spot Instance column of the resource group.

      After you enable the spot instance feature, Spark jobs that run in the job resource group attempt to use the spot instance resources.

(Optional) Disable the usage of spot instance resources for a Spark job

If you do not want a Spark job to use spot instance resources, you can configure the following parameter in the CONF parameter for the Spark job:

After you run the Spark job, you can view the value of the Spot Instance Resource Usage parameter to check whether the usage of spot instance resources is disabled. If the value of the Spot Instance Resource Usage parameter is 0, the Spark job does not use spot instance resources.

View the usage of spot instance resources

After a Spark job is complete, you can check the job resource group's Monitoring page to view the usage of spot instance resources.

Disable the spot instance feature

1. Log on to the AnalyticDB for MySQL console. In the upper-left corner of the console, select a region. In the left-side navigation pane, click Clusters. Find the cluster that you want to manage and click the cluster ID.

2. In the navigation pane on the left, choose Cluster Management > Resource Management. On the page that appears, click the Resource Groups tab.

3. Find the job resource group that you want to manage and click Modify in the Actions column.

4. In the Modify Resource Group panel, turn off Spot Instance.

5. Click OK.

6. Check whether the spot instance feature is disabled in the Spot Instance column of the job resource group.

Related operations