Use ECI to elastically schedule Spark jobs - E-MapReduce

You can use Elastic Container Instance (ECI) to schedule Spark jobs without being limited by the compute capacity of your ACK cluster nodes. ECI dynamically creates pods on demand to help you reduce compute costs. This topic describes how to use ECI to elastically schedule Spark jobs.

Background information

To use more advanced ECI features, you can set additional annotations to configure parameters for ECI on demand. For more information, see ECI pod annotation.

Prerequisites

You have created a Spark cluster in the EMR on ACK console. For more information, see Get started with EMR on ACK.
The Elastic Container Instance service is activated. For more information, see Workflow.

Procedure

Install the virtual nodes required by ECI in your ACK cluster. For more information, see Step 1: Deploy the ack-virtual-node component in an ACK cluster.

When you submit a Spark job on EMR on ACK, you can enable ECI scheduling by configuring a label, an annotation, or Spark Conf.

For more information about how to submit a Spark job, see Submit a Spark job.

Note

The examples in this topic use Spark 3.1.1 (EMR-5.2.1-ack). If you use a different version, modify the sparkVersion and mainApplicationFile configurations. For descriptions of the parameters in the examples, see spark-on-k8s-operator.

Method 1: Configure a pod label

Set the alibabacloud.com/eci label to true to schedule specified pods to run on ECI. The following example shows the configuration.

apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
  name: spark-pi-eci
spec:
  type: Scala
  sparkVersion: 3.1.1
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: "local:///opt/spark/examples/jars/spark-examples_2.12-3.1.1.jar"
  arguments:
    - "1000000"
  driver:
    cores: 2
    coreLimit: 2000m
    memory: 4g
  executor:
    cores: 4
    coreLimit: 4000m
    memory: 8g
    instances: 10
    # By configuring this label, all executors use ECI.
    labels:
      alibabacloud.com/eci: "true"
    # (Optional) Enable the ECI image cache to improve performance.
    annotations:
      k8s.aliyun.com/eci-image-cache: "true"

Method 2: Configure a pod annotation

Set the alibabacloud.com/burst-resource annotation to eci to schedule specified pods to run on ECI. This annotation has two valid values:

eci: Use ECI when regular cluster nodes have insufficient resources.
eci_only: Use only ECI.

The following example shows the configuration.

apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
  name: spark-pi-eci
spec:
  type: Scala
  sparkVersion: 3.1.1
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: "local:///opt/spark/examples/jars/spark-examples_2.12-3.1.1.jar"
  arguments:
    - "1000000"
  driver:
    cores: 2
    coreLimit: 2000m
    memory: 4g
  executor:
    cores: 4
    coreLimit: 4000m
    memory: 8g
    instances: 10
    # By configuring this annotation, executors use ECI when regular node resources are insufficient.
    annotations:
      alibabacloud.com/burst-resource: "eci"
      # (Optional) Enable the ECI image cache to improve performance.
      k8s.aliyun.com/eci-image-cache: "true"

Method 3: Configure Spark Conf

Alternatively, you can enable ECI scheduling by configuring pod annotations in Spark Conf. The annotation values are the same as those in Method 2: Configure a pod annotation.

Go to the spark-defaults.conf tab.
1. Log on to the EMR on ACK console.
2. On the EMR on ACK page, find the target cluster and click Configure in the Actions column.
3. On the Configure page, click the spark-defaults.conf tab.

Configure the Spark cluster to enable ECI.

Click Add Configuration Item.

In the Add Configuration Item dialog box, add the following configuration items.

Parameter	Description
spark.kubernetes.driver.annotation.alibabacloud.com/burst-resource	Specifies whether the Spark driver uses ECI. Valid values: `eci` and `eci_only`.
spark.kubernetes.driver.annotation.k8s.aliyun.com/eci-image-cache	Specifies whether the Spark driver uses the ECI image cache. Set this parameter to `true` to improve performance.
spark.kubernetes.executor.annotation.alibabacloud.com/burst-resource	Specifies whether Spark executors use ECI. Valid values: `eci` and `eci_only`.
spark.kubernetes.executor.annotation.k8s.aliyun.com/eci-image-cache	Specifies whether Spark executors use the ECI image cache. Set this parameter to `true` to improve performance.
spark.kubernetes.driver.annotation.k8s.aliyun.com/eci-ram-role-name	Specifies the RAM role name to bind when the Spark driver pod is created. Set this value to AliyunECSInstanceForEMRRole.

Click OK.
In the dialog box that appears, enter an Execution Reason and click Save.

Apply the configuration.
1. Click Deploy Client Configuration.
2. In the dialog box that appears, enter an Execution Reason and click OK.
3. In the Confirm dialog box, click OK.

Optional: If your job needs to read from or write to OSS, or uses DLF metadata, you need to grant ECI permissions to access these cloud services. You can grant the permissions in one of the following ways:
- Method 1: Assign a RAM role to ECI for password-free access.
  1. In the RAM console, create a RAM role whose trusted entity is an Alibaba Cloud service. For more information, see Create a regular service role.
    
    Note
    Select ECS as the trusted service.
  2. Attach the AliyunOSSFullAccess and AliyunDLFFullAccess policies to the RAM role.
    
    For more information, see Manage the permissions of a RAM role.
  3. In your Spark job, add the following annotation to specify the RAM role.
```
annotations:
  k8s.aliyun.com/eci-ram-role-name: <your-ram-role-name>
```
- Method 2: Configure an OSS AccessKey or a DLF AccessKey
  - If your job needs to read from or write to OSS, add the AccessKey configuration to hadoopConf as shown below.
```
hadoopConf:
  fs.jfs.cache.oss.accessKeyId: <yourAccessKeyId>
  fs.jfs.cache.oss.accessKeySecret: <yourAccessKeySecret>
```
  - If your job uses DLF, add the AccessKey configuration to hadoopConf as shown below.
```
hadoopConf:
  dlf.catalog.accessKeyId: <yourAccessKeyId>
  dlf.catalog.accessKeySecret: <yourAccessKeySecret>
  dlf.catalog.akMode: "MANUAL"
```