Submit a Spark job on EMR on ACK - E-MapReduce - Alibaba Cloud - E-MapReduce

EMR on ACK supports three methods for submitting jobs: using a Custom Resource Definition (CRD), the spark-submit command, or the console terminal. This topic describes these methods.

Prerequisites

Create a Spark cluster in the EMR on ACK console. For more information, see Create a cluster.

Notes

In the examples in this topic, the JAR file is packaged directly into the image. If you use your own JAR file, you can upload it to Object Storage Service (OSS). For more information, see Simple upload.

Replace local:///opt/spark/examples/spark-examples.jar in the command with the OSS path to your JAR file. The path format is oss://<yourBucketName>/<path>.jar.

Submit a job

Method 1: CRD submission

Use kubectl to connect to the Kubernetes cluster. For more information, see Obtain the KubeConfig file of a cluster and use kubectl to connect to the cluster.

Create a file named spark-pi.yaml with the following content.

apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
  name: spark-pi-simple
spec:
  type: Scala
  sparkVersion: 3.2.1
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: "local:///opt/spark/examples/spark-examples.jar"
  arguments:
    - "1000"
  driver:
    cores: 1
    coreLimit: 1000m
    memory: 4g
  executor:
    cores: 1
    coreLimit: 1000m
    memory: 8g
    memoryOverhead: 1g
    instances: 1

For descriptions of the parameters in this example, see spark-on-k8s-operator.

Note

You can use a custom file name. This topic uses spark-pi.yaml as an example.
The example uses Spark 3.2.1 (EMR-5.6.0). If you use a different version, modify the sparkVersion parameter.

Run the following command to submit the job.
```
kubectl apply -f spark-pi.yaml --namespace <cluster-namespace>
```
Replace <cluster-namespace> with your cluster's namespace. You can find the namespace on the Cluster Details page in the EMR on ACK console.
The following output is returned:
```
sparkapplication.sparkoperator.k8s.io/spark-pi-simple created
```
Note
spark-pi-simple is the job name in this example.
Optional: View the created job on the Job Details page.

Method 2: spark-submit command

Use kubectl to connect to the Kubernetes cluster. For more information, see Obtain the KubeConfig file of a cluster and use kubectl to connect to the cluster.

Run the following commands to install the emr-spark-ack tool from Alibaba Cloud E-MapReduce and set its permissions.

wget https://ecm-repo-cn-hangzhou.oss-cn-hangzhou.aliyuncs.com/emr-on-ack/util/emr-spark-ack
chmod 755 emr-spark-ack

Use the emr-spark-ack tool to submit the job.

Use the following syntax.

 ./emr-spark-ack -n <cluster-namespace> <spark-command>

Note

The <spark-command> placeholder supports spark-submit, spark-sql, spark-shell, and pyspark. The syntax is identical to the native Spark commands.

Example: cluster mode

Use spark-submit to submit a spark-pi job.

./emr-spark-ack -n <cluster-namespace> spark-submit \
    --name spark-pi-submit \
    --deploy-mode cluster \
    --class org.apache.spark.examples.SparkPi \
    local:///opt/spark/examples/spark-examples.jar \
    1000

Example: client mode

Using the spark-sql command

# Prepare a local SQL file.
echo "select 1+1">test.sql
# Submit the job.
./emr-spark-ack -n <cluster-namespace> spark-sql -f test.sql

For clusters that run Spark 3 or later (EMR-5.X), the emr-spark-ack tool automatically uploads local dependencies. The tool uploads local files specified by parameters like --jars, --files, and -f to the EMR on ACK cluster before submitting the job in the Kubernetes environment.

The following code provides an example and shows the output:

[user@hostname ~]$ echo "select 1+1">test.sql
[user@hostname ~]$ ./emr-spark-ack -n <cluster-namespace> spark-sql -f test.sql
uploading test.sql to c-xxx/spark-submit-gateway-865bbf56f6-sfb95:/opt/spark/work-dir/test.sql
cmd: kubectl exec spark-submit-gateway-865bbf56f6-sfb95 -i -n <cluster-namespace> -- /bin/bash spark-sql -f test.sql
Warning: Ignoring non-Spark config property: ack.clusterid.for.rss.linked
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark master: k8s://https://kubernetes.default:443, Application Id: spark-application-1653639654376
Time taken: 4.738 seconds, Fetched 1 row(s)
2
22/05/27 16:21:04 WARN [shutdown-hook-0] ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed.

Using the spark-shell command

./emr-spark-ack -n <cluster-namespace> spark-shell

The following code provides an example and shows the output:

[user@hostname ~]$ ./emr-spark-ack -n <cluster-namespace> spark-shell
cmd: kubectl exec spark-submit-gateway-865bbf56f6-sfb95 -it -n <cluster-namespace> -- /bin/bash spark-shell
Warning: Ignoring non-Spark config property: ack.clusterid.for.rss.linked
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://10.210.xxx.xxx:4041
Spark context available as 'sc' (master = k8s://https://kubernetes.default:443, app id = spark-application-16536387xxx).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.2.1
      /_/
Using Scala version 2.12.15 (OpenJDK 64-Bit Server VM, Java 1.8.0_332)
Type in expressions to have them evaluated.
Type :help for more information.
scala>

Optional: View the created job on the Job Details page.
Optional: Use the emr-spark-ack tool to terminate a job.
Use the following syntax.
```
 ./emr-spark-ack -n <cluster-namespace> kill <Spark_app_id>
```
Note
The emr-spark-ack tool generates the <Spark_app_id> when you submit the job. This ID is available in the output log.

Method 3: Console terminal

Go to the Access Links and Ports page.
1. Log on to the EMR on ACK console.
2. On the EMR on ACK page, click your target cluster's name.
3. Click the Access Links and Ports tab.
On the Access Links and Ports page, click the link for SparkSubmitGateway UI.

This opens the shell terminal.

In the shell terminal, run Spark commands in one of the following two ways.

Using the spark-sql command

spark-sql

After entering the spark-sql shell, you can run Spark commands for interactive queries.

[root@spark-submit-gateway-865bb... work-dir]# spark-sql
Warning: Ignoring non-Spark config property: ack.clusterid.for.rss.linked
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark master: k8s://https://kubernetes.default:443, Application Id: spark-application-1654656192425
spark-sql> select 1+1;
2
Time taken: 5.277 seconds, Fetched 1 row(s)
spark-sql>

Using the spark-submit command

spark-submit \
    --name spark-pi-submit \
    --deploy-mode cluster \
    --class org.apache.spark.examples.SparkPi \
    local:///opt/spark/examples/spark-examples.jar \
    1000

Optional: View the created job on the Job Details page.

E-MapReduce:Submit a Spark job