E-MapReduce:Submit a Spark job using the spark-submit command line interface (CLI)

Prerequisites

Before you begin, make sure you have:

• Java 1.8 or later installed on your ECS instance

• (For RAM users) The RAM user added to the Serverless Spark workspace with the Developer role or higher. See Manage users and roles

Placeholders in this topic

Replace the following placeholders with your actual values before running the commands:

Step 1: Download and install the CLI

1. Download the installation package: emr-serverless-spark-tool-0.11.3-SNAPSHOT-bin.zip.

2. Upload the package to your ECS instance. See Upload or download files.

3. Decompress the package:

   unzip emr-serverless-spark-tool-0.11.3-SNAPSHOT-bin.zip

   After decompression, a directory named emr-serverless-spark-tool-0.11.3-SNAPSHOT is created in your current working directory.

Step 2: Configure the connection

1. Open the configuration file:

   Important

   If the SPARK_CONF_DIR environment variable is set in your environment (for example, in an EMR cluster, this is typically /etc/taihao-apps/spark-conf), place connection.properties in that directory instead. Otherwise, the system returns an error.

   vim emr-serverless-spark-tool-0.11.3-SNAPSHOT/conf/connection.properties

2. Set the following required parameters using key=value format:

   Important

   The RAM user or role associated with the AccessKey must be granted RAM authorization and added to the Serverless Spark workspace. See RAM user authorization and Manage users and roles.

   Parameter	Description
   accessKeyId	AccessKey ID of the Alibaba Cloud account or RAM user used to run Spark jobs. The associated user must have read and write permissions on the Object Storage Service (OSS) bucket bound to the workspace. To find the bound OSS bucket, go to the Spark page and click Details in the Actions column.
   accessKeySecret	AccessKey secret paired with accessKeyId.
   regionId	Region ID of your workspace. This example uses cn-hangzhou (China (Hangzhou)).
   endpoint	EMR Serverless Spark service endpoint. This example uses the public endpoint for China (Hangzhou): emr-serverless-spark.cn-hangzhou.aliyuncs.com. For all endpoints, see Service endpoints. If your ECS instance cannot access the public network, use the VPC endpoint instead.
   workspaceId	ID of the EMR Serverless Spark workspace.

   accessKeyId=<ALIBABA_CLOUD_ACCESS_KEY_ID>
   accessKeySecret=<ALIBABA_CLOUD_ACCESS_KEY_SECRET>
   regionId=<region-id>
   endpoint=emr-serverless-spark.<region-id>.aliyuncs.com
   workspaceId=<workspace-id>

   The following table describes all parameters:

Step 3: Submit a Spark job

Navigate to the tool directory:

cd emr-serverless-spark-tool-0.11.3-SNAPSHOT

Choose a submission method based on your job type.

./bin/spark-submit --name SparkPi \
  --queue dev_queue \
  --num-executors 5 \
  --driver-memory 1g \
  --executor-cores 2 \
  --executor-memory 2g \
  --class org.apache.spark.examples.SparkPi \
  oss://<your-bucket>/path/to/spark-examples_2.12-3.5.2.jar \
  10000

./bin/spark-submit --name PySpark \
  --queue dev_queue \
  --num-executors 5 \
  --driver-memory 1g \
  --executor-cores 2 \
  --executor-memory 2g \
  --conf spark.tags.key=value \
  oss://<your-bucket>/path/to/DataFrame.py \
  oss://<your-bucket>/path/to/employee.csv

spark-sql -e "SHOW TABLES"

spark-sql -f oss://<your-bucket>/path/to/your/example.sql

CREATE TABLE IF NOT EXISTS employees (
    id INT,
    name STRING,
    age INT,
    department STRING
);

INSERT INTO employees VALUES
(1, 'Alice', 30, 'Engineering'),
(2, 'Bob', 25, 'Marketing'),
(3, 'Charlie', 35, 'Sales');

SELECT * FROM employees;

spark-submit parameters

Step 4: Query a Spark job

cd emr-serverless-spark-tool-0.11.3-SNAPSHOT
./bin/spark-submit --status <job-run-id>

cd emr-serverless-spark-tool-0.11.3-SNAPSHOT
./bin/spark-submit --detail <job-run-id>

Step 5 (optional): Stop a Spark job

cd emr-serverless-spark-tool-0.11.3-SNAPSHOT
./bin/spark-submit --kill <job-run-id>

FAQ

How do I specify network connectivity when submitting a batch job?

First, set up a network connection. See Add a network connection.

Then pass the connection name in your spark-submit command:

--conf spark.emr.serverless.network.service.name=<network-connection-name>