E-MapReduce:Submit a Spark job using spark-submit

Step 1: Download and install the EMR Serverless spark-submit tool

1. Click emr-serverless-spark-tool-1.16.0-bin.zip to download the installation package.

2. Upload the installation package to your ECS instance. For more information, see Upload or download files.

3. Run the following command to decompress the EMR Serverless spark-submit tool.

   unzip emr-serverless-spark-tool-1.16.0-bin.zip

Step 2: Configure parameters

1. Run the following command to modify the configuration in the connection.properties file.

   vim emr-serverless-spark-tool-1.16.0/conf/connection.properties

2. We recommend that you configure the file as shown in the following example, with parameters in the key=value format.

   accessKeyId=<ALIBABA_CLOUD_ACCESS_KEY_ID>
   accessKeySecret=<ALIBABA_CLOUD_ACCESS_KEY_SECRET>
   regionId=cn-hangzhou
   endpoint=emr-serverless-spark.cn-hangzhou.aliyuncs.com
   workspaceId=w-xxxxxxxxxxxx

   Important

   The RAM user or role for this AccessKey must be authorized in RAM and added to the EMR Serverless Spark workspace.

   • For RAM authorization, see Grant permissions to a RAM user.

   • For user and role management in an EMR Serverless Spark workspace, see Manage users and roles.

   The following table describes the parameters.

   Parameter	Required	Description
   accessKeyId	Yes	The AccessKey ID and AccessKey Secret of the Alibaba Cloud account or RAM user that runs the Spark job. Important When you configure the accessKeyId and accessKeySecret parameters, ensure that the user associated with the AccessKey has read and write permissions for the OSS Bucket that is bound to the workspace. You can view the OSS Bucket that is bound to the workspace on the Spark page by clicking Details in the Actions column of the workspace.
   accessKeySecret	Yes
   regionId	Yes	The region ID. This example uses the China (Hangzhou) region.
   endpoint	Yes	The endpoint of EMR Serverless Spark. For more information, see Endpoints. This example uses the public endpoint for the China (Hangzhou) region. The parameter value is emr-serverless-spark.cn-hangzhou.aliyuncs.com. Note If your ECS instance does not have internet access, you must use a VPC endpoint.
   workspaceId	Yes	The ID of the EMR Serverless Spark workspace.

Step 3: Submit a Spark job

1. Run the following command to go to the EMR Serverless spark-submit tool directory.

   cd emr-serverless-spark-tool-1.16.0

2. Select a submission method based on your job type.

   When you submit a job, you must specify the file resources that the job depends on, such as a JAR package or Python script. You can store these resources in OSS or locally. This topic uses OSS resources in all examples.

   spark-submit

   spark-submit is a general-purpose task submission tool provided by Spark for Java/Scala and PySpark tasks.

   Java/Scala jobs

   This example uses spark-examples_2.12-3.5.2.jar. You can click spark-examples_2.12-3.5.2.jar to download the test JAR package, and then upload the JAR package to OSS. This JAR package is a simple example built into Spark that calculates the value of pi (π).

   Note

   The spark-examples_2.12-3.5.2.jar file must be used with an esr-4.x engine version to submit jobs. If you are using an esr-5.x engine version, download spark-examples_2.13-4.0.1.jar for this example.

   ./bin/spark-submit  --name SparkPi \
   --queue dev_queue  \
   --num-executors 5 \
   --driver-memory 1g \
   --executor-cores 2 \
   --executor-memory 2g \
   --class org.apache.spark.examples.SparkPi \
    oss://<yourBucket>/path/to/spark-examples_2.12-3.5.2.jar \
   10000

   PySpark jobs

   This example uses DataFrame.py and employee.csv. You can click DataFrame.py and employee.csv to download the test files, and then upload them to OSS.

   Note

   • DataFrame.py is a code snippet that uses the Apache Spark framework to process data in OSS.

   • employee.csv is a data file that contains employee names, departments, and salaries.

   ./bin/spark-submit --name PySpark \
   --queue dev_queue  \
   --num-executors 5 \
   --driver-memory 1g \
   --executor-cores 2 \
   --executor-memory 2g \
   --conf spark.tags.key=value \
   oss://<yourBucket>/path/to/DataFrame.py \
   oss://<yourBucket>/path/to/employee.csv

   The parameters are described below.

   • Supported open source parameters

     Parameter	Example	Description
     --name	SparkPi	The application name for the Spark job.
     --class	org.apache.spark.examples.SparkPi	The entry class for the Spark job. This parameter is required for Java or Scala applications but is not required for Python applications.
     --num-executors	5	The number of executors for the Spark job.
     --driver-cores	1	The number of driver cores for the Spark job.
     --driver-memory	1g	The amount of driver memory for the Spark job.
     --executor-cores	2	The number of executor cores for the Spark job.
     --executor-memory	2g	The amount of executor memory for the Spark job.
     --files	oss://<yourBucket>/file1,oss://<yourBucket>/file2	The resource files required by the Spark job. The files can be stored in OSS or locally. Separate multiple files with commas (,).
     --py-files	oss://<yourBucket>/file1.py,oss://<yourBucket>/file2.py	The Python scripts required by the Spark job. The scripts can be stored in OSS or locally. Separate multiple files with commas (,). This parameter is only for PySpark applications.
     --jars	oss://<yourBucket>/file1.jar,oss://<yourBucket>/file2.jar	The JAR packages required by the Spark job. The resources can be stored in OSS or locally. Separate multiple files with commas (,).
     --archives	oss://<yourBucket>/archive.tar.gz#env,oss://<yourBucket>/archive2.zip	The archive files required by the Spark job. The resources can be stored in OSS or locally. Separate multiple files with commas (,).
     --queue	root_queue	The name of the queue where the Spark job runs. This name must match the queue name in the queue management section of your EMR Serverless Spark workspace.
     --proxy-user	test	The set value will override the HADOOP_USER_NAME environment variable, and the behavior is consistent with the open-source version.
     --conf	spark.tags.key=value	A custom parameter for the Spark job.
     --status	jr-8598aa9f459d****	Checks the status of the Spark job.
     --kill	jr-8598aa9f459d****	Terminates the Spark job.

   • Enhanced parameters

     Parameter	Example	Description
     --detach	No value required	Causes spark-submit to exit immediately after submitting the job, without waiting for the job status.
     --detail	jr-8598aa9f459d****	Displays the details of the Spark job.
     --release-version	esr-4.1.1 (Spark 3.5.2, Scala 2.12)	Specifies the Spark version. Enter the engine version number displayed in the console.
     --enable-template	No value required	Enables the template feature. The job will then use the default configuration template of the workspace. If you created a Configurations in Configuration management, you can specify its ID by adding spark.emr.serverless.templateId to the --conf parameter. The job then directly applies the specified template. For more information about creating templates, see Configuration management. If you specify only --enable-template, the task automatically applies the workspace's default configuration template. Use only --conf to specify the template ID: The task directly applies the specified template ID. Specifying both --enable-template and --conf: If you specify both --enable-template and --conf spark.emr.serverless.templateId, the template ID in --conf overrides the default template. No parameters specified: If you do not use --enable-template or specify --conf spark.emr.serverless.templateId, the job will not apply any template configuration.
     --timeout	60	The timeout period for the job, in seconds.
     --workspace-id	w-4b4d7925a797****	Specifies the workspace ID at the job level. This overrides the workspaceId parameter in the connection.properties file.

   • Unsupported open source parameters

     • --deploy-mode

     • --master

     • --repositories

     • --keytab

     • --principal

     • --total-executor-cores

     • --driver-library-path

     • --driver-class-path

     • --supervise

     • --verbose

   spark-sql

   spark-sql is a tool specifically for running SQL queries or scripts directly.

   • Example 1: Run an SQL statement directly

     spark-sql -e "SHOW TABLES"

     This command lists all tables in the current database.

   • Example 2: Run an SQL script file

     spark-sql -f oss://<yourBucketname>/path/to/your/example.sql

     This example uses example.sql. You can click example.sql to download the test file and then upload it to OSS.

     Example content of the example.sql file

     CREATE TABLE IF NOT EXISTS employees (
         id INT,
         name STRING,
         age INT,
         department STRING
     );
     INSERT INTO employees VALUES
     (1, 'Alice', 30, 'Engineering'),
     (2, 'Bob', 25, 'Marketing'),
     (3, 'Charlie', 35, 'Sales');
     SELECT * FROM employees;
     

   The following table describes the parameters.

   Parameter	Example	Description
   -e "<sql>"	-e "SELECT * FROM table"	Executes an inline SQL statement from the command line.
   -f <path>	-f oss://path/script.sql	Executes an SQL script file from a specified path.

Step 4: Query a Spark job

cd emr-serverless-spark-tool-1.16.0
./bin/spark-submit --status <jr-8598aa9f459d****>

cd emr-serverless-spark-tool-1.16.0
./bin/spark-submit --detail <jr-8598aa9f459d****>

(Optional) Step 5: Terminate a Spark job

cd emr-serverless-spark-tool-1.16.0
./bin/spark-submit --kill <jr-8598aa9f459d****>