E-MapReduce:Configure a Spark SQL job

Prerequisites

Before you begin, make sure you have:

• An EMR project. See Manage projects to create one.

How it works

EMR submits Spark SQL jobs using the following command:

spark-sql [options] [cli options] {SQL_CONTENT}

By default, a Spark SQL job is submitted in yarn-client mode.

Create a Spark SQL job

1. Log on to the Alibaba Cloud EMR console.

2. In the top navigation bar, select the region where your cluster resides and select a resource group.

3. Click the Data Platform tab.

4. In the Projects section, find the project you want to edit and click Edit Job in the Actions column.

5. In the Edit Job pane on the left, right-click the folder where you want to add the job and select Create Job.

6. In the Create Job dialog box, fill in the following fields and click OK:

   • Name: Enter a name for the job.

   • Description: Add a description.

   • Job Type: Select SparkSQL from the drop-down list.

7. In the Content field, enter your Spark SQL statements. For example:

   Note SQL statements cannot exceed 64 KB. EMR automatically appends LIMIT 2000 to SELECT statements.

   -- Show all databases
   show databases;
   
   -- Show all tables
   show tables;
   
   -- Query a table (LIMIT 2000 is automatically added to SELECT statements)
   select * from test1;

8. Click Save.

Pass Spark configuration flags

To customize executor resources or other Spark settings for this job, set the SPARK_CLI_PARAMS environment variable:

1. On the job page, click Job Settings in the upper-right corner.

2. In the Job Settings panel, click the Advanced Settings tab.

3. In the Environment Variables section, click the + icon and add the variable. For example:

   SPARK_CLI_PARAMS="--executor-memory 1g --executor-cores"

These flags are passed as the options argument when EMR submits the job.

What's next

After saving the job, you can:

• Add it to a workflow and schedule it to run automatically.

• Run it manually from the job page to verify the SQL output.

• Check execution logs in the EMR console to confirm the job completed successfully.