This topic describes how to configure a Spark SQL job.

Prerequisites

A project is created. For more information, see Manage projects.

Procedure

  1. Create a job.
    1. You have logged on to the Alibaba Cloud EMR console by using your Alibaba Cloud account.
    2. In the top navigation bar, select the region where your cluster resides and select a resource group based on your business requirements.
    3. Click the Data Platform tab.
    4. In the Projects section of the page that appears, find the project you want to edit and click Edit Job in the Actions column.
    5. In the Edit Job pane on the left, right-click the folder on which you want to perform operations and select Create Job.
  2. Configure the job.
    1. In the Create Job dialog box, specify Name and Description, and select SparkSQL from the Job Type drop-down list.
      Note By default, a Spark SQL job is submitted in yarn-client mode.
      You can use the following command syntax to submit a Spark SQL job:
      spark-sql [options] [cli options] {SQL_CONTENT}                
      Parameter description:
      • options: the setting of the SPARK_CLI_PARAMS parameter that you configure by performing the following operations: Click Job Settings in the upper-right corner of the job page. In the Job Settings panel, click the Advanced Settings tab. Click the add icon in the Environment Variables section and add the setting of the SPARK_CLI_PARAMS parameter, for example, SPARK_CLI_PARAMS="--executor-memory 1g --executor-cores".
      • cli options: For example, -e <quoted-query-string> indicates that the SQL statements within quotation marks are executed. -f <filename> indicates that the SQL statements in the file are executed.
      • SQL_CONTENT: the SQL statements that you enter.
    2. Click OK.
    3. Enter the Spark SQL statements in the Content field.
      Example:
      -- SQL statement example
      -- The size of SQL statements cannot exceed 64 KB.
      show databases;
      show tables;
      -- LIMIT 2000 is automatically used for the SELECT statement.
      select * from test1;
  3. Click Save.