E-MapReduce:Spark SQL job configuration

Procedure

1. Go to the Projects page in Data Platform.

   1. Log in to the E-MapReduce console with your Alibaba Cloud account.

   2. In the top navigation bar, select the region and resource group.
      

   3. Click the Data Development tab.

2. In the row of the project you want to edit, click Edit Job.

3. Create a Spark SQL job.
   1. On the left side of the page, right-click the target folder and select Create Job.
   2. In the Create Job dialog box, enter a Job Name and Description. From the Job Type drop-down list, select Spark SQL.
      Note By default, the platform submits Spark SQL jobs in yarn-client mode.

      The platform submits this type of job with the following command:

      spark-sql [options] [cli options] {SQL_CONTENT}                

      The following table describes the parameters.

      Parameter	Description
      options	On the Advanced Settings tab of the Job Settings panel, click the icon in the Environment Variable row to add the SPARK_CLI_PARAMS environment variable. Example: SPARK_CLI_PARAMS="--executor-memory 1g --executor-cores".
      cli options	Examples: -e <quoted-query-string> : runs the SQL query string in quotation marks. -f <filename>: runs the SQL statements in the specified file.
      SQL_CONTENT	The SQL statements you enter.

   3. Click OK.
4. Edit the job content.
   1. In the Content field, enter the Spark SQL statements.
      Example:

      -- Example SQL statements.
      -- SQL statements cannot exceed 64 KB.
      show databases;
      show tables;
      -- The system automatically appends 'limit 2000' to SELECT statements.
      select * from test1;

   2. Click Save.