Spark SQL job configuration

Note By default, the Spark SQL mode to submit a job is Yarn mode.

Procedure

  1. Log on to the Alibaba Cloud E-MapReduce Console to enter the Cluster List page with primary account.
  2. Click the Data Platform tab on the top to enter the Project List page.
  3. Click Design Workflow of the specified project in the Operation column.
  4. On the left side of the Job Editing page, right-click on the folder you want to operate and select New Job.
  5. In the New Job dialog box, enter the job name, job description.

    Once the job type is selected, it cannot be modified.

  6. Click OK.
    Note You can also create subfolder, rename folder, and delete folder by right-clicking on the folder.
  7. Select the Spark SQL job type to create a Spark SQL job. This type of job is submitted in the background by using the following process:
    spark-sql [options] [cli option]
  8. Enter the parameters in the Content box with parameters subsequent to Spark SQL commands.
    • -e option
      Directly write running SQL for -e options by inputting it into the Parameters box of the job, for example:
      -e "show databases;"
    • -f option
      -f options can be used to specify a Spark SQL script file. Loading well prepared Spark SQL script files on OSS can give more flexibility. We recommend that you use this operation mode, for example:
      -f ossref://your-bucket/your-spark-sql-script.sql
  9. Click Save to complete Spark SQL job configuration.