DataWorks:EMR Spark node

Option 1a: Upload a JAR smaller than 500 MB to DataWorks

1. Create an EMR JAR resource. After creating the resource, submit it. For more information, see Create and use an EMR resource.

   1. Upload the JAR package from your Local machine to the directory where JAR resources are stored. For more information, see Resource management.

   2. Click Upload.

   3. Select the Storage Path, Data Source, and Resource Group.

   4. Click Save.

   

2. Reference the EMR JAR resource in the node.

   1. Open the created EMR Spark node to go to the code editor.

   2. In the left-side navigation pane, right-click the resource you want to reference and select Reference Resource.

   3. A reference is automatically added to the code editor:

      ##@resource_reference{"spark-examples_2.12-1.0.0-SNAPSHOT-shaded.jar"}
      spark-examples_2.12-1.0.0-SNAPSHOT-shaded.jar

   4. Add the spark-submit command. For example:

      ##@resource_reference{"spark-examples_2.11-2.4.0.jar"}
      spark-submit --class org.apache.spark.examples.SparkPi --master yarn spark-examples_2.11-2.4.0.jar 100

      Parameter	Description
      org.apache.spark.examples.SparkPi	The main class of the job in your compiled JAR package
      spark-examples_2.11-2.4.0.jar	The name of the EMR JAR resource you uploaded

      Additional notes:

      • EMR Spark nodes only support YARN cluster mode. Set --deploy-mode to cluster, not client. In cluster mode, the Spark driver runs inside the cluster; in client mode, it runs on the submitting machine, which is not supported for scheduled jobs in DataWorks.

      • To add simplified parameters, include them in the command. For example: --executor-memory 2G.

      • Run spark-submit --help to view all available parameters.

Option 1b: Reference a JAR 500 MB or larger from HDFS

If the JAR is 500 MB or larger, you cannot upload it from your local machine. Store the JAR in HDFS on EMR and reference it by path instead.

EMR on ACK and EMR Serverless Spark clusters do not support uploading resources to HDFS. Use Option 2 (OSS REF) for those cluster types.

1. Create an EMR JAR resource in DataWorks.

   1. Upload the JAR package from your Local machine to the directory where JAR resources are stored. For more information, see Resource management.

   2. Click Upload.

   3. Select the Storage Path, Data Source, and Resource Group.

   4. Click Save.

   

2. Reference the JAR by its HDFS path.

   1. Double-click the created EMR Spark node to open the code editor.

   2. Write the spark-submit command with the HDFS path:

      spark-submit --master yarn \
      --deploy-mode cluster \
      --name SparkPi \
      --driver-memory 4G \
      --driver-cores 1 \
      --num-executors 5 \
      --executor-memory 4G \
      --executor-cores 1 \
      --class org.apache.spark.examples.JavaSparkPi \
      hdfs:///tmp/jars/spark-examples_2.11-2.4.8.jar 100

      Parameter	Description
      hdfs:///tmp/jars/spark-examples_2.11-2.4.8.jar	The actual path of the JAR package in HDFS
      org.apache.spark.examples.JavaSparkPi	The main class of the job in your compiled JAR package

      Additional notes:

      • Configure other parameters based on your actual cluster settings.

      • EMR Spark nodes only support YARN cluster mode. Set --deploy-mode to cluster, not client.

      • To add simplified parameters, include them in the command. For example: --executor-memory 2G.

      • Run spark-submit --help to view all available parameters.

Option 2: Reference an OSS resource directly

Reference an OSS resource in the node using OSS REF. When the EMR node runs, DataWorks automatically loads the referenced OSS resource for the job. Use this method when running JAR dependencies in an EMR job or when an EMR job depends on a script.

1. Develop the JAR resource.

   1. Find the required dependencies at /usr/lib/emr/spark-current/jars/ on the master node of your EMR cluster. The following example uses Spark 3.4.2. In your IDEA project, add the pom dependencies and reference the plug-ins.

      Add pom dependencies

      <dependencies>
              <dependency>
                  <groupId>org.apache.spark</groupId>
                  <artifactId>spark-core_2.12</artifactId>
                  <version>3.4.2</version>
              </dependency>
              <!-- Apache Spark SQL -->
              <dependency>
                  <groupId>org.apache.spark</groupId>
                  <artifactId>spark-sql_2.12</artifactId>
                  <version>3.4.2</version>
              </dependency>
      </dependencies>

      Reference plug-ins

      <build>
          <sourceDirectory>src/main/scala</sourceDirectory>
          <testSourceDirectory>src/test/scala</testSourceDirectory>
          <plugins>
              <plugin>
                  <groupId>org.apache.maven.plugins</groupId>
                  <artifactId>maven-compiler-plugin</artifactId>
                  <version>3.7.0</version>
                  <configuration>
                      <source>1.8</source>
                      <target>1.8</target>
                  </configuration>
              </plugin>
              <plugin>
                  <artifactId>maven-assembly-plugin</artifactId>
                  <configuration>
                      <descriptorRefs>
                          <descriptorRef>jar-with-dependencies</descriptorRef>
                      </descriptorRefs>
                  </configuration>
                  <executions>
                      <execution>
                          <id>make-assembly</id>
                          <phase>package</phase>
                          <goals>
                              <goal>single</goal>
                          </goals>
                      </execution>
                  </executions>
              </plugin>
              <plugin>
                  <groupId>net.alchim31.maven</groupId>
                  <artifactId>scala-maven-plugin</artifactId>
                  <version>3.2.2</version>
                  <configuration>
                      <recompileMode>incremental</recompileMode>
                  </configuration>
                  <executions>
                      <execution>
                          <goals>
                              <goal>compile</goal>
                              <goal>testCompile</goal>
                          </goals>
                          <configuration>
                              <args>
                                  <arg>-dependencyfile</arg>
                                  <arg>${project.build.directory}/.scala_dependencies</arg>
                              </args>
                          </configuration>
                      </execution>
                  </executions>
              </plugin>
          </plugins>
      </build>

   2. Write the Scala job code. The following is an example:

      package com.aliyun.emr.example.spark
      
      import org.apache.spark.sql.SparkSession
      
      object SparkMaxComputeDemo {
          def main(args: Array[String]): Unit = {
              // Create a SparkSession.
              val spark = SparkSession.builder()
                  .appName("HelloDataWorks")
                  .getOrCreate()
              // Print the Spark version.
              println(s"Spark version: ${spark.version}")
          }
      }

   3. Generate a JAR package from the Scala code. The generated JAR package in this example is SparkWorkOSS-1.0-SNAPSHOT-jar-with-dependencies.jar.

2. Upload the JAR to OSS.

   1. Log on to the OSS console. In the navigation pane, click Bucket List.

   2. Click the name of the destination bucket to go to the File Management page. This example uses the onaliyun-bucket-2 bucket.

   3. Click Create Directory. Set the Directory Name to emr/jars.

   4. Go to the directory and click Upload File. In the Files to Upload section, click Scan Files, select SparkWorkOSS-1.0-SNAPSHOT-jar-with-dependencies.jar, and then click Upload File.

3. Reference the JAR resource.

   1. On the editing page of the EMR Spark node, write the spark-submit command referencing the OSS path:

      spark-submit --class com.aliyun.emr.example.spark.SparkMaxComputeDemo --master yarn ossref://onaliyun-bucket-2/emr/jars/SparkWorkOSS-1.0-SNAPSHOT-jar-with-dependencies.jar

      The ossref:// scheme tells DataWorks to load the resource from OSS before running the job. The format is ossref://{endpoint}/{bucket}/{object}.

      Parameter	Description
      class	The full name of the main class to run
      master	The mode in which the Spark application runs
      endpoint	The public endpoint for OSS. If left blank, the OSS bucket must be in the same region as the EMR cluster
      bucket	The OSS bucket name. Log on to the OSS console to view all buckets under your account
      object	The specific object (file name or path) stored in the bucket

   2. Click the icon and select the Serverless resource group to run the job. After the job completes, record the applicationId printed in the console — for example, application_1730367929285_xxxx.

   3. To view the result, create an EMR Shell node and run:

      yarn logs -applicationId application_1730367929285_xxxx

DataLake and custom (ECS)

Spark (ACK)

Hadoop (ECS)

EMR Serverless Spark

For parameter details, see Set parameters for submitting a Spark job.