E-MapReduce:Submit jobs through Alibaba Cloud DataWorks

Method 1 (recommended): Upload a JAR resource and reference it

Use this method when the JAR package is less than 200 MB.

Upload the JAR package to DataWorks as an EMR JAR resource, then reference it in the node code.

1. Create an EMR JAR resource.

Upload the JAR package from your machine to the DataWorks console. After creating the resource, commit it. See Create and use an EMR resource.

On the first upload, authorize DataWorks to store the JAR package in OSS as prompted.

2. Reference the JAR resource in the node.

1. Open the EMR Spark node configuration tab.

2. Under Resource in the EMR folder, right-click the JAR resource and select Insert Resource Path. DataWorks automatically adds a resource reference line to the editor:

   ##@resource_reference{"spark-examples_2.12-1.0.0-SNAPSHOT-shaded.jar"}
   spark-examples_2.12-1.0.0-SNAPSHOT-shaded.jar

3. Add the spark-submit command below the reference line. Example:

   • org.apache.spark.examples.SparkPi: the main class in the JAR package.

   • spark-examples_2.11-2.4.0.jar: the name of the uploaded JAR package.

   - Spark nodes on YARN only support cluster mode. Set --deploy-mode cluster (not client): in cluster mode, the driver runs on the cluster and its logs appear in the YARN container logs; in client mode, the driver runs on the submitting machine, which is not supported for scheduled YARN jobs. - To add spark-submit options like --executor-memory 2G, include them directly in the node code. - Run spark-submit --help to view all available options.

   ##@resource_reference{"spark-examples_2.11-2.4.0.jar"}
   spark-submit --class org.apache.spark.examples.SparkPi --master yarn spark-examples_2.11-2.4.0.jar 100

   Key components:

   

Method 2: Reference a JAR file stored in HDFS

Use this method when the JAR package is 200 MB or larger. Store the JAR in HDFS and reference it by path in the spark-submit command.

1. Upload the JAR to HDFS on your EMR cluster and record the storage path.

2. Open the EMR Spark node configuration tab and write the spark-submit command with the HDFS path. Example:

   • hdfs:///tmp/jars/spark-examples_2.11-2.4.8.jar: the HDFS path of the JAR package.

   • org.apache.spark.examples.JavaSparkPi: the main class in the JAR package.

   • Other parameters are set based on your cluster configuration.

   Important

   - Spark nodes on YARN only support cluster mode (--deploy-mode cluster). - To add spark-submit options like --executor-memory 2G, include them directly in the node code.

   spark-submit --master yarn
   --deploy-mode cluster
   --name SparkPi
   --driver-memory 4G
   --driver-cores 1
   --num-executors 5
   --executor-memory 4G
   --executor-cores 1
   --class org.apache.spark.examples.JavaSparkPi
   hdfs:///tmp/jars/spark-examples_2.11-2.4.8.jar 100

   Key parameters:

   

Method 3: Reference a JAR file stored in OSS (OSS REF)

Use the OSS REF method to reference a JAR file stored in OSS. When the task runs, DataWorks automatically downloads the OSS resource. This method works for JAR dependencies and scripts.

1. Build the JAR package.

Check the required code dependencies in /usr/lib/emr/spark-current/jars/ on the EMR cluster's master node. The following example uses Spark 3.4.2.

Add POM dependencies to your IntelliJ IDEA project:

<dependencies>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.12</artifactId>
        <version>3.4.2</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.12</artifactId>
        <version>3.4.2</version>
    </dependency>
</dependencies>

Add build plugins:

<build>
    <sourceDirectory>src/main/scala</sourceDirectory>
    <testSourceDirectory>src/test/scala</testSourceDirectory>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.7.0</version>
            <configuration>
                <source>1.8</source>
                <target>1.8</target>
            </configuration>
        </plugin>
        <plugin>
            <artifactId>maven-assembly-plugin</artifactId>
            <configuration>
                <descriptorRefs>
                    <descriptorRef>jar-with-dependencies</descriptorRef>
                </descriptorRefs>
            </configuration>
            <executions>
                <execution>
                    <id>make-assembly</id>
                    <phase>package</phase>
                    <goals>
                        <goal>single</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <groupId>net.alchim31.maven</groupId>
            <artifactId>scala-maven-plugin</artifactId>
            <version>3.2.2</version>
            <configuration>
                <recompileMode>incremental</recompileMode>
            </configuration>
            <executions>
                <execution>
                    <goals>
                        <goal>compile</goal>
                        <goal>testCompile</goal>
                    </goals>
                    <configuration>
                        <args>
                            <arg>-dependencyfile</arg>
                            <arg>${project.build.directory}/.scala_dependencies</arg>
                        </args>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

Sample Scala code:

package com.aliyun.emr.example.spark

import org.apache.spark.sql.SparkSession

object SparkMaxComputeDemo {
  def main(args: Array[String]): Unit = {
    // Create a Spark session.
    val spark = SparkSession.builder()
      .appName("HelloDataWorks")
      .getOrCreate()

    // Print the Spark version.
    println(s"Spark version: ${spark.version}")
  }
}

Package the code. This example produces SparkWorkOSS-1.0-SNAPSHOT-jar-with-dependencies.jar.

2. Upload the JAR to OSS.

1. Log on to the OSS console and click Buckets in the left-side navigation pane.

2. Click your bucket (for example, onaliyun-bucket-2) to open the Objects page.

3. Click Create Directory, set Directory Name to emr/jars, and click OK.

4. Go to the new directory, click Upload Object, select SparkWorkOSS-1.0-SNAPSHOT-jar-with-dependencies.jar, and click Upload Object.

3. Reference the JAR in the node.

On the EMR Spark node configuration tab, write the spark-submit command using the ossref:// protocol:

spark-submit --class com.aliyun.emr.example.spark.SparkMaxComputeDemo --master yarn ossref://onaliyun-bucket-2/emr/jars/SparkWorkOSS-1.0-SNAPSHOT-jar-with-dependencies.jar

Key parameters:

4. Run the task and view results.

1. Click the icon and select a serverless resource group to run the task. After it completes, note the application ID displayed in the console (for example, application_1730367929285_xxxx).

2. Create an EMR Shell node and run the following command to view logs:

   yarn logs -applicationId application_1730367929285_xxxx