DataWorks:Create an EMR Spark node

This node can directly reference OSS resources using OSS REF. When you run an EMR node, DataWorks automatically loads the OSS resources from the code to your local machine. This method is often used in scenarios where an EMR task needs to run JAR dependencies or depends on scripts.

1. Develop the JAR resource.

   1. Prepare code dependencies.

      Go to your EMR cluster. In the /usr/lib/emr/spark-current/jars/ path of the cluster's master node, view your required code dependencies. The following example uses Spark 3.4.2. You must open the created IntelliJ IDEA project, add pom dependencies, and reference the related plug-ins.

      Add pom dependencies

      <dependencies>
              <dependency>
                  <groupId>org.apache.spark</groupId>
                  <artifactId>spark-core_2.12</artifactId>
                  <version>3.4.2</version>
              </dependency>
              <!-- Apache Spark SQL -->
              <dependency>
                  <groupId>org.apache.spark</groupId>
                  <artifactId>spark-sql_2.12</artifactId>
                  <version>3.4.2</version>
              </dependency>
      </dependencies>

      Reference related plug-ins

      <build>
              <sourceDirectory>src/main/scala</sourceDirectory>
              <testSourceDirectory>src/test/scala</testSourceDirectory>
              <plugins>
                  <plugin>
                      <groupId>org.apache.maven.plugins</groupId>
                      <artifactId>maven-compiler-plugin</artifactId>
                      <version>3.7.0</version>
                      <configuration>
                          <source>1.8</source>
                          <target>1.8</target>
                      </configuration>
                  </plugin>
                  <plugin>
                      <artifactId>maven-assembly-plugin</artifactId>
                      <configuration>
                          <descriptorRefs>
                              <descriptorRef>jar-with-dependencies</descriptorRef>
                          </descriptorRefs>
                      </configuration>
                      <executions>
                          <execution>
                              <id>make-assembly</id>
                              <phase>package</phase>
                              <goals>
                                  <goal>single</goal>
                              </goals>
                          </execution>
                      </executions>
                  </plugin>
                  <plugin>
                      <groupId>net.alchim31.maven</groupId>
                      <artifactId>scala-maven-plugin</artifactId>
                      <version>3.2.2</version>
                      <configuration>
                          <recompileMode>incremental</recompileMode>
                      </configuration>
                      <executions>
                          <execution>
                              <goals>
                                  <goal>compile</goal>
                                  <goal>testCompile</goal>
                              </goals>
                              <configuration>
                                  <args>
                                      <arg>-dependencyfile</arg>
                                      <arg>${project.build.directory}/.scala_dependencies</arg>
                                  </args>
                              </configuration>
                          </execution>
                      </executions>
                  </plugin>
              </plugins>
          </build>

   2. Write sample code.

      package com.aliyun.emr.example.spark
      
      import org.apache.spark.sql.SparkSession
      
      object SparkMaxComputeDemo {
        def main(args: Array[String]): Unit = {
          // Create a SparkSession
          val spark = SparkSession.builder()
            .appName("HelloDataWorks")
            .getOrCreate()
      
          // Print the Spark version
          println(s"Spark version: ${spark.version}")
        }
      }

   3. Package the code into a JAR file.

      After you edit and save the Scala code, package it into a JAR file. The JAR package generated in this example is SparkWorkOSS-1.0-SNAPSHOT-jar-with-dependencies.jar.

2. Upload the JAR resource.

   1. After you finish code development, log on to the OSS console. In the navigation pane on the left, click Buckets.

   2. Click the name of the target bucket to open the Object Management page.

      The bucket used in this example is onaliyun-bucket-2.

   3. Click Create Directory to create a JAR resource directory.

      Set Directory Name to emr/jars to create a directory for your JAR resources.

   4. Upload the JAR resource to the directory.

      Navigate to the directory and click Upload Object. In the Files to Upload area, click Select Files. Select the SparkWorkOSS-1.0-SNAPSHOT-jar-with-dependencies.jar file and click Upload Object.

3. Reference the JAR resource.

   1. Edit the code to reference the JAR resource.

      On the editor page of the created EMR Spark node, edit the code to reference the JAR resource.

      spark-submit --class com.aliyun.emr.example.spark.SparkMaxComputeDemo --master yarn ossref://onaliyun-bucket-2/emr/jars/SparkWorkOSS-1.0-SNAPSHOT-jar-with-dependencies.jar

      Parameter description:

      Parameter	Description
      class	The full name of the main class to run.
      master	The running mode of the Spark application.
      ossref file path	The format is ossref://{endpoint}/{bucket}/{object} endpoint: The endpoint for accessing OSS. If the endpoint is empty, you can only use the OSS in the same region as the EMR cluster. This means the OSS bucket must be in the same region as the EMR cluster. Bucket: The OSS container used to store objects. Each bucket has a unique name. Log on to the OSS console to view all buckets under the current account. object: A specific object (file name or path) stored in a bucket.

   2. Run the EMR Spark node task.

      After you finish editing, click the icon. Select the Serverless resource group that you created to run the EMR Spark node. After the task is complete, record the applicationId that is printed in the console, for example, application_1730367929285_xxxx.

   3. View the result.

      Create an EMR Shell node and run the yarn logs -applicationId application_1730367929285_xxxx command in the node to view the result: