All Products
Search
Document Center

DataWorks:Create an EMR Spark node

Last Updated:Jan 23, 2024

Spark is a general-purpose big data analytics engine. Spark features high performance, ease of use, and widespread use. You can use Spark to perform complex memory data analysis and build large, low-latency data analysis applications. For more information about Spark, see Overview. DataWorks provides E-MapReduce (EMR) Spark nodes that you can use to develop and periodically schedule Spark tasks in DataWorks. This topic describes how to create an EMR Spark node and provides examples on how the features of an EMR Spark node work.

Background information

Prerequisites

  • A workflow is created in DataStudio.

    Development operations in different types of compute engines are performed based on workflows in DataStudio. Therefore, before you create a node, you must create a workflow. For more information, see Create a workflow.

  • An Alibaba Cloud EMR cluster is created and registered to DataWorks.

    Before you create an EMR node and use the EMR node to develop EMR tasks in DataWorks, you must register a desired EMR cluster to a DataWorks workspace. For more information, see Register an EMR cluster to DataWorks.

  • (Required if you use a RAM user to develop tasks) The RAM user is added to the DataWorks workspace as a member and is assigned the Development or Workspace Administrator role. The Workspace Administrator role has more permissions than necessary. Exercise caution when you assign the Workspace Administrator role. For more information about how to add a member, see Add workspace members and assign roles to them.

  • An exclusive resource group for scheduling is purchased and configured. The configurations include association with a workspace and network configuration. For more information, see Create and use an exclusive resource group for scheduling.

Limits

  • Tasks on this type of node cannot be run on the shared resource group for scheduling. You can run the tasks only on exclusive resource groups that are created after December 1, 2023.

  • If you want to manage metadata for a DataLake or custom cluster in DataWorks, you must configure EMR-HOOK in the cluster first. If you do not configure EMR-HOOK in the cluster, metadata cannot be displayed in real time, audit logs cannot be generated, and data lineages cannot be displayed in DataWorks. In addition, the related EMR governance tasks cannot be run. For more information about how to configure EMR-HOOK, see Use the Hive extension feature to record data lineage and historical access information.

Preparations: Prepare Spark task code and obtain a JAR package

Before you use DataWorks to schedule an EMR Spark task, you must prepare Spark task code in EMR and compile the task code to generate a JAR package. For more information about preparation of Spark task code, see Overview.

Note

You must upload the obtained JAR package to the DataWorks console. This way, DataWorks can periodically schedule EMR Spark tasks.

Create an EMR Spark node

  1. Go to the DataStudio page.

    Log on to the DataWorks console. In the left-side navigation pane, choose Data Modeling and Development > DataStudio. On the page that appears, select the desired workspace from the drop-down list and click Go to DataStudio.

  2. Create a workflow.

  3. If you have an existing workflow, skip this step.

    1. Move the pointer over the 新建 icon and select Create Workflow.

    2. In the Create Workflow dialog box, configure the Workflow Name parameter.

    3. Click Create.

  4. Create an EMR Spark node.

    1. Move the pointer over the image.png icon and choose Create Node > EMR > EMR Spark.

      Alternatively, you can find the desired workflow, right-click the name of the workflow, and then choose Create Node > EMR > EMR Spark.

    2. In the Create Node dialog box, configure the Name, Engine Instance, Node Type, and Path parameters.

      Note

      The node name must be 1 to 128 characters in length and can contain only letters, digits, underscores (_), and periods (.).

    3. Click Confirm. The configuration tab of the EMR Spark node appears.

Develop a Spark task

Use one of the following methods to develop a Spark task on the configuration tab of the EMR Spark node:

Method 1: Reference an OSS resource

The current node can reference an OSS resource by using the OSS REF method. When you run a task on the node, DataWorks automatically loads the OSS resource specified in the node code. This method is commonly used in scenarios in which JAR dependencies are required in EMR tasks or EMR tasks need to depend on scripts. Reference format:

ossref://{endpoint}/{bucket}/{object}
  • endpoint: the endpoint of OSS. If the endpoint parameter is left empty, only a resource in an OSS bucket that resides in the same region as the current EMR cluster can be referenced.

  • bucket: a container that is used to store objects in OSS. Each bucket has a unique name. You can log on to the OSS console to view all buckets within the current logon account.

  • object: a file name or path that is stored in a bucket.

Note

This feature is unavailable for the shared resource group for scheduling. To use this feature on a resource group that is created before December 1, 2023, you must contact technical support to upgrade the configurations of the resource group.

Method 2: Upload and reference an EMR JAR resource

DataWorks allows you to upload a resource from your on-premises machine to DataStudio before you reference the resource. You must obtain and store the JAR package that is generated after the code of a Spark task is compiled in EMR. The method for storing a JAR package varies based on the size of the JAR package. You can upload the JAR package to the DataWorks console as an EMR JAR resource and commit the resource. You can also store the JAR package in Hadoop Distributed File System (HDFS) of EMR.

A JAR package is less than 200 MB in size

  1. Create an EMR JAR resource.

    You can upload the JAR package from your on-premises machine to the DataWorks console as an EMR JAR resource. This way, you can manage the JAR package in the DataWorks console in a visualized manner. After you create an EMR JAR resource, you must commit the resource. For more information, see Create and use an EMR JAR resource.

    image.png

    Note

    The first time you create an EMR JAR resource, you must perform authorization as prompted first if you want the JAR package to be stored in Object Storage Service (OSS) after the JAR package is uploaded.

  2. Reference the EMR JAR resource.

    1. Double-click the name of the created EMR Spark node to go to the configuration tab of the node.

    2. Find the desired EMR JAR resource under Resource in the EMR folder, right-click the resource name, and then select Insert Resource Path.

    3. Resource reference code is automatically added to the configuration tab of the EMR Spark node. Sample code:

      ##@resource_reference{"spark-examples_2.12-1.0.0-SNAPSHOT-shaded.jar"}
      spark-examples_2.12-1.0.0-SNAPSHOT-shaded.jar

      If the automatic addition of the preceding code is successful, the resource is referenced. spark-examples_2.12-1.0.0-SNAPSHOT-shaded.jar is the name of the JAR package that you uploaded.

    4. Rewrite the code of the EMR Spark node and add the spark-submit command. The following sample code is only for reference.

      Important

      You cannot add comments when you write code for an EMR Spark node. If you add comments, an error is reported when you run the EMR Spark node. You can refer to the following sample code to rewrite the code of an EMR Spark node.

      ##@resource_reference{"spark-examples_2.11-2.4.0.jar"}
      spark-submit --class org.apache.spark.examples.SparkPi --master yarn  spark-examples_2.11-2.4.0.jar 100

      Parameter description:

      • org.apache.spark.examples.SparkPi: the main class of the task in the compiled JAR package.

      • spark-examples_2.11-2.4.0.jar: the name of the JAR package that you uploaded.

      • You can keep the settings of other parameters unchanged. You can also run the following command to view the help documentation for using the spark-submit command and modify the spark-submit command based on your business requirements.

        Important
        • If you want to use a parameter that is simplified by running the spark-submit command, such as --executor-memory 2G, in an EMR Spark node, you need to add the parameter to the code of the EMR Spark node.

        • You can use Spark nodes on YARN to submit jobs only if your nodes are in cluster mode.

        • If you commit a node by using spark-submit, we recommend that you set deploy-mode to cluster rather than client.

        spark-submit --help

        image.png

A JAR package is greater than or equal to 200 MB in size

  1. Store the JAR package in HDFS of EMR.

    You cannot upload the JAR package from your on-premises machine to the DataWorks console as a DataWorks resource. We recommend that you store the JAR package in HDFS of EMR and record the storage path of the JAR package. This way, you can reference the JAR package in this path when you use DataWorks to schedule Spark tasks.

  2. Reference the JAR package.

    You can reference the JAR package by specifying the storage path of the JAR package in the code of an EMR Spark node.

    1. Double-click the name of the created EMR Spark node to go to the configuration tab of the node.

    2. Write the spark-submit command. Example:

      spark-submit --master yarn
      --deploy-mode cluster
      --name SparkPi
      --driver-memory 4G
      --driver-cores 1
      --num-executors 5
      --executor-memory 4G
      --executor-cores 1
      --class org.apache.spark.examples.JavaSparkPi
      hdfs:///tmp/jars/spark-examples_2.11-2.4.8.jar 100

      Parameter description:

      • hdfs:///tmp/jars/spark-examples_2.11-2.4.8.jar: the storage path of the JAR package in HDFS.

      • org.apache.spark.examples.JavaSparkPi: the main class of the task in the compiled JAR package.

      • Other parameters are configured in the EMR cluster that is used. You can modify the parameters based on your business requirements. You can also run the following command to view the help documentation for using the spark-submit command and modify the spark-submit command based on your business requirements.

        Important
        • If you want to use a parameter that is simplified by running the spark-submit command, such as --executor-memory 2G, in an EMR Spark node, you need to add the parameter to the code of the EMR Spark node.

        • You can use Spark nodes on YARN to submit jobs only if your nodes are in cluster mode.

        • If you commit a node by using spark-submit, we recommend that you set deploy-mode to cluster rather than client.

        spark-submit --help

        image.png

Configure advanced parameters

The following table describes the advanced parameters that can be configured for different types of EMR clusters. For more information about how to configure the parameters, see Spark Configuration.

DataLake cluster or custom cluster: created on the EMR on ECS page

Advanced parameter

Description

"queue"

The scheduling queue to which jobs are committed. Default value: default.

If you have configured a workspace-level YARN queue when you register an EMR cluster to a DataWorks workspace, the following configurations apply:

  • If you select Yes for Global Settings Task Precedence, the YARN queue that is configured when you register the EMR cluster is used to run Spark tasks.

  • If you do not select Yes for Global Settings Task Precedence, the YARN queue that is configured for the EMR Spark node is used to run Spark tasks.

For information about EMR YARN, see YARN schedulers. For information about queue configuration when you register an EMR cluster, see Configure a global YARN queue.

"priority"

The priority. Default value: 1.

Others

  • You can also add a custom Spark parameter for the EMR Spark node on the Advanced Settings tab, such as "spark.eventLog.enabled" : false. When you commit the code of the EMR Spark node, DataWorks adds the custom parameter to the code in the --conf key=value format.

  • You can also configure global Spark parameters. For more information, see Configure global Spark parameters.

Hadoop cluster: created on the EMR on ECS page

Advanced parameter

Description

"queue"

The scheduling queue to which jobs are committed. Default value: default.

If you have configured a workspace-level YARN queue when you register an EMR cluster to a DataWorks workspace, the following configurations apply:

  • If you select Yes for Global Settings Task Precedence, the YARN queue that is configured when you register the EMR cluster is used to run Spark tasks.

  • If you do not select Yes for Global Settings Task Precedence, the YARN queue that is configured for the EMR Spark node is used to run Spark tasks.

For information about EMR YARN, see YARN schedulers. For information about queue configuration when you register an EMR cluster, see Configure a global YARN queue.

"priority"

The priority. Default value: 1.

"FLOW_SKIP_SQL_ANALYZE"

The manner in which SQL statements are executed. Valid values:

  • true: Multiple SQL statements are executed at a time.

  • false: Only one SQL statement is executed at a time.

Note

This parameter is available only for testing in the development environment of a DataWorks workspace.

"USE_GATEWAY"

Specifies whether to use a gateway cluster to commit jobs on the current node. Valid values:

  • true: Use a gateway cluster to commit jobs.

  • false: Use no gateway cluster to commit jobs. Jobs are automatically committed to the master node.

Note

If the EMR cluster to which the node belongs is not associated with a gateway cluster but the USE_GATEWAY parameter is set to true, jobs may fail to be committed.

Others

  • You can also add a custom Spark parameter for the EMR Spark node on the Advanced Settings tab, such as "spark.eventLog.enabled" : false. When you commit the code of the EMR Spark node, DataWorks adds the custom parameter to the code in the --conf key=value format.

  • You can also configure global Spark parameters. For more information, see Configure global Spark parameters.

Spark cluster: created on the EMR on ACK page

  • You can also add a custom Spark parameter for the EMR Spark node on the Advanced Settings tab, such as "spark.eventLog.enabled" : false. When you commit the code of the EMR Spark node, DataWorks adds the custom parameter to the code in the --conf key=value format.

  • You can also configure global Spark parameters. For more information, see Configure global Spark parameters.

Configure scheduling properties

If you want the system to periodically run the node, you can click Properties in the right-side navigation pane on the configuration tab of the node to configure properties for the node based on your business requirements.

Debug the node

  1. Optional. Select a resource group and assign custom parameters to variables.

  2. Save and execute the SQL statements.

    In the top toolbar, click the Save icon to save the SQL statements. Then, click the Run icon to execute the SQL statements.

  3. Optional. Perform smoke testing.

    You can perform smoke testing on the node in the development environment when you commit the node or after you commit the node. For more information, see Perform smoke testing.

What to do next

  1. Commit and deploy the node.

    1. Click the Save icon in the top toolbar to save the node.

    2. Click the Submit icon in the top toolbar to commit the node.

    3. In the Commit Node dialog box, configure the Change description parameter.

    4. Click OK.

    If you use a workspace in standard mode, you must deploy the node in the production environment after you commit the node. On the left side of the top navigation bar, click Deploy. For more information, see Deploy nodes.

  2. View the node.

    1. Click Operation Center in the upper-right corner of the configuration tab of the node to go to Operation Center in the production environment.

    2. View the scheduled node. For more information, see View and manage auto triggered nodes.

    To view more information about the node, click Operation Center in the top navigation bar of the DataStudio page. For more information, see Overview.

References