All Products
Search

This Product

    DataWorks:CDH Spark node

    Document Center

    DataWorks:CDH Spark node

    Last Updated:Oct 09, 2026

    Spark is a general-purpose big data analytics engine known for high performance, ease of use, and broad applicability. It supports complex in-memory analysis and large, low-latency analytics applications. DataWorks provides CDH Spark nodes so you can develop Spark tasks and schedule them periodically in DataWorks. This topic describes how to configure and use a CDH Spark node.

    Prerequisites

    • You have created an Alibaba Cloud CDH cluster and associated it with a DataWorks workspace. For more information, see Associate a CDH compute engine.

      Important

      Spark is installed on the CDH cluster, and the Spark information was configured when the cluster was bound.

    • (Optional, for RAM users) The RAM user for task development has been added to the corresponding workspace and granted the Development or Workspace Administrator role. The Workspace Administrator role has extensive permissions, so grant it with caution. For more information about adding members, see Add members to a workspace.

      Note

      If you use an Alibaba Cloud account, you can skip this step.

    • You have configured a Hive data source in DataWorks and verified that the data source passed the connectivity test. For more information, see Manage data sources.

    Prepare: Develop a Spark task and obtain its JAR package

    Before scheduling a CDH Spark task with DataWorks, develop and compile the Spark task code in CDH to generate a JAR package. For instructions on developing CDH Spark tasks, see Overview.

    Create a CDH JAR resource

    Upload the task JAR package to DataWorks so you can periodically schedule the Spark task on CDH through DataWorks.

    1. For details, see Resource Management Configuration. You can upload the JAR package from your local computer to the directory that stores JAR resources. Click Click Upload to upload the JAR resource.

    2. Select Storage Path, Data Sources, and Resource Group.

    3. Click Save to save.

    Create a node

    For instructions, see Create a node.

    Develop the node

    On the CDH Spark node editing page, perform the following development steps.

    Reference a CDH JAR resource

    1. Open the created CDH Spark node and stay on the code editing page.

    2. Find the resource to reference under Resource Management in the left-side navigation pane, right-click it, and select Insert Resource Path.

    3. After you reference the resource, a statement in the format ##@resource_reference{""} appears on the CDH Spark node code editing page. This indicates that the code resource is referenced successfully. Then run the following command. The resource package, bucket name, path, and other values in the command are examples. Replace them with your actual values.

    ##@resource_reference{"spark_examples_2.11_2.4.0.jar"}
spark_examples_2.11_2.4.0.jar

    Edit CDH Spark node code

    Modify the CDH Spark node code to include the Spark Submit command. The following example shows the modified code:

    Important

    CDH Spark node code does not support comment statements. Rewrite the task code based on the following example. Do not add comments arbitrarily, or the node will fail when it runs.

    ##@resource_reference{"spark-examples_2.11-2.4.0.jar"}
spark-submit --class org.apache.spark.examples.SparkPi --master yarn  spark-examples_2.11-2.4.0.jar 100
    Note

    • org.apache.spark.examples.SparkPi: The main class of the task in the JAR package that you compiled.

    • spark-examples_2.11-2.4.0.jar: The name of the CDH JAR resource that you uploaded.

    Debug the node

    1. Under Run Configuration > Compute Resource, configure Compute Resource and Resource Group.

      1. For the compute resource, select the name of the CDH cluster registered in DataWorks.

      2. For the resource group, select a scheduling resource group that passed the connectivity test with the data source. For details, see Network connectivity solutions.

      Note

      You can also adjust CUs for Scheduling based on the resources required by the task. The default CU is 0.5.

    2. In the toolbar at the top of the node editing page, click Run to run the task.

    Next steps

    • Configure node scheduling: If you need to run a node periodically, configure its Scheduling Policy in the Scheduling Settings panel on the right.

    • Publish a node: To run a task in the production environment, click the image icon to publish the node. A node runs on schedule only after it is published to the production environment.

    • Task O&M: After a task is published, monitor its periodic runs in the Operation Center. For more information, see Get started with Operation Center.