DataWorks:Create an EMR Spark Streaming node

Reference an EMR JAR resource (DataLake clusters)

If your EMR Spark Streaming node uses a DataLake cluster, follow these steps to reference an EMR JAR resource.

Before referencing, create the EMR JAR resource. The first time you use an EMR JAR resource, click Authorize to grant DataWorks access.

If the resources are too large to upload through the DataWorks console, store them in Hadoop Distributed File System (HDFS) and reference them directly in the node code. Example:

spark-submit --master yarn
--deploy-mode cluster
--name SparkPi
--driver-memory 4G
--driver-cores 1
--num-executors 5
--executor-memory 4G
--executor-cores 1
--class org.apache.spark.examples.JavaSparkPi
hdfs:///tmp/jars/spark-examples_2.11-2.4.8.jar 100

1. Open the EMR Spark Streaming node to display its configuration tab.

2. Under Resource in the EMR folder, right-click the resource name and select Insert Resource Path. When the ##@resource_reference{""} clause appears on the configuration tab, the resource is referenced.

3. Write your task code. Replace the placeholder values with actual values for your environment.

   ##@resource_reference{"examples-1.2.0-shaded.jar"}
   --master yarn-cluster --executor-cores 2 --executor-memory 2g --driver-memory 1g --num-executors 2 --class com.aliyun.emr.example.spark.streaming.JavaLoghubWordCount examples-1.2.0-shaded.jar <logService-project> <logService-store> <group> <endpoint> <access-key-id> <access-key-secret>

Write the spark-submit code

On the configuration tab, write the spark-submit command for the node. Example:

spark-submit --master yarn-cluster --executor-cores 2 --executor-memory 2g --driver-memory 1g --num-executors 2 --class com.aliyun.emr.example.spark.streaming.JavaLoghubWordCount examples-1.2.0-shaded.jar <logService-project> <logService-store> <group> <endpoint> <access-key-id> <access-key-secret>

Replace the placeholder values:

To get your AccessKey ID and AccessKey secret: log on to the DataWorks console, move the pointer over your profile picture in the upper-right corner, and select AccessKey Management.

In this example, the examples-1.2.0-shaded.jar JAR package is uploaded in the DataWorks console.

EMR Spark Streaming nodes do not support comments in node code.

If multiple EMR compute engines are associated with DataStudio, select the one to use. If only one is associated, no selection is needed.

(Optional) Override Spark parameters

Configure advanced parameters on the Advanced Settings tab to override default Spark behavior on a per-node basis. For the full list of configurable properties, see Spark Configuration.

The following parameters are available for DataLake cluster: EMR on ECS:

Run the task

1. In the toolbar, click the icon. In the Parameters dialog box, select a resource group from the Resource Group Name drop-down list and click Run.

   - To access a compute resource over the Internet or a virtual private cloud (VPC), use the resource group for scheduling that is connected to that compute resource. For more information, see Network connectivity solutions. - To change the resource group in a later run, click the (Run with Parameters) icon and select a different resource group in the Parameters dialog box.

2. Click the icon in the top toolbar to save the code.

3. (Optional) Perform smoke testing on the node in the development environment before or after committing it. For more information, see Perform smoke testing.