DataWorks:Create an EMR Spark SQL node

Prerequisites

Before you begin, make sure you have:

• An EMR cluster registered with DataWorks. See Legacy Data Development: Attach EMR computing resources.

• A serverless resource group or exclusive resource group for scheduling, purchased and configured (attached to a workspace with network connectivity). See Add and use a Serverless resource group.

• A RAM user added to the workspace with the Developer or Workspace Administrator role. Grant Workspace Administrator only when necessary—it carries extensive permissions. See Add workspace members and assign roles to them.

• A business flow created in DataStudio. All node development is organized within business flows. See Create a business flow.

• (Optional) A custom image, if you need a specific component environment (for example, to replace Spark JAR packages or bundle specific libraries). Build a custom image from the official dataworks_emr_base_task_pod image. See Custom images.

Limitations

• EMR Spark SQL nodes run only on a serverless resource group or an exclusive resource group for scheduling. Use a serverless resource group unless you have a specific reason to use an exclusive resource group. If you use a custom image in DataStudio, use a serverless computing resource group.

• To manage metadata for DataLake or custom clusters, configure EMR-HOOK on the cluster. Without it, DataWorks cannot display real-time metadata, generate audit logs, show data lineage, or perform EMR-related administration. See Configure EMR-HOOK for Spark SQL.

The following table summarizes feature availability by cluster type:

Usage notes

These notes apply if Ranger access control is enabled for Spark in the EMR cluster associated with your workspace.

• Default image: Ranger access control is enabled automatically.

• Custom image: Submit a support ticket to request an image upgrade that supports Ranger access control.

Step 1: Create an EMR Spark SQL node

1. Log on to the DataWorks console. In the top navigation bar, select your region. In the left-side navigation pane, choose Data Development and O&M > Data Development. Select your workspace from the drop-down list and click Go to Data Development.

2. In DataStudio, right-click the target business flow and choose New Node > EMR > EMR Spark SQL. Alternatively, hover over New and choose New Node > EMR > EMR Spark SQL.

3. In the New Node dialog box, set Name, Engine Instance, Node Type, and Path, then click Confirm. The node name can contain uppercase letters, lowercase letters, Chinese characters, digits, underscores (_), and periods (.).

The configuration tab for the new EMR Spark SQL node opens.

Step 2: Develop the SQL task

SHOW TABLES;

-- Use ${var} to define a variable.
-- Assign ${yyyymmdd} to var to append a date suffix to table names at runtime.
CREATE TABLE IF NOT EXISTS userinfo_new_${var} (
  ip  STRING COMMENT 'IP address',
  uid STRING COMMENT 'User ID'
)
PARTITIONED BY (dt STRING);

Step 3: Configure scheduling

To run the node on a recurring schedule, click Properties in the right-side pane and configure the scheduling options.

Configure Rerun and Parent Nodes before committing the task. If you need a custom component environment, create a custom image from the official dataworks_emr_base_task_pod image and use it in DataStudio.

For the full scheduling configuration reference, see Overview.

Step 4: Publish the node

After configuring the node, commit and deploy it. The system then runs it on the schedule you configured.

1. Click the icon to save.

2. Click the icon to commit. In the Submit dialog box, fill in the Change description and decide whether to trigger a code review.

   Configure Rerun and Parent Nodes on the Properties tab before committing. If code review is enabled, the committed code must pass review before it can be deployed. See Code review.

3. If your workspace is in standard mode, deploy the node to the production environment after committing. Click Deploy in the upper-right corner of the configuration tab. See Deploy nodes.

What's next

After deploying, the node runs automatically on its configured schedule. To monitor execution, click Operation Center in the upper-right corner of the configuration tab. See View and manage auto triggered tasks.

FAQ

The EMR Spark task must be submitted to SQL Compute, but the task fails immediately.

Check whether the SQL Compute service is in Running status. If it is stopped or starting, the submission fails. See Manage SQL sessions to view and start the session.

A `DlfMetaStoreClientFactory not found` error occurs after enabling Kerberos on the EMR cluster.

This error appears when using YARN-Cluster mode with spark-submit in an EMR Spark node after Kerberos is enabled. See the troubleshooting entry for the resolution steps.