DataWorks:Flink SQL batch node

Prerequisites

Before you begin, make sure you have:

• A DataWorks workspace with Realtime Compute for Apache Flink computing resources associated in Management Center. For details, see Associate a computing resource.

• A Flink SQL Batch node created in your workspace. For details, see Node development.

Step 1: Write SQL code

Open the configuration tab of the Flink SQL Batch node and write your task code in the SQL editor.

Using variables in your code

Define variables in the ${Variable name} format anywhere in your SQL. Then, in the Properties tab, go to the Scheduling Parameters section and assign values to those variables. When the task runs, DataWorks substitutes the scheduling parameter values into your code automatically.

For details on available expressions, see Sources and expressions of scheduling parameters.

Example

The following SQL creates a source table using the datagen connector and inserts rows into a blackhole result table. The ${var} placeholder is replaced at runtime by a scheduling parameter.

-- Create a source table named datagen_source.
CREATE TEMPORARY TABLE datagen_source_${var}(
  name VARCHAR
) WITH (
  'connector' = 'datagen',
  'number-of-rows' = '1000'
);

-- Create a result table named blackhole_sink.
CREATE TEMPORARY TABLE blackhole_sink_${var}(
  name VARCHAR
) WITH (
  'connector' = 'blackhole'
);

-- Insert data from the source table into the result table.
INSERT INTO blackhole_sink_${var}
SELECT
  name
FROM datagen_source_${var};

Step 2: Configure the task

In the Properties tab, configure the resource and scheduling settings for the task.

Step 3: Deploy and monitor the task

1. Commit and deploy the configured node. For details, see Node and workflow deployment.

2. After deployment, click Perform O&M under Prod Online to open Operation Center. For more information, see Getting started with Operation Center.

3. In Operation Center, check the running status of your task.

What's next

• Getting started with batch processing of Realtime Compute for Apache Flink