DataWorks:Flink SQL batch node

Develop SQL code

Develop your task code in the SQL editing area. In your code, you can define variables by using the ${variable_name} format. Then, on the right side of the node editing page, assign a value to the variable in the Scheduling Parameters section of the Scheduling Settings pane. This allows you to dynamically pass parameters to your code in scheduling scenarios. For more information about how to use scheduling parameters, see Scheduling parameter sources and their expressions. The following is an example.

-- Create a source table named datagen_source.
CREATE TEMPORARY TABLE datagen_source_${var}(
  name VARCHAR
) WITH (
  'connector' = 'datagen',
  'number-of-rows' = '1000' 
);

-- Create a result table named blackhole_sink.
CREATE TEMPORARY TABLE blackhole_sink_${var}(
  name  VARCHAR
) WITH (
  'connector' = 'blackhole'
);

-- Insert data from the source table into the result table.
INSERT INTO blackhole_sink_${var}
SELECT
  name
FROM datagen_source_${var};

Configure Flink resources

You can configure the following parameters on the right side of the edit page in the Flink resource information section under Scheduling Settings. For more information, see Configure schedule settings.

(Optional) Configure scheduling parameters

On the right side of the edit page, in the Scheduling Parameters section under Scheduling Settings, click Add parameters, and then edit the Parameter name and Parameter Value to dynamically use them in your code.

(Optional) Configure Flink runtime parameters

You can configure runtime parameters on the right side of the edit page in the Flink running parameters section under Scheduling Settings. For more information, see Configure schedule settings.

When configuring Flink runtime parameters, the syntax is compatible with VVP (Ververica Platform). You can write configurations directly in YAML format without adding semicolons or other special characters for line breaks.

Configure Flink resource information

On the right side of the node editing page, in the Flink resource information section of the Run Configuration pane, configure the parameters as described in the following table.

Prepare debug data

In the Debug Data section of the Run Configuration pane, prepare mock data for the source tables referenced in your code.

1. Click Generate Template. The system parses the source tables referenced in the current SQL code and generates corresponding table name records in the list below. Previously uploaded data is not cleared.

2. In the Actions column of the table name record, click Download Template to download a CSV template that matches the structure of the source table.

3. Fill in the debug data in the downloaded template following the column order and save the file in CSV format.

4. In the Actions column of the table name record, click Upload and select the completed CSV file to upload. After the upload succeeds, the Status column displays Enabled.

5. (Optional) After the upload succeeds, you can click Preview in the bottom panel to view the data. To modify the data, re-upload a CSV file to overwrite the existing data.

6. If you do not want the mock data of a specific source table to be used in the current debug session, click Disable to change the status to Disabled. To re-enable the data, click Enable. Only data with the Enabled status is used in the current debug session.

Run the debug task

After the debug data is ready, click the Run button on the editor toolbar (or press F8). The system submits the code, mock data, and Flink resource information to the selected session cluster for execution.

If your code uses parameters in the ${variable_name} format, make sure that you have assigned values to the variables in the Scheduling Parameters section. During debugging, the system replaces the placeholders in the code with the assigned values before submission.

View the debug results

After the debug task runs, the result area at the bottom of the node provides the following information to help you quickly identify issues:

• Code: The SQL code submitted to the Flink engine for this run (with variable substitutions applied).

• Logs: The runtime log and error information of the debug task.

• Query results: The output data of the debug task.