Realtime Compute for Apache Flink:Quick start with Flink SQL jobs

Prerequisites

Before you begin, ensure that you have:

• The required permissions if you access Realtime Compute for Apache Flink as a Resource Access Management (RAM) user or a RAM role. For more information, see Permission management.

• A Flink workspace. For more information, see Activate Realtime Compute for Apache Flink.

Step 1: Create a job

1. Navigate to the SQL job creation page.

   1. Log on to the Realtime Compute for Apache Flink console.

   2. Click Console in the Actions column for the target workspace.

   3. In the navigation pane on the left, choose Data Development > ETL.

2. Click the icon, select New Stream Job, enter a File Name, and select an Engine Version.

   Realtime Compute for Apache Flink provides code and data synchronization templates. Each template covers a specific scenario with code examples and instructions, making it easier to learn Flink features and syntax. For more information, see Code Template and Data synchronization templates.

   Parameter	Description	Example
   File Name	The name of the job. Must be unique within the current project.	flink-test
   Engine Version	The Flink engine version for the job. Select a version tagged Recommended or Stable — these versions offer higher reliability and performance. For details, see Release notes and Engine versions.	vvr-8.0.8-flink-1.17

   

3. Click Create.

Step 2: Write the SQL job and view configuration information

1. Write the SQL job. Copy the following SQL into the SQL editor. This example uses the Datagen connector to generate a random data stream and the Print connector to print results to the development console. For a full list of supported connectors, see Supported connectors.

   INSERT INTO supports writing to a single sink or multiple sinks. For more information, see INSERT INTO statement.

   For production jobs, minimize the use of temporary tables. Instead, use tables that are registered in Data Management. For more information, see Data Management.

   -- Create a temporary source table named datagen_source.
   CREATE TEMPORARY TABLE datagen_source(
     randstr VARCHAR
   ) WITH (
     'connector' = 'datagen' -- Datagen connector
   );
   
   -- Create a temporary sink table named print_table.
   CREATE TEMPORARY TABLE print_table(
     randstr  VARCHAR
   ) WITH (
     'connector' = 'print',   -- Print connector
     'logger' = 'true'        -- Display the results in the console.
   );
   
   -- Truncate the randstr field and print the result.
   INSERT INTO print_table
   SELECT SUBSTRING(randstr,0,8) from datagen_source;

2. Review the configuration tabs. On the tabs to the right of the SQL editor, view or upload additional configurations.

   Tab	Description
   More Configurations	Configure Engine Version, Additional Dependencies (such as temporary functions), and Kerberos authentication. Engine version tags: Recommend (latest minor version of the latest major version), Stable (latest minor version of a major version still in service), Normal (other versions still in service), Deprecated (out of service). For details on version management, see Engine versions and Lifecycle policy. To enable Kerberos authentication, configure the registered Kerberos cluster and principal. See Register a Hive Kerberos cluster if needed.
   Code Structure	View the Data Flow or Tree Structure of your SQL statements.
   Version Information	View job version history and manage versions. See Manage job versions.

Step 3 (Optional): Perform a deep check and debug the job

The RPC times out maybe because the SQL parsing is too complicated. Please consider enlarging the `flink.sqlserver.rpc.execution.timeout` option in flink-configuration, which by default is `120 s`.

SET 'flink.sqlserver.rpc.execution.timeout' = '600s';

Step 4: Deploy the job

In the upper-right corner of the SQL editor, click Deploy. In the Deploy New Version dialog box, configure the parameters and click OK.

Set Deployment Target based on your environment:

Step 5: Start the job and view the results

1. In the navigation pane on the left, choose Operation Center > Job Operations.

2. Click Start in the Actions column for the target job.

3. Select Stateless Start and click Start. The job is running when its status shows Running. For details on startup parameters, see Start a job.

4. View the computation results.

   1. Go to Operation Center > Job O&M and click the target job.

   2. On the Job Logs tab, click the Running Task Managers tab, then click a task in the Path, ID column.

   3. Click Logs and search for PrintSinkOutputWriter.

Step 6 (Optional): Stop the job

Stop and restart a job when:

• The SQL code changes

• WITH parameters are added or removed

• The job version changes

• The job cannot reuse its state

• You want to start a new job

• Parameters that do not take effect dynamically are updated

To stop the job:

1. On the Operation Center > Job O&M page, click Stop in the Actions column for the target job.

2. Click OK.

What's next

• Configure resources: Allocate job resources in basic mode (coarse-grained) or expert mode (fine-grained) before starting, or adjust them after the job is online. See Configure job resources.

• Configure logging: Set log levels and outputs for different severity levels. See Configure job log outputs.

• Troubleshoot: Browse common questions for Job development and Job O&M.

• Try other job types:

  • Quick start for Flink JAR jobs

  • Quick start for Flink Python jobs

  • Quick start for real-time ingestion of database data

• Explore best practices:

  • Build a real-time data warehouse in Hologres

  • Build a streaming data lakehouse using Paimon and StarRocks