DataWorks:Lindorm Flink JAR Streaming node

Overview

The Lindorm stream engine provides low-latency, high-throughput stream computing. It is compatible with Flink APIs and supports submitting Flink Streaming jobs as JAR packages, which makes it suitable for scenarios such as real-time data processing, real-time ingestion, and real-time analysis. With the Lindorm Flink JAR Streaming node in DataWorks, you can reference a Flink job JAR package, configure a submission command, and develop, debug, and periodically schedule streaming jobs. You can also manage the running status of streaming tasks in Operation Center.

Limitations

• Resource groups: Lindorm Flink JAR Streaming nodes can run only on a Serverless resource group.

• Engine: Lindorm Flink JAR Streaming nodes depend on the Lindorm stream engine. You must first enable the stream engine for your instance and create a stream engine resource group in the Lindorm console.

• Job type: Lindorm Flink JAR Streaming nodes currently support only Flink Streaming jobs submitted as JAR packages.

Create a Lindorm Flink JAR Streaming node

For instructions on how to create a node, see Create a Lindorm Flink JAR Streaming node.

Develop a Lindorm Flink JAR Streaming node

Developing a Lindorm Flink JAR Streaming node involves two parts: uploading the Flink job JAR package as a DataWorks resource file, and configuring the stream engine submission command in the node. The following example uses a DataGenerator streaming job to describe how to configure and use a Lindorm Flink JAR Streaming node.

##@resource_reference{"datagen.jar"}
lstream-cli run \
--class com.alibaba.lindorm.stream.demo.DataGenerator \
--parallelism 1 \
--detached \
datagen.jar

Debug a Lindorm Flink JAR Streaming node

1. Configure run properties.

   In the Run Configuration pane on the right side of the node, configure the Compute Resource, Lindorm Resource Group, and Resource Group. The following table describes these parameters.

   Parameter	Description
   Compute Resource	Select the associated Lindorm compute resource.
   Lindorm Resource Group	Select the stream engine resource group you created in the Lindorm console.
   Resource Group	Select a Serverless resource group that has passed the network connectivity test. Lindorm Flink JAR Streaming nodes support only Serverless resource groups.
   Runtime duration	A streaming job is a long-running task. You can set the maximum duration for a debug run. When the specified duration is reached, the debug run stops automatically to prevent the debug job from occupying resources for an extended period.

2. Run and debug the node.

   To execute the node task, click Save and then click Run. After the job runs, you can view the submission and running information in the run log. The run log also prints a URL that links to the job in the Lindorm stream engine. Click the URL to go to the job management page in the Lindorm console and observe the running status of the job in real time.

Deploy the Lindorm Flink JAR Streaming node

After the node passes debugging, configure the runtime resources and run properties of the task in the Real-time Configuration pane on the right side of the node, and then deploy the node to the production environment. After the node is deployed, the task runs in the production environment based on the selected startup method, and you can manage it in Operation Center.

1. Configure the real-time configuration.

   Configure the following items in the Real-time Configuration pane on the right side of the node.

   Item	Description
   Compute Resource	On the Compute resource tab, select the Lindorm compute resource that you have bound.
   Lindorm Resource Group	On the Compute resource tab, select the corresponding Lindorm stream engine resource group.
   Resource Group	On the Resource group tab, select a Serverless resource group that has passed the network connectivity test. Lindorm Flink JAR Streaming nodes support only Serverless resource groups.
   Script parameters	On the Script parameters tab, add runtime parameters as required by the job.
   Startup method	On the Time properties tab, select the startup method after the task is deployed: Start immediately after deployment or Do not start after deployment.
   Auto-rerun upon failure	On the Time properties tab, after you enable this option, the task is automatically rerun when it fails. You can further set the Number of reruns (the maximum number of automatic reruns) and the Rerun interval (the interval between two reruns, in minutes).

2. Deploy the node.

   After you complete the real-time configuration, click Deploy on the toolbar to complete the deployment package build, production inspector, and deployment to the production environment in sequence. For details about the deployment process, see Deploy the node. After the node is deployed, the task runs in the production environment based on the selected startup method.

Manage streaming tasks in Operation Center

After you deploy the Lindorm Flink JAR Streaming node to the production environment, you can manage streaming tasks in Operation Center:

• View the running status of streaming tasks in Operation Center, perform operations such as stopping and starting tasks, and use the log link to go to the job management page in the Lindorm console.

• The job status is synchronized in both directions between DataWorks Operation Center and the Lindorm console. When you cancel or start a job in the Lindorm console, Operation Center displays the latest job status (such as Stopped or Running), and vice versa.