How to run through your first DataStream job

Preparation

Jar is packaged and uploaded. Here is an example from the community. You can use your own development application in actual operation.

On the left side of the VVP page, click "Resource Upload", and then click "Upload Resource" in the upper right corner to upload the packaged Jar.

Create and start jobs

There are two ways to create a job. The first is to click Create Job directly through the uploaded Jar, and you will jump directly to the creation page;

The second is to directly click Create Job on the left menu of VVP, and then select the jar file to be executed in Jar Uri.

Add the job name and other information, and the job is created.

After creating a job, you can see some basic operations and display basic information on the job details page. The status of the newly created job is Canceled, and you can click "Start" on the right side of the status bar to start the job.

During startup, the page will display the current instance, and each instance has a unique ID.

In "Running Events", you can see some more detailed staged Events, which are also a type of resource. The position selected by the cursor in the figure below is the error message of the Event alarm. If you click the "+" sign on the left, you can see the log details.

When the status of Deployment in "Running Event" changes to Running, it means that the job starts successfully. Note: Only one Deployment instance can run at a time. That is to say, if you want to reconfigure the job, you need to stop the running job first, and restart the new job after reconfiguring.

Basic operations of Deployment

Start, pause, and stop are the basic operations for this job, so I won’t go into details here. The reminder delete function can only be operated on the canceled job, and the running and failed jobs cannot be deleted. It cannot be restored after deletion, so be careful.

Cloning means that if you want to do some tentative configuration or comparison without modifying the existing Deloyment, you can clone the job first.

Deployment configuration

The configuration of Deployment can be modified when creating a new job, or it can be modified during operation.

The "upgrade strategy" in the above figure refers to whether a restart is required when the Deployment is modified, and if a restart is required, whether to restart with status or without status. If you select None as shown in the figure below, the currently running job will not be affected when the job configuration is modified.

If you need to restart the job, there are two options. Stateless is without state, that is, stops the current job and restarts a new job after adjusting the configuration; Stateful is with state, that is, before stopping the job, make a savepoint for the job, and then use The latest configuration starts a new job and resumes from this savepoint.

"Recovery Strategy" has three options, None means to restore the job without state; Latest Savepoint means to use the latest Savepoint to restore the job; Latest State means to use the latest one of Checkpoint or Savepoint to restore the job. Recovery strategies are used when we need to restart a job that has been stopped.

"Maximum number of retries to create a Savepoint" on the left side of the figure below, because the Savepoint is written in OSS, if you encounter a problem and fail to create a Savepoint, the number of retries needs to comply with the configuration here, and the operation of the Savepoint exceeding the maximum number of configurations will fail.

"Maximum number of retries for instance creation" on the right refers to the maximum number of retries for starting the Deployment. When a problem occurs and the number of retries of the Deployment exceeds the configured number, the status of starting the Deployment will change to red Failed.

Stop with Drain refers to whether the stop interface has the Drain parameter; label means that multiple tags can be added to the Deployment, so that the required Deployment can be searched more quickly in the job list.

"Additional Dependency Jar", here you can not only put additional Jar, but also some additional resource files.

"Flink version", now only 1.10 and 1.11, two versions to choose from. The following figure is a common configuration of Flink, so I won’t repeat it here.

Deployment other operations

Use of Job Snapshots

Job snapshots can be used in several ways:

The first one clicks Savepoint once to trigger Savepoint and leave a snapshot;
The second type is a running job. By default, Checkpoint will leave a snapshot every 10 seconds;

Job snapshots can also be used to restore a job by selecting an earlier snapshot instead of the most recent one. Click "Copy jobs from this snapshot" to folk a Deployment, which is similar to cloning, but this specifies a Savepoint. If this job is started after copying, the job will be resumed from the specified savepoint.

Secret Values

Secret Value can only be used in Jar URI and Flink Configuration.

It is usually used in some sensitive scenarios. For example, the host part in the Jar URI does not want to be exposed, so how to do it? Click "Variable Configuration" on the left side of the VVP page, click Add Variable, and complete the variable name and variable value in the pop-up window.

After the creation is completed, you can see that only the new variable name is displayed, and the variable value is hidden and not displayed. Then go to the assignment to apply this variable.

Open the job list, select the application job, and set variables in the Jar URI.

UI / Metrics view

There is "Indicator" on the right side of the page, click the indicator, jump to the page, and you can view the indicator information of the job.

Click on the "Flink UI" page. Flink UI has a skill to use. After starting the job, if there is a problem that causes the VVP to always show that it is starting but the actual job has started, then you can see the first-hand information on the Flink page, and you can click to view it through the Flink UI.