How to migrate Flink tasks to real-time computing

1. How to migrate: from Flink run to stream computing platform

How to migrate offline commands to online as shown in the figure below. First open the VVP UI and add basic configuration, mainly including: job name, Jar URI, some main parameters and parallelism. You can also click Advanced Configuration to configure more information.

For example, the behavior configuration in the advanced configuration is divided into three parts: upgrade strategy, initial state, and recovery strategy. Generally select Stateless for the upgrade strategy, Running for the initial state, and Latest State for the recovery strategy.

Select Stateless for the upgrade strategy, which means that after editing and saving the configuration for the job that is already running, the original job will be stopped directly, and then restart a newer job; if you choose Stateble, edit and save the configuration for the running job, The job will do a Savepoint first, then load the new configuration and use the Savepoint to start a new job.

If you select LatestSavepoint for the recovery strategy, it means that if the job is suspended, if it is restarted, it will start with the latest Savepoint.

The above is the more important part of the Flink configuration. The picture below shows another part of the configuration that can set the checkpoint interval, etc., and the next is the resource configuration. In the log configuration part, you can choose the log storage location, so that if there is a problem with the job in the future, you can easily troubleshoot the problem.

After the Flink job is configured and started running, if an exception occurs, you can check the running status and problems through running events. You can also open the Job Manager of the Flink UI to view the logs.

2. In the case of multiple Jars, how do users add other dependent Jars

Some users have custom dependencies and can't solve the problem with fat jar. For example, user A has his own main Jar and some other Jars. The application scenarios of these Jars are different, so the Jar package needs to be uploaded on the resource management page first, and the file can be used on the page after the upload is completed.

After uploading, go to the advanced configuration, find "Extra Dependent Jar", pull down and select the Jar package just uploaded.

3. How users define jobs through configuration files

Upload the file in the resource management, then select Add dependency in the advanced configuration, and select the required dependency. If in the startup function, the main class needs to read a certain file, first add the file through dependencies, and then read the file according to the prompt.

The above two methods are for placing user Jar, jar and other files can be stored here.

4. How to reuse the original state to speed up job recovery

If you find a problem while running a job, it's generally a good idea to stop and find a fix before restarting. Users need to specify the OSS required by Checkpoint when creating it. If Checkpoint is enabled, subsequent jobs can be resumed directly from Checkpoint.

First confirm that the recovery policy in the advanced configuration selects LatestState or LatestSavepoint. Under this premise, you can pause the job (do not stop the job), the pause will trigger the Savepoint, and when you click Start again, it will resume from the Savepoint point.

Open the Flink UI, and then click the job snapshot. In the job snapshot, you can view the Flink Checkpoint indicator, which will display information such as the number of Restores and the address of the last Restore, so that you can confirm that the job is restored from the latest State.

When the job is running and you want to copy the current job, how to do it? Click Checkpoint on the job control page, find the historical snapshot and click "Copy job from this snapshot" on the far right corresponding to it, then the operation of copying the current job is completed, and the copied content includes the Jar package configuration of the running job, etc. After clicking Start, you can see from the job snapshot that the new job is copied and started from the latest Savepoint of the previous job.