All Products
Search
Document Center

Realtime Compute for Apache Flink:View the logs of a historical job

Last Updated:Dec 22, 2023

Fully managed Flink allows you to view the logs of a historical job on the Diagnostics tab in the console of fully managed Flink. You can also view the logs in the Object Storage Service (OSS) or Simple Log Service console. This topic describes how to use the three methods to view the logs of a historical job.

Background information

You need to view the logs of a historical job in the following scenarios:

  • You want to view, search for, or analyze the operational logs of a historical job. By default, the system retains only 5 MB of the operational logs.

  • The Flink web UI cannot be accessed and you want to check the logs of the JobManager to identify the error.

Prerequisites

  • Before you view the logs of a historical job on the Diagnostics tab in the console of fully managed Flink, you have turned on Allow Log Archives in the Logging section of the Configuration tab. By default, Allow Log Archives is turned on.Image 6.png

  • Before you view the logs of a historical job in the OSS console or the Simple Log Service console, you have configured the logs of the historical job to be exported to OSS or Simple Log Service. For more information, see Configure parameters to export logs of a deployment.

Limits

  • In Ververica Runtime (VVR) 3.X, only VVR 3.0.7 and later allow you to view the logs of the TaskManagers of a historical job.

  • In VVR 4.X, only VVR 4.0.11 and later allow you to view the logs of the TaskManagers of a historical job.

Precautions

  • If a TaskManager or the JobManager is started, logs are written to Simple Log Service or OSS. However, if no TaskManagers are started and the JobManager is also not started, logs are not written to Simple Log Service or OSS.

  • If you cannot find the relevant log file on OSS because the JobManager and TaskManagers fail to start, contact Alibaba Cloud technical support.

  • Two log files are retained in the configurations of fully managed Flink. The default size of each of the log files is 5 MB. We recommend that you do not modify the file size. If the file size does not meet your business requirements, you can export the logs to OSS.log

    Note

    The log files that are shown on the Configuration tab of the Deployments page do not include the log files to which data is being written. If you set the DefaultRolloverStrategy max parameter to 1, a maximum of two files are saved. One file is saved in the memory, and the other is saved to your local disk.

View the logs of a historical job on the Diagnostics tab

  1. Log on to the Realtime Compute for Apache Flink console.

  2. On the Fully Managed Flink tab, find the workspace that you want to manage and click Console in the Actions column.

  3. In the left-side navigation pane, click Deployments.

  4. Click the name of the desired deployment.

  5. Click the Diagnostics tab.

  6. On the left side of the Diagnostics tab, click the Logs tab. On the Logs tab, select a job from the Job drop-down list.

    image.png

  7. View the logs of the JobManager and TaskManagers

    You can click the Job Manager tab to view the logs of the JobManager and click the Task Managers tab to view the logs of the TaskManagers.

View the logs of a historical job in the OSS console

  1. Log on to the OSS console.

  2. In the left-side navigation pane, click Buckets.

  3. Find the bucket that you configure when you activate fully managed Flink and click the name of the bucket in the Bucket Name column.

  4. In the Objects pane of the page that appears, view the logs in the related directories.

    Folder

    Description

    artifacts

    Stores the uploaded JAR packages.

    flink-jobs

    Stores the checkpoints of fully managed Flink deployments.

    flink-savepoints

    Saves the final savepoint file that is generated by the savepoint operation when you click Savepoint in the upper-right corner of the console of fully managed Flink.

    logs

    Stores the logs of the deployment for which Logging Profile is set to OSS.

    Note

    The logs of deployments in the logs folder are stored in the following path: logs/${Workspace ID}/${Deployment ID}/${Job ID}. You can find the logs of the JobManager and TaskManagers of the related job based on this directory structure.

View the logs of a historical job in the Simple Log Service console

  1. Log on to the Simple Log Service console.

  2. On the Projects tab, click the project whose logs you want to view.

  3. On the Logstores tab, click the Logstore in which all logs are stored.

  4. Enter the query and analysis statements, specify a time range, and then click Search & Analyze.

    For more information, see Query and analyze logs.