Fully managed Flink allows you to view the logs of a historical job instance on the Logs tab in the console of fully managed Flink. You can also view the logs on Object Storage Service (OSS) or Log Service. This topic describes how to use the three methods to view the logs of a historical job instance.

Background information

You need to view the logs of a historical job instance in the following scenarios:
  • You want to view, search for, or analyze the operational logs of a historical job instance. By default, the system retains only 5 MB of the operational logs.
  • The Flink web UI cannot be accessed and you want to check the logs of the JobManager to identify the error.

Prerequisites

  • Before you view the logs of a historical job instance on the Logs tab in the console of fully managed Flink, you must click the Advanced tab on the right side of the Draft Editor page and turn on Allow Log Archives in the Logging section. By default, Allow Log Archives is turned on. Log Archiving
  • Before you view the logs of a historical job instance in the OSS console or the Log Service console, you must configure the logs of the historical job instance to be exported to OSS or Log Service. For more information, see Configure the logs of a historical job instance to be exported.

Limits

  • In Ververica Runtime (VVR) 3.X, only VVR 3.0.7 and later allow you to view the logs of the TaskManagers of a historical job instance.
  • In VVR 4.X, only VVR 4.0.11 and later allow you to view the logs of the TaskManagers of a historical job instance.

Precautions

  • If a TaskManager or the JobManager is started, logs are written to Log Service or OSS. However, if no TaskManagers are started and the JobManager is also not started, logs are not written to Log Service or OSS.
  • If you cannot find the relevant log file on OSS because the JobManager and TaskMangers fail to start, submit a ticket.
  • Two log files are retained in the configurations of fully managed Flink. The default size of each of the log files is 5 MB. We recommend that you do not modify the file size. If the file size does not meet your business requirements, you can export the logs to OSS. Log template
    Note The log files that are shown on the Logging Configuration | default tab of the Configuration page do not include the log files to which data is being written. If you set the DefaultRolloverStrategy max parameter to 1, a maximum of two files are saved. One file is saved in the memory, and the other is saved to your local disk.

View the logs of a historical job instance on the Logs tab

  1. Log on to the Realtime Compute for Apache Flink console.
  2. On the Fully Managed Flink tab, find the workspace that you want to manage and click Console in the Actions column.
  3. In the left-side navigation pane, choose Applications > Deployments.
  4. Click the name of the desired job.
  5. Click the Logs tab.
  6. Click the Running Logs tab. Then, select the running instance of the job from the Selected Job drop-down list.
    In the Selected Job drop-down list, the option marked with a gray dot indicates that the job instance that you selected is a historical job instance. History Jobs
  7. View the logs of the JobManager and TaskManagers
    You can click the Job Manager tab to view the logs of the JobManager and click the Task Managers tab to view the logs of the TaskMangers. Sample log

View the logs of a historical job instance on OSS

  1. Log on to the OSS console.
  2. In the left-side navigation pane, click Buckets.
  3. Find the bucket that you configured when you activate fully managed Flink and click the name of the bucket in the Bucket Name column.
  4. In the left-side navigation pane, click Files.
  5. View the log information in the related directory. Log directories
    Folder Description
    artifacts The uploaded JAR packages are stored in this directory.
    flink-jobs The checkpoints of fully managed Flink jobs are stored in this directory.
    flink-savepoints If you click Savepoint in the upper-right corner of the console of fully managed Flink, the Savepoint operation is triggered and the final Savepoint file is stored in this directory.
    logs If you set Logging Profile to OSS for your job, the logs of your job are stored in this directory.
    Note In the logs directory, jobs are stored in the following path: logs/${Workspace ID}/${Job ID}/${Instance ID}. You can find the logs of the JobManager and TaskManagers of the related job based on this directory structure.

View the logs of a historical job instance on Log Service

  1. Log on to the Log Service console.
  2. On the Projects tab, click the project whose logs you want to view.
  3. On the Logstores tab, click the Logstore in which all logs are stored.
  4. Enter a query and analysis statement, specify a time range, and then click Search & Analyze.
    For information, see Query and analyze logs.