View historical job logs - Realtime Compute for Apache Flink

Background

You may need to view historical job logs in the following scenarios:

You need to view, search for, and analyze the run logs of a historical job. By default, the system retains the latest 5 MB of run logs.
You need to check JobManager logs to troubleshoot issues when the Flink UI is inaccessible.

This topic describes three ways to view historical job logs:

To view historical job logs on the Logs tab, ensure that log archiving is enabled. This feature is enabled by default, and its archived logs are retained for 7 days. On the Configuration tab of a Flink deployment, expand the Logging section, turn on the Log Archiving switch, and set the Archived Log Retention Period, for example, to 7 days.
To view the logs of historical jobs in OSS or SLS, you must have configured log output to OSS or SLS for these jobs in Realtime Compute for Apache Flink. For more information, see Configure job log output.

When a log file reaches the specified size, the system automatically archives it and periodically deletes log files that exceed their retention period. Archived logs are retained even after a job stops. If a deployment is deleted, its associated logs are also deleted.

Note
After the new version is released in November 2024, all existing job logs in the Realtime Compute for Apache Flink console, including archived logs from stopped jobs, will be deleted when their retention period expires.
Logs are written to SLS or OSS only if a TaskManager or the JobManager starts successfully.
Contact us if you cannot find logs in OSS, which can happen if the JobManager and TaskManagers fail to start.
By default, the logging configuration retains two log files, each with a size of 5 MB. We recommend not changing this value; if you need more log capacity, export the logs to OSS instead. On the Configuration tab, in the Logging section, the default Log4j2 XML configuration sets the size of SizeBasedTriggeringPolicy to 5 MB. This triggers rollover archiving when a log file reaches this size. The max attribute of DefaultRolloverStrategy is set to 1.

Note
The number of files specified in this configuration does not include the file that is currently being written. For example, if max is set to 1, a maximum of two files are retained: one active file and one archived file.

Log in to the Realtime Compute for Apache Flink console.
In the Actions column of the target workspace, click Console.
In the left-side navigation pane, choose O&M > Deployments.
Click the name of the target deployment.
Click the Logs tab.
On the Logs tab on the left, select a historical job from the Job drop-down list.

The left panel also contains the Startup Logs and Exceptions tabs. The drop-down list separates jobs into Running jobs and Historical jobs.
Select a log to view.

On the Job Manager or Running Task Managers tab, click the name of a log to view its content.

In the Artifacts section, view the log information in the corresponding directory.

Folder	Description
artifacts	Stores the uploaded JAR packages.
flink-jobs	This directory stores the checkpoints for a Flink job.
flink-savepoints	This directory stores the savepoint file created when you click Create Savepoint for a deployment on the O&M > Deployments page of the Realtime Compute for Apache Flink console.
logs	If you set the Logging Profile for your job to OSS, this directory stores the job logs. Note The storage path for job logs in the logs directory is logs/${workspace-id}/${job-id}/${instance-id}. You can use this directory structure to find the JobManager and TaskManager logs for the corresponding job.

Log in to the Simple Log Service console.
In the Projects section, click the target project.
On the Logstores tab, click the target Logstore.
Enter a query, select a time range, and click Search & Analyze.

For more information, see Quick start for search and analysis.