Learn how to identify the causes of checkpoint and savepoint timeouts in Flink and apply tuning strategies to resolve them.
How it works
Flink state management uses the Chandy-Lamport algorithm for data consistency. Creating a checkpoint or savepoint involves two phases:
-
Synchronous phase: Barriers flow between operators to align data. Alignment time is proportional to data record latency.
-
Asynchronous phase: Each operator uploads its local state to remote persistent storage. Upload time is proportional to state size.
Backpressure slows barrier alignment, which causes checkpoint and savepoint timeouts. If you detect backpressure, resolve it first: Tune SQL jobs with large state to resolve backpressure and Tune DataStream jobs with large state to resolve backpressure.
Diagnosis methods
After resolving backpressure, check whether the synchronous-phase alignment time is too long, and then whether large state data is causing the delay.
Checkpoint UI
Go to , click the Logs tab, and navigate to . Analyze checkpoint metrics at the job, operator, and subtask levels to identify the timeout cause.
In the Operators section, click an operator to expand its Subtasks list. Key subtask-level metrics include End to End Duration, Checkpointed Data Size, Full Checkpoint Data Size, Sync Duration, Async Duration, Processed (persisted) Data, and Alignment Duration.
Focus on operators of failed or in-progress checkpoints:
-
High
Sync DurationandAlignment Durationindicate a synchronous-phase bottleneck. Resolve this first. -
High
Async Durationwith largeCheckpointed Data Sizeindicates an asynchronous-phase state upload bottleneck.
Checkpoint metrics
Go to and click the Alarm tab. Use the lastCheckpointDuration and lastCheckpointSize metrics to analyze historical checkpoint duration and size trends.
Tuning strategies
Before tuning checkpoints, ensure the job's runtime performance meets expectations. Then apply the following strategies.
|
Strategy |
Description |
Use case |
Configuration |
Notes |
|
Use unaligned checkpoints and buffer debloating |
Resolves timeouts caused by data alignment delays. Suitable for jobs of all sizes. |
Checkpoint or savepoint timeout in the synchronous phase |
Configure runtime parameters as described in How to use unaligned checkpoints and buffer debloating. |
Review the Limitations. |
|
Increase runtime parallelism |
Reduces per-subtask state size, which speeds up the asynchronous phase. |
Checkpoint or savepoint timeout in the asynchronous phase |
Increase parallelism in the resource configuration or fine-grained resource configuration. For details, see Configure job resources. |
None. |
|
Use native savepoints |
Native savepoints are faster to create and use less storage than standard savepoints. |
Savepoint timeout in the asynchronous phase |
Create a savepoint in native format for a running job. Manually create a job savepoint. |
Native savepoints may not be compatible across major versions. |