Realtime Compute for Apache Flink:Best practices for monitoring and alerting

Use case

Metric(s)/event(s)

Rule configuration

Severity level

Recommended actions

Job failures

System event: JOB_FAILED

Upon event occurrence

P0

1. Check the configured restart policy.
2. Determine the cause of the failure (e.g., JobManager/TaskManager issue or a deliberate stop).
3. Restore the job from the latest savepoint or checkpoint.

Frequent restarts

Overview > NumOfRestart

• ≥ 1 restart

• 1 consecutive period

P0

1. Identify the cause.

• Analyze failover, JobManager, and TaskManager logs to find the cause of the failure.

• Ignore infrequent, auto-recoverable machine failures.

• Fix code bugs, resource bottlenecks, or configuration errors.

2. Recover the job from the latest savepoint or checkpoint.

Consecutive checkpoint failures

Checkpoint > NumOfCheckpoints

• ≤ 0 checkpoints

• 1 consecutive period

P0

1. See System checkpoints to troubleshoot the cause of checkpoint failures.

2. Identify the issue.

• Parameter issues (e.g., timeout): Adjust the checkpoint configuration.

• Resource issues (e.g., backpressure): Use dynamic scaling to add resources to the affected operator.

3. Dynamically update the configuration or restore the job from the latest checkpoint.

Latency

Overview > CurrentEmitEventTimeLag & NumOfRecordsInFromSourcePerSecond

• Maximum latency ≥ 180,000

• Input records ≥ 0

• 3 consecutive periods

P1

1. See Metric description to investigate the cause of the latency.

• Data-related: Check if event times are out of order.

• Traffic-related: Check if there is an upstream data surge or backpressure from a downstream system.

2. Take action based on the cause.

• Internal cause: Adjust connector options or scale up the bottleneck operator.

• External cause: Optimize external service configurations (e.g., adjust the traffic throttling strategy, increase connection counts).

Upstream data flow interruptions

Overview > NumOfRecordsInFromSourcePerSecond &
SourceIdleTime

• Input records ≤ 0 (depending on your business logic)

• Maximum source idle time ≥ 60,000 ms

• 5 consecutive periods

P1

1. Check taskmanager.log, flame graphs, and upstream metrics to identify the cause, such as a stalled thread, an upstream issue, rate throttling, or an error.

2. Take action based on the cause.

• Connector issues: Adjust connector options (e.g., timeout, concurrency) or increase TaskManager resources.

• Upstream/downstream service issues: Notify the upstream service owner to resolve the issue.

• Internal Flink bottlenecks (e.g., backpressure or system freeze): First, resolve the root cause of the bottleneck (e.g., fix the downstream issue), then restart the job from the latest checkpoint.

Output issues

Overview > NumOfRecordsOutToSinkPerSecond

• Output records ≤ 0

• 5 consecutive periods

P1

1. Verify if data is reaching the sink operator.

• Data filtering: Check logs/metrics for any data filtering that might be dropping records.

• Late data handling: Review watermark and window configurations to ensure late data isn't being dropped.

2. Verify the sink operator sends data to the destination.

• Connection: Check for full connection pools. Assess network connection stability.

• Destination system: Investigate downstream system for table locks, disk space, write throttling, or other errors.

3. Temporarily enable dual-write to a backup system.

CPU performance bottlenecks

CPU > TMCPUUsage

• TM CPU usage ≥ 85%

• 10 consecutive periods

P2

1. Use the flame graph or Flink UI to identify the hotspot operator.

• Business logic: Check for complex computations, JSON parsing, or inefficient UDFs.

• Data skew: Determine if a hot key is overloading a single task due to an excessive volume of data.

• Insufficient resources: Check for the parallelism and TaskManager resources. Investigate backpressure severity.

• Frequent GC: Check logs or JVM metrics to see if memory pressure is causing frequent full GCs, which consume significant CPU.

2. Increase the parallelism for the bottleneck operator or allocate more vCPUs to TaskManagers.

Memory performance bottlenecks

Memory > TMHeapMemoryUsed

• TM heap memory usage ≥ 90%

• 10 consecutive periods

P2

1. Analyze GC logs to identify the problem.

• Memory leak: In the Flink UI or monitoring dashboards, check if heap memory usage fails to return to its baseline after GC and continues to rise.

• Insufficient capacity: Determine heap memory usage. High usage can trigger frequent full GCs, degrading performance.

• Sudden OOM: Verify if memory is exhausted instantly when processing a specific record or batch of data, directly causing an OOM.

2. Take action based on the cause: increase the heap size or increase the parallelism.