Realtime Compute for Apache Flink:Alert configurations and templates

Scenario

Metric/event name

Rule configuration

Severity

Actions

Job failure

Job run status event

= FAILED (Event-based alert)

P0

1. Check if the restart policy is misconfigured. Using the default configuration is recommended.

2. Determine if the failure is caused by the restart policy or a JobManager/TaskManager exception.

3. Restore the job from the latest savepoint or successful checkpoint.

Spike in failovers

Overview / NumOfRestart

≥ 1 for 1 consecutive period

P0

1. Identify the root cause.

• Analyze failover, JobManager, and TaskManager logs for error details.

• Ignore: Infrequent, auto-recoverable machine failures.

• Fix: Code bugs, resource bottlenecks, or configuration errors.

2. Restore the job from the latest savepoint or successful checkpoint.

Consecutive checkpoint failures

NumOfCheckpoints (5-minute aggregate)

≤ 0 for 1 consecutive period

P0

1. Refer to System checkpoints to troubleshoot the root cause of checkpoint failures.

2. Identify and resolve the issue.

• Parameter issues (e.g., timeout): Adjust checkpoint-related configurations.

• Resource scaling (e.g., backpressure): Use dynamic scaling to add resources to the backpressured operator.

3. Dynamically update the configuration or restore the job from the latest successful checkpoint.

High latency (with data ingress)

Overview / CurrentEmitEventTimeLag && NumOfRecordsInFromSourcePerSecond

Maximum latency ≥ 180,000 ms

Input records > 0

for 3 consecutive periods

P1

1. Refer to Monitor metrics to identify the cause of the latency.

• Data: Are event times out of order?

• Traffic: Is there a surge in upstream traffic or backpressure from a downstream system?

Step 2: Adjust based on the specific cause.

• Internal: Adjust connector WITH options or scale up the bottleneck operator.

• External: Optimize external service configurations, such as adjusting rate-limiting policies or increasing connection counts.

Upstream data flow interruption

Overview / NumOfRecordsInFromSourcePerSecond &&

SourceIdleTime

Input records ≤ 0 (depending on your business logic)

Maximum idle time ≥ 60,000 ms

for 5 consecutive periods

P1

1. Check taskmanager.log, flame graphs, and upstream service metrics to confirm if the issue is no upstream data, rate limiting, or an exception, or a stalled thread stack.

Step 2: Adjust based on the specific cause.

• Connector issues: Optimize connector parameters (e.g., timeout, concurrency), or add more TaskManager resources.

• Upstream service issues: Notify the upstream service owner to resolve the issue.

• Internal Flink bottlenecks (e.g., backpressure or system freeze): Resolve the root cause of the bottleneck (such as a downstream issue), then restart the job from the latest checkpoint.

No data output

Overview / NumOfRecordsOutToSinkPerSecond

≤ 0 for 5 consecutive periods

P1

1. Verify if data is reaching the sink operator.

• Business logic filtering: Use logs or metrics to confirm if all input data is being filtered out because it does not meet certain conditions.

• Late data dropped: Check watermark and window configurations to confirm if data is being dropped for being late.

2. Verify if the sink can write to the external system.

• Connection level: Is the sink connection pool exhausted? Is the network connection stable?

• Destination system level: Check the downstream database or service for table locks, insufficient disk space, write throttling, or other errors.

3. For a temporary fallback, implement a dual-write to a backup storage system.

CPU performance bottleneck

CPU / TMCPUUsage

≥ 85% for 10 consecutive periods

P2

1. Use a flame graph or the Flink UI to identify the hotspot operator.

• Business logic: Complex computations, JSON parsing, or inefficient user-defined functions (UDFs).

• Data skew: A hot key is causing a single task to be overloaded with a large volume of data.

• Insufficient resources: Does the current parallelism and TaskManager resources match the data traffic? Is there severe backpressure?

• Frequent GC: Use logs or JVM metrics to check if memory pressure is causing frequent Full GCs, which consume significant CPU.

2. Increase the parallelism of the bottleneck operator or allocate more CPU cores to the TaskManager.

Memory performance bottleneck

TMHeapMemoryUsed

≥ 90% for 10 consecutive periods

P2

1. Analyze GC logs to identify the problem.

• Memory leak: Use the Flink UI or monitoring dashboards to observe if the heap memory fails to return to its baseline after GC and the baseline continues to rise.

• Insufficient capacity: If heap memory usage is consistently high, it can trigger frequent Full GCs and degrade performance.

• Sudden OOM: Check if processing a specific record or batch of data causes memory to be exhausted instantly, leading directly to an OutOfMemoryError.

2. Increase the heap size or parallelism to reduce the data volume per slot.