Realtime Compute for Apache Flink:Recommended monitoring configurations

Job failure alert

Configure a P0 alert that fires immediately when a job transitions to the FAILED state, so you can restore service before the failure cascades.

Metric: Job status event = FAILED

Notification: Phone call, text message, email, and webhook (Critical)

When this alert fires:

1. Check whether the restart strategy is misconfigured. Use the default settings unless you have a specific reason to override them.

2. Determine whether the failure is caused by the restart strategy or by an abnormal JobManager or TaskManager.

3. Restore the job from the latest snapshot or successful checkpoint.

Configure in ARMS

1. Log in to the Realtime Compute for Apache Flink console. In the Actions column of your workspace, click Console.

2. On the Operation Center > Job O&M page, click the target job.

3. Click the Alert Configuration tab.

Configure in CloudMonitor

1. Log in to the CloudMonitor console.

2. In the left navigation pane, choose Event Center > Event Subscription.

3. On the Subscription Policy tab, click Create Subscription Policy.

4. Configure the parameters. For details, see Manage event subscriptions (Recommended).

Failover surge

Frequent restarts indicate an underlying problem — code bugs, resource bottlenecks, or configuration errors — that auto-recovery cannot fix. Configure this alert to catch surge patterns before they exhaust retry budgets.

Metric: Number of error recoveries per minute for the job

Recommended configuration:

• Metric value ≥ 1

• Period: 1 minute

• Notification: Phone call, text message, email, and webhook (Critical)

When this alert fires:

1. Identify the root cause by analyzing failover, JobManager, and TaskManager logs.

   • Ignore: Infrequent, auto-recoverable machine failures.

   • Fix: Code bugs, resource bottlenecks, or configuration errors.

2. Restore the job from the latest snapshot or successful checkpoint.

Consecutive checkpoint failures

When no checkpoint succeeds within 5 minutes, the job has no recent recovery point.

Metric: Number of completed checkpoints per minute

Recommended configuration:

• Metric value ≤ 0

• Period: 5 minutes

• Notification: Phone call, text message, email, and webhook (Critical)

When this alert fires:

1. See System checkpoints to identify the root cause.

2. Act on the cause:

   • Configuration issue (such as timeout): Adjust the checkpoint settings.

   • Backpressure: Use dynamic scaling to add resources to the backpressured operator.

3. Update the configuration dynamically or restore the job from the latest successful checkpoint.

Ensure SLA for latency

Alert when the job is receiving data but processing lags behind by more than 5 minutes. Adjust the threshold and alert level to match your SLA.

Metrics: Business latency, Records in from source per second

Recommended configuration:

• Business latency Maximum ≥ 300000

• Records in from source per second Metric value > 0

• Period: 5 minutes

When this alert fires:

1. See Metric description to investigate the cause.

   • Data plane: Are event timestamps out of order?

   • Traffic: Is there an upstream surge or downstream backpressure?

2. Act on the cause:

   • Internal: Adjust connector WITH parameters and scale out the bottleneck operator.

   • External: Optimize external service settings, such as adjusting throttling policies or increasing connection limits.

Upstream data interruption

Alert when there is inbound data and the service latency exceeds 5 minutes. You can adjust the threshold and alert level as needed.

Metrics: Records in from source per second, Age of unprocessed data at the source

Recommended configuration:

• Records in from source per second Metric value ≤ 0

• Age of unprocessed data at the source Maximum > 60000

• Period: 5 minutes

When this alert fires:

1. Check taskmanager.log, flame graphs, and upstream service metrics to confirm the cause: no upstream data, throttling, an error, or a stalled thread stack.

2. Act on the cause:

   • Connector issue: Adjust connector parameters such as timeout or concurrency, or add TaskManager resources.

   • Upstream or downstream service issue: Notify the upstream team to investigate.

   • Flink internal bottleneck (such as backpressure or a freeze): Resolve the root cause first, then restart the job from the latest checkpoint.

No downstream output

Alert when the sink stops emitting data for more than 5 minutes. This can indicate a business logic issue, a late-data configuration problem, or a failure in the downstream system.

Metric: Records out to sink per second

Recommended configuration:

• Metric value ≤ 0

• Period: 5 minutes

When this alert fires:

1. Confirm whether data reaches the sink operator:

   • Business logic filtering: Check logs or metrics to determine whether all input was filtered out for not meeting conditions.

   • Late data discard: Check watermark and window settings to determine whether data was discarded as late arrivals.

2. Confirm whether the sink can write to the external system:

   • Connection layer: Is the connection pool full? Is network connectivity normal?

   • Target system: Does the downstream database or service have a locked table, insufficient disk space, write throttling, or other errors?

3. As a temporary measure, enable dual-write to a backup storage system.

CPU bottleneck

Alert when a single TaskManager's CPU stays above 85% for 10 consecutive minutes.

Metric: CPU utilization of a single TM

Recommended configuration:

• Maximum ≥ 85

• Period: 10 minutes

When this alert fires:

1. Use flame graphs or the Flink UI to locate the hot-spot operator.

   • Business logic: Check for complex calculations, JSON parsing, or inefficient UDFs.

   • Data skew: Check whether a hot-spot key is overloading a single task with excessive data volume.

   • Insufficient resources: Determine whether the current degree of parallelism and TaskManager resources can handle the traffic, and whether there is severe backpressure.

   • Frequent GC: Check logs or JVM metrics to determine whether memory pressure is triggering frequent Full GCs that consume CPU.

2. Increase the degree of parallelism for the bottleneck operator, or allocate more CPU cores to the TaskManager.

Memory bottleneck

Alert when a TaskManager's heap memory exceeds 90% of capacity for 10 consecutive minutes. Derive the absolute threshold from the actual heap size shown on the Job O&M > Job Log page. For example, if the usage reads 194 MB / 413 MB, set the threshold to 372 MB (90% of 413 MB).

Metric: TM heap memory usage

Recommended configuration:

• Maximum ≥ Threshold (90% of total heap)

• Period: 10 minutes

When this alert fires:

1. Check GC logs to identify the problem type:

   • Memory leak: Heap does not return to baseline after GC and the baseline keeps rising.

   • Insufficient capacity: Heap usage stays consistently high, triggering frequent Full GCs and degrading performance.

   • Sudden OutOfMemoryError (OOM): Memory fills instantly when processing a specific record or batch.

2. Increase heap size or increase the degree of parallelism to reduce data volume per slot.