Application Real-Time Monitoring Service:Alerting best practices for Application Monitoring

Call volume

Interface call volume is the primary indicator of business health. A sudden drop usually signals an outage; a spike beyond capacity signals overload. Set a data range to trigger alerts when call volume falls outside the expected band.

During off-peak hours (such as midnight), absolute call volume is naturally low, which can trigger false positives on a static lower bound. Use a lower limit with link relative to detect rapid decreases caused by anomalies rather than normal traffic patterns.

Error rate

Set an upper limit on the error rate of interface calls to catch failures before they affect a significant portion of users.

Response time and slow calls

If your business requires low latency, add alerts for response time or the number of slow calls.

Exceptions

Exceptions typically spike during deployments or when a downstream service degrades. To catch these spikes, set an upper limit with link relative so that you are alerted on rapid increases rather than a constant background level.

An increase in exceptions does not always mean the application is broken -- graceful degradation can handle exceptions without affecting the response. However, some exceptions propagate to the interface as errors. Set an upper limit on the exception error rate to catch those cases.

HTTP 5xx status codes

For HTTP services, 4xx codes generally indicate client-side issues, while 5xx codes indicate server errors. Set an upper limit with link relative on 5xx codes to detect surges.

Response time

When an application encounters problems or traffic surges, overall response time often increases sharply. Set an upper limit for the average response time within a time window. The following example uses a 1-minute window. If your traffic fluctuates frequently, use a longer window (5 or 10 minutes) to smooth out noise.

Database and downstream service latency

Slow responses often come from either database queries or downstream service calls.

Database call response time: Database issues can increase latency by tens or even hundreds of times, so set a relatively high upper limit to avoid false positives on normal query variance.

Downstream service call response time: Set an upper limit to detect increasing latency from services your application depends on.

Downstream service call error rate: Set an upper limit to detect rising failure rates in downstream dependencies.

JVM health

For internal application issues, monitor JVM and thread pool metrics.

Full GC events: Traverse all nodes and configure two alerting conditions -- one for continuous Full GC events and one for frequent Full GC bursts in a short period -- because both patterns indicate abnormal memory behavior.

Runnable JVM threads: Excessive threads consume memory and can destabilize the process. Conversely, if no threads are runnable, the service is unresponsive. Add alerting conditions for both extremes.

Thread pool usage: To prevent thread pool exhaustion, alert on thread pool usage, active thread count, or maximum thread count over a time window.

If the thread pool size is not explicitly configured, it defaults to 2147483647 (the maximum 32-bit signed integer). In this case, do not use the maximum thread count as an alerting condition because it is not meaningful.

ECS node metrics

For applications deployed on Elastic Compute Service (ECS) instances, ARMS collects node-level metrics. Configure upper limits for CPU utilization, memory usage, and disk utilization.

CPU utilization: CPU readings fluctuate frequently. Rather than alerting on a single spike, check whether utilization continuously exceeds the threshold over a sustained period.

Memory usage:

Disk utilization:

ACK cluster metrics

For applications deployed in Container Service for Kubernetes (ACK) clusters:

• With Managed Service for Prometheus: Use Prometheus alert rules. For more information, see Monitor an ACK cluster.

• Without Managed Service for Prometheus: ARMS agent V4.1.0 and later collects CPU and memory data from ACK clusters. Set thresholds based on your cluster's request volume and resource allocation -- no universal default applies.

CPU utilization of ACK clusters:

Memory usage of ACK clusters:

Filter conditions

Filter conditions control which dimension of a metric is evaluated.

Alerting conditions

Recommended thresholds

Application Monitoring provides recommended thresholds for common scenarios. Use these as a starting point and adjust based on your application's baseline behavior.

To tune thresholds over time:

• Tighten thresholds that never fire. If a threshold is set too high, you may miss real incidents.

• Loosen thresholds that fire too often. Frequent false positives cause alert fatigue and slow response times.

• Temporarily widen thresholds during planned changes, such as deployments or migrations, to avoid a flood of expected alerts.

PromQL reference

All PromQL statements below use two placeholder variables:

• $dims -- grouping dimensions (similar to SQL GROUP BY). Replace with dimension keys such as service, pid, serverIp, or rpc.

• $labelFilters -- label selectors (similar to SQL WHERE). Replace with label key-value pairs such as "serverIp"="127.0.0.1".

sum by (rpc) (sum_over_time_lorc(arms_http_requests_count{"serverIp"="127.0.0.1"}[1m]))

sum by (serverIp) (sum_over_time_lorc(arms_http_requests_count{"rpc"="mall/pay"}[1m]))