Designing Better Alerting Workflows with Alibaba Cloud Observability Tools

An SRE perspective on reducing alert noise, improving signal quality, and making alerts more useful during production incidents with Alibaba Cloud observability tools.

By Sai Joshitha Kathari

A monitoring system can collect plenty of useful data and still make an incident harder to handle. During a production issue, several alerts may arrive within a short period of time. Latency increases, CPU rises, pods restart, error rates change, and a dependency may begin timing out. Each signal can be valid, but sending every one of them independently to the on-call engineer does not necessarily improve the response.

This is where alerting becomes part of reliability engineering. The goal is not simply to detect every unusual condition. The more important work is deciding which conditions need attention, how long they should persist before triggering a notification, and what context an engineer needs when that notification arrives.

Alibaba Cloud provides several services that can support this workflow. Managed Service for Prometheus can be used for metric-based monitoring and PromQL-driven alerts, Cloud Monitor provides alerting across cloud resources and applications, and Simple Log Service (SLS) includes alert monitoring, alert management, and notification capabilities.

Start with conditions that require action

One of the easiest ways to create alert noise is to turn every useful metric into a notification. Dashboards and alerts serve different purposes even when they rely on the same monitoring data.

Take CPU utilization as an example. A Kubernetes workload may briefly use a large amount of CPU during a traffic burst and then return to normal without affecting users. That behavior is still useful to see on a dashboard, especially during troubleshooting, but it may not justify paging someone.

The same CPU behavior becomes more important if it continues for several minutes while application latency also starts increasing. At that point, there is more evidence that the service may need attention.

I prefer to keep dashboards broad and paging conditions selective. Kubernetes environments naturally produce pod restarts, scaling events, scheduling activity, and short-lived resource changes. If every one of those events becomes an alert, the notification channel becomes harder to trust.

Duration matters along with the threshold

Thresholds usually get most of the attention when an alert rule is created, but the amount of time a condition remains abnormal can be just as important.

CPU above 80 percent for thirty seconds and CPU above 80 percent for fifteen minutes describe very different situations. The same applies to latency, queue depth, error rate, or resource pressure.

For this reason, I would not use the same evaluation window for every type of alert. An availability failure may require a quick response, while a resource-utilization alert can often tolerate a short observation period so that normal traffic bursts do not create unnecessary pages.

The timing should reflect the behavior of the workload and how quickly the condition can affect users.

Several signals may belong to the same incident

A slow backend dependency is a good example of how one problem can create many different signals.

Application response time may increase first. Requests begin waiting longer, some eventually time out, retries create additional traffic, and resource utilization starts changing. Later, pods may restart or autoscaling may respond.

A monitoring system can detect each of those symptoms separately, but from the engineer's point of view they may all belong to the same production incident.

SLS alert management can help reduce this type of noise by processing alerts before notifications are sent. Related alerts can be grouped, deduplicated, silenced, suppressed, or routed based on the alert-management configuration.

Labels such as service, cluster, namespace, environment, and severity can make grouping more useful. The important part is to keep the grouping aligned with the architecture. If it is too broad, unrelated failures may be combined. If it is too narrow, one service problem can still create a large number of independent notifications.

Severity should reflect the expected response

Severity becomes less useful when almost every notification is marked critical.

I find it more practical when the severity reflects what the engineer is expected to do. A critical alert may represent active user impact that requires immediate investigation. A warning may indicate degradation that should be reviewed but does not require waking someone overnight. Other conditions may be worth recording without generating a direct notification.

This also makes routing easier. When teams have a shared definition for each severity level, the monitoring system can send the right notifications to the right people without requiring the on-call engineer to reinterpret every alert.

Give the engineer enough context to get started

An alert saying that an error rate crossed a threshold is useful, but it still leaves the engineer with work to do before the investigation really begins.

A more useful notification identifies the affected service and environment, shows when the problem started, and provides a clear place to continue the investigation. Depending on the environment, that might be a dashboard, a log query, or a short runbook.

SLS action management can be used with notification policies, templates, user groups, and different notification channels. That makes it possible to include useful operational context without putting an entire dashboard into the notification.

I would keep the message focused. The engineer does not need every metric in the alert itself. They need enough information to know where to begin.

Use metrics and logs together

Metrics and logs usually answer different parts of the same incident.

Prometheus metrics may show that the error rate or latency changed. Logs can then help narrow down whether the change came from a specific exception, response code, dependency, or request pattern.

This is one reason I find it useful to connect metric-based monitoring with log investigation instead of treating them as completely separate workflows.

At the same time, individual error logs usually should not become individual pages. Production applications generate too much detail for that to remain useful. Log-based alerting becomes more practical when the events are aggregated into something operational, such as a sustained increase in a specific error pattern or the appearance of a condition that should be rare during normal operation.

Do not ignore missing telemetry

Monitoring data disappearing can also be a problem.

If a service normally exposes metrics continuously and the monitoring system suddenly stops receiving them, some existing alerts may stop evaluating in a useful way. The graph becomes quiet, but the service may not actually be healthy.

For important workloads, I would decide explicitly how missing data should be handled. A batch job may legitimately produce no metrics between runs, while a continuously serving API should normally keep producing telemetry.

Monitoring the health of the observability path itself can prevent a collection failure from looking like application recovery.

Use silences carefully

Silencing is useful during planned maintenance and during incidents that are already understood. The problem starts when a temporary silence is forgotten.

When possible, I would give each silence a reason, an owner, and an expiration time. This makes it less likely that a real problem will be hidden weeks later because of a change made during an earlier incident.

If an alert needs to remain silenced because it fires constantly, the rule itself probably deserves another look. The threshold may be too sensitive, the duration may be too short, or the condition may belong on a dashboard instead of in the paging path.

Fixing the alert rule is generally more useful than hiding its output permanently.

Review alert behavior after incidents

Post-incident reviews are a useful time to look at the monitoring timeline along with the application timeline.

I usually want to know which alert first pointed engineers toward the problem and whether it arrived early enough to help. From there, I look at the notifications that followed. Some may have added useful context, while others may simply have repeated symptoms that the team already understood.

It is also worth looking at what engineers had to discover manually. If a particular signal repeatedly becomes important during troubleshooting but is missing from the alerting workflow, that may be worth improving.

The result does not always need to be another alert. Sometimes increasing an evaluation period, changing the severity, consolidating duplicate notifications, or adding better context produces a bigger improvement.

Making alerts easier to trust

Alibaba Cloud provides the building blocks for metric monitoring, log analysis, alert management, and notification through services such as Managed Service for Prometheus, Cloud Monitor, and Simple Log Service.

From an SRE perspective, the useful part is being able to treat detection and notification as separate decisions. Detecting an unusual condition is only the first step. The alert still needs the right timing, severity, context, grouping, and destination before it becomes useful to the person responding.

As an alerting environment matures, it may actually become quieter. Engineers receive fewer notifications, but the alerts that remain are easier to understand and more likely to require action. During an incident, that helps keep attention on the system that needs to be fixed instead of on the monitoring system reporting it.

Disclaimer: The views expressed herein are for reference only and don't necessarily represent the official views of Alibaba Cloud.

Community

Designing Better Alerting Workflows with Alibaba Cloud Observability Tools

Read previous post:

Read next post:

Sai Joshitha Kathari

You may also like

Comments

Sai Joshitha Kathari

Related Products

Managed Service for Prometheus

CloudMonitor

AgentLoop

Simple Log Service