Building Kubernetes Observability with Alibaba Cloud Managed Service for Prometheus

This article shares an SRE perspective on building actionable Kubernetes observability and incident workflows with Alibaba Cloud Managed Service for Prometheus.

By Sai Joshitha Kathari

A Kubernetes incident rarely stays inside one layer of the stack. A service starts responding slowly, but the first useful clue might be a pod restart, CPU throttling on a node, a change in request volume, or a dependency that has become slow. By the time an engineer is looking at the problem, there may already be dozens of graphs and alerts available. The difficult part is deciding which signals actually help explain what changed.

That is the lens I use when thinking about observability. I care less about how many metrics a platform can collect and more about whether those metrics help an engineer move from a symptom to a useful hypothesis. Alibaba Cloud Managed Service for Prometheus fits naturally into that workflow for ACK because it keeps the Prometheus model familiar: metrics, labels, PromQL, dashboards, alert rules, and resources such as ServiceMonitor and PodMonitor.

What I want to see first during an incident

Imagine an API running in ACK. Response time has increased and users are beginning to notice. My first pass is usually simple: confirm whether the service is still handling requests, check the error rate, look at latency, and then compare that with pod and resource behavior. If latency rose at the same time as restarts, throttling, or replica loss, the investigation already has a direction. If those signals stayed normal, I move outward toward dependencies or traffic changes.

This is why a dashboard should reflect how an engineer investigates rather than how a monitoring product organizes its metric catalog. For a service dashboard, I would usually want request rate, errors, latency, replica availability, pod restarts, and CPU or memory behavior close together. Those signals answer different questions, but during an incident they are often part of the same story.

Cluster health is not service health

Kubernetes can look healthy while an application is not. Nodes may be Ready, the deployment may show the expected number of replicas, and every pod may be Running while the application still returns errors or times out. That gap matters because platform metrics and application metrics describe different parts of the system.

At the infrastructure level, node availability, CPU, memory, and resource pressure are useful. At the Kubernetes workload level, I look at restarts, scheduling problems, unavailable replicas, and resource requests or limits. At the application level, the useful signals depend on the service, but request rate, errors, response time, and a few application-specific indicators are usually more meaningful than generic container health alone.

Managed Service for Prometheus makes it possible to bring these layers into the same monitoring workflow. The point is not to collapse everything into one dashboard. It is to make correlation easier. If application latency begins rising at 2:15 PM, I want to know whether something else changed at 2:15 PM as well.

PromQL is most useful when the question is clear

PromQL can become complicated very quickly, but complicated queries are not automatically better queries. In production monitoring, I try to keep the operational question visible. A CPU graph by itself tells me utilization. CPU throttling on a latency-sensitive workload, especially when response time rises at the same moment, tells me much more. A total restart count is useful for history; a sudden increase in restart rate after a release is useful for investigation.

The same idea applies to latency. An average can look acceptable while a smaller set of requests becomes very slow. Looking at latency distributions or percentiles alongside traffic and error behavior gives a clearer picture of whether the issue is broad or concentrated in a subset of requests.

Alert only when someone should act

Most teams do not struggle because they have too few alerts. They struggle because too many alerts do not require action. A temporary CPU spike may recover on its own. One pod restart may be routine. Neither should automatically wake someone up in the middle of the night.

I would rather page on behavior that is sustained, user-relevant, or clearly moving toward service impact. That might be a persistent rise in error rate, a loss of available replicas, or latency that remains elevated long enough to matter. Duration is part of the design as well. Some conditions need time to prove they are real; others, especially availability failures, justify faster escalation.

Prometheus-based alerting is useful here because the alert can be defined around the same metrics and queries engineers already use during investigation. That keeps monitoring and incident response connected instead of treating alerting as a separate system with its own logic.

The monitoring path can fail too

One issue that deserves more attention is telemetry failure itself. An empty graph is ambiguous. It can mean there is no traffic, but it can also mean a target stopped being scraped or metrics stopped arriving. If alerting depends on those metrics, losing telemetry can create a false sense of calm.

For important services, I want at least a basic way to verify that expected targets are still being collected and that key metrics are arriving. This becomes especially useful after monitoring configuration changes, exporter failures, or workload moves. Observability should help identify application failures, but the observability path also needs enough visibility that engineers can recognize when it is the thing that broke.

Where the managed service helps

The practical advantage of a managed Prometheus service is that teams can keep familiar Prometheus concepts while spending less time operating the monitoring backend itself. That still leaves the part that requires engineering judgment: deciding what healthy behavior looks like, which signals deserve attention, how dashboards should support an investigation, and when an alert should interrupt someone.

For ACK workloads, that combination is useful because Kubernetes already gives engineers a large amount of telemetry. The challenge is turning it into a smaller set of signals that explain what is happening. A good monitoring setup does not need to show everything at once. It needs to help the on-call engineer find the next useful clue.

Disclaimer: The views expressed herein are for reference only and don't necessarily represent the official views of Alibaba Cloud.

Community

Building Kubernetes Observability with Alibaba Cloud Managed Service for Prometheus

What I want to see first during an incident

Cluster health is not service health

PromQL is most useful when the question is clear

Alert only when someone should act

The monitoring path can fail too

Where the managed service helps

Read next post:

Sai Joshitha Kathari

You may also like

Comments

Sai Joshitha Kathari

Related Products

CloudMonitor

Application Real-Time Monitoring Service

Simple Log Service

Managed Service for Prometheus