Different monitoring tools report different memory values for the same container because they measure different things. The most common source of confusion is the discrepancy between kubectl top pod output and the container_memory_usage_bytes metric in Container Service.
How memory usage is calculated
kubectl top pod reports container_memory_working_set_bytes, not container_memory_usage_bytes. The two metrics are related but include different memory components:
| Metric | Formula | What it measures |
|---|---|---|
container_memory_usage_bytes | container_memory_rss + container_memory_cache + kernel memory | All memory charged to the container, including file-backed pages that have not been accessed recently |
container_memory_working_set_bytes | container_memory_usage_bytes - total_inactive_file | Memory actively in use that cannot be reclaimed under memory pressure |
Because container_memory_usage_bytes includes inactive file cache, it is always equal to or greater than container_memory_working_set_bytes. This difference explains why the two values never match.
container_memory_working_set_bytes represents the actual memory consumption of the container. Kubernetes compares this value against the configured memory limit to decide whether to OOM-kill a container.
Which metric to use
| Scenario | Recommended metric | Reason |
|---|---|---|
| Monitor OOM risk | container_memory_working_set_bytes | Kubernetes compares this metric against the memory limit |
| Set memory requests and limits | container_memory_working_set_bytes | Reflects actual memory demand |
| Investigate high memory allocation | container_memory_usage_bytes | Helps identify whether inactive file cache is inflating the total |
Diagnose OOM-killed containers
If a container restarts unexpectedly, check its termination reason:
kubectl describe pod <pod-name>In the output, look for:
Last State: Terminated
Reason: OOMKilled
Exit Code: 137Exit Code: 137 confirms that the kernel terminated the container because container_memory_working_set_bytes exceeded the configured memory limit.