etcd component metrics and dashboard - Container Service for Kubernetes

Before you begin

Access

See View monitoring dashboards for control plane components.

List of metrics

The following table lists etcd component metrics.

Metric	Type	Description
cpu_utilization_core	Gauge	CPU usage, in cores.
etcd_server_has_leader	Gauge	In the Raft consensus algorithm, one etcd member is elected as leader. The leader sends periodic heartbeats to maintain cluster stability. Indicates whether a leader exists among etcd members. 1: A leader exists. 0: No leader exists.
etcd_server_is_leader	Gauge	Indicates whether this etcd member is the leader. 1: Yes. 0: No.
etcd_server_leader_changes_seen_total	Counter	Total leader changes observed by this etcd member.
etcd_mvcc_db_total_size_in_bytes	Gauge	The total size of the etcd member database.
etcd_mvcc_db_total_size_in_use_in_bytes	Gauge	The used size of the etcd member database.
etcd_disk_backend_commit_duration_seconds_bucket	Histogram	Time for a data change to be written to backend storage and committed. The bucket boundaries are `[0.001, 0.002, 0.004, 0.008, 0.016, 0.032, 0.064, 0.128, 0.256, 0.512, 1.024, 2.048, 4.096, 8.192]`.
etcd_debugging_mvcc_keys_total	Gauge	The total number of keys stored in etcd.
etcd_server_proposals_committed_total	Gauge	In Raft, state changes are submitted as proposals. Total proposals committed to the Raft log.
etcd_server_proposals_applied_total	Gauge	Total applied proposals.
etcd_server_proposals_pending	Gauge	Number of pending proposals.
etcd_server_proposals_failed_total	Counter	Number of failed proposals.
memory_utilization_byte	Gauge	Memory usage, in bytes.
resource_utilization_level	Gauge	Resource utilization level. resource: The resource type. Valid values: `cpu` and `memory`. utilization_level: The utilization level. Valid values: `high` (utilization ≥ 80%) and `normal` (utilization < 80%). container: The target container. Valid values: `kube-apiserver`, `kube-scheduler`, `kube-controller-manager`, `cloud-controller-manager`, and `etcd`.

Note

The following resource utilization metrics are no longer in use. Remove any alerts or monitoring rules that rely on these metrics:

cpu_utilization_ratio: CPU utilization.
memory_utilization_ratio: Memory utilization.

Dashboard guide

The dashboard uses PromQL queries to visualize etcd metrics.

Dashboard overview

Panel descriptions

Parameter	PromQL	Description
etcd liveness status	etcd_server_has_leader `etcd_server_is_leader == 1`	Indicates if the etcd members are alive. A value of 3 is normal. Indicates which member is the leader. A healthy cluster must have one leader.
Leader changes in the last day	`changes(etcd_server_leader_changes_seen_total{job="etcd"}[1d])`	Number of etcd leader changes over the last day.
Memory usage	`memory_utilization_byte{container="etcd"}`	Memory usage, in bytes.
CPU usage	`cpu_utilization_core{container="etcd"}*1000`	CPU usage, in millicores.
Memory usage level	`resource_utilization_level{resource="memory",container="etcd",utilization_level="high"}` `resource_utilization_level{resource="memory",container="etcd",utilization_level="normal"}`	If `resource_utilization_level{utilization_level="high",...}` is 1, the container resource utilization level is ≥ 80%. If `resource_utilization_level{utilization_level="normal",...}` is 1, the container resource utilization level is < 80%.
CPU usage level	`resource_utilization_level{resource="cpu",container="etcd",utilization_level="high"}` `resource_utilization_level{resource="cpu",container="etcd",utilization_level="normal"}`
Disk size	etcd_mvcc_db_total_size_in_bytes	The total size of the etcd backend database.
Disk size	etcd_mvcc_db_total_size_in_use_in_bytes	The used size of the etcd backend database.
Total key-value pairs	etcd_debugging_mvcc_keys_total	The total number of key-value pairs in the etcd cluster.
Backend commit latency	`histogram_quantile(0.99, sum(rate(etcd_disk_backend_commit_duration_seconds_bucket{job="etcd"}[5m])) by (instance, le))`	The time required to persist a proposal in the etcd database.
Raft proposal status	`rate(etcd_server_proposals_failed_total{job="etcd"}[1m])`	The rate of failed Raft proposals per minute.
	`etcd_server_proposals_pending{job="etcd"}`	The total number of pending Raft proposals.
	`etcd_server_proposals_committed_total{job="etcd"} - etcd_server_proposals_applied_total{job="etcd"}`	Number of committed but unapplied proposals.

Common metric anomalies

etcd liveness status

Normal case

Abnormal case

Description

All three etcd members have a leader, and one must be the leader:sum(etcd_server_has_leader)=3, and only one member satisfies member etcd_server_is_leader == 1.

A single member is abnormal.

The corresponding member etcd_server_has_leader!=1 does not affect overall etcd service.

More than one member is abnormal.

Multiple members report member etcd_server_has_leader!=1, preventing the etcd cluster from providing services.

Also check whether any member has etcd_server_is_leader == 1. If not, etcd is leaderless and cannot provide services.

Backend commit latency

Normal case	Abnormal case	Description
The latency is typically between a few and tens of milliseconds.	Sustained latency of hundreds of milliseconds or more.	Indicates disk I/O issues.

Raft proposal anomalies

Normal case	Abnormal case	Description
The rate of failed Raft proposals is 0.	The rate of failed Raft proposals is greater than 0.	Some proposals fail to commit. Investigate high failure rates.
The total number of pending Raft proposals is 0.	A non-zero number of pending Raft proposals persists.	Backlog of Raft proposals, often caused by slow apply speed. Correlate with backend commit latency.
The gap between committed and applied Raft proposals is 0.	The gap between committed and applied proposals is greater than 0.	May indicate excessive client request volume on etcd. If this value exceeds 5000, etcd rejects requests with `too many requests` until backlogged proposals are processed.

References

See Monitoring metrics for kube-apiserver, Monitoring metrics for kube-scheduler, Monitoring metrics for kube-controller-manager, and Monitoring metrics for cloud-controller-manager.