This topic describes the key metrics for Milvus and open-source Milvus.
Milvus metrics
You can view these metrics on the Monitoring & Alerting tab of your target instance.
Cluster overview
Resources
Parameter
Description
Dimension
Search resource usage (QueryNode)
The memory usage of the Milvus QueryNode.
We recommend scaling out if QueryNode memory usage exceeds 70%.
Per pod
Write resource usage (DataNode/IndexNode)
The CPU utilization of the Milvus DataNode and IndexNode.
We recommend scaling out if CPU utilization exceeds 90%.
Per pod
Write resource usage reflects only the CPU utilization of write-related nodes and does not include the memory dimension. If the background compaction task of a node stays fully loaded for a long time but CPU utilization shows no anomaly, check whether insufficient memory is causing an out-of-memory (OOM) event. Follow these steps to troubleshoot:
On the Monitoring & Alerting tab, go to Monitoring dashboards and check the Memory Util metric under the Runtime group to confirm memory usage. You can use the Component drop-down list to filter by the components actually deployed in the instance. The available components vary by instance version: 2.6 instances include Streaming Node, Query Node, Data Node, and Proxy, while 2.5 and earlier versions include Index Node.
On the Monitoring & Alerting tab, go to Alert settings and configure a memory utilization alert rule (with the metric set to
ProcessResidentMemoryUtilizationV2) so that you can detect high memory utilization in time.Use Log query to check the logs of the corresponding node and confirm whether an OOM event occurred. You must enable Simple Log Service before you can query logs.
If OOM is confirmed due to insufficient memory, we recommend upgrading the specification of the corresponding node (for example, from 4 vCPU 16 GiB to 8 vCPU 32 GiB) to resolve the issue.
Data
Parameter
Description
Dimension
Collection Count
The total number of collections created in the current Milvus cluster. This metric helps you evaluate the cluster scale and metadata management overhead.
Per cluster
Entity Count
The total number of entities stored in each collection. This metric helps you monitor data scale and performance bottlenecks.
Per collection
Monitoring dashboards
Proxy
Chart name
Chart description
Dimension
Search Vector Count Rate
The number of vector search requests processed per second, which reflects the throughput of the search service.
Per pod + sum of all pods
Insert Vector Count Rate
The number of vector insert requests processed per second, which reflects the throughput of data writes.
Per pod
Search Latency
The P99 and average latency of search requests, which measures the stability of the search service.
Per pod + request type (search/query)
Mutation Latency
The P99 and average latency of mutation operations (such as insert and delete), which measures data write performance.
Per pod + request type
Success Request Rate
The number of successfully processed requests per second, which reflects service availability.
Per pod + function type (search/insert, etc.)
Failed Request Rate
The number of failed requests per second, which helps you locate anomalies or bottlenecks.
Per pod + function type
Query Node
Chart name
Chart description
Dimension
Search Request Rate
The number of search requests processed per second, which reflects the load of search tasks.
Per pod + request type (search/query) + status
Search Request Latency
The P99 and average latency of search requests, which measures the stability of search performance.
Per pod + request type
Queryable Entity Num
The number of loaded queryable/searchable entities, which reflects data availability.
Per collection + sum of all pods
Segment Loaded Num
The number of currently loaded segments. A segment is the smallest unit of data sharding.
Per pod + segment status (loaded/unloaded)
Data Node
Chart name
Chart description
Dimension
Msg Rows Consumed Rate
The number of message rows processed per second (insert/delete only), which reflects data write throughput.
Per pod + message type (insert/delete)
Flush Data Size Rate
The amount of data flushed to disk per second (insert/delete only), which reflects persistence efficiency.
Per pod + message type
Compaction Latency
The P99 and average latency of segment compaction tasks, which measures storage optimization efficiency.
Per pod + sum of all pods
Query Coord
Chart name
Chart description
Dimension
Collection Loaded Num
The total number of loaded collections currently managed by QueryCoord, which reflects the activity level of the data service.
Sum of all pods
Root Coord
Chart name
Chart description
Dimension
DDL Request Rate
The number of DDL operations processed per second, which reflects the metadata management load.
DDL status + function name (CreateCollection/DescribeCollection, etc.)
DDL Request Latency
The P99 and average latency of DDL requests, which measures the stability of metadata operations.
Per DDL function name
ID Alloc Rate
The number of unique IDs allocated per second, which reflects the concurrency capability of data writes.
Per cluster
Index Node
Chart name
Chart description
Dimension
Build Index Latency
The P99 and average latency of building vector indexes, which measures index generation efficiency.
Per pod
Open-source metrics
Component | Metric | Metric description |
Proxy | milvus_proxy_search_vectors_count | The cumulative number of vector search requests processed, which reflects the query load of the system. |
milvus_proxy_insert_vectors_count | The cumulative number of vector inserts processed, used to evaluate data write throughput. | |
milvus_proxy_sq_latency | The latency of search and query requests. | |
milvus_proxy_mutation_latency | The latency of data mutation operations, which measures write performance. | |
milvus_proxy_req_count | The number of received requests of all types. | |
milvus_proxy_cache_hit_count | Statistics on the hit and miss rates of each cache read operation. | |
Query Node | milvus_querynode_sq_req_count | The cumulative number of search and query requests. |
milvus_querynode_sq_req_latency | The query request latency of the query node. | |
milvus_querynode_entity_num | The number of queryable and searchable entities on each query node. | |
milvus_querynode_segment_num | The number of segments loaded on each query node. | |
Data Node | milvus_datanode_msg_rows_count | The number of rows of stream messages consumed by the data node. Currently, the stream messages counted by the data node include only insert and delete messages, used to evaluate data throughput. |
milvus_datanode_flushed_data_size | The size of each flushed message. Currently, the stream messages counted by the data node include only insert and delete messages. This metric reflects the amount of data involved in each flush operation, usually in bytes. | |
milvus_datanode_unflushed_segment_num | The number of unflushed segments created on each data node. This metric counts segments that have not yet been persisted. Too many can cause memory pressure. | |
milvus_datanode_compaction_latency | The time spent by each data node executing compaction tasks. | |
Query Coord | milvus_querycoord_collection_num | The number of collections currently loaded by Milvus. |
Root Coord | milvus_rootcoord_ddl_req_count | The cumulative number of all DDL requests (including CreateCollection, DescribeCollection, etc.), used to evaluate the metadata management load. |
milvus_rootcoord_ddl_req_latency | The latency of all types of DDL requests. This metric reflects the response time of DDL operations, in milliseconds. | |
milvus_rootcoord_id_alloc_count | The cumulative number of IDs allocated by Root Coord. | |
Index Node | milvus_indexnode_build_index_latency | The time spent building indexes. |