Application Monitoring metrics
This topic describes metrics commonly used in Application Monitoring, a sub-service of the Application Real-Time Monitoring Service (ARMS). You can use them to build custom Grafana dashboards and PromQL queries.
If your application is ingested through Managed Service for OpenTelemetry, only business metrics are available. JVM metrics, system metrics, and other categories are not supported.
Business metrics
Common dimensions
Dimension name | Dimension key |
Service name | service |
Service PID | pid |
Host IP | serverIp |
Interface | rpc |
Metric list
The following metrics apply to every access type. In queries, replace $callType with the actual access type. For the list of access types, see Service access types and available dimensions.
For example, to query HTTP request counts, change arms_$callType_requests_count to arms_http_requests_count.
Metric name | Metric | Metric type | Collection interval | Unit | Dimensions |
Request count | arms_$callType_requests_count | Gauge | 15 seconds | None | Different service access types have different dimensions. For details, see Service access types and available dimensions. |
Failed request count | arms_$callType_requests_error_count | Gauge | 15 seconds | None | |
Request latency | arms_$callType_requests_seconds | Gauge | 15 seconds | Seconds | |
Slow request count | arms_$callType_requests_slow_count | Gauge | 15 seconds | None | |
Request latency quantiles | arms_$callType_requests_latency_seconds | Summary | 15 seconds | Seconds | Available only when the access type is HTTP and quantile statistics are enabled. To turn them on, see Advanced settings. Quantile dimensions:
|
Except for quantile metrics, the metrics above are Gauge types: each point is the cumulative value over the collection interval, which differs from typical open-source metrics. For example, to estimate a 1-minute average QPS, use ARMS PromQL such as sum_over_time(arms_$callType_requests_count[1m])/60, whereas open-source frameworks often use rate(http_server_requests_count[1m]).
Metric storage
Business metrics are stored in the detailed metric store (metricstore-apm-metrics-detail).
Aggregated business metrics
Because each call type uses its own metric name, PromQL can become very long when one application uses multiple call types (for example, both HTTP and Dubbo).
Business metrics retain full observation dimensions, but some analytics scenarios do not need every dimension. Querying raw business metrics can be slow in those cases.
To address these issues, ARMS provides aggregated business metrics.
Metric description
Aggregated business metrics fall into these categories:
General
Records request count, failed request count, slow request count, and average request latency across all access types.
Database types
Same as General, scoped to database access types.
SQL
Same metrics as the database category, with an additional SQL dimension.
Exception
Records request count and average request latency when exceptions occur across all access types.
Status code
Records request count per HTTP status code.
Quantile
Records request latency quantiles across all access types.
Except for the quantile category, each category offers two metric shapes: a standard aggregate whose name ends with xxx_raw, and a reduced-dimension variant named xxx_ign_x_y, where x and y are dimensions rolled up (omitted from the metric series).
Metric type and collection interval
Unless otherwise noted, aggregated business metrics are Gauge metrics with a collection interval of 15 seconds.
Common dimensions
Every aggregated metric includes the following dimensions:
Dimensions | Description |
pid | Application PID |
service | Application name |
serverIp | Instance IP |
source | Metric source:
|
Metric list
Metric category | Metric name | Metric | Unit | Additional dimensions |
General | Request count | arms_app_requests_count_raw | Count |
|
arms_app_requests_count_ign_destid_endpoint_parent_ppid_prpc_rpc | Count | Excludes the destId, endpoint, parent, ppid, prpc, and rpc dimensions. | ||
arms_app_requests_count_ign_destid_endpoint_prpc_rpc | Count | Excludes the destId, endpoint, rpc, and prpc dimensions. | ||
arms_app_requests_count_ign_destid_endpoint_parent_ppid_prpc | Count | Excludes destId, endpoint, parent, ppid, prpc dimensions. | ||
arms_app_requests_count_ign_parent_ppid_prpc_rpc | Count | Excludes parent, ppid, prpc, rpc dimensions. | ||
Failed request count | arms_app_requests_error_count_raw | Count |
| |
arms_app_requests_error_count_ign_destid_endpoint_prpc_rpc | Count | Excludes the destId, endpoint, rpc, and prpc dimensions. | ||
arms_app_requests_error_count_ign_destid_endpoint_parent_ppid_prpc | Count | Excludes destId, endpoint, parent, ppid, prpc dimensions. | ||
arms_app_requests_error_count_ign_parent_ppid_prpc_rpc | Count | Excludes parent, ppid, prpc, rpc dimensions. | ||
arms_app_requests_error_count_ign_parent_ppid_prpc_rpc | Count | Excludes parent, ppid, prpc, rpc dimensions. | ||
Slow request count | arms_app_requests_slow_count_raw | Count |
| |
arms_app_requests_slow_count_ign_destid_endpoint_parent_ppid_prpc_rpc | Count | Excludes the destId, endpoint, parent, ppid, prpc, and rpc dimensions. | ||
arms_app_requests_slow_count_ign_destid_endpoint_prpc_rpc | Count | Excludes the destId, endpoint, rpc, and prpc dimensions. | ||
arms_app_requests_slow_count_ign_destid_endpoint_parent_ppid_prpc | Count | Excludes destId, endpoint, parent, ppid, prpc dimensions. | ||
arms_app_requests_slow_count_ign_parent_ppid_prpc_rpc | Count | Excludes parent, ppid, prpc, rpc dimensions. | ||
Request latency | arms_app_requests_seconds_raw | Seconds |
| |
arms_app_requests_seconds_ign_destid_endpoint_parent_ppid_prpc_rpc | Seconds | Excludes the destId, endpoint, parent, ppid, prpc, and rpc dimensions. | ||
arms_app_requests_seconds_ign_destid_endpoint_prpc_rpc | Seconds | Excludes the destId, endpoint, rpc, and prpc dimensions. | ||
arms_app_requests_seconds_ign_destid_endpoint_parent_ppid_prpc | Seconds | Excludes destId, endpoint, parent, ppid, prpc dimensions. | ||
arms_app_requests_seconds_ign_parent_ppid_prpc_rpc | Seconds | Excludes parent, ppid, prpc, rpc dimensions. | ||
Database types | Database request count | arms_db_requests_count_raw | Count |
|
arms_db_requests_count_ign_rpc | Count | Excludes interface dimension. | ||
Failed database request count | arms_db_requests_error_count_raw | Count |
| |
arms_db_requests_error_count_ign_rpc | Count | Excludes interface dimension. | ||
Slow database request count | arms_db_requests_slow_count_raw | Count |
| |
arms_db_requests_slow_count_ign_rpc | Count | Excludes interface dimension. | ||
Database request latency | arms_db_requests_seconds_raw | Seconds |
| |
arms_db_requests_seconds_ign_rpc | Seconds | Excludes interface dimension. | ||
SQL | SQL request count | arms_sql_requests_count_raw |
| |
arms_sql_requests_count_ign_rpc | Excludes interface dimension. | |||
Failed SQL request count | arms_sql_requests_error_count_raw | Count |
| |
arms_sql_requests_error_count_ign_rpc | Count | Excludes interface dimension. | ||
Slow SQL request count | arms_sql_requests_slow_count_raw | Count |
| |
arms_sql_requests_slow_count_ign_rpc | Count | Excludes interface dimension. | ||
SQL request latency | arms_sql_requests_seconds_raw | Seconds |
| |
arms_sql_requests_seconds_ign_rpc | Seconds | Excludes interface dimension. | ||
Exception | Abnormal request count | arms_exception_requests_count_raw | Count |
|
arms_exception_requests_count_ign_destid_endpoint_rpc | Count | Excludes destid, endpoint, and rpc dimensions. | ||
arms_exception_requests_count_ign_destid_endpoint | Count | Excludes destid and endpoint dimensions. | ||
Exception request latency | arms_exception_requests_seconds_raw | Seconds |
| |
arms_exception_requests_seconds_ign_destid_endpoint_rpc | Seconds | Excludes destid, endpoint, and rpc dimensions. | ||
arms_exception_requests_seconds_ign_destid_endpoint | Seconds | Excludes destid and endpoint dimensions. | ||
Status code | Request count by status code | arms_requests_by_status_count_raw | Count |
|
arms_requests_by_status_count_ign_rpc | Count | Excludes interface dimension. | ||
Quantile | Request latency quantiles Note Only supported by agent versions 4.x and later. | arms_uni_requests_latency_seconds |
|
Usage example
Choose a metric when using PromQL to sum requests across all interfaces of an application
You need a metric that exposes per-interface request counts. The General category in this topic satisfies that requirement.
If you only need the interface dimension—and not upstream interfaces, upstream applications, or remote hosts—choose a metric that keeps the interface dimension while omitting as many other dimensions as possible.
In practice, arms_app_requests_count_ign_destid_endpoint_parent_ppid_prpc is a good choice.
Metric storage
Metrics stored in the core metric store (metricstore-apm-metrics):
arms_app_requests_{metric}_ign_destid_endpoint_parent_ppid_prpc_rpc
arms_app_requests_{metric}_ign_destid_endpoint_rpc_prpc
arms_app_requests_{metric}_ign_destid_endpoint_parent_ppid_prpc
The remaining aggregate business metrics are stored in the detailed metric store (metricstore-apm-metrics-detail).
JVM metrics
Common dimensions
Dimension name | Dimension key |
Service name | service |
Service PID | pid |
Host IP | serverIp |
Metric list
Metric name | Metric | Metric type | Collection interval | Unit | Dimensions |
GC count (cumulative) | arms_jvm_gc_total | Counter | 15 seconds | None | Gen GC occurrence area:
Cause GC triggering reason (agent version 4.4.0 and later): System.gc(), Heap Dump Initiated GC, Allocation Failure, etc. |
GC time (cumulative) | arms_jvm_gc_seconds_total | Counter | 15 seconds | Seconds | |
GC count between collection intervals | arms_jvm_gc_delta | Gauge | 15 seconds | None | |
GC duration between collection intervals | arms_jvm_gc_seconds_delta | Gauge | 15 seconds | Seconds | |
JVM thread count | arms_jvm_threads_count | Gauge | 15 seconds | None | State thread status:
|
JVM memory initial size | arms_jvm_mem_init_bytes | Gauge | 15 seconds | Bytes | Area:
ID area breakdown:
|
JVM memory maximum size | arms_jvm_mem_max_bytes | Gauge | 15 seconds | Bytes | |
JVM memory used size | arms_jvm_mem_used_bytes | Gauge | 15 seconds | Bytes | |
JVM memory committed size | arms_jvm_mem_committed_bytes | Gauge | 15 seconds | Bytes | |
JVM memory usage ratio | arms_jvm_mem_usage_ratio | Gauge | 15 seconds | Ratio (0–1) | |
JVM loaded class count | arms_class_load_loaded | Counter | 15 seconds | None | None |
JVM unloaded class count | arms_class_load_un_loaded | Counter | 15 seconds | None | None |
JVM buffer pool size | arms_jvm_buffer_pool_total_bytes | Gauge | 15 seconds | Bytes | ID area:
|
JVM buffer pool used size | arms_jvm_buffer_pool_used_bytes | Gauge | 15 seconds | Bytes | |
JVM buffer pool count | arms_jvm_buffer_pool_count | Gauge | 15 seconds | None | |
Open file descriptor count | arms_file_desc_open_count | Gauge | 15 seconds | None | None |
File descriptor usage ratio (opened / limit) | arms_file_desc_open_ratio | Gauge | 15 seconds | Ratio (0–1) | None |
Metric storage
JVM metrics are stored in the core metric store (metricstore-apm-metrics).
System metrics
Common dimensions
Dimension name | Dimension key |
Service name | service |
Service PID | pid |
Host IP | serverIp |
Metric list
Metric name | Metric | Metric type | Collection interval | Unit |
CPU idle ratio | arms_system_cpu_idle | Gauge | 15 seconds | Percentage |
CPU I/O wait ratio | arms_system_cpu_io_wait | Gauge | 15 seconds | Percentage |
CPU system ratio | arms_system_cpu_system | Gauge | 15 seconds | Percentage |
CPU user ratio | arms_system_cpu_user | Gauge | 15 seconds | Percentage |
System load (1 minute) | arms_system_load | Gauge | 15 seconds | None |
Free disk space | arms_system_disk_free_bytes | Gauge | 15 seconds | Bytes |
Total disk space | arms_system_disk_total_bytes | Gauge | 15 seconds | Bytes |
Disk usage ratio | arms_system_disk_used_ratio | Gauge | 15 seconds | Ratio (0–1) |
System memory buffer size | arms_system_mem_buffers_bytes | Gauge | 15 seconds | Bytes |
System memory cache size | arms_system_mem_cached_bytes | Gauge | 15 seconds | Bytes |
Free system memory | arms_system_mem_free_bytes | Gauge | 15 seconds | Bytes |
Free swap space | arms_system_mem_swap_free_bytes | Gauge | 15 seconds | Bytes |
Total swap space | arms_system_mem_swap_total_bytes | Gauge | 15 seconds | Bytes |
Total system memory | arms_system_mem_total_bytes | Gauge | 15 seconds | Bytes |
Used system memory | arms_system_mem_used_bytes | Gauge | 15 seconds | Bytes |
Network ingress traffic | arms_system_net_in_bytes | Gauge | 15 seconds | Bytes |
Network egress traffic | arms_system_net_out_bytes | Gauge | 15 seconds | Bytes |
Network ingress error count | arms_system_net_in_err | Gauge | 15 seconds | None |
Network egress error count | arms_system_net_out_err | Gauge | 15 seconds | None |
Metric storage
System metrics are stored in the core metric store (metricstore-apm-metrics).
Thread pool/connection pool metrics
Common dimensions
Dimension name | Dimension key |
Service name | service |
Service PID | pid |
Host IP | serverIp |
Thread pool name (supported by agent versions earlier than 4.1.x) | name |
Thread pool type (supported by agent versions earlier than 4.1.x) | type |
Metric list
Agent version 4.1.x and later
Thread pool metrics
Metric name | Metric | Metric type | Collection interval | Dimensions |
Thread pool core thread count | arms_thread_pool_core_pool_size | Gauge | 15 seconds |
|
Thread pool maximum thread count | arms_thread_pool_max_pool_size | Gauge | 15 seconds |
|
Thread pool active thread count | arms_thread_pool_active_thread_count | Gauge | 15 seconds |
|
Thread pool current thread count | arms_thread_pool_current_thread_count | Gauge | 15 seconds |
|
Thread pool peak thread count (since pool creation) | arms_thread_pool_max_thread_count | Gauge | 15 seconds |
|
Thread pool scheduled task count | arms_thread_pool_scheduled_task_count | Counter | 15 seconds |
|
Thread pool completed task count | arms_thread_pool_completed_task_count | Counter | 15 seconds |
|
Thread pool rejected task count | arms_thread_pool_rejected_task_count | Counter | 15 seconds |
|
Thread pool task queue size | arms_thread_pool_queue_size | Gauge | 15 seconds |
|
Connection pool metrics
Metric description | Metric name | Metric type | Collection interval | Dimensions |
Connection count | arms_connection_pool_connection_count | Gauge | 15 seconds |
|
Minimum idle connection count | arms_connection_pool_connection_min_idle_count | Gauge | 15 seconds |
|
Maximum idle connection count | arms_connection_pool_connection_max_idle_count | Gauge | 15 seconds |
|
Maximum connection count | arms_connection_pool_connection_max_count | Gauge | 15 seconds |
|
Blocked connection request count | arms_connection_pool_pending_request_count | Counter | 15 seconds |
|
Agent versions earlier than 4.1.x
Metric name | Metric | Metric type | Collection interval | Dimensions |
Thread pool core thread count | arms_threadpool_core_size | Gauge | 15 seconds | None |
Thread pool maximum thread count | arms_threadpool_max_size | Gauge | 15 seconds | None |
Thread pool active thread count | arms_threadpool_active_size | Gauge | 15 seconds | None |
Thread pool queue size | arms_threadpool_queue_size | Gauge | 15 seconds | None |
Thread pool size | arms_threadpool_current_size | Gauge | 15 seconds | None |
Thread pool task count by state | arms_threadpool_task_total | Gauge | 15 seconds | StatusTask status:
|
Metric storage
Thread pool/connection pool metrics are stored in the core metric store (metricstore-apm-metrics).
Scheduled task metrics
The following metrics only exist for scheduled tasks.
Common dimensions
Dimension name | Dimension key |
Service name | service |
Service PID | pid |
Host IP | serverIp |
Task ID | rpc |
Metric list
Metric name | Metric | Metric type | Collection interval | Unit |
Scheduling delay | arms_$callType_delay_milliseconds | Gauge | 15 seconds | Milliseconds |
Metric storage
Scheduled task metrics are stored in the detailed metric store (metricstore-apm-metrics-detail).
Go runtime metrics
Metric list
Metric name | Metric | Metric type | Collection interval |
Application startup time (ms) | arms_golang_runtime_uptime | Int64Counter | 15 seconds |
Current Goroutine count | arms_golang_process_runtime_go_goroutines | Gauge | 15 seconds |
Heap object memory (bytes) | arms_golang_process_runtime_go_mem_heap_alloc | Gauge | 15 seconds |
Unallocated or reclaimed heap memory | arms_golang_process_runtime_go_mem_heap_idle | Gauge | 15 seconds |
Heap memory used | arms_golang_process_runtime_go_mem_heap_inuse | Gauge | 15 seconds |
allocated live heap objects | arms_golang_process_runtime_go_mem_heap_objects | Gauge | 15 seconds |
Heap memory released to the operating system | arms_golang_process_runtime_go_mem_heap_released | Gauge | 15 seconds |
Requested virtual memory size | arms_golang_process_runtime_go_mem_heap_sys | Gauge | 15 seconds |
Live object count | arms_golang_process_runtime_go_mem_live_objects | Gauge | 15 seconds |
GC count since program start | arms_golang_process_runtime_go_gc_count | Gauge | 15 seconds |
Cumulative GC pause time (stop-the-world) | arms_golang_process_runtime_go_gc_pause_total_ns | Int64Counter | 15 seconds |
GC time distribution | arms_golang_process_runtime_go_gc_pause_ns | Int64Histogram | 15 seconds |
Metric storage
Go runtime metrics are stored in the core metric store (metricstore-apm-metrics).
Service access types and available dimensions
Client-side types
Access type
http_client
dubbo_client
hsf_client
dsf_client
notify_client
grpc_client
thrift_client
sofa_client
mq_client
kafka_client
Dimensions
parent: the name of the upstream service
ppid: upstream service PID
destId: Request peer extension information
endpoint: request peer address
excepType: exception ID
excepInfo: Exception ID encoding rules
excepName: exception name
stackTraceId: exception stack ID
Database types
Access type
mysql
oracle
mariadb
postgresql
ppas
sqlserver
mongodb
dmdb
Dimensions
parent: the name of the upstream service
ppid: upstream service PID
destId: database name
endpoint: database address
excepType: exception ID
excepInfo: Exception ID encoding rules
excepName: exception name
stackTraceId: exception stack ID
sqlId: SQL statement ID
Server-side types
Access type
http
dubbo
hsf
dsf
user_method
mq
kafka
grpc
thrift
sofa
Dimensions
prpc: upstream interface
parent: the name of the upstream service
ppid: upstream service PID
endpoint: service address
excepType: exception ID
excepInfo: Exception ID encoding rules
excepName: exception name
stackTraceId: exception stack ID
Scheduled task types
Access type
xxl_job
spring_scheduled
quartz
elasticjob
jdk_timer
schedulerx
Dimensions
prpc: upstream interface
parent: the name of the upstream service
ppid: upstream service PID
excepType: exception ID
excepInfo: Exception ID encoding rules
excepName: exception name
stackTraceId: exception stack ID