This topic describes the common metrics in ARMS Application Monitoring. You can use these metrics to create custom Grafana dashboards.
Applications connected through Managed Service for OpenTelemetry support only business metrics. Other metrics, such as JVM metrics and system metrics, are not supported.
Business metrics
Common dimensions
Dimension name | Dimension key |
Service name | service |
Service PID | pid |
Machine IP | serverIp |
Interface | rpc |
Metric list
All access types include the following metrics. When you run a query, replace $callType with a specific access type. For more information about access types, see Service access types and available dimensions.
For example, to query the number of requests for an HTTP service, change arms_$callType_requests_count to arms_http_requests_count.
Metric name | Metric | Metric type | Collection interval | Unit | Dimensions |
Request count | arms_$callType_requests_count | Gauge | 15 s | None | The dimensions vary based on the service access type. For more information, see Service access types and available dimensions. |
Error request count | arms_$callType_requests_error_count | Gauge | 15 s | None | |
Request duration | arms_$callType_requests_seconds | Gauge | 15 s | Seconds | |
Slow request count | arms_$callType_requests_slow_count | Gauge | 15 s | None | |
Request duration quantiles | arms_$callType_requests_latency_seconds | Summary | 15 s | Seconds | This dimension is available only when the service access type is HTTP and quantile statistics are enabled. To enable quantile statistics, see Advanced Settings. Quantile dimension:
|
Except for quantile metrics, the preceding metrics are all of the Gauge type. This means the value of each data point represents the cumulative total value within the collection interval. This is different from metrics generated by open source frameworks. For example, to calculate the average queries per second (QPS) over one minute, the Prometheus Query Language (PromQL) expression for an ARMS metric is sum_over_time(arms_$callType_requests_count[1m])/60. For an open source framework, the expression is typically rate(http_server_requests_count[1m]).
Metric storage instance description
Business metrics are stored in the detailed metric library (metricstore-apm-metrics-detail).
Aggregated business metrics
Business metrics create a separate metric for each call type. This makes the PromQL expression long and complex when a single application has multiple call types, such as HTTP and Dubbo.
Business metrics record all observation dimensions. However, not all dimensions are needed in some statistical scenarios. This causes poor performance when you directly query business metrics in some scenarios.
To resolve these issues, ARMS provides aggregated business metrics.
Metric description
Aggregated business metrics are classified into the following categories:
General
Records the request count, error count, slow request count, and average request duration for all access types.
Database
Records the request count, error count, slow request count, and average request duration for database access types.
SQL class
Records the request count, error count, slow request count, and average request duration for database access types. Compared with the database category, this category adds the SQL dimension.
Exception
Records the request count and average request duration for all access types when an exception occurs.
Status code
Records the request count for different status codes in HTTP scenarios.
Quantile
Records the request duration quantiles for all access types.
Except for the quantile category, each major category has two types of metrics. One is the normal aggregated metric, named in the xxx_raw format. The other is the reduced-dimension metric, named in the xxx_ign_x_y format, where x and y are the aggregated dimensions excluded from the metric.
Metric type and collection interval
Unless otherwise specified, all aggregated business metrics are of the Gauge type, and the collection interval is 15 seconds.
Common dimensions
The following dimensions exist in every aggregated metric:
Dimension | Description |
pid | Application PID |
service | Application name |
serverIp | Instance IP |
source | Metric source:
|
Metric list
Metric category | Metric name | Metric | Unit | Other dimensions |
General | Request count | arms_app_requests_count_raw | Unit |
|
arms_app_requests_count_ign_destid_endpoint_parent_ppid_prpc_rpc | Unit | Excludes the destId, endpoint, parent, ppid, prpc, and rpc dimensions. | ||
arms_app_requests_count_ign_destid_endpoint_rpc_prpc | Unit | Excludes the destId, endpoint, rpc, and prpc dimensions. | ||
arms_app_requests_count_ign_destid_endpoint_parent_ppid_prpc | Item | Excludes the destId, endpoint, parent, ppid, and prpc dimensions. | ||
arms_app_requests_count_ign_parent_ppid_prpc_rpc | Unit | Excludes the parent, ppid, prpc, and rpc dimensions. | ||
Error request count | arms_app_requests_error_count_raw | item |
| |
arms_app_requests_error_count_ign_destid_endpoint_rpc_prpc | Unit | Excludes the destId, endpoint, rpc, and prpc dimensions. | ||
arms_app_requests_error_count_ign_destid_endpoint_parent_ppid_prpc | Unit | Excludes the destId, endpoint, parent, ppid, and prpc dimensions. | ||
arms_app_requests_error_count_ign_parent_ppid_prpc_rpc | Unit | Excludes the parent, ppid, prpc, and rpc dimensions. | ||
arms_app_requests_error_count_ign_parent_ppid_prpc_rpc | Count | Excludes the parent, ppid, prpc, and rpc dimensions. | ||
Slow request count | arms_app_requests_slow_count_raw | unit |
| |
arms_app_requests_slow_count_ign_destid_endpoint_parent_ppid_prpc_rpc | Item | Excludes the destId, endpoint, parent, ppid, prpc, and rpc dimensions. | ||
arms_app_requests_slow_count_ign_destid_endpoint_rpc_prpc | Unit | Excludes the destId, endpoint, rpc, and prpc dimensions. | ||
arms_app_requests_slow_count_ign_destid_endpoint_parent_ppid_prpc | Item | Excludes the destId, endpoint, parent, ppid, and prpc dimensions. | ||
arms_app_requests_slow_count_ign_parent_ppid_prpc_rpc | Item | Excludes the parent, ppid, prpc, and rpc dimensions. | ||
Request duration | arms_app_requests_seconds_raw | Seconds |
| |
arms_app_requests_seconds_ign_destid_endpoint_parent_ppid_prpc_rpc | Seconds | Excludes the destId, endpoint, parent, ppid, prpc, and rpc dimensions. | ||
arms_app_requests_seconds_ign_destid_endpoint_rpc_prpc | Seconds | Excludes the destId, endpoint, rpc, and prpc dimensions. | ||
arms_app_requests_seconds_ign_destid_endpoint_parent_ppid_prpc | Seconds | Excludes the destId, endpoint, parent, ppid, and prpc dimensions. | ||
arms_app_requests_seconds_ign_parent_ppid_prpc_rpc | Seconds | Excludes the parent, ppid, prpc, and rpc dimensions. | ||
Database | Database request count | arms_db_requests_count_raw | Unit |
|
arms_db_requests_count_ign_rpc | Item | Excludes the interface dimension. | ||
Database request error count | arms_db_requests_error_count_raw | Unit |
| |
arms_db_requests_error_count_ign_rpc | Unit | Excludes the interface dimension. | ||
Database slow request count | arms_db_requests_slow_count_raw | Item |
| |
arms_db_requests_slow_count_ign_rpc | Count | Excludes the interface dimension. | ||
Database request duration | arms_db_requests_seconds_raw | Seconds |
| |
arms_db_requests_seconds_ign_rpc | Seconds | Excludes the interface dimension. | ||
SQL Class | SQL request count | arms_sql_requests_count_raw |
| |
arms_sql_requests_count_ign_rpc | Excludes the interface dimension. | |||
SQL request error count | arms_sql_requests_error_count_raw | Unit |
| |
arms_sql_requests_error_count_ign_rpc | Item | Excludes the interface dimension. | ||
SQL slow request count | arms_sql_requests_slow_count_raw | Count |
| |
arms_sql_requests_slow_count_ign_rpc | Unit | Excludes the interface dimension. | ||
SQL request duration | arms_sql_requests_seconds_raw | Seconds |
| |
arms_sql_requests_seconds_ign_rpc | Seconds | Excludes the interface dimension. | ||
Exception | Exception request count | arms_exception_requests_count_raw | Item |
|
arms_exception_requests_count_ign_destid_endpoint_rpc | Unit | Excludes the destid, endpoint, and rpc dimensions. | ||
arms_exception_requests_count_ign_destid_endpoint | Unit | Excludes the destid and endpoint dimensions. | ||
Exception request duration | arms_exception_requests_seconds_raw | Seconds |
| |
arms_exception_requests_seconds_ign_destid_endpoint_rpc | Seconds | Excludes the destid, endpoint, and rpc dimensions. | ||
arms_exception_requests_seconds_ign_destid_endpoint | Seconds | Excludes the destid and endpoint dimensions. | ||
Status code | Request count by status code | arms_requests_by_status_count_raw | Count |
|
arms_requests_by_status_count_ign_rpc | Item | Excludes the interface dimension. | ||
Quantile | Request duration quantiles Note Supported by agent V4.x and later. | arms_uni_requests_latency_seconds |
|
Usage example
How to select a metric to count the requests of all application interfaces using PromQL
Find a metric that provides the request count for interfaces. The general category of metrics meets this requirement.
The statistical result requires only the interface dimension. Other dimensions, such as upstream interface, upstream application, and remote address, are not required. When you select a specific metric, make sure that the metric includes the interface dimension and as few other dimensions as possible.
Based on the preceding analysis, the optimal metric is arms_app_requests_count_ign_destid_endpoint_parent_ppid_prpc.
Metric storage instance description
The following metrics are stored in the core metric library (metricstore-apm-metrics):
arms_app_requests_{metric}_ign_destid_endpoint_parent_ppid_prpc_rpc
arms_app_requests_{metric}_ign_destid_endpoint_rpc_prpc
arms_app_requests_{metric}_ign_destid_endpoint_parent_ppid_prpc
Other aggregated business metrics are stored in the detailed metric library (metricstore-apm-metrics-detail).
JVM metrics
Common dimensions
Dimension name | Dimension key |
Service name | service |
Service PID | pid |
Machine IP | serverIp |
Metric list
Metric name | Metric | Metric type | Collection interval | Unit | Dimensions |
Total GC count | arms_jvm_gc_total | Counter | 15 s | None | Generational GC area:
Cause (GC cause) (supported by agent V4.4.0 and later): System.gc(), Heap Dump Initiated GC, Allocation Failure, and more. |
Total GC duration | arms_jvm_gc_seconds_total | Counter | 15 s | Seconds | |
GC count per interval | arms_jvm_gc_delta | Gauge | 15 s | None | |
GC duration per interval | arms_jvm_gc_seconds_delta | Gauge | 15 s | Seconds | |
JVM thread count | arms_jvm_threads_count | Gauge | 15 s | None | State (thread state):
|
Initial size of the JVM memory region | arms_jvm_mem_init_bytes | Gauge | 15 s | Bytes | Area:
ID (region breakdown):
|
Maximum size of the JVM memory region | arms_jvm_mem_max_bytes | Gauge | 15 s | Bytes | |
Used size of the JVM memory region | arms_jvm_mem_used_bytes | Gauge | 15 s | Bytes | |
Committed size of the JVM memory region | arms_jvm_mem_committed_bytes | Gauge | 15 s | Bytes | |
Usage ratio of the JVM memory region | arms_jvm_mem_usage_ratio | Gauge | 15 s | Ratio (0 to 1) | |
JVM loaded classes | arms_class_load_loaded | Counter | 15 s | None | None |
JVM unloaded classes | arms_class_load_un_loaded | Counter | 15 s | None | None |
JVM buffer pool size | arms_jvm_buffer_pool_total_bytes | Gauge | 15 s | Bytes | ID (region):
|
Used size of the JVM buffer pool | arms_jvm_buffer_pool_used_bytes | Gauge | 15 s | Bytes | |
Number of JVM buffer pools | arms_jvm_buffer_pool_count | Gauge | 15 s | None | |
Number of open file descriptors | arms_file_desc_open_count | Gauge | 15 s | None | None |
Ratio of open file descriptors (open/maximum allowed) | arms_file_desc_open_ratio | Gauge | 15 s | Ratio (0 to 1) | None |
Metric storage instance description
JVM metrics are stored in the core metric library (metricstore-apm-metrics).
System metrics
Common dimensions
Dimension name | Dimension key |
Service name | service |
Service PID | pid |
Machine IP | serverIp |
Metric list
Metric name | Metric | Metric type | Collection interval | Unit |
CPU idle percentage | arms_system_cpu_idle | Gauge | 15 s | Percentage |
CPU I/O wait percentage | arms_system_cpu_io_wait | Gauge | 15 s | Percentage |
CPU system percentage | arms_system_cpu_system | Gauge | 15 s | Percentage |
CPU user percentage | arms_system_cpu_user | Gauge | 15 s | Percentage |
System load (1 minute) | arms_system_load | Gauge | 15 s | None |
Free disk space | arms_system_disk_free_bytes | Gauge | 15 s | Bytes |
Total disk space | arms_system_disk_total_bytes | Gauge | 15 s | Bytes |
Disk usage | arms_system_disk_used_ratio | Gauge | 15 s | Ratio (0 to 1) |
Memory buffer size | arms_system_mem_buffers_bytes | Gauge | 15 s | Bytes |
Memory cache size | arms_system_mem_cached_bytes | Gauge | 15 s | Bytes |
Free memory size | arms_system_mem_free_bytes | Gauge | 15 s | Bytes |
Free swap memory size | arms_system_mem_swap_free_bytes | Gauge | 15 s | Bytes |
Swap memory size | arms_system_mem_swap_total_bytes | Gauge | 15 s | Bytes |
Memory size | arms_system_mem_total_bytes | Gauge | 15 s | Bytes |
Used memory size | arms_system_mem_used_bytes | Gauge | 15 s | Bytes |
Inbound network traffic size | arms_system_net_in_bytes | Gauge | 15 s | Bytes |
Outbound network traffic size | arms_system_net_out_bytes | Gauge | 15 s | Bytes |
Inbound network errors | arms_system_net_in_err | Gauge | 15 s | None |
Outbound network errors | arms_system_net_out_err | Gauge | 15 s | None |
Metric storage instance description
System metrics are stored in the core metric library (metricstore-apm-metrics).
Thread pool/Connection pool metrics
Common dimensions
Dimension name | Dimension key |
Service name | service |
Service PID | pid |
Machine IP | serverIp |
Thread pool name (supported by agent versions earlier than 4.1.x) | name |
Thread pool type (supported by agent versions earlier than 4.1.x) | type |
Metric list
Agent versions 4.1.x and later
Thread pool metrics
Metric name | Metric | Metric type | Collection interval | Dimensions |
Core thread count | arms_thread_pool_core_pool_size | Gauge | 15 s |
|
Maximum thread count | arms_thread_pool_max_pool_size | Gauge | 15 s |
|
Active thread count | arms_thread_pool_active_thread_count | Gauge | 15 s |
|
Current thread count | arms_thread_pool_current_thread_count | Gauge | 15 s |
|
Peak thread count in the thread pool (the highest number of threads since the pool was created) | arms_thread_pool_max_thread_count | Gauge | 15 s |
|
Scheduled task count | arms_thread_pool_scheduled_task_count | Counter | 15 s |
|
Completed task count | arms_thread_pool_completed_task_count | Counter | 15 s |
|
Rejected task count | arms_thread_pool_rejected_task_count | Counter | 15 s |
|
Task queue size | arms_thread_pool_queue_size | Gauge | 15 s |
|
Connection pool metrics
Metric description | Metric name | Metric type | Collection interval | Dimensions |
Connections | arms_connection_pool_connection_count | Gauge | 15 s |
|
Minimum idle connections | arms_connection_pool_connection_min_idle_count | Gauge | 15 s |
|
Maximum idle connections | arms_connection_pool_connection_max_idle_count | Gauge | 15 s |
|
Maximum connections | arms_connection_pool_connection_max_count | Gauge | 15 s |
|
Pending connection requests | arms_connection_pool_pending_request_count | Counter | 15 s |
|
Agent versions earlier than 4.1.x
Metric name | Metric | Metric type | Collection interval | Dimensions |
Thread pool core size | arms_threadpool_core_size | Gauge | 15 s | None |
Thread pool maximum size | arms_threadpool_max_size | Gauge | 15 s | None |
Thread pool active size | arms_threadpool_active_size | Gauge | 15 s | None |
Thread pool queue size | arms_threadpool_queue_size | Gauge | 15 s | None |
Thread pool current size | arms_threadpool_current_size | Gauge | 15 s | None |
Number of tasks in different states in the thread pool | arms_threadpool_task_total | Gauge | 15 s | Status (task state):
|
Metric storage instance description
Thread pool and connection pool metrics are stored in the core metric library (metricstore-apm-metrics).
Scheduled task metrics
The following metrics are available only for scheduled tasks.
Common dimensions
Dimension name | Dimension key |
Service name | service |
Service PID | pid |
Machine IP | serverIp |
Task ID | rpc |
Metric list
Metric name | Metric | Metric type | Collection interval | Unit |
Scheduling delay | arms_$callType_delay_milliseconds | Gauge | 15 s | ms |
Metric storage instance description
Scheduled task metrics are stored in the detailed metric library (metricstore-apm-metrics-detail).
Go runtime metrics
Metric list
Metric name | Metric | Metric type | Collection interval |
Application uptime (ms) | arms_golang_runtime_uptime | Int64Counter | 15 s |
Number of goroutines in the current application | arms_golang_process_runtime_go_goroutines | Gauge | 15 s |
Heap object memory (bytes) | arms_golang_process_runtime_go_mem_heap_alloc | Gauge | 15 s |
Unallocated or reclaimed heap memory | arms_golang_process_runtime_go_mem_heap_idle | Gauge | 15 s |
Used heap memory | arms_golang_process_runtime_go_mem_heap_inuse | Gauge | 15 s |
Allocated and live heap objects | arms_golang_process_runtime_go_mem_heap_objects | Gauge | 15 s |
Memory in HeapIdle that has been released to the operating system | arms_golang_process_runtime_go_mem_heap_released | Gauge | 15 s |
Size of virtual memory requested from the system | arms_golang_process_runtime_go_mem_heap_sys | Gauge | 15 s |
Number of current live objects | arms_golang_process_runtime_go_mem_live_objects | Gauge | 15 s |
Number of GC cycles since the program started | arms_golang_process_runtime_go_gc_count | Gauge | 15 s |
Cumulative stop-the-world garbage collection pause time, which is the unavailable time | arms_golang_process_runtime_go_gc_pause_total_ns | Int64Counter | 15 s |
GC pause time distribution | arms_golang_process_runtime_go_gc_pause_ns | Int64Histogram | 15 s |
Metric storage instance description
Go runtime metrics are stored in the core metric library (metricstore-apm-metrics).
Service access types and available dimensions
Client-side types
Access types
http_client
dubbo_client
hsf_client
dsf_client
notify_client
grpc_client
thrift_client
sofa_client
mq_client
kafka_client
Dimensions
parent: Upstream service name
ppid: Upstream service PID
destId: Extended information about the request peer
endpoint: Request peer address
excepType: Exception ID
excepInfo: Exception ID encoding rule
excepName: Exception name
stackTraceId: Stack trace ID
DB types
Access types
mysql
oracle
mariadb
postgresql
ppas
sqlserver
mongodb
dmdb
Dimensions
parent: Upstream service name
ppid: Upstream service PID
destId: Database name
endpoint: Database address
excepType: Exception ID
excepInfo: Exception ID encoding rule
excepName: Exception name
stackTraceId: Stack trace ID
sqlId: SQL statement ID
Server-side types
Access types
http
dubbo
hsf
dsf
user_method
mq
kafka
grpc
thrift
sofa
Dimensions
prpc: Upstream interface
parent: Upstream service name
ppid: Upstream service PID
Endpoint: The address of the service.
excepType: Exception ID
excepInfo: Exception ID encoding rule
excepName: Exception name
stackTraceId: Stack trace ID
Scheduled task types
Access types
xxl_job
spring_scheduled
quartz
elasticjob
jdk_timer
schedulerx
Dimensions
prpc: Upstream interface
parent: Upstream service name
ppid: Upstream service PID
excepType: Exception ID
excepInfo: Exception ID encoding rule
excepName: Exception name
stackTraceId: Stack trace ID