customize metrics in ASM - Alibaba Cloud Service Mesh - Alibaba Cloud Documentation Center

After you enable Prometheus metric collection for a Service Mesh ASM instance, its Envoy proxies begin outputting a default set of metrics. ASM allows you to customize these metrics at the mesh, Service Mesh, or Service Mesh level for more targeted observability. This topic describes how to customize metrics in Service MeshASM.

Prerequisites

The cluster is added to the ASM instance. For more information, see Add a cluster to an ASM instance.

Background

Istio generates the following metrics:

For HTTP, HTTP/2, and gRPC traffic, Istio generates the following metrics:

Request Count (istio_requests_total): A COUNTER incremented for every request handled by an Istio proxy.
Request Duration (istio_request_duration_milliseconds): A DISTRIBUTION that measures the duration of requests.
Request Size (istio_request_bytes): A DISTRIBUTION that measures HTTP request body sizes.
Response Size (istio_response_bytes): A DISTRIBUTION that measures HTTP response body sizes.
gRPC Request Message Count (istio_request_messages_total): A COUNTER incremented for every gRPC message sent from a client.
gRPC Response Message Count (istio_response_messages_total): A COUNTER incremented for every gRPC message sent from a server.

For TCP traffic, Istio generates the following metrics:

Tcp Bytes Sent (istio_tcp_sent_bytes_total): A COUNTER that measures the total bytes sent in TCP responses.
Tcp Bytes Received (istio_tcp_received_bytes_total): A COUNTER that measures the total bytes received in TCP requests.
Tcp Connections Opened (istio_tcp_connections_opened_total): A COUNTER incremented for every opened connection.
Tcp Connections Closed (istio_tcp_connections_closed_total): A COUNTER incremented for every closed connection.

Each Istio metric includes the following default labels:

DefaultStatTags=["reporter","source_namespace","source_workload","source_workload_namespace","source_principal","source_app","source_version","source_cluster","destination_namespace","destination_principal","destination_app","destination_version","destination_service","destination_service_name","destination_service_namespace","destination_port","destination_cluster","request_protocol","request_operation","response_flags","grpc_response_status","connection_security_policy","source_canonical_service","destination_canonical_service","source_canonical_revision","destination_canonical_revision"]

Label reference

Label	Description
reporter	The entity that reports the request. The value of this label varies based on the reporter: `destination` if a server-side Istio proxy is the reporter. `source` if a client-side Istio proxy or gateway is the reporter.
source_workload	The name of the source workload. Defaults to `unknown` if the value cannot be collected.
source_workload_namespace	The namespace of the source workload. Defaults to `unknown` if the value cannot be collected.
source_principal	The peer principal of the traffic source. This value is collected only if peer authentication is configured.
source_app	The source application, identified by the application label of the source workload. Defaults to `unknown` if the value cannot be collected.
source_version	The version of the source workload. Defaults to `unknown` if the value cannot be collected.
source_cluster	The cluster of the source workload.
destination_namespace	The namespace of the destination workload. Defaults to `unknown` if the value cannot be collected.
destination_workload	The name of the destination workload. Defaults to `unknown` if the value cannot be collected.
destination_workload_namespace	The namespace of the destination workload. Defaults to `unknown` if the value cannot be collected.
destination_principal	The peer principal of the traffic destination. This value is collected only if peer authentication is configured.
destination_app	The destination application, identified by the application label of the destination workload. Defaults to `unknown` if the value cannot be collected.
destination_version	The version of the destination workload. Defaults to `unknown` if the value cannot be collected.
destination_service	The host of the destination service that handles the request.
destination_service_name	The name of the destination service.
destination_service_namespace	The namespace of the destination service.
destination_port	The destination port.
destination_cluster	The cluster of the destination workload.
request_protocol	The protocol of the request.
request_operation	The operation of the request.
response_flags	Additional details about the response or connection from the proxy.
grpc_response_status	The gRPC response status.
connection_security_policy	The service authentication policy of the request.

Procedure

Before v1.15.3.65

Optional: Enable Prometheus monitoring.

Note
If you have already enabled Prometheus monitoring, skip this step.
1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
2. On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Instance > Base Information.
3. On the Basic Information page, click Settings in the upper-right corner.
4. In the Settings Update panel, select Enable Prometheus monitoring, select an option as needed, and then click OK.

Customize metrics.

In the left-side navigation pane, choose Observability Management Center > Customized Metrics, and then click Create.

On the Create page, configure the parameters, and then click Create.

Parameter	Description
Match Label	Set Name to the label name and Value to the application name. The custom metrics apply only to applications that match this label.
inboundSidecar	Click inboundSidecar, and then click Add Metrics to configure custom labels for inbound traffic. Metric Name: Leave this blank to apply the configuration to all metrics. Metric Dimension: Click Add Dimension to configure a Tag and an Expression. Tags_to_remove: Specify the dimension labels to remove.
outboundSidecar	Click outboundSidecar, and then click Add Metrics to configure custom labels for outbound traffic. Metric Name: Leave this blank to apply the configuration to all metrics. Metric Dimension: Click Add Dimension to configure a Tag and an Expression. Tags_to_remove: Specify the dimension labels to remove.
gateway	Click gateway, and then click Add Metrics to configure custom labels for the gateway. Metric Name: Leave this blank to apply the configuration to all metrics. Metric Dimension: Click Add Dimension to configure a Tag and an Expression. Tags_to_remove: Specify the dimension labels to remove.

Note

If you configure only inboundSidecar and do not configure outboundSidecar and gateway, custom labels apply only to inbound traffic. Default metrics are generated for outbound and gateway traffic.

v1.15.3.65 to v1.17.2.35

ASM replaced the Customized Metrics page with the Monitoring metrics page. The new UI streamlines configuration. To use this feature, your ASM instance must be v1.15.3.65-g97cc18dd-aliyun or later. To learn how to upgrade an instance, see Upgrade an ASM instance.

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Observability Management Center > Monitoring metrics.

On the Monitoring metrics page, click the Metric setting tab to configure Global or custom metrics as needed.

Configure global metrics

ASM provides a set of default global metrics, which are enabled by default. You can enable or disable related metrics or modify their dimensions as needed. For more information about the metrics, see Metric reference.

Actions	Description
Enable or disable global metrics	To enable a global metric: Click the Global tab. Find the target metric and select the Enabled checkbox in the client-side metric or server-side metric column, and then click preserve. To disable a global metric: Click the Global tab. Find the target metric and clear the Enabled checkbox in the client-side metric or server-side metric column, and then click preserve.
Modify metric dimensions	Click the Global tab. Find the target metric and click Edit dimension in the client-side metric or server-side metric column. In the Customize CLIENT dimension configuration dialog box, select the target dimensions, click Confirm, and then click preserve.

Global metrics include ALL_METRICS, REQUEST_COUNT, REQUEST_DURATION, REQUEST_SIZE, and RESPONSE_SIZE. Removed dimensions, such as source_app and source_version, are displayed in the corresponding metric rows.

Configure custom metrics

Click the custom tab and click Create. Configure the parameters, and then click Create at the bottom of the page. For more information about the metrics, see Metric reference.

Parameter	Description
Namespaces	Select the namespace where the custom metric applies.
Name	Enter a name for the custom metric configuration.
Workload selection	Click Add Matching Label to configure the label name and workload name. ASM matches applications based on the configuration, and the custom metrics apply only to the matched application.
Metric setting	ASM provides a set of default global metrics, which are enabled by default. You can enable or disable related metrics or modify their dimensions. Enable or disable global metrics: To enable a global metric: Find the target metric and select the Enabled checkbox in the client-side metric or server-side metric column. To disable a global metric: Find the target metric and clear the Enabled checkbox in the client-side metric or server-side metric column. Modify dimensions: Find the target metric and click Edit dimension in the client-side metric or server-side metric column. In the Customize CLIENT dimension configuration dialog box, select the target dimensions, and then click Confirm.

Important

You can use a selector to configure metrics for a specific workload in a namespace. The selector allows you to select workloads based on their labels.
The following configurations are invalid in the same namespace:
- Using a selector to create multiple custom metrics (Telemetry CRs) for the same workload.
- Creating multiple custom metrics without specifying a selector.

For more information, see Workload selection.

v1.17.2.35 and later

For more information, see Configure observability settings.

Metric reference

The following table describes the metrics in ASM. For more information, see MetricSelector.IstioMetric.

Note

ASM gateways export only client-side metrics. Sidecars can export both client-side and server-side metrics.

Metric	Description	Prometheus metric
ALL_METRICS	Represents all default Istio metrics.	-
REQUEST_COUNT	An application request counter for HTTP, HTTP/2, and gRPC traffic.	`istio_requests_total`
REQUEST_DURATION	A histogram of request durations for HTTP, HTTP/2, and gRPC traffic.	`istio_request_duration_milliseconds`
REQUEST_SIZE	A histogram of request body sizes for HTTP, HTTP/2, and gRPC traffic.	`istio_request_bytes`
RESPONSE_SIZE	A histogram of response body sizes for HTTP, HTTP/2, and gRPC traffic.	`istio_response_bytes`
TCP_OPENED_CONNECTIONS	A counter of TCP connections opened over the lifetime of a workload.	`istio_tcp_connections_opened_total`
TCP_CLOSED_CONNECTIONS	A counter of TCP connections closed over the lifetime of a workload.	`istio_tcp_connections_closed_total`
TCP_SENT_BYTES	A counter of bytes sent during responses over a TCP connection.	`istio_tcp_sent_bytes_total`
TCP_RECEIVED_BYTES	A counter of bytes received during requests over a TCP connection.	`istio_tcp_received_bytes_total`
GRPC_REQUEST_MESSAGES	A counter that increments for each gRPC message a client sends.	`istio_request_messages_total`
GRPC_RESPONSE_MESSAGES	A counter that increments for each gRPC message a server sends.	`istio_response_messages_total`