Customize monitoring metrics in ASM - Alibaba Cloud Service Mesh

After you enable Prometheus monitoring for a Service Mesh (ASM) instance, Envoy generates default metrics that you can customize for a specific mesh, namespace, or workload. This topic describes how to customize these monitoring metrics in Service Mesh (ASM).

Prerequisites

The cluster is added to the ASM instance. For more information, see Add a cluster to an ASM instance.

Background information

Istio generates default metrics, and each metric contains tags. This section describes the default metrics that Istio generates.

For HTTP, HTTP/2, and gRPC traffic, Istio generates the following metrics:

Request Count (istio_requests_total): This is a COUNTER incremented for every request handled by an Istio proxy.
Request Duration (istio_request_duration_milliseconds): This is a DISTRIBUTION which measures the duration of requests.
Request Size (istio_request_bytes): This is a DISTRIBUTION which measures HTTP request body sizes.
Response Size (istio_response_bytes): This is a DISTRIBUTION which measures HTTP response body sizes.
gRPC Request Message Count (istio_request_messages_total): This is a COUNTER incremented for every gRPC message sent from a client.
gRPC Response Message Count (istio_response_messages_total): This is a COUNTER incremented for every gRPC message sent from a server.

For TCP traffic, Istio generates the following metrics:

Tcp Bytes Sent (istio_tcp_sent_bytes_total): This is a COUNTER which measures the size of total bytes sent during response in case of a TCP connection.
Tcp Bytes Received (istio_tcp_received_bytes_total): This is a COUNTER which measures the size of total bytes received during request in case of a TCP connection.
Tcp Connections Opened (istio_tcp_connections_opened_total): This is a COUNTER incremented for every opened connection.
Tcp Connections Closed (istio_tcp_connections_closed_total): This is a COUNTER incremented for every closed connection.

Each metric that Istio generates contains the following default tags:

DefaultStatTags=["reporter","source_namespace","source_workload","source_workload_namespace","source_principal","source_app","source_version","source_app","source_version","source_cluster","destination_namespace","destination_principal","destination_app","destination_version","destination_service","destination_service_name","destination_service_namespace","destination_port","destination_cluster","request_protocol","request_operation","response_flags","grpc_response_status","connection_security_policy","source_canonical_service","destination_canonical_service","source_canonical_revision","destination_canonical_revision"]

Tag descriptions

Tag	Description
reporter	The reporter of the request. The tag value varies with the reporter. If the reporter is a server-side Istio proxy, the tag value is `destination`. If the reporter is a client-side Istio proxy or a gateway, the tag value is `source`.
source_workload	The name of the source workload. If no tag value is collected, the value is `unknown`.
source_workload_namespace	The namespace of the source workload. If no tag value is collected, the value is `unknown`.
source_principal	The peer principal of the traffic source. The value of this tag can be collected only if peer authentication is created.
source_app	The source application identified by the application tag of the source workload. If no tag value is collected, the value is `unknown`.
source_version	The version of the source workload. If no tag value is collected, the value is `unknown`.
source_cluster	The cluster of the source workload.
destination_namespace	The namespace of the destination workload. If no tag value is collected, the value is `unknown`.
destination_workload	The name of the destination workload. If no tag value is collected, the value is `unknown`.
destination_workload_namespace	The namespace of the destination workload. If no tag value is collected, the value is `unknown`.
destination_principal	The peer principal of the traffic destination. The value of this tag can be collected only if peer authentication is created.
destination_app	The destination application identified by the application tag of the destination workload. If no tag value is collected, the value is `unknown`.
destination_version	The version of the destination workload. If no tag value is collected, the value is `unknown`.
destination_service	The destination service host that is responsible for an incoming request.
destination_service_name	The name of the destination service.
destination_service_namespace	The namespace of the destination service.
destination_port	The destination port.
destination_cluster	The cluster of the destination workload.
request_protocol	The protocol of the request.
request_operation	The requested operation.
response_flags	Additional details about the response or connection from the proxy.
grpc_response_status	The gRPC response status.
connection_security_policy	The service authentication policy of the request.

Procedure

For ASM instances earlier than V1.15.3.65

(Optional) Enable Prometheus monitoring.
Note
If you have enabled Prometheus monitoring, skip this step.
1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
2. On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Instance > Base Information.
3. On the Basic Information page, click Feature Settings in the upper-right corner.
4. In the Update Feature Settings panel, select Enable Prometheus Metric Collection, select an enabling method as needed, and then click OK.

Customize metrics.

In the navigation pane on the left, choose Observability Management Center > Custom Metrics, and then click Create.

On the Create page, configure the parameters and click Create.

Configuration item	Description
Match Labels	Set Name to the tag name and Value to the application name. ASM matches the application based on the specified value. The custom metrics take effect only on this application.
inboundSidecar	Click inboundSidecar, and then click Add Metric to configure custom tags for inbound traffic. Metric Name: Leave this empty to apply to all metrics. Metric Dimension: Click Add Metric Dimension to configure Tag and Expression. Tags To Remove: Specify the tags to remove from the metric dimension.
outboundSidecar	Click outboundSidecar, and then click Add Metric to configure custom tags for outbound traffic. Metric Name: Leave this empty to apply to all metrics. Metric Dimension: Click Add Metric Dimension to configure Tag and Expression. Tags To Remove: Specify the tags to remove from the metric dimension.
gateway	Click gateway, and then click Add Metric to configure custom tags for the gateway. Metric Name: Leave this empty to apply to all metrics. Metric Dimension: Click Add Metric Dimension to configure Tag and Expression. Tags To Remove: Specify the tags to remove from the metric dimension.

配置inboundSidecar

Note

If you configure only inboundSidecar and leave outboundSidecar and gateway unconfigured, custom tags are used only for inbound traffic. Metrics for outbound traffic and the gateway are generated using default methods.

For ASM instances that are V1.15.3.65 or later and earlier than V1.17.2.35

ASM has upgraded the Custom Metrics page and renamed it to Monitoring Metrics. The new UI simplifies configuration and improves efficiency. To use this feature, your ASM instance must be V1.15.3.65-g97cc18dd-aliyun or later. For more information about how to upgrade an ASM instance, see Upgrade an ASM instance.

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Observability Management Center > Monitoring metrics.

On the Monitoring metrics page, click the Metric Settings tab. Configure Global or custom metrics as needed.

Configure global metrics

ASM provides some global metrics that are enabled by default. You can enable or disable these metrics or modify their dimensions as needed. For more information about the metrics, see Metric descriptions.

Operation	Description
Enable or disable a global metric	To enable a global metric: Click the Global tab. For the target metric, select Enable in the CLIENT side indicator or SERVER side index column, and then click Save. To disable a global metric: Click the Global tab. For the target metric, clear Enable in the CLIENT side indicator or SERVER side index column, and then click Save.
Modify metric dimensions	Click the Global tab. For the target metric, click Edit dimensions in the CLIENT side indicator or SERVER side index column. In the Configure Custom CLIENT side dimension dialog box, select the target dimensions, click OK, and then click Save.

全局指标

Configure custom metrics

Click the custom tab, click Create, configure the parameters, and then click Create at the bottom of the page. For more information about these metrics, see Metric descriptions.

Configuration item	Description
Namespace	Select the namespace where the custom monitoring metrics take effect.
Name	Enter a name for the custom metric.
Workload Selector	Click Add Match Label to configure the tag name and workload name. ASM matches an application based on the configuration. The custom metrics take effect only on this application.
Metric Settings	ASM provides some global metrics that are enabled by default. You can enable or disable the metrics or modify their dimensions. Enable or disable a global metric: To enable a global metric: For the target metric, select Enable in the CLIENT-side Metrics or SERVER-side Metrics column. To disable a global metric: For the target metric, clear Enable in the CLIENT-side Metrics or SERVER-side Metrics column. Modify dimensions: For the target metric, click Edit Dimensions in the CLIENT-side Metrics or SERVER-side Metrics column. In the Configure Custom CLIENT-side Dimensions dialog box, select the target dimensions, and then click OK.

自定义指标

Important

You can use a selector to configure metrics for a single workload in a namespace. The selector supports selecting workloads based on tags.
In the same namespace, the following configurations are invalid:
- Using a selector to create multiple custom metrics (Telemetry custom resources) for the same workload.
- Creating multiple custom metrics without specifying a selector.

For more information, see Workload Selection.

For ASM instances V1.17.2.35 or later

For more information, see Configure observability settings.

Metric descriptions

The following table describes the ASM monitoring metrics. For more information, see MetricSelector.IstioMetric.

Note

ASM gateways support exporting only client-side metrics. Sidecars can export both client-side and server-side metrics.

Metric	Description	Corresponding metric in Prometheus
ALL_METRICS	All default Istio metrics.	-
REQUEST_COUNT	A counter for application requests generated for HTTP, HTTP/2, and gRPC traffic.	istio_requests_total
REQUEST_DURATION	A histogram of request durations for HTTP, HTTP/2, and gRPC traffic.	istio_request_duration_milliseconds
REQUEST_SIZE	A histogram of request body sizes for HTTP, HTTP/2, and gRPC traffic.	istio_request_bytes
RESPONSE_SIZE	A histogram of response body sizes for HTTP, HTTP/2, and gRPC traffic.	istio_response_bytes
TCP_OPENED_CONNECTIONS	A counter for TCP connections opened during the lifecycle of a workload.	istio_tcp_connections_opened_total
TCP_CLOSED_CONNECTIONS	A counter for TCP connections closed during the lifecycle of a workload.	istio_tcp_connections_closed_total
TCP_SENT_BYTES	A counter for the bytes sent during a response over a TCP connection.	istio_tcp_sent_bytes_total
TCP_RECEIVED_BYTES	A counter for the bytes received during a request over a TCP connection.	istio_tcp_received_bytes_total
GRPC_REQUEST_MESSAGES	A counter that increments for each gRPC message sent from a client.	istio_request_messages_total
GRPC_RESPONSE_MESSAGES	A counter that increments for each gRPC message sent from a server.	istio_response_messages_total