Create custom metrics in ASM - Alibaba Cloud Service Mesh

After you enable Prometheus monitoring for a Service Mesh (ASM) instance, Envoy generates default metrics. You can create custom metrics for a specific ASM instance, namespace, or workload. This topic describes how to create custom metrics that apply to an ASM instance, namespace, or workload.

Prerequisites

The cluster is added to the ASM instance. For more information, see Add a cluster to an ASM instance.

Background information

Istio generates default metrics. Each metric contains labels. This section describes the default metrics that Istio generates.

For HTTP, HTTP/2, and gRPC traffic, Istio generates the following metrics:

Request Count (istio_requests_total): This is a COUNTER incremented for every request handled by an Istio proxy.
Request Duration (istio_request_duration_milliseconds): This is a DISTRIBUTION which measures the duration of requests.
Request Size (istio_request_bytes): This is a DISTRIBUTION which measures HTTP request body sizes.
Response Size (istio_response_bytes): This is a DISTRIBUTION which measures HTTP response body sizes.
gRPC Request Message Count (istio_request_messages_total): This is a COUNTER incremented for every gRPC message sent from a client.
gRPC Response Message Count (istio_response_messages_total): This is a COUNTER incremented for every gRPC message sent from a server.

For TCP traffic, Istio generates the following metrics:

Tcp Bytes Sent (istio_tcp_sent_bytes_total): This is a COUNTER which measures the size of total bytes sent during response in case of a TCP connection.
Tcp Bytes Received (istio_tcp_received_bytes_total): This is a COUNTER which measures the size of total bytes received during request in case of a TCP connection.
Tcp Connections Opened (istio_tcp_connections_opened_total): This is a COUNTER incremented for every opened connection.
Tcp Connections Closed (istio_tcp_connections_closed_total): This is a COUNTER incremented for every closed connection.

Each metric that Istio generates contains the following default labels:

DefaultStatTags=["reporter","source_namespace","source_workload","source_workload_namespace","source_principal","source_app","source_version","source_app","source_version","source_cluster",destination_namespace","destination_principal","destination_app","destination_version","destination_service","destination_service_name","destination_service_namespace","destination_port","destination_cluster","request_protocol","request_operation","request_host","response_flags","grpc_response_status","connection_security_policy","source_canonical_service","destination_canonical_service","source_canonical_revision","destination_canonical_revision"]

Label description

Label	Description
reporter	The reporter of the request. The label value varies with the reporter. If the reporter is a server sidecar proxy, the label is set to `destination`. If the reporter is a client sidecar proxy or an ASM gateway, the label is set to `source`.
source_workload	The name of the source workload. If no value is collected for the label, the label is set to `unknown`.
source_workload_namespace	The namespace of the source workload. If no value is collected for the label, the label is set to `unknown`.
source_principal	The peer principal of the traffic source. Prometheus monitoring can collect the value of the label only if peer authentication is used.
source_app	The source application that is identified based on the application label of the source workload. If no value is collected for the label, the label is set to `unknown`.
source_version	The version of the source workload. If no value is collected for the label, the label is set to `unknown`.
source_cluster	The cluster of the source workload.
destination_namespace	The namespace of the destination workload. If no value is collected for the label, the label is set to `unknown`.
destination_workload	The name of the destination workload. If no value is collected for the label, the label is set to `unknown`.
destination_workload_namespace	The namespace of the destination workload. If no value is collected for the label, the label is set to `unknown`.
destination_principal	The peer principal of the traffic destination. Prometheus monitoring can collect the value of the label only if peer authentication is used.
destination_app	The destination application that is identified based on the application label of the destination workload. If no value is collected for the label, the label is set to `unknown`.
destination_version	The version of the destination workload. If no value is collected for the label, the label is set to `unknown`.
destination_service	The destination service host that is responsible for an incoming request.
destination_service_name	The name of the destination service.
destination_service_namespace	The namespace of the destination service.
destination_port	The port of the destination service.
destination_cluster	The cluster of the destination workload.
request_protocol	The protocol of the request.
request_operation	The requested operation.
request_host	The host of the request.
response_flags	Additional details about the response or connection from the Envoy proxy.
grpc_response_status	The response status of gRPC.
connection_security_policy	The service authentication policy of the request.

Procedure

Operations if the version of your ASM instance is earlier than 1.15.3.65

(Optional) Enable Prometheus monitoring.
Note
If you have enabled Prometheus monitoring, skip this step.
1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
2. On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Instance > Base Information.
3. On the Basic Information page, click Settings in the upper-right corner.
4. In the Settings Update panel, select Enable Managed Service for Prometheus, select an Application Real-Time Monitoring Service (ARMS) Prometheus instance or a self-managed Prometheus instance, and then click OK.

Create custom metrics.

In the left-side navigation pane, choose Observability Management Center > Customized Metrics. Then, click Create.

On the Create page, configure the related parameters and click Create.

Configuration item	Description
Matching Label	Set the Name parameter to the name of the label and the Value parameter to the name of the application. ASM matches an application based on the specified value. The custom metrics take effect only for the application.
inboundSidecar	Click the inboundSidecar tab, and then click Add Metrics to configure a custom label for inbound traffic. Metric Name: If you do not set this parameter, the label applies to all metrics. Metric Dimension: Click Add Dimension and configure the Tag and Expression parameters. Tags_to_remove: Configure the labels whose metric dimensions need to be deleted.
outboundSidecar	Click the outboundSidecar tab, and then click Add Metrics to configure a custom label for outbound traffic. Metric Name: If you do not set this parameter, the label applies to all metrics. Metric Dimension: Click Add Dimension and configure the Tag and Expression parameters. Tags_to_remove: Configure the labels whose metric dimensions need to be deleted.
gateway	Click the gateway tab, and then click Add Metrics to configure a custom label for the gateway. Metric Name: If you do not set this parameter, the label applies to all metrics. Metric Dimension: Click Add Dimension and configure the Tag and Expression parameters. Tags_to_remove: Configure the labels whose metric dimensions need to be deleted.

Note

If you configure parameters on the inboundSidecar tab and do not configure parameters on the outboundSidecar and gateway tabs, the custom labels are used only for inbound traffic. The default labels are used to generate the metrics for the outbound traffic and gateway.

Operations if the version of your ASM instance is 1.15.3.65 or later or earlier than 1.17.2.35

In the ASM console, the Customized Metrics user interface is updated and renamed as Monitoring indicators. This new user interface makes your configurations easier and more efficient. To use this feature, the version of your ASM instance must be V1.15.3.65-g97cc18dd-aliyun or later. For more information about how to update an ASM instance, see Update an ASM instance.

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Observability Management Center > Monitoring indicators.

On the Monitoring indicators page, click the Indicator setting tab. Then, click the Global or custom tab to configure global or custom metrics based on your business requirements.

Configure global metrics

ASM provides some global metrics that are enabled by default. You can enable or disable global metrics or modify the dimensions of the metrics based on your business requirements. For more information about metrics, see the Metric description section of this topic.

Operation	Description
Enable or disable a global metric	Enable a global metric: Click the Global tab. Find the metric that you want to enable, select Enabled in the CLIENT side Indicator or SERVER side index column, and then click preserve. Disable a global metric: Click the Global tab. Find the metric that you want to disable, deselect Enabled in the CLIENT side Indicator or SERVER side index column, and then click preserve.
Modify the dimensions of a global metric	Click the Global tab. Find the metric whose dimensions you want to modify, and click Edit dimension in the CLIENT side Indicator or SERVER side index column. In the Customize CLIENT dimension configuration dialog box, select the dimensions that you want to modify, click Submit, and then click preserve.

Configure custom metrics

Click the custom tab. On the custom tab, click Create, configure the related parameters, and then click Create in the lower part of the page. For more information about metrics, see the Metric description section of this topic.

Configuration item	Description
Namespace	The namespace in which the custom metric takes effect.
Name	The name of the custom metric.
Workload selection	Click Add Matching Label and then specify the label name and the workload name. ASM matches an application based on the configuration items. The custom metrics take effect only for the application.
Indicator setting	ASM provides some global metrics that are enabled by default. You can enable the metrics or modify the dimensions of the metrics. Enable or disable a global metric: Enable a global metric: Find the metric that you want to enable, and select Enabled in the CLIENT side Indicator or SERVER side index column. Disable a global metric: Find the metric that you want to disable, and deselect Enabled in the CLIENT side Indicator or SERVER side index column. Modify the dimensions of a metric: Find the metric whose dimensions you want to modify, and click Edit dimension in the CLIENT side Indicator or SERVER side index column. In the Customize CLIENT dimension configuration dialog box, select the dimensions that you want to modify and click Submit.

Important

You can use a selector to select a single workload within a namespace. You can use a selector to select workloads based on labels.
The following operations are invalid in the same namespace:
- Create multiple custom metrics for a workload by using a selector. These custom metrics are configured by using Telemetry custom resources.
- Create multiple custom metrics without specifying a selector.

For more information, see Workload Selection.

Operations if the version of your ASM instance is 1.17.2.35 or later

For more information about how to create custom metrics, see Configure Observability Settings.

Metric description

The following table describes the metrics of ASM. For more information, see MetricSelector.IstioMetric.

Note

If the reporter is an ASM gateway, only metrics on the client side are exported. If the reporter is a sidecar proxy, metrics on the client and server sides are exported.

Metric	Description	Metric in Prometheus
ALL_METRICS	All default Istio metrics.	-
REQUEST_COUNT	This counter is incremented for every request generated by an application for HTTP, HTTP/2, and gRPC traffic.	istio_requests_total
REQUEST_DURATION	This counter is a histogram and measures the duration of requests generated for HTTP, HTTP/2, and gRPC traffic.	istio_request_duration_milliseconds
REQUEST_SIZE	This counter is a histogram and measures body sizes of requests generated for HTTP, HTTP/2, and gRPC traffic.	istio_request_bytes
RESPONSE_SIZE	This counter is a histogram and measures body sizes of responses generated for HTTP, HTTP/2, and gRPC traffic.	istio_response_bytes
TCP_OPENED_CONNECTIONS	This counter is incremented for every opened TCP connection during the lifetime of the workload.	istio_tcp_connections_opened_total
TCP_CLOSED_CONNECTIONS	This counter is incremented for every closed TCP connection during the lifetime of the workload.	istio_tcp_connections_closed_total
TCP_SENT_BYTES	This counter measures the number of total bytes sent during a response over a TCP connection.	istio_tcp_sent_bytes_total
TCP_RECEIVED_BYTES	This counter measures the number of total bytes received during a request over a TCP connection.	istio_tcp_received_bytes_total
GRPC_REQUEST_MESSAGES	This counter is incremented for every gRPC message sent from a client.	istio_request_messages_total
GRPC_RESPONSE_MESSAGES	This counter is incremented for every gRPC message sent from a server.	istio_response_messages_total