After you enable Prometheus monitoring for an Alibaba Cloud Service Mesh (ASM) instance, Envoy generates default metrics. Custom metrics are supported in ASM. You can customize metrics for a specified ASM instance, namespace, or workload. This topic describes how to customize metrics in ASM.

Background information

Istio generates default metrics. Each metric contains labels. This section describes the default metrics that Istio generates.
  • For HTTP, HTTP/2, and gRPC traffic, Istio generates the following metrics:
    Request Count (istio_requests_total): This is a COUNTER incremented for every request handled by an Istio proxy.
    Request Duration (istio_request_duration_milliseconds): This is a DISTRIBUTION which measures the duration of requests.
    Request Size (istio_request_bytes): This is a DISTRIBUTION which measures HTTP request body sizes.
    Response Size (istio_response_bytes): This is a DISTRIBUTION which measures HTTP response body sizes.
    gRPC Request Message Count (istio_request_messages_total): This is a COUNTER incremented for every gRPC message sent from a client.
    gRPC Response Message Count (istio_response_messages_total): This is a COUNTER incremented for every gRPC message sent from a server.
  • For TCP traffic, Istio generates the following metrics:
    Tcp Bytes Sent (istio_tcp_sent_bytes_total): This is a COUNTER which measures the size of total bytes sent during response in case of a TCP connection.
    Tcp Bytes Received (istio_tcp_received_bytes_total): This is a COUNTER which measures the size of total bytes received during request in case of a TCP connection.
    Tcp Connections Opened (istio_tcp_connections_opened_total): This is a COUNTER incremented for every opened connection.
    Tcp Connections Closed (istio_tcp_connections_closed_total): This is a COUNTER incremented for every closed connection.
Each metric that Istio generates contains the following default labels:
DefaultStatTags=["reporter","source_namespace","source_workload","source_workload_namespace","source_principal","source_app","source_version","source_app","source_version","source_cluster",destination_namespace","destination_principal","destination_app","destination_version","destination_service","destination_service_name","destination_service_namespace","destination_port","destination_cluster","request_protocol","request_operation","request_host","response_flags","grpc_response_status","connection_security_policy","source_canonical_service","destination_canonical_service","source_canonical_revision","destination_canonical_revision"]

Labels

Label Description
reporter The reporter of the request. The label value varies with the reporter.
  • If the reporter is a server Istio proxy, the label is set to destination.
  • If the reporter is a client Istio proxy or a gateway, the label is set to source.
source_workload The name of the source workload that controls the source. If no value is collected for the label, the label is set to unknown.
source_workload_namespace The namespace of the source workload. If no value is collected for the label, the label is set to unknown.
source_principal The peer principal of the traffic source. You can collect the value of the label only if peer authentication is used.
source_app The source application that is identified based on the application label of the source workload. If no value is collected for the label, the label is set to unknown.
source_version The version of the source workload. If no value is collected for the label, the label is set to unknown.
source_cluster The cluster of the source workload.
destination_namespace The namespace of the destination workload. If no value is collected for the label, the label is set to unknown.
destination_workload The name of the destination workload. If no value is collected for the label, the label is set to unknown.
destination_workload_namespace The namespace of the destination workload. If no value is collected for the label, the label is set to unknown.
destination_principal The peer principal of the traffic destination. You can collect the value of the label only if peer authentication is used.
destination_app The destination application that is identified based on the application label of the destination workload. If no value is collected for the label, the label is set to unknown.
destination_version The version of the destination workload. If no value is collected for the label, the label is set to unknown.
destination_service The destination service host responsible for an incoming request.
destination_service_name The name of the destination service.
destination_service_namespace The namespace of the destination service.
destination_port The port of the destination service.
destination_cluster The cluster of the destination workload.
request_protocol The protocol of the request.
request_operation The requested operation.
request_host The host of the request.
response_flags Additional details about the response or connection from the proxy.
grpc_response_status The response status of gRPC.
connection_security_policy The service authentication policy of the request.

Customize metrics for a specified ASM instance

After you customize metrics for a specified ASM instance, ASM generates the custom metrics from the perspective of the entire ASM instance.

  1. Enable Prometheus monitoring.
    Note If you have enabled Prometheus monitoring, skip this step.
    1. Log on to the ASM console.
    2. In the left-side navigation pane, choose Service Mesh > Mesh Management.
    3. On the Mesh Management page, find the ASM instance that you want to configure. Click the name of the ASM instance or click Manage in the Actions column.
    4. On the details page of the ASM instance, choose ASM Instance > Base Information in the left-side navigation pane.
    5. In the Basic Information section, click Settings in the upper-right corner.
    6. In the Settings Update panel, select Enable Prometheus and then Confirm that the added ACK cluster has installed the Alibaba Cloud ARMS Prometheus monitoring plug-in, and use ARMS Prometheus to collect monitoring indicators. Then, click OK.
  2. On the details page of the ASM instance, choose Observability Management > Customized Metrics in the left-side navigation pane. On the page that appears, click Create.
  3. On the Create page, set the Namespace parameter to istio-system.
  4. Click the Show icon icon to the right of inboundSidecar and click Add Metrics.
  5. Leave the Metric Name parameter empty, click Add Dimension, set the Tag and Expression parameters, and then specify the labels that you want to delete.
    After the Metric Name parameter is left empty, the specified labels are collected for all metrics.
  6. Set outboundSidecar and gateway by referring to Step 4 and Step 5, and then click Create.
    If you set inboundSidecar but not outboundSidecar or gateway, the custom labels are used only for the inbound traffic. The default labels are used to generate the metrics for the outbound traffic and gateway.

Customize metrics for a specified namespace

After you customize metrics for a specified namespace, ASM generates the custom metrics only for the specified namespace. The default metrics are generated for the other namespaces.

  1. Disable Prometheus monitoring.
    If you enable Prometheus monitoring and customize metrics, duplicate metrics are generated.
    Note If you have disabled Prometheus monitoring, skip this step.
    1. Log on to the ASM console.
    2. In the left-side navigation pane, choose Service Mesh > Mesh Management.
    3. On the Mesh Management page, find the ASM instance that you want to configure. Click the name of the ASM instance or click Manage in the Actions column.
    4. On the details page of the ASM instance, choose ASM Instance > Base Information in the left-side navigation pane.
    5. In the Basic Information section, click Settings in the upper-right corner.
    6. In the Settings Update panel, clear Enable Prometheus. Then, click OK.
  2. On the details page of the ASM instance, choose Observability Management > Customized Metrics in the left-side navigation pane. On the page that appears, click Create.
  3. On the Create page, select a namespace from the Namespace drop-down list and set the Name parameter.
  4. Click the Show icon icon to the right of inboundSidecar and click Add Metrics.
  5. Set the Metric Name parameter, click Add Dimension, set the Tag and Expression parameters, and then specify the labels that you want to delete.
  6. Set outboundSidecar by referring to Step 4 and Step 5, and then click Create.
    If you set inboundSidecar but not outboundSidecar, the custom labels are used only for the inbound traffic. The default labels are used to generate the metrics for the outbound traffic.

Customize metrics for a specified workload

After you customize metrics for a specified workload, ASM generates the custom metrics only for the specified workload. The default metrics are generated for the other workloads.

  1. Disable Prometheus monitoring.
    If you enable Prometheus monitoring and customize metrics, duplicate metrics are generated.
    Note If you have disabled Prometheus monitoring, skip this step.
    1. Log on to the ASM console.
    2. In the left-side navigation pane, choose Service Mesh > Mesh Management.
    3. On the Mesh Management page, find the ASM instance that you want to configure. Click the name of the ASM instance or click Manage in the Actions column.
    4. On the details page of the ASM instance, choose ASM Instance > Base Information in the left-side navigation pane.
    5. In the Basic Information section, click Settings in the upper-right corner.
    6. In the Settings Update panel, clear Enable Prometheus. Then, click OK.
  2. On the details page of the ASM instance, choose Observability Management > Customized Metrics in the left-side navigation pane. On the page that appears, click Create.
  3. On the Create page, select a namespace from the Namespace drop-down list, set the Name parameter, and then click Add Matching Label. Set the Name parameter to app and the Value parameter to the name of the application.
    ASM matches the application based on the specified value. This indicates that the custom metrics take effect only for the application.
  4. Click the Show icon icon to the right of inboundSidecar and click Add Metrics.
  5. Set the Metric Name parameter, click Add Dimension, set the Tag and Expression parameters, and then specify the labels that you want to delete.
  6. Set outboundSidecar by referring to Step 4 and Step 5, and then click Create.
    If you set inboundSidecar but not outboundSidecar, the custom labels are used only for the inbound traffic. The default labels are used to generate the metrics for the outbound traffic.