how to use observability configuration - Alibaba Cloud Service Mesh

Service Mesh (ASM) provides observability configurations for logs, metrics, and tracing. You can use the ASM console to customize these configurations for the entire mesh, a specific namespace, or a specific workload. For example, you can set the log output format, define metric dimensions, enable or disable specific metrics, and set the sampling rate for tracing. This topic describes how to use the observability configuration feature.

Prerequisites

An ASM instance of version 1.17.2.35 or later is created. For more information, see Create an ASM instance or Upgrade an ASM instance.

Applicable scope

Type	Description
Global	Global configurations support settings for logs, metrics, and tracing. Only one global configuration exists, and it cannot be deleted. Only global configurations support tracing settings.
Namespace	Create a dedicated observability configuration for a namespace. Each namespace can have only one namespace-level observability configuration.
Custom	Use a workload selector to define the scope of a custom configuration. Each workload can be selected by at most one custom configuration.

Procedure

Global

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Observability Management Center > Observability Settings.

On the Observability Settings page, click the Global tab, configure logs, metrics, and Tracing Analysis as needed, and then click Submit.

Click the links in the following table to view detailed descriptions of the configurations.

Configuration area	Description
Log settings	Enable or disable access log output Set the log output format Customize the log format Log filtering
Metric settings	Enable or disable metric generation Metric dimensions
Tracing settings	Sampling percentage Custom tags

Namespace

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Observability Management Center > Observability Settings.

On the Observability Configuration page, click the Namespace tab, click Create, select the target Namespace, configure logs and metrics as needed, and then click Create.

Click the links in the following table to view detailed descriptions of the configurations.

Configuration area	Description
Log settings	Enable or disable access log output Set the log output format Customize the log format Log filtering
Metric settings	Enable or disable metric generation Metric dimensions

Custom

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Observability Management Center > Observability Settings.

On the Observability Configuration page, click the Custom tab, select the target Namespace, click Create, enter a Name and Matching Labels, configure logs and metrics as needed, and then click Create.

Click the links in the following table to view detailed descriptions of the configurations.

Configuration area	Description
Log settings	Enable or disable access log output Set the log output format Customize the log format Log filtering
Metric settings	Enable or disable metric generation Metric dimensions

Log settings

Log settings include enabling or disabling access log output, setting the log output format, customizing the log format, and log filtering.

Enable or disable access log output

In the Log Settings area, turn on or off the Enable Log Output switch as needed.
- If you turn on the switch, the Envoy proxy on the data plane of the service mesh outputs access logs to the standard output of the container.
- If you turn off the switch, the Envoy proxy on the data plane of the service mesh stops outputting logs to the standard output of the container.

View Waypoint logs.

The following example shows how to view access logs using kubectl.

Run the following command to view Waypoint logs.

kubectl -n istio-egress logs deployments/waypoint  | tail -1 | jq

View sample output

{
  "authority_for": "httpbin.test.com",
  "bytes_received": 0,
  "bytes_sent": 0,
  "downstream_local_address": "240.240.0.5:80",
  "downstream_remote_address": "10.18.190.138:49132",
  "duration": 346,
  "istio_policy_status": null,
  "method": "HEAD",
  "path": "/",
  "protocol": "HTTP/1.1",
  "request_id": "d3f48714-0e88-9534-a3f9-a639fd22defe",
  "requested_server_name": null,
  "response_code": 0,
  "response_flags": "DC",
  "route_name": "default",
  "start_time": "2025-08-18T11:01:13.249Z",
  "trace_id": null,
  "upstream_cluster": "inbound-vip|80|http|httpbin.test.com;",
  "upstream_host": null,
  "upstream_local_address": null,
  "upstream_response_time": null,
  "upstream_service_time": null,
  "upstream_transport_failure_reason": null,
  "user_agent": "curl/8.1.2",
  "x_forwarded_for": null
}

Run the following command to view ingress gateway logs.

kubectl -n istio-system logs istio-ingressgateway-6cff9b6b58-r**** --tail 1

View sample output

{
    "authority_for":"47.110.XX.XXX",
    "bytes_received":"0",
    "bytes_sent":"22382",
    "downstream_local_address":"192.168.0.63:80",
    "downstream_remote_address":"221.220.XXX.XXX:64284",
    "duration":"81",
    "istio_policy_status":"-",
    "method":"GET",
    "path":"/static/favicon.ico",
    "protocol":"HTTP/1.1",
    "request_id":"0f2cf829-3da5-4810-a618-08d9745d****",
    "requested_server_name":"-",
    "response_code":"200",
    "response_flags":"-",
    "route_name":"httpbin",
    "start_time":"2023-06-30T04:00:36.841Z",
    "trace_id":"-",
    "upstream_cluster":"outbound|8000||httpbin.default.svc.cluster.local",
    "upstream_host":"192.168.0.29:80",
    "upstream_local_address":"192.168.0.63:36140",
    "upstream_response_time":"81",
    "upstream_service_time":"81",
    "upstream_transport_failure_reason":"-",
    "user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.X.X Safari/537.36",
    "x_forwarded_for":"221.220.XXX.XXX"
}

(Optional) View access logs in the Container Service for Kubernetes console.
If you use an Alibaba Cloud Container Service for Kubernetes (ACK) cluster, you can also view access logs in the ACK console.
1. Log on to the ACK console. In the navigation pane on the left, choose Clusters.
2. On the Clusters page, click the name of the target cluster. In the navigation pane on the left, choose Workloads > Pods.
3. On the Pods page, click the name of the target pod, and then click the Logs tab at the bottom of the page to view the access log.

Set the log output format

Note

This feature is available only for ASM instances of version 1.20.6.36 or later. For more information about how to upgrade an instance, see Upgrade an ASM instance.

In the Log Settings area, set Log Output Format to JSON or TEXT as needed.

If you set this option to JSON, the access log is output to the standard output of the container as a JSON string.
If you set this option to TEXT, the access log is output to the standard output of the container as a plain text string.

Customize the log format

In the Log Settings area, select fields as needed, modify custom field information, or click the icon to the right of the log fields at the bottom to add a log field.
You can customize the log format only after you enable Enable Log Output. In the Log Format area, the default log fields are required and cannot be modified. Log fields can retrieve values from request headers, response headers, or Envoy built-in values.
For example, to print the accept-encoding header of a request, set accessLogFormat key to accept-encoding, Type to Request Properties, and accessLogFormat value to Accept-Encoding.

Run the following command to view the logs of the data plane component in the service mesh.

kubectl logs httpbin-5c5944c58c-w**** -c istio-proxy --tail 1|grep accept-encoding --color=auto

View sample output

{
    "bytes_received":"0",
    "bytes_sent":"9593",
    "downstream_local_address":"192.168.0.29:80",
    "downstream_remote_address":"69.164.XXX.XX:0",
    "duration":"2",
    "istio_policy_status":"-",
    "method":"GET",
    "path":"/",
    "protocol":"HTTP/1.1",
    "request_id":"29939dc9-62be-4ddf-acf6-32cb098d****",
    "requested_server_name":"outbound_.8000_._.httpbin.default.svc.cluster.local",
    "response_code":"200",
    "response_flags":"-",
    "route_name":"default",
    "start_time":"2023-06-30T04:18:19.734Z",
    "trace_id":"-",
    "upstream_cluster":"inbound|80||",
    "upstream_host":"192.168.0.29:80",
    "upstream_local_address":"127.0.X.X:34723",
    "upstream_service_time":"2",
    "upstream_transport_failure_reason":"-",
    "user_agent":"Mozilla/5.0 zgrab/0.x",
    "x_forwarded_for":"69.164.XXX.XX",
    "authority_for":"47.110.XX.XXX",
    "upstream_response_time":"2",
    "accept-encoding":"gzip"
}

The value of the Accept-Encoding header that you added in Step 1 is output to the access log.

Log filtering

Below the Log Settings area, select Enable Log Filter and enter a filtering expression in the text box. Access logs for requests that do not match the expression are not output.

For example, to output logs only for requests with a Response HTTP Status >=400, use the expression response.code >= 400. For more information, see CEL expressions and common fields.

CEL expressions and common fields

The log filtering expression is a standard Common Expression Language (CEL) expression. The following table describes the common fields for CEL expressions. For more information, see CEL and Envoy.

Attribute	Type	Description
request.path	string	The path of the request.
request.url_path	string	The path of the request, without the query string.
request.host	string	The hostname part of the URL.
request.method	string	The method of the request.
request.headers	map<string, string>	All request headers, indexed by the lowercase header name.
request.useragent	string	The value of the User-Agent header.
request.time	timestamp	The time when the first byte of the request arrives.
request.id	string	The ID of the request.
request.protocol	string	The protocol of the request. Valid values: `HTTP/1.0`, `HTTP/1.1`, `HTTP/2`, or `HTTP/3`.
request.query	string	The query string in the request URL.
response.code	int	The return code of the HTTP response.
response.code_details	string	Details about the response code.
response.grpc_status	int	The gRPC status code in the response.
response.headers	map<string, string>	All response headers, indexed by the lowercase header name.
response.size	int	The size of the response body, in bytes.
response.total_size	int	The total size of the response message, in bytes.

Metric settings

Metric settings include enabling or disabling metric generation and metric dimensions.

Enable or disable metric generation

Metrics are divided into client-side metrics and server-side metrics.

Client-side metrics: Metric data generated when the Envoy proxy acts as a client to initiate requests. This includes:
- Sidecar egress traffic (traffic from an application that accesses other applications through a sidecar proxy).
- Gateway traffic.
Server-side metrics: Metric data generated when the Envoy proxy acts as a server to receive requests. This includes:
- Sidecar ingress traffic (traffic from other applications that access the proxied application through a sidecar).
- Waypoint traffic.

In the Client-side Metrics (Gateway and Sidecar Outbound) or Server-side Metrics (Sidecar Inbound) column of the Metric Settings area, select or clear the Enable checkbox for a target metric as needed.
- If you enable a metric, the Envoy proxy on the data plane of the service mesh exposes the metric through the /stats/prometheus path on port 15020.
- If you disable a metric, the metric is not exposed through the port.

Run the following command to view the metrics exposed by the Envoy proxy.

You can run the curl command in the Envoy proxy container using kubectl to access the /stats/prometheus path on the local port 15020 and view the exported metrics.

kubectl exec httpbin-5c5944c58c-w**** -c istio-proxy -- curl 127.0.0.1:15020/stats/prometheus|head -n 10

Sample output:

# TYPE istio_agent_cert_expiry_seconds gauge
istio_agent_cert_expiry_seconds{resource_name="default"} 46725.287654548
# HELP istio_agent_endpoint_no_pod Endpoints without an associated pod.
# TYPE istio_agent_endpoint_no_pod gauge
istio_agent_endpoint_no_pod 0
# HELP istio_agent_go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE istio_agent_go_gc_duration_seconds summary
istio_agent_go_gc_duration_seconds{quantile="0"} 5.0149e-05
istio_agent_go_gc_duration_seconds{quantile="0.25"} 9.8807e-05
......

Metric dimensions

Metric dimensions can provide richer information. You can use these dimensions to filter for target metrics in Prometheus. For example, you can use the source_app dimension to filter for metrics where the request client is a specific application.

Edit default dimensions

Edit the default dimensions as follows.

In the Metric Settings area, in the Client-side Metrics (Gateway and Sidecar Outbound) or Server-side Metrics (Sidecar Inbound) column, click Edit dimension for the enabled metric.
In the Customize CLIENT dimension configuration or Customize SERVER dimension configuration dialog box, select or deselect the metric dimensions that you want to export, and click Submit.

For example, if no dimensions are disabled, run the curl command in the Envoy proxy container using kubectl to access the /stats/prometheus path on the local port 15020 and view the exported metrics.

kubectl exec httpbin-5c5944c58c-w**** -c istio-proxy -- curl 127.0.0.1:15020/stats/prometheus

For example, istio_request_bytes_sum (which corresponds to the REQUEST_SIZE metric on the dashboard) contains all the dimensions.

istio_request_bytes_sum{reporter="destination",source_workload="istio-ingressgateway",source_canonical_service="unknown",source_canonical_revision="latest",source_workload_namespace="istio-system",source_principal="spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway",source_app="istio-ingressgateway",source_version="unknown",source_cluster="c479fc4abd2734bfaaa54e9e36fb26c01",destination_workload="httpbin",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/httpbin",destination_app="httpbin",destination_version="v1",destination_service="httpbin.default.svc.cluster.local",destination_canonical_service="httpbin",destination_canonical_revision="v1",destination_service_name="httpbin",destination_service_namespace="default",destination_cluster="c479fc4abd2734bfaaa54e9e36fb26c01",request_protocol="http",response_code="200",grpc_response_status="",response_flags="-",connection_security_policy="mutual_tls"} 18000

Modify the default server-side REQUEST_SIZE metric to retain only the response_code dimension. Use kubectl to run a curl command in the Envoy proxy container to access the /stats/prometheus path on local port 15020 and view the exported metrics. You can see that the metric includes only the response_code dimension.

istio_request_bytes_sum{response_code="200"} 16550

Add custom dimensions

Add custom dimensions as follows:

In the Client-side Metrics (Gateway and Sidecar Outbound) or Server-side Metrics (Sidecar Inbound) column of the Metric Settings area, click Edit dimension for the enabled metric.
In the Customize CLIENT dimension configuration or Customize SERVER dimension configuration dialog box, under Custom Dimension, edit the dimension's name and value, and then click Submit.

For example, after you edit the custom dimension for the REQUEST_SIZE metric on the server-side and add request_path as the dimension name and request.path as the dimension value, use the kubectl command to run a curl command in the Envoy proxy container to access the /stats/prometheus path on local port 15020 and view the exported metrics. You can see that the metric now contains the custom dimension request_path.

istio_request_bytes_sum{response_code="200",request_path="/spec.json"} 5800

Important

You can reduce the memory consumption of Envoy and Prometheus by removing default dimensions that are not required for your business. However, most dimensions are typically retained. Therefore, the Metric Settings area displays only the dimensions that have been removed.

Tracing settings

Tracing settings include sampling percentage and custom tags. Tracing requires consistent reporting configurations across the entire call chain to build a complete trace. If the reporting endpoints or sampling rates are inconsistent, the call chain may be incomplete. For this reason, in versions earlier than 1.24.6.83, you cannot configure tracing at the namespace or workload level. In versions 1.24.6.83 and later, ASM supports namespace-level and workload-level tracing configurations by modifying the Telemetry resource through the Kubernetes API. For more information about how to configure the Telemetry resource, see Telemetry CRD.

Sampling percentage

You can customize the sampling percentage for tracing. This value represents the percentage of requests that trigger tracing reports. A value of 0 disables tracing, and no requests trigger reports.

Custom tags

You can customize the tags for reported tracing analysis spans. In the Tracing Analysis Settings section, click Add Custom Tags, and configure the Name, Type, and Value.

The type can be a static field, request header, or environment variable. The following table describes the types and provides tag configuration examples.

Type	Description	Tag configuration example
Fixed Value	The value of a static field tag is fixed to the value that you set.	Name: env Type: Static value Value: prod
Request Header	The value of a request header tag is the value of the request header that you specify. If the header does not exist in the request, the default value is used as the tag value. For example, get the tag value from the User-Agent header. If the header does not exist, set the tag value to the default value `unknow`.	Name: useragent Type: Request Header Value: Header Name: User-Agent Default Value: unknow
Environment Variable	An environment variable tag gets its value from a specified environment variable of the workload. If the environment variable does not exist in the workload, the default value is used as the tag value. For example, get the tag value from the ENV environment variable. If the environment variable does not exist, set the tag value to the default value `unknow`.	Name: env Type: environment variable Value: Environment Variable Name: ENV Default Value: unknown