Configure observability settings for logs metrics and traces - Alibaba Cloud Service Mesh

Service Mesh (ASM) lets you control what telemetry data your sidecar proxies and gateways collect and export: access logs, metrics, and distributed traces. Apply these settings at the global, namespace, or workload level to match each environment's observability needs.

Prerequisites

Before you begin, make sure that you have:

An ASM instance running version 1.17.2.35 or later. To upgrade, see Update an ASM instance. To create a new instance, see Create an ASM instance

Configuration scopes

ASM supports three configuration scopes.

Scope	What it controls	Limit
Global	Log Settings, Metric Settings, and Tracing Analysis Settings for the entire mesh.	One global configuration per mesh. Required.
Namespace	Log Settings and Metric Settings for a specific namespace.	One configuration per namespace.
Workload (Custom)	Log Settings and Metric Settings for workloads matching a label selector.	Each workload can match only one custom configuration.

Note

Tracing Analysis Settings are available only at the global scope.

Navigate to observability settings

All three configuration scopes share the same starting point in the console:

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Observability Management Center > Observability Settings.

From here, select the tab for your target scope:

Global -- Configure Log Settings, Metric Settings, and Tracing Analysis Settings, then click submit.
Namespace -- Click Create, select a namespace from the Namespace drop-down list, configure Log Settings and Metric Settings, then click Create.
Custom -- Select the target namespace from the Namespace drop-down list, click Create, set Name and Matching Label to identify the workloads, configure Log Settings and Metric Settings, then click Create.

Log Settings

The Log Settings section controls how access logs are collected, formatted, and filtered.

Enable or disable access log output

In the Log Settings section, toggle the Enable Log Output switch:

On: Sidecar proxies and gateways on the data plane send access logs to their containers, which emit logs to stdout.
Off: Access log output stops. No logs are emitted to stdout.

Verify access log output

After you enable log output, verify that logs appear in the sidecar proxy or gateway container.

View sidecar proxy logs:

kubectl logs <pod-name> -c istio-proxy --tail 1

Sample output:

{
    "authority_for":"47.110.XX.XXX",
    "bytes_received":"0",
    "bytes_sent":"22382",
    "downstream_local_address":"192.168.0.29:80",
    "downstream_remote_address":"221.220.XXX.XXX:0",
    "duration":"80",
    "istio_policy_status":"-",
    "method":"GET",
    "path":"/static/favicon.ico",
    "protocol":"HTTP/1.1",
    "request_id":"0f2cf829-3da5-4810-a618-08d9745d****",
    "requested_server_name":"outbound_.8000_._.httpbin.default.svc.cluster.local",
    "response_code":"200",
    "response_flags":"-",
    "route_name":"default",
    "start_time":"2023-06-30T04:00:36.841Z",
    "trace_id":"-",
    "upstream_cluster":"inbound|80||",
    "upstream_host":"192.168.0.29:80",
    "upstream_local_address":"127.0.X.X:55879",
    "upstream_response_time":"79",
    "upstream_service_time":"79",
    "upstream_transport_failure_reason":"-",
    "user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.X.X Safari/537.36",
    "x_forwarded_for":"221.220.XXX.XXX"
}

View ingress gateway logs:

kubectl -n istio-system logs <gateway-pod-name> --tail 1

Sample output:

{
    "authority_for":"47.110.XX.XXX",
    "bytes_received":"0",
    "bytes_sent":"22382",
    "downstream_local_address":"192.168.0.63:80",
    "downstream_remote_address":"221.220.XXX.XXX:64284",
    "duration":"81",
    "istio_policy_status":"-",
    "method":"GET",
    "path":"/static/favicon.ico",
    "protocol":"HTTP/1.1",
    "request_id":"0f2cf829-3da5-4810-a618-08d9745d****",
    "requested_server_name":"-",
    "response_code":"200",
    "response_flags":"-",
    "route_name":"httpbin",
    "start_time":"2023-06-30T04:00:36.841Z",
    "trace_id":"-",
    "upstream_cluster":"outbound|8000||httpbin.default.svc.cluster.local",
    "upstream_host":"192.168.0.29:80",
    "upstream_local_address":"192.168.0.63:36140",
    "upstream_response_time":"81",
    "upstream_service_time":"81",
    "upstream_transport_failure_reason":"-",
    "user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.X.X Safari/537.36",
    "x_forwarded_for":"221.220.XXX.XXX"
}

View logs in the ACK console (optional):

If your data plane runs on a Container Service for Kubernetes (ACK) cluster, access logs are also available from the ACK console:

Log on to the ACK console. In the left-side navigation pane, click Clusters.
On the Clusters page, click the cluster name, then choose Workloads > Pods in the left-side navigation pane.
On the Pods page, click the pod name and click the Logs tab.

Set the log output format

Note

Requires ASM v1.20.6.36 or later. To upgrade, see Update an ASM instance.

In the Log Settings section, set Log Output Format to one of the following:

Format	Description
JSON	Outputs access logs as JSON strings. Easier to parse programmatically.
TEXT	Outputs access logs as plain text strings.

Customize log fields

Add custom fields to include additional information (such as specific request or response headers) in your access logs.

In the Log Settings section, select custom fields or modify existing ones. Click the icon to add new log fields.
Note
Log output must be enabled to customize log fields. Default log fields are mandatory and cannot be removed. Each field value can come from a request header, response header, or Envoy built-in value.
Set the field properties. For example, to include the Accept-Encoding request header in access logs:
- accessLogFormat key: accept-encoding
- Type: Request Properties
- accessLogFormat value: Accept-Encoding

Verify the custom field appears in the logs:

kubectl logs <pod-name> -c istio-proxy --tail 1 | grep accept-encoding --color=auto

The output should include the new field:

{
       "bytes_received":"0",
       "bytes_sent":"9593",
       "downstream_local_address":"192.168.0.29:80",
       "downstream_remote_address":"69.164.XXX.XX:0",
       "duration":"2",
       "istio_policy_status":"-",
       "method":"GET",
       "path":"/",
       "protocol":"HTTP/1.1",
       "request_id":"29939dc9-62be-4ddf-acf6-32cb098d****",
       "requested_server_name":"outbound_.8000_._.httpbin.default.svc.cluster.local",
       "response_code":"200",
       "response_flags":"-",
       "route_name":"default",
       "start_time":"2023-06-30T04:18:19.734Z",
       "trace_id":"-",
       "upstream_cluster":"inbound|80||",
       "upstream_host":"192.168.0.29:80",
       "upstream_local_address":"127.0.X.X:34723",
       "upstream_service_time":"2",
       "upstream_transport_failure_reason":"-",
       "user_agent":"Mozilla/5.0 zgrab/0.x",
       "x_forwarded_for":"69.164.XXX.XX",
       "authority_for":"47.110.XX.XXX",
       "upstream_response_time":"2",
       "accept-encoding":"gzip"
   }

Filter access logs

Use Common Expression Language (CEL) expressions to filter access logs. Only requests matching the filter expression produce log entries.

In the Log Settings section, select Enable Log Filter and enter a CEL expression.

For example, to print the logs of requests whose HTTP status code in the response is greater than or equal to 400, set the expression to response.code >= 400.

CEL expression attribute reference

The following tables list commonly used attributes in CEL expressions. For full details, see the CEL specification and Envoy attributes reference.

Request attributes:

Attribute	Type	Description
`request.path`	string	Request path, including query string.
`request.url_path`	string	Request path without the query string.
`request.host`	string	Host name from the URL.
`request.method`	string	HTTP method (GET, POST, etc.).
`request.headers`	map<string, string>	All request headers, indexed by lowercase name.
`request.useragent`	string	User-Agent header value.
`request.time`	timestamp	Time the first byte of the request arrived.
`request.id`	string	Request ID.
`request.protocol`	string	Protocol: `HTTP/1.0`, `HTTP/1.1`, `HTTP/2`, or `HTTP/3`.
`request.query`	string	Query portion of the URL.

Response attributes:

Attribute	Type	Description
`response.code`	int	HTTP status code.
`response.code_details`	string	Response code details.
`response.grpc_status`	int	gRPC status code.
`response.headers`	map<string, string>	All response headers, indexed by lowercase name.
`response.size`	int	Response body size in bytes.
`response.total_size`	int	Total response size in bytes.

Metric Settings

The Metric Settings section controls which metrics your sidecar proxies and gateways export, and the dimensions (labels) attached to each metric.

Client-side vs. server-side metrics

ASM categorizes metrics by the role of the sidecar proxy at the time the metric is generated:

Category	When generated	Includes
Client side	The sidecar proxy initiates an outbound request.	Application-to-service traffic, gateway traffic.
Server side	The sidecar proxy receives an inbound request.	Inbound traffic to the application.

Enable or disable specific metrics

In the Metric Settings section, select or clear the Enabled check box for each metric in the CLIENT side Indicator or SERVER side index column.

Enabled: The sidecar proxy or gateway exposes the metric at /stats/prometheus on port 15020.
Disabled: The metric is not exposed.

Verify exported metrics:

kubectl exec <pod-name> -c istio-proxy -- curl -s 127.0.0.1:15020/stats/prometheus | head -n 10

Sample output:

# TYPE istio_agent_cert_expiry_seconds gauge
istio_agent_cert_expiry_seconds{resource_name="default"} 46725.287654548
# HELP istio_agent_endpoint_no_pod Endpoints without an associated pod.
# TYPE istio_agent_endpoint_no_pod gauge
istio_agent_endpoint_no_pod 0
# HELP istio_agent_go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE istio_agent_go_gc_duration_seconds summary
istio_agent_go_gc_duration_seconds{quantile="0"} 5.0149e-05
istio_agent_go_gc_duration_seconds{quantile="0.25"} 9.8807e-05
......

Edit metric dimensions

Dimensions are Prometheus labels attached to each metric. Use them to filter and group metrics -- for example, filter by source_app to isolate traffic from a specific application.

In the Metric Settings section, click Edit dimension in the CLIENT side Indicator or SERVER side index column.
In the Customize CLIENT dimension configuration dialog box, select or clear dimensions, then click Submit.

Note

Adding new dimensions is not supported. Remove unnecessary dimensions to reduce Prometheus storage usage. The Metric Settings section displays only removed dimensions, since most dimensions are retained by default.

Tracing Analysis Settings

Tracing Analysis Settings control distributed tracing for the entire mesh. These settings are available only at the global scope.

Sampling percentage

Sampling Percentage sets the percentage of requests that generate traces. Set to 0 to disable tracing entirely.

Custom tags

Add custom tags to enrich the spans your mesh generates. Tags make it easier to filter and search traces in your tracing backend.

In the Tracing Analysis Settings section, click Add Custom Tags and set Name, Type, and Value.

Type	Behavior	Example
Fixed Value	Every span carries this tag with the specified value.	Name: `env` Value: `prod`
Request Header	Tag value is extracted from the specified request header. Falls back to the default value if the header is missing.	Name: `useragent` Header name: `User-Agent` Default value: `unknow`
Environment Variable	Tag value is read from the specified environment variable on the workload. Falls back to the default value if the variable is missing.	Name: `env` Environment Variable name: `ENV` Default value: `unknow`