Observability configuration - Alibaba Cloud Service Mesh

Prerequisites

An ASM instance of version 1.17.2.35 or later. For more information, see Create an ASM instance or Upgrade an ASM instance.

Scopes

Scope	Description
Global	Global settings apply to logs, metrics, and tracing. You can have only one global configuration, which cannot be deleted. Tracing settings can only be configured at the global scope.
Namespace	You can create dedicated observability settings for a namespace. Each namespace can have only one namespace-level observability configuration.
Custom	Use a workload selector to define the scope for a custom configuration. Each workload can belong to only one custom configuration.

Procedure

Global

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Observability Management Center > Observability Settings.

On the Observability Settings page, click the Global tab. Configure settings for logs, metrics, and tracing as needed, and then click Submission.

For detailed descriptions of the settings, see the following sections.

Area	Description
Log settings	Enable or disable access log output Set the log output format Customize the log format Log filtering
Metrics settings	Enable or disable metrics generation Metric dimensions
Tracing settings	Sampling percentage Custom tags

Namespace

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Observability Management Center > Observability Settings.

On the Observability Settings page, click the Namespace tab, and then click Create. Select the target Namespace, configure settings for logs and metrics as needed, and then click Create.

For detailed descriptions of the settings, see the following sections.

Area	Description
Log settings	Enable or disable access log output Set the log output format Customize the log format Log filtering
Metrics settings	Enable or disable metrics generation Metric dimensions

Custom

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Observability Management Center > Observability Settings.

On the Observability Settings page, click the Custom tab. Select the target Namespace and click Create. Enter a Name and a Label selector, configure settings for logs and metrics as needed, and then click Create.

For detailed descriptions of the settings, see the following sections.

Area	Description
Log settings	Enable or disable access log output Set the log output format Customize the log format Log filtering
Metrics settings	Enable or disable metrics generation Metric dimensions

Log settings

Log settings include enabling or disabling access log output, setting the log output format, customizing the log format, and filtering logs.

Access log output

In the Log Settings section, turn the Enable Log Output switch on or off as needed.
- If enabled, data plane sidecars or gateways send access logs to the container's standard output.
- If disabled, data plane sidecars or gateways stop sending logs to the container's standard output.

View logs in the standard output of the data plane sidecar container.

The following examples show how to view access logs by using kubectl.

Run the following command to view sidecar logs.

kubectl logs <Pod-name> -c istio-proxy --tail 1

Sample output

{
    "authority_for":"47.110.XX.XXX",
    "bytes_received":"0",
    "bytes_sent":"22382",
    "downstream_local_address":"192.168.0.29:80",
    "downstream_remote_address":"221.220.XXX.XXX:0",
    "duration":"80",
    "istio_policy_status":"-",
    "method":"GET",
    "path":"/static/favicon.ico",
    "protocol":"HTTP/1.1",
    "request_id":"0f2cf829-3da5-4810-a618-08d9745d****",
    "requested_server_name":"outbound_.8000_._.httpbin.default.svc.cluster.local",
    "response_code":"200",
    "response_flags":"-",
    "route_name":"default",
    "start_time":"2023-06-30T04:00:36.841Z",
    "trace_id":"-",
    "upstream_cluster":"inbound|80||",
    "upstream_host":"192.168.0.29:80",
    "upstream_local_address":"127.0.X.X:55879",
    "upstream_response_time":"79",
    "upstream_service_time":"79",
    "upstream_transport_failure_reason":"-",
    "user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.X.X Safari/537.36",
    "x_forwarded_for":"221.220.XXX.XXX"
}

Run the following command to view ingress gateway logs.

kubectl -n istio-system logs <ingress-gateway-Pod-name> --tail 1

Sample output

{
    "authority_for":"47.110.XX.XXX",
    "bytes_received":"0",
    "bytes_sent":"22382",
    "downstream_local_address":"192.168.0.63:80",
    "downstream_remote_address":"221.220.XXX.XXX:64284",
    "duration":"81",
    "istio_policy_status":"-",
    "method":"GET",
    "path":"/static/favicon.ico",
    "protocol":"HTTP/1.1",
    "request_id":"0f2cf829-3da5-4810-a618-08d9745d****",
    "requested_server_name":"-",
    "response_code":"200",
    "response_flags":"-",
    "route_name":"httpbin",
    "start_time":"2023-06-30T04:00:36.841Z",
    "trace_id":"-",
    "upstream_cluster":"outbound|8000||httpbin.default.svc.cluster.local",
    "upstream_host":"192.168.0.29:80",
    "upstream_local_address":"192.168.0.63:36140",
    "upstream_response_time":"81",
    "upstream_service_time":"81",
    "upstream_transport_failure_reason":"-",
    "user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.X.X Safari/537.36",
    "x_forwarded_for":"221.220.XXX.XXX"
}

(Optional) View access logs in the ACK console.

If you use an ACK cluster, you can also view access logs in the ACK console.
1. Log on to the ACK console. In the left navigation pane, click Clusters.
2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Workloads > Pods.
3. On the Pods page, click the name of the target Pod, and then click the Logs tab at the bottom of the page to view the access log.

Log output format

Note

This feature requires an ASM instance of version 1.20.6.36 or later. For information about how to upgrade an instance, see Upgrade an ASM instance.

In the Log Settings section, set Log Output Format to JSON or TEXT as needed.

If you select JSON, access logs are sent to the container's standard output as JSON strings.
If you select TEXT, access logs are sent to the container's standard output as plain text strings.

Custom log format

In the Log Settings section, select fields to include, modify custom field information, or click the icon to add a new log field.

You can customize the log format only when Enable Log Output is turned on. In the Log Format section, the default log fields are mandatory and cannot be modified. Log fields can retrieve values from request headers, response headers, or built-in Envoy values.

The following example shows how to print the accept-encoding header of a request. Set accessLogFormat key to accept-encoding, Type to Request Properties, and accessLogFormat value to Accept-Encoding. For example, the built-in field authority_for corresponds to the format variable %REQ(:AUTHORITY)%, and request_duration corresponds to %REQUEST_DURATION%. For a custom field such as upstream_response_time, if you select Response Attribute as the attribute type, the format variable is %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%. For a custom field such as accept-encoding, if you select Request Attribute as the attribute type, the format variable is %REQ(Accept-Encoding)%.

Run the following command to check the logs of the data plane components for the new field.

kubectl logs <Pod-name> -c istio-proxy --tail 1|grep accept-encoding --color=auto

Sample output

{
    "bytes_received":"0",
    "bytes_sent":"9593",
    "downstream_local_address":"192.168.0.29:80",
    "downstream_remote_address":"69.164.XXX.XX:0",
    "duration":"2",
    "istio_policy_status":"-",
    "method":"GET",
    "path":"/",
    "protocol":"HTTP/1.1",
    "request_id":"29939dc9-62be-4ddf-acf6-32cb098d****",
    "requested_server_name":"outbound_.8000_._.httpbin.default.svc.cluster.local",
    "response_code":"200",
    "response_flags":"-",
    "route_name":"default",
    "start_time":"2023-06-30T04:18:19.734Z",
    "trace_id":"-",
    "upstream_cluster":"inbound|80||",
    "upstream_host":"192.168.0.29:80",
    "upstream_local_address":"127.0.X.X:34723",
    "upstream_service_time":"2",
    "upstream_transport_failure_reason":"-",
    "user_agent":"Mozilla/5.0 zgrab/0.x",
    "x_forwarded_for":"69.164.XXX.XX",
    "authority_for":"47.110.XX.XXX",
    "upstream_response_time":"2",
    "accept-encoding":"gzip"
}

The output shows that the Accept-Encoding header, added in Step 1, has been written to the access log.

Log filtering

In the Log Settings section, select Enable Log Filter to activate log filtering. Then, enter a log filter expression in the text box. Access logs are generated only for requests that match the expression.

For example, if you want to log only requests with a response HTTP status of 400 or higher, use the expression response.code >= 400. For more information, see CEL expressions and common fields.

CEL expressions and fields

Log filter expressions use the Common Expression Language (CEL). The following table lists common fields for CEL expressions. For more information, see CEL and Envoy.

Attribute	Type	Description
request.path	string	The request path.
request.url_path	string	The request path, excluding the query string.
request.host	string	The host portion of the URL.
request.method	string	The request method.
request.headers	map<string, string>	All request headers, indexed by the lowercase header name.
request.useragent	string	The value of the User-Agent header.
request.time	timestamp	The time when the first byte of the request arrives.
request.id	string	The request ID.
request.protocol	string	The request protocol. Valid values: `HTTP/1.0`, `HTTP/1.1`, `HTTP/2`, or `HTTP/3`.
request.query	string	The query string in the request URL.
response.code	int	The HTTP response status code.
response.code_details	string	Details about the response code.
response.grpc_status	int	The gRPC status code in the response.
response.headers	map<string, string>	All response headers, indexed by the lowercase header name.
response.size	int	The size of the response body in bytes.
response.total_size	int	The total size of the response message in bytes.

Metrics settings

Metrics settings include enabling or disabling metrics generation and configuring metric dimensions.

Metrics generation

Metrics are divided into client-side metrics and server-side metrics.

Client-side metrics are metrics generated when a sidecar acts as a client to initiate requests. Gateway metrics also belong to the CLIENT category.
Server-side metrics: Metrics generated when a sidecar is accessed as a server.

In the Metric Settings section, in the Client-Side Metrics or Server-Side Metrics column, select or clear the Enabled checkbox for the target metric as needed.
- If enabled, the data plane sidecar or gateway exposes the metric through the /stats/prometheus path on port 15020.
- If disabled, the metric is not exposed on this port.

Run the following command to view the metrics exposed by the sidecar or gateway.

You can use the kubectl command to execute a curl command in the Sidecar or gateway container, accessing the /stats/prometheus path on local port 15020 to view the exported metrics.

kubectl exec <Pod-name> -c istio-proxy -- curl 127.0.0.1:15020/stats/prometheus | head -n 10

Sample output:

# TYPE istio_agent_cert_expiry_seconds gauge
istio_agent_cert_expiry_seconds{resource_name="default"} 46725.287654548
# HELP istio_agent_endpoint_no_pod Endpoints without an associated pod.
# TYPE istio_agent_endpoint_no_pod gauge
istio_agent_endpoint_no_pod 0
# HELP istio_agent_go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE istio_agent_go_gc_duration_seconds summary
istio_agent_go_gc_duration_seconds{quantile="0"} 5.0149e-05
istio_agent_go_gc_duration_seconds{quantile="0.25"} 9.8807e-05
......

Metric dimensions

Metric dimensions add context, allowing you to filter metrics in Prometheus. For example, use the source_app dimension to filter metrics for requests from a specific client application.

Edit default dimensions

Follow these steps to edit the default dimensions:

In the Metric Settings section, in the Client-Side Metrics or Server-Side Metrics column, click Edit dimension for an enabled metric.
In the Customize CLIENT dimension configuration or Customize SERVER dimension configuration dialog box, select or clear the checkboxes for the dimensions to export, and then click Confirm.

For example, if no dimensions are disabled, you can run a curl command inside the sidecar or gateway container by using kubectl to access the local /stats/prometheus path on port 15020 and view the exported metrics.

kubectl exec <Pod-name> -c istio-proxy -- curl 127.0.0.1:15020/stats/prometheus

For example, consider the istio_request_bytes_sum metric, which corresponds to the REQUEST_SIZE metric in the console. The output shows that all dimensions are included.

istio_request_bytes_sum{reporter="destination",source_workload="istio-ingressgateway",source_canonical_service="unknown",source_canonical_revision="latest",source_workload_namespace="istio-system",source_principal="spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway",source_app="istio-ingressgateway",source_version="unknown",source_cluster="c479fc4abd2734bfaaa54e9e36fb26c01",destination_workload="httpbin",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/httpbin",destination_app="httpbin",destination_version="v1",destination_service="httpbin.default.svc.cluster.local",destination_canonical_service="httpbin",destination_canonical_revision="v1",destination_service_name="httpbin",destination_service_namespace="default",destination_cluster="c479fc4abd2734bfaaa54e9e36fb26c01",request_protocol="http",response_code="200",grpc_response_status="",response_flags="-",connection_security_policy="mutual_tls"} 18000

Modify the default server-side REQUEST_SIZE metric to retain only the response_code dimension. Then, run a curl command to view the exported metrics from the /stats/prometheus path. The output shows that only the response_code dimension is included.

istio_request_bytes_sum{response_code="200"} 16550

Add custom dimensions

Follow these steps to add custom dimensions:

In the Metric Settings section, in the Client-Side Metrics or Server-Side Metrics column, click Edit dimension for an enabled metric.
In the Customize CLIENT dimension configuration or Customize SERVER dimension configuration dialog box, in the Custom Dimensions section, configure the dimension name and value, and then click Confirm.

For example, edit the custom dimensions for the server-side REQUEST_SIZE metric. Add a dimension with the name request_path and the value request.path. Then, run a curl command to view the exported metrics from the /stats/prometheus path. The output shows that the metric now includes the custom dimension request_path.

istio_request_bytes_sum{response_code="200",request_path="/spec.json"} 5800

Important

Removing default dimensions that your services do not require can reduce Envoy and Prometheus memory consumption. However, since most dimensions are typically needed, the Metric Settings section displays only the removed dimensions.

Tracing settings

To construct a complete trace, tracing requires a consistent reporting configuration across the entire call chain. Inconsistent reporting endpoints or sampling percentages can lead to incomplete traces. For this reason, ASM instances earlier than version 1.24.6.83 do not allow namespace-level or workload-level tracing configurations. Starting from version 1.24.6.83, ASM supports modifying the Telemetry resource through the Kubernetes API to implement namespace-level and workload-level tracing configurations. For more information about how to configure Telemetry resources, see Telemetry CRD Description.

Sampling percentage

You can customize the sampling percentage for tracing, which is the proportion of requests that trigger trace reporting. A value of 0 disables tracing, and no requests trigger trace reporting.

Custom tags

You can customize the tags on reported tracing spans. In the Tracing Analysis Settings section, click Add Custom Tags, and configure the Name, Type, and Value.

The Type can be Fixed Value, Request Header, and Environment Variable. The following table describes each type and provides configuration examples.

Type	Description	Example
Fixed Value	A Fixed Value tag uses the value you specify.	Name: env Type: Fixed Value Value: prod
Request Header	A Request Header tag reads its value from a specified request header. If the header is not present in the request, the default value is used. For example, you can get the tag value from the User-Agent header. If the header does not exist, the tag value is set to the default value `unknown`.	Name: useragent Type: Request Header Value: Header name: User-Agent Default value: unknown
Environment Variable	An Environment Variable tag reads its value from a specified environment variable in the workload. If the environment variable does not exist in the workload, the default value is used. For example, you can get the tag value from the `ENV` environment variable. If the environment variable does not exist, the tag value is set to the default value `unknown`.	Name: env Type: Environment Variable Value: Environment Variable Name: ENV Default value: unknown