In a service mesh, different services may require different observability data. This necessitates defining distinct collection rules for sidecar proxies and gateway pods and standardizing these configurations to better support observability for cloud-native applications. Observability is vital for cloud-native applications. It helps you monitor service health and performance in real time, detect and resolve faults and bottlenecks, and ultimately improve application reliability and performance. Alibaba Cloud Service Mesh (ASM) provides a unified, standardized model for configuring telemetry data generation and collection to enhance the observability of your cloud-native applications. This topic covers observability concepts and features.
Observability
As application systems grow in complexity, it becomes increasingly difficult to ensure that all components remain stable. Parts of a system may operate in a degraded state due to underlying issues. Therefore, in addition to building reliable and resilient applications, you must use observability tools to understand the runtime behavior of your services and infrastructure. This visibility allows you to detect failures, debug unexpected issues, and ultimately reduce the mean time to recovery (MTTR), minimizing business impact.
Observability is a property of a system that involves gathering data from multiple levels, including application metrics, network metrics, and infrastructure telemetry. By correlating this vast amount of data, you can build a complete picture to investigate unpredictable events. A service mesh significantly enhances the collection of application-level network metrics. From a practical standpoint, the key is to understand your system's stability—knowing when it operates correctly versus when it experiences issues. This enables faster error identification and lets you implement automated or manual controls to maintain availability.
The data plane proxies of a service mesh are positioned in the network path between services. By capturing telemetry from these proxies, you gain runtime visibility into the behavior of your application network and the mesh itself.
Achieving observability in a service mesh involves configuring the generation and collection of observability data, such as logs, metrics, and distributed tracing, and then sending this data to either cloud-hosted or self-managed services. You also need to define separate collection configurations for sidecar proxies and gateway pods to support different scenarios. ASM provides a unified and converged configuration model for data generation and collection to better support the observability of cloud-native applications.
Built-in best practices
The Telemetry CRD lets you create multiple objects in different namespaces. However, defining them arbitrarily can lead to conflicts and unexpected behavior. Follow these best practices to ensure your configurations work as intended:
-
You cannot define multiple mesh-wide Telemetry objects in the istio-system root namespace; only one such object can exist. ASM enforces this best practice by allowing only a single Telemetry object named default in the istio-system namespace.
-
Each namespace allows only one Telemetry object with an empty workload selector, which must be named default.
-
You can use a workload selector to apply a new Telemetry object in a specific namespace, which overrides the settings for the target workload.
-
If two Telemetry objects with the same workload selector target the same workload, the behavior is undefined.
-
If the global Telemetry object in the istio-system root namespace does not define a metrics configuration, metric generation defaults to disabled.
Logs
In a service mesh, log collection is a primary way to achieve observability. Aggregating logs from all services into a central location simplifies management and search. To achieve this, each service's logs should be written to stdout or stderr and collected by a log agent into a central logging system. ASM provides log filtering and formatting capabilities, allowing you to filter and format logs as needed for easier retrieval and analysis.
Log format rules
In a real-world environment, services often have different log formats. You can set generation rules to control how logs are produced. The Envoy proxy, deployed on the data plane (in the Kubernetes cluster added to the mesh), can output a comprehensive access log for all traffic. ASM lets you customize the content of these access logs.
Using the Telemetry CRD, ASM provides a graphical user interface (GUI) to simplify configuring log data formats. For detailed steps, see Customize access logs on the data plane.
The graphical configuration interface for log format rules contains three scope tabs: Global, Namespace, and Custom. On the Global tab, after you turn on the Enable Log Output switch, you can select the required log variables in the variable configuration table. The variable types are classified into Envoy Built-in Attributes and Request Attributes.
The following is the equivalent YAML configuration:
envoyFileAccessLog:
logFormat:
text: '{"bytes_received":"%BYTES_RECEIVED%","bytes_sent":"%BYTES_SENT%","downstream_local_address":"%DOWNSTREAM_LOCAL_ADDRESS%","downstream_remote_address":"%DOWNSTREAM_REMOTE_ADDRESS%","duration":"%DURATION%","istio_policy_status":"%DYNAMIC_METADATA(istio.mixer:status)%","method":"%REQ(:METHOD)%","path":"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%","protocol":"%PROTOCOL%","request_id":"%REQ(X-REQUEST-ID)%","requested_server_name":"%REQUESTED_SERVER_NAME%","response_code":"%RESPONSE_CODE%","response_flags":"%RESPONSE_FLAGS%","route_name":"%ROUTE_NAME%","start_time":"%START_TIME%","trace_id":"%REQ(X-B3-TRACEID)%","upstream_cluster":"%UPSTREAM_CLUSTER%","upstream_host":"%UPSTREAM_HOST%","upstream_local_address":"%UPSTREAM_LOCAL_ADDRESS%","upstream_service_time":"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%","upstream_transport_failure_reason":"%UPSTREAM_TRANSPORT_FAILURE_REASON%","user_agent":"%REQ(USER-AGENT)%","x_forwarded_for":"%REQ(X-FORWARDED-FOR)%","authority_for":"%REQ(:AUTHORITY)%","upstream_response_time":"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%","xff":"%REQ(X-FORWARDED-FOR)%","app_service_name":"%UPSTREAM_CLUSTER%"}'
path: /dev/stdoutThe following is an example of a filter condition:
accessLogging:
- disabled: false
filter:
expression: response.code >= 400
providers:
- name: envoyData plane log collection
When collecting data plane logs to Simple Log Service (SLS), configure collection rules to control how logs are collected and retained. Container Service for Kubernetes (ACK) integrates with SLS, allowing you to collect the access log from clusters on the data plane. For detailed steps, see Use Simple Log Service to collect access logs on the data plane.
The configuration page contains the following settings:
-
Log Service Project: Select Use Default or Use Existing.
-
Gateway Access Log Storage Duration: The default value is 30 days.
-
Sidecar Access Log Storage Duration: The default value is 30 days.
After you complete the settings, click Enable Data Plane Log Collection.
Control plane logs and alerts
ASM supports collecting control plane logs and configuring log-based alerts. For example, you can collect logs about the ASM control plane pushing configurations to data plane sidecar proxies. A primary function of the control plane is to deliver mesh rule configurations to data plane sidecar proxies and gateways. If a configuration conflict causes a push to fail, the proxy or gateway will not receive the latest rules. While it can continue operating with its last known good configuration, a pod restart will likely cause the proxy or gateway to fail. Misconfigurations that make gateways or proxies unavailable are a common issue. Enabling control plane log alerts is essential for quickly identifying and resolving these problems. For detailed instructions, see Enable control-plane log collection and log-based alerting in an ASM instance of a version earlier than 1.17.2.35 or Enable control-plane log collection and log-based alerting in an ASM instance of version 1.17.2.35 or later.
Metrics
Metrics are another key dimension of observability in a service mesh, describing request handling, service-to-service communication, and more. Istio uses Prometheus to collect and store metrics. The Envoy proxy for each service generates a large volume of metrics. You can use these metrics for real-time monitoring of service health and performance, as well as for scenarios like anomaly detection and auto-scaling.
Metric generation rules
Enabling service mesh data plane metrics lets gateways and sidecar proxies generate data related to their operational status. You can send these metrics to Managed Service for Prometheus to view monitoring dashboards directly (which may incur costs), or you can use a self-managed Prometheus instance to scrape metrics from the ASM data plane.
Using the Telemetry CRD, ASM provides a GUI to simplify custom metric configuration. For detailed steps, see Create custom metrics in ASM.
The following is the equivalent YAML configuration:
Metric considerations
-
Initial enablement: Managed Service for Prometheus is a paid service. To avoid excessive costs, carefully define the scope of metric generation based on your actual needs. For example, to monitor a gateway, you must enable client-side metrics. If you have previously configured metrics, your settings are preserved when you re-enable them.
-
Mesh Topology settings: The Mesh Topology feature depends on metrics reported by the sidecar proxy. If you enable Mesh Topology, disabling certain metrics can impair its functionality or make it unavailable.
-
If you do not enable the server-side metric for REQUEST_COUNT, topology graphs for HTTP or gRPC services cannot be generated.
-
If you do not enable the server-side metric for TCP_SENT_BYTES, topology graphs for TCP services cannot be generated.
-
Disabling the server-side metrics for REQUEST_SIZE and REQUEST_DURATION, or the client-side metric for REQUEST_SIZE, makes some monitoring information on topology nodes unavailable.
-
Metric collection
Enable data collection in Prometheus to send the collected metrics for storage and analysis. ASM integrates with Managed Service for Prometheus to enable service mesh monitoring. For detailed steps, see Integrate Managed Service for Prometheus to monitor ASM instances.
The Prometheus scraping interval significantly affects the overhead of metric collection. A longer interval means fewer data points are scraped, which reduces processing, storage, and computation costs. The default interval is 15 seconds, which may be too frequent for production environments. Adjust the interval on the Prometheus side based on your needs. If you are using Managed Service for Prometheus, make this configuration in the ARMS console. For detailed steps, see Configure data collection rules.
Histogram-related metrics, including istio_request_duration_milliseconds_bucket, istio_request_bytes_bucket, and istio_response_bytes_bucket, are typically high-cardinality and can be costly. To avoid ongoing charges for these custom metrics, you can discard them. If you are using Managed Service for Prometheus, configure this in the ARMS console. For detailed steps, see Configure metrics.
ASM also supports integration with a self-managed Prometheus for mesh monitoring. For detailed steps, see Monitor ASM instances by using a self-managed Prometheus instance.
As shown in the following figure, you can view the corresponding dashboard in Grafana.
Merge Istio and application metrics
For application services that already expose a Prometheus metrics endpoint, you can enable the metric merging feature to allow the sidecar proxy to export your existing business metrics. When this feature is enabled, ASM merges application metrics with Istio metrics. ASM adds the corresponding prometheus.io annotations to all data plane pods to enable metric scraping by Prometheus. If these annotations already exist, ASM overwrites them. The sidecar proxy merges the application and Istio metrics, and Prometheus can then scrape the merged metrics from the :15020/stats/prometheus endpoint. For detailed steps, see Merge Istio metrics with application metrics.
Mesh topology
Mesh Topology is an observability tool for service meshes. It provides a visual interface to view related services and configurations. As shown in the following figure, ASM includes a built-in Mesh Topology. For detailed steps, see Enable Mesh Topology to improve observability.
Service level objective (SLO)
A service level indicator (SLI) is a measure of service health. A service level objective (SLO) is a target value or range for one or more SLIs.
A service level objective (SLO) provides a formal way to describe, measure, and monitor the performance, quality, and reliability of microservice applications. SLOs offer a shared quality baseline for application development, platform, and operations teams, serving as a reference for measuring service level quality and driving continuous improvement. Combining SLIs to define an SLO helps teams describe service health more precisely.
The following are examples of an SLO:
-
Average QPS per minute > 100,000/s
-
99th percentile access latency < 500 ms
-
99th percentile bandwidth per minute > 200 MB/s
ASM provides out-of-the-box monitoring and alerting capabilities based on service level objectives (SLOs), allowing you to monitor characteristics such as latency and error rates for calls between application services.
The following SLI types are supported in ASM:
-
Availability: The proportion of successful responses that a service returns. The corresponding SLI plugin type is availability. HTTP status code 429 or 5XX (status codes that start with 5) are treated as unavailable.
-
Latency: The time (in milliseconds) a service takes to return a response. The corresponding SLI plugin type is latency. You can customize the upper latency threshold. Responses that exceed the threshold are treated as non-compliant.
ASM provides the UI for defining SLO configurations.
The SLO creation page contains the following configuration items:
-
Basic Information: Set Namespace to
default, set Target Service tohttpbin. Name is auto-generated asasm-slo-default-httpbin. Set Duration to 30 Days. -
SLO Rules: Set Name to
asm-slo, set Target Value to99, and set Plugin Type to availability. -
Alert Rules: Turn on the alert rule switch, set Alert Rule Name to
asm-alert, and enable both Critical and Warning level alert rules.
When you use ASM to define an application-level service level objective (SLO), ASM automatically generates Prometheus rules. After you import these rules into Prometheus, it can enforce the SLO. In the Prometheus framework, the Alertmanager component is responsible for collecting alerts generated by the Prometheus server and sending them to various receivers based on your configuration. When an alert is triggered, you can see the custom alert information collected on the Alertmanager page. For more information about SLOs, see SLO management.
The triggered ASM SLO alert is named asm-alert and contains two records: slo_severity="page" (slo_window="30m") and slo_severity="ticket" (slo_window="2h").
Distributed tracing
Distributed tracing is a critical component of observability in a service mesh. It is a method for profiling and monitoring applications, especially those built with a microservices architecture. In a microservices architecture, communication between services occurs over the network, which makes distributed tracing technology necessary to track and monitor the call relationships between services. In Istio, you can use distributed tracing tools like Jaeger and Zipkin to achieve this. Distributed tracing involves two key concepts: a trace and a span.
-
Span: The basic unit of distributed tracing. It represents a single unit of work in a distributed system. Each span can contain references to other spans. Multiple spans together form a trace.
-
Trace: A record of the complete execution path of a request through a microservices system. A complete trace consists of one or more spans.
Although Istio proxies can automatically send span information, the application must still propagate the appropriate HTTP headers. This enables the proxies to correctly correlate individual spans into a single, complete trace when they are sent. Therefore, your application must collect the following headers from incoming requests and propagate them to any outgoing requests:
-
x-request-id
-
x-b3-traceid
-
x-b3-spanid
-
x-b3-parentspanid
-
x-b3-sampled
-
x-b3-flags
-
x-ot-span-context
Tracing data generation rules
Based on the Telemetry CRD, ASM provides a GUI to simplify the configuration of rules for generating distributed tracing data.
The following is the equivalent YAML configuration:
tracing:
- customTags:
mytag1:
literal:
value: fixedvalue
mytag2:
header:
defaultValue: value1
name: myheader1
mytag3:
environment:
defaultValue: value1
name: myenv1
providers:
- name: zipkin
randomSamplingPercentage: 90Tracing data collection
To send collected data to a managed cloud service or a self-managed backend, use one of the following approaches:
-
Managed service: Use a cloud-native application management service to collect and analyze data. For detailed steps, see Enable distributed tracing in ASM.
-
Self-managed service: Use open-source collection and analysis tools such as Zipkin or Jaeger. For detailed steps, see Export ASM tracing data to a self-managed system.