Configure periodic ASM metric cleanup - Alibaba Cloud Service Mesh

Service Mesh (ASM) generates monitoring metrics for all traffic that flows into, out of, and within the service mesh to help you monitor service behavior. These metrics include information such as traffic volume, error rate, and response time. Over time, large amounts of metric data can accumulate, increasing the resource consumption of Envoy and Prometheus. To manage this, ASM provides a feature to periodically clean up monitoring metrics. This feature automatically removes unused metrics cached in Envoy, reducing Envoy's memory consumption and lowering the network load when Prometheus scrapes metrics. This topic describes how to configure this feature and provides a working example.

Prerequisites

You have added a cluster to an ASM instance of version 1.18 or later. For more information, see Add a cluster to an ASM instance.
Automatic sidecar proxy injection is enabled.

Procedure

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Observability Management Center > Observability Settings.
On the Observability Settings page, click the Global tab. In the Metric Settings section, enter a value for Periodic Cleaning Time of Metrics and then click Submit.

Turn on the Enable Periodic Cleaning of Metrics switch.

Important
Set the periodic cleanup time to at least twice the Prometheus scrape_interval. This ensures that Prometheus can successfully scrape the metrics before they are cleared.

Example

Deploy the sample application

Use the following YAML content to deploy a sample application on the data plane cluster. For more information, see Create a stateless workload Deployment or Create resources by using a YAML file.

Click to view the YAML

apiVersion: v1
kind: Service
metadata:
  name: httpbin
  labels:
    app: httpbin
    service: httpbin
spec:
  ports:
  - name: http
    port: 8000
    targetPort: 80
  selector:
    app: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
      version: v1
  template:
    metadata:
      labels:
        app: httpbin
        version: v1
    spec:
      containers:
      - image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/httpbin:0.1.0
        imagePullPolicy: IfNotPresent
        name: httpbin
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: sleep
  labels:
    app: sleep
    service: sleep
spec:
  ports:
  - port: 80
    name: http
  selector:
    app: sleep
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sleep
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sleep
  template:
    metadata:
      labels:
        app: sleep
    spec:
      terminationGracePeriodSeconds: 0
      containers:
      - name: sleep
        image: registry.cn-hangzhou.aliyuncs.com/acs/curl:8.1.2
        command: ["/bin/sleep", "infinity"]
        imagePullPolicy: IfNotPresent

Run the following command to check the application status:

kubectl get pod

Example output:

NAME                       READY   STATUS    RESTARTS   AGE
httpbin-846bxxx694-xxxxx   2/2     Running   0          44s
sleep-866xxx97f9-xxxxx     2/2     Running   0          44s

Enable metrics and test

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Observability Management Center > Observability Settings.
On the Observability Settings page, click the Global tab. In the Metric Settings section, select the Enabled checkbox for the REQUEST_COUNT metric in the Server-side Metrics section. Then, click Submit.

Run the following command to access the httpbin application:

 kubectl exec -it deploy/sleep -- sh -c 'for i in $(seq 1 10); do curl -s httpbin:8000/status/418 > /dev/null; done'

Run the following command to view the monitoring metrics of the httpbin application:

kubectl exec -it deploy/httpbin -c istio-proxy -- curl 0.0.0.0:15020/stats/prometheus | grep istio_requests_total{

Example output:

istio_requests_total{reporter="destination",source_workload="sleep",source_canonical_service="sleep",...,request_protocol="http",response_code="418",grpc_response_status="",response_flags="-",connection_security_policy="mutual_tls"} 20

Configure periodic metric cleanup

Follow the steps in the Procedure section to set the periodic cleanup time for monitoring metrics to 10s.

Metric cleanup may be delayed by a few seconds. During verification, you can adjust the cleanup time as needed.

After completing the configuration, run the following command to view the monitoring metrics of the httpbin application:

kubectl exec -it deploy/httpbin -c istio-proxy -- sh -c ' for i in $(seq 1 2); do echo "# Current content of the istio_requests_total metric: "; curl -s 0.0.0.0:15020/stats/prometheus | grep istio_requests_total{; sleep 15; done'

Example output:

# Current content of the istio_requests_total metric: 
istio_requests_total{reporter="destination",source_workload="sleep",source_canonical_service="sleep",...connection_security_policy="mutual_tls"} 30
# Current content of the istio_requests_total metric:

The test command makes two polling requests. The first request, made before cleanup, returns one line of metric output and one line of custom output. After 15 seconds, the metric data is cleared, and the subsequent request returns only one line of custom output.