The Mixerless Telemetry technology of Alibaba Cloud Service Mesh (ASM) allows you to obtain telemetry data on containers in a non-intrusive manner. Telemetry data is collected by Prometheus Service or self-managed Prometheus as monitoring metrics. You can use the telemetry data to observe ASM instances. This topic describes how to use ASM to obtain application monitoring metrics that are collected by self-managed Prometheus to observe ASM instances.

Prerequisites

Step 1: Install Prometheus

  1. Download and decompress the installation package of Istio. To download the installation package of Istio, go to the Download Istio page.
  2. Use kubectl to connect to an ACK cluster. For more information, see Connect to Kubernetes clusters by using kubectl.
  3. Run the following command to install Prometheus:
    kubectl --kubeconfig <Path of the kubeconfig file> apply -f <Path to which the installation package of Istio is decompressed>/samples/addons/prometheus.yaml

Step 2: Create an Envoy filter

  1. Log on to the ASM console.
  2. In the left-side navigation pane, choose Service Mesh > Mesh Management.
  3. On the Mesh Management page, find the ASM instance that you want to configure. Click the name of the ASM instance or click Manage in the Actions column of the ASM instance.
  4. On the details page of the ASM instance, click Settings in the upper-right corner.
  5. In the Settings Update panel, select Enable Prometheus and then Enable Self-managed Prometheus. In the field that appears, enter a Prometheus endpoint. In this example, the default endpoint http://prometheus:9090 is used. Then, click OK.
    Note This topic uses self-managed Prometheus as an example. If you use Prometheus Service, see Monitor service meshes based on ARMS Prometheus.
    In the left-side navigation pane of the details page, choose Traffic Management > EnvoyFilter. On the EnvoyFilter page, you can view the created Envoy filter.

Step 3: Configure Prometheus

  1. Configure the monitoring metrics of Istio.
    1. Log on to the ACK console.
    2. In the left-side navigation pane of the ACK console, click Clusters.
    3. On the Clusters page, find the cluster that you want to manage and click the name of the cluster or click Details in the Actions column. The details page of the cluster appears.
    4. In the left-side navigation pane of the details page, choose Configurations > ConfigMaps.
    5. In the upper part of the ConfigMap page, select istio-system from the Namespace drop-down list. Find the item that is named prometheus and click Edit in the Actions column.
    6. In the Edit panel, enter configuration information in the Value field and click OK. To obtain the configuration information, visit GitHub.
  2. Delete the pod of Prometheus to make Prometheus configurations take effect.
    1. Log on to the ACK console.
    2. In the left-side navigation pane of the ACK console, click Clusters.
    3. On the Clusters page, find the cluster that you want to manage and click the name of the cluster or click Details in the Actions column. The details page of the cluster appears.
    4. In the left-side navigation pane of the details page, choose Workloads > Pods.
    5. On the Pods page, find the pod that is named Prometheus and click Delete in the Actions column.
    6. In the Delete Pod message, click Confirm.
  3. Run the following command to view job_name in the Prometheus configurations:
    kubectl --kubeconfig <Path of the kubeconfig file> get cm prometheus -n istio-system -o jsonpath={.data.prometheus\\.yml} | grep job_name

    Expected output:

    - job_name: 'istio-mesh'
    - job_name: 'envoy-stats'
    - job_name: 'istio-policy'
    - job_name: 'istio-telemetry'
    - job_name: 'pilot'
    - job_name: 'sidecar-injector'
    - job_name: prometheus
      job_name: kubernetes-apiservers
      job_name: kubernetes-nodes
      job_name: kubernetes-nodes-cadvisor
    - job_name: kubernetes-service-endpoints
    - job_name: kubernetes-service-endpoints-slow
      job_name: prometheus-pushgateway
    - job_name: kubernetes-services
    - job_name: kubernetes-pods
    - job_name: kubernetes-pods-slow

Step 4: Generate monitoring data

  1. Deploy the podinfo application in the ACK cluster.
    1. Download the required YAML files of the podinfo application. For more information, visit GitHub.
    2. Run the following commands to deploy the podinfo application in the ACK cluster:
      kubectl --kubeconfig <Path of the kubeconfig file> apply -f <Path of the podinfo application>/kustomize/deployment.yaml -n test
      kubectl --kubeconfig <Path of the kubeconfig file> apply -f <Path of the podinfo application>/kustomize/service.yaml -n test
  2. Run the following command to request the podinfo application to generate monitoring data:
    podinfo_pod=$(k get po -n test -l app=podinfo -o jsonpath={.items..metadata.name})
    for i in {1..10}; do
       kubectl --kubeconfig "$USER_CONFIG" exec $podinfo_pod -c podinfod -n test -- curl -s podinfo:9898/version
      echo
    done
  3. Check whether monitoring data is generated in the Envoy container.
    1. Run the following command to request Envoy to check whether the monitoring data of the istio_requests_total metric is generated:
      kubectl --kubeconfig <Path of the kubeconfig file> exec $podinfo_pod -n test -c istio-proxy -- curl -s localhost:15090/stats/prometheus | grep istio_requests_total

      Expected output:

      :::: istio_requests_total ::::
      # TYPE istio_requests_total counter
      istio_requests_total{response_code="200",reporter="destination",source_workload="podinfo",source_workload_namespace="test",source_principal="spiffe://cluster.local/ns/test/sa/default",source_app="podinfo",source_version="unknown",source_cluster="c199d81d4e3104a5d90254b2a210914c8",destination_workload="podinfo",destination_workload_namespace="test",destination_principal="spiffe://cluster.local/ns/test/sa/default",destination_app="podinfo",destination_version="unknown",destination_service="podinfo.test.svc.cluster.local",destination_service_name="podinfo",destination_service_namespace="test",destination_cluster="c199d81d4e3104a5d90254b2a210914c8",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="podinfo",destination_canonical_service="podinfo",source_canonical_revision="latest",destination_canonical_revision="latest"} 10
      
      istio_requests_total{response_code="200",reporter="source",source_workload="podinfo",source_workload_namespace="test",source_principal="spiffe://cluster.local/ns/test/sa/default",source_app="podinfo",source_version="unknown",source_cluster="c199d81d4e3104a5d90254b2a210914c8",destination_workload="podinfo",destination_workload_namespace="test",destination_principal="spiffe://cluster.local/ns/test/sa/default",destination_app="podinfo",destination_version="unknown",destination_service="podinfo.test.svc.cluster.local",destination_service_name="podinfo",destination_service_namespace="test",destination_cluster="c199d81d4e3104a5d90254b2a210914c8",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="podinfo",destination_canonical_service="podinfo",source_canonical_revision="latest",destination_canonical_revision="latest"} 10
    2. Run the following command to request Envoy to check whether the monitoring data of the istio_request_duration metric is generated:
      kubectl --kubeconfig <Path of the kubeconfig file> exec $podinfo_pod -n test -c istio-proxy -- curl -s localhost:15090/stats/prometheus | grep istio_request_duration

      Expected output:

      :::: istio_request_duration ::::
      # TYPE istio_request_duration_milliseconds histogram
      istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="podinfo",source_workload_namespace="test",source_principal="spiffe://cluster.local/ns/test/sa/default",source_app="podinfo",source_version="unknown",source_cluster="c199d81d4e3104a5d90254b2a210914c8",destination_workload="podinfo",destination_workload_namespace="test",destination_principal="spiffe://cluster.local/ns/test/sa/default",destination_app="podinfo",destination_version="unknown",destination_service="podinfo.test.svc.cluster.local",destination_service_name="podinfo",destination_service_namespace="test",destination_cluster="c199d81d4e3104a5d90254b2a210914c8",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="podinfo",destination_canonical_service="podinfo",source_canonical_revision="latest",destination_canonical_revision="latest",le="0.5"} 10
      
      istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="podinfo",source_workload_namespace="test",source_principal="spiffe://cluster.local/ns/test/sa/default",source_app="podinfo",source_version="unknown",source_cluster="c199d81d4e3104a5d90254b2a210914c8",destination_workload="podinfo",destination_workload_namespace="test",destination_principal="spiffe://cluster.local/ns/test/sa/default",destination_app="podinfo",destination_version="unknown",destination_service="podinfo.test.svc.cluster.local",destination_service_name="podinfo",destination_service_namespace="test",destination_cluster="c199d81d4e3104a5d90254b2a210914c8",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="podinfo",destination_canonical_service="podinfo",source_canonical_revision="latest",destination_canonical_revision="latest",le="1"} 10
      ...

Verify the result

  1. Expose Prometheus by using a Server Load Balancer (SLB) instance. For more information, see Manage Services.
  2. Log on to the ACK console.
  3. In the left-side navigation pane of the ACK console, click Clusters.
  4. On the Clusters page, find the cluster that you want to manage and click the name of the cluster or click Details in the Actions column. The details page of the cluster appears.
  5. In the left-side navigation pane of the details page, choose Network > Services.
  6. On the Services page, find the service that is named Prometheus and click the IP address in the External Endpoint column.
  7. On the Prometheus page, enter istio_requests_total in the search box and click Execute.
    If the page shown in the following figure appears, application monitoring metrics are collected by Prometheus. prometheus