Prometheus is an open source tool that is used to monitor cloud-native applications. This topic shows you how to deploy a self-managed Prometheus instance to monitor Alibaba Cloud Service Mesh (ASM) instances.

Prerequisites

Step 1: Configure monitoring metrics

Copy the following content to the configuration file of the Prometheus instance. Then, restart the Prometheus instance for the configuration to take effect. 
scrape_configs:
# Mixer scrapping. Defaults to Prometheus and mixer on same namespace.
- job_name: 'istio-mesh'
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - istio-system
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
    action: keep
    regex: istio-telemetry;prometheus
# Scrape config for envoy stats
- job_name: 'envoy-stats'
  metrics_path: /stats/prometheus
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_container_port_name]
    action: keep
    regex: '.*-envoy-prom'
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:15090
    target_label: __address__
  - action: labeldrop
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: namespace
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: pod_name
- job_name: 'istio-policy'
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - istio-system
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
    action: keep
    regex: istio-policy;http-policy-monitoring
- job_name: 'istio-telemetry'
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - istio-system
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
    action: keep
    regex: istio-telemetry;http-monitoring
- job_name: 'pilot'
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - istio-system
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
    action: keep
    regex: istiod;http-monitoring
  - source_labels: [__meta_kubernetes_service_label_app]
    target_label: app
- job_name: 'sidecar-injector'
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - istio-system
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
    action: keep
    regex: istio-sidecar-injector;http-monitoring

Step 2: Enable Prometheus

  1. Log on to the ASM console.
  2. In the left-side navigation pane, choose Service Mesh > Mesh Management.
  3. On the Mesh Management page, find the ASM instance that you want to configure. Click the name of the ASM instance or click Manage in the Actions column.
  4. On the details page of the ASM instance, choose ASM Instance > Base Information. On the Basic Information page, click Settings.
    Note Make sure that the Istio version of the ASM instance is 1.6.8.4 or later.
  5. In the Settings Update panel, select Enable Prometheus, select Enable Self-managed Prometheus, enter the endpoint of the Prometheus instance, and then click OK. Kiali uses the endpoint of the Prometheus instance to display the topology of the ASM instance. You need to enter the endpoint of the Prometheus instance only if you select Enable Kiali.
    After you enable Prometheus for the ASM instance, ASM automatically configures the Envoy filters that are required for Prometheus.

Step 3: Generate monitoring metric data

Send requests to services on the data plane. In this example, the Bookinfo application is used. For more information, see Overview. Go to the Productpage page of Bookinfo. Refresh the page multiple times. Monitoring metric data

Verify the result

  • Run the following commands to check whether Envoy proxies collect data of the specified monitoring metrics.

    Data in the command output indicates the monitoring metric data that is collected by Envoy proxies. If the command output contains no data, Envoy proxies do not collect monitoring metric data. 

    details=$(kubectl get pod -l app=details -o jsonpath={.items..metadata.name})
kubectl exec $details -c istio-proxy -- curl -s localhost:15090/stats/prometheus grep istio
  • View the metrics that are monitored by Prometheus.
    1. Log on to the ACK console.
    2. In the left-side navigation pane of the ACK console, click Clusters.
    3. On the Clusters page, find the cluster that you want to manage and click its name or click Details in the Actions column.
    4. In the left-side navigation pane of the details page, choose Network > Services.
    5. On the Services page, click the public endpoint of Prometheus.
    6. On the Prometheus page, enter istio_requests_total in the search box and click Execute. The metrics that are monitored by Prometheus are displayed. peomethus
  • View the Grafana report of the monitoring metric data.
    1. Log on to the ACK console.
    2. In the left-side navigation pane of the ACK console, click Clusters.
    3. On the Clusters page, find the cluster that you want to manage and click its name or click Details in the Actions column.
    4. In the left-side navigation pane of the details page, choose Network > Services.
    5. On the Services page, click the public endpoint of Grafana.
    6. On the Grafana page, select Istio Workload Dashboard in the upper-left corner. The following figure shows the Grafana report that is displayed. 3