Prometheus is an open source tool that is used to monitor cloud-native applications. This topic describes how to deploy a custom Prometheus instance to monitor Alibaba Cloud Service Mesh (ASM) instances.

Prerequisites

Step 1: Configure monitoring metrics

Copy the following content to the configuration file of the Prometheus instance. Then, restart the Prometheus instance for the configuration to take effect.
scrape_configs:
# Mixer scrapping. Defaults to Prometheus and mixer on same namespace.
- job_name: 'istio-mesh'
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - istio-system
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
    action: keep
    regex: istio-telemetry;prometheus
# Scrape config for envoy stats
- job_name: 'envoy-stats'
  metrics_path: /stats/prometheus
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_container_port_name]
    action: keep
    regex: '.*-envoy-prom'
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:15090
    target_label: __address__
  - action: labeldrop
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: namespace
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: pod_name
- job_name: 'istio-policy'
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - istio-system
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
    action: keep
    regex: istio-policy;http-policy-monitoring
- job_name: 'istio-telemetry'
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - istio-system
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
    action: keep
    regex: istio-telemetry;http-monitoring
- job_name: 'pilot'
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - istio-system
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
    action: keep
    regex: istiod;http-monitoring
  - source_labels: [__meta_kubernetes_service_label_app]
    target_label: app
- job_name: 'sidecar-injector'
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - istio-system
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
    action: keep
    regex: istio-sidecar-injector;http-monitoring

Step 2: Create Envoy filters

  1. Log on to the ASM console.
  2. In the left-side navigation pane, choose Service Mesh > Mesh Management.
  3. On the Mesh Management page, find the ASM instance that you want to configure. Click the name of the ASM instance or click Manage in the Actions column of the ASM instance.
  4. On the management page of the ASM instance, click Settings in the upper-right corner.
    Note Make sure that the Istio version of the ASM instance is 1.6.8.4 or later.
  5. In the Settings Update panel, select Enable Prometheus. Then, click OK.
    After you enable Prometheus for the ASM instance, ASM automatically configures the Envoy filters that are required for Prometheus.

Step 3: Generate monitoring data

Send requests to services on the data plane. In this example, the Bookinfo application is used. For more information, see Overview. Go to the Productpage page of Bookinfo. Refresh the page multiple times. Monitoring data

View the details of the custom egress gateway service

  • Run the following commands to check whether Envoy proxies collect data of the specified monitoring metrics.

    Data in the command output indicates the monitoring data that is collected by Envoy proxies. If the command output contains no data, Envoy proxies do not collect monitoring data.

    details=$(kubectl get pod -l app=details -o jsonpath={.items..metadata.name})
    kubectl exec $details -c istio-proxy -- curl -s localhost:15090/stats/prometheus |grep istio
  • View the metrics that are monitored by Prometheus.
    1. Log on to the Container Service console.
    2. In the left-side navigation pane, click Clusters.
    3. On the Clusters page, click the name of the cluster where the Bookinfo application is deployed. Alternatively, click Details in the Actions column of the cluster where the Bookinfo application is deployed.
    4. In the left-side navigation pane of the details page, choose Network > Services.
    5. On the Services page, click the public endpoint of Prometheus.
    6. On the Prometheus page, enter istio_requests_total in the search box and click Execute. The metrics that are monitored by Prometheus are displayed. peomethus
  • View the Grafana report of the monitoring data.
    1. Log on to the Container Service console.
    2. In the left-side navigation pane, click Clusters.
    3. On the Clusters page, click the name of the cluster where the Bookinfo application is deployed. Alternatively, click Details in the Actions column of the cluster where the Bookinfo application is deployed.
    4. In the left-side navigation pane of the details page, choose Network > Services.
    5. On the Services page, click the public endpoint of Grafana.
    6. On the Grafana page, select Istio Workload Dashboard in the upper-left corner. The following figure shows the Grafana report that is displayed. 3