Collect metrics for mesh applications over mTLS - Alibaba Cloud Service Mesh

When you enable mutual TLS (mTLS) in Service Mesh (ASM), the sidecar proxy intercepts all inbound traffic to your applications. Consequently, the port that exposes application metrics must also be accessed over mTLS. For critical services, securing metrics collection is just as important as encrypting service-to-service communication. This topic uses a Service Mesh (ASM) instance deployed with Prometheus Operator to demonstrate how to collect metrics from applications in a service mesh over mTLS.

Important

If you use Alibaba Cloud Application Real-Time Monitoring Service (ARMS) to collect metrics, the agent version must be 1.1.20 or later. Log on to the ARMS console. In the left-side navigation pane, click Integration Management. Find your cluster in the list, and in the Actions column, click Configure Agent to view the Prometheus agent version and upgrade it if necessary.

Prerequisites

The Bookinfo sample application is deployed. For more information, see Deploy an application in the cluster associated with your ASM instance.

Configure Prometheus to scrape metrics over TLS

To pass the TLS authentication that the sidecar proxy enforces, Prometheus must use a certificate issued by the ASM instance's root certificate. You can achieve this by using the sidecar proxy's certificate mounting capability. By defining specific annotations for the Prometheus pod, you configure the sidecar proxy to mount the certificate issued by the ASM control plane to a shared volume. The Prometheus container can then access the certificate and key by mounting this shared volume. The following steps describe this process:

Add the istio-certs volume to the Prometheus pod.

volumes:
- emptyDir:
    medium: Memory
  name: istio-certs

Add the following two annotations to the Prometheus pod.
```
annotations:
  proxy.istio.io/config: |
    proxyMetadata:
      OUTPUT_CERTS: /etc/istio-output-certs
  sidecar.istio.io/userVolumeMount: '[{"name": "istio-certs", "mountPath": "/etc/istio-output-certs"}]'
```
- proxy.istio.io/config: Specifies the proxy configuration, where proxyMetadata.OUTPUT_CERTS stores the certificates and keys in the /etc/istio-output-certs path.
- sidecar.istio.io/userVolumeMount: Mounts a volume to the /etc/istio-output-certs path in the sidecar proxy container.
Mount the istio-certs volume to the /etc/prom-certs/ path in the Prometheus container to allow Prometheus to obtain the certificates and keys that the sidecar proxy writes to this path.
```
volumeMounts:
- mountPath: /etc/prom-certs/
  name: istio-certs
```
In the scraping configuration, specify which workloads use TLS to initiate metric scraping requests and provide the certificate path. Only workloads with an injected sidecar proxy require this configuration.
This topic uses a Prometheus Operator environment as an example.
```
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: productpage
  labels:
    app: productpage
    team: bookinfo
spec:
  selector:
    matchLabels:
      app: productpage
  endpoints:
  - port: http-9080
    interval: 30s
    path: /metrics
    scheme: https
    tlsConfig:
      caFile: /etc/prom-certs/root-cert.pem
      certFile: /etc/prom-certs/cert-chain.pem
      keyFile: /etc/prom-certs/key.pem
      insecureSkipVerify: true
```
In the preceding YAML, the labels field specifies metrics collection for the productpage application, and its scrape endpoint is defined. In the endpoint definition, the TLS-related configurations are as follows:
- scheme: https: Specifies that the request is sent over HTTPS.
- tlsConfig: Specifies the file paths for the certificate, CA certificate, and key.
- insecureSkipVerify: true: Allows insecure authentication access because Prometheus does not support Istio's identity naming scheme.

The preceding configuration allows Prometheus to mount the certificate and key provided by the sidecar proxy and use them to initiate TLS requests.

Procedure

Step 1: Install Prometheus Operator

Run the following command to clone the Prometheus Operator source repository from GitHub to your local machine.
```
git clone https://github.com/prometheus-operator/prometheus-operator.git
```
Run the following command to install Prometheus Operator.
```
cd prometheus-operator/
kubectl create -f bundle.yaml
```

Run the following command to check the pod status.

kubectl get pods

Expected output:

NAME                                     READY   STATUS        RESTARTS   
prometheus-operator-58dd988c9c-qhrrp     2/2     Running       0

This output indicates that Prometheus Operator is installed successfully.

Step 2: Deploy Prometheus by using a CR

Save the following YAML to a local file named prometheus.yaml.

The YAML includes the Prometheus instance declaration and the required ServiceAccount, ClusterRole, and ClusterRoleBinding. The Deployment configuration contains the certificate volume mount settings described in Configure Prometheus to scrape metrics over TLS.

Note

The Prometheus-related custom resources (CRs) provided in this topic are for demonstration purposes only. Adjust them based on your production environment.

Prometheus.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus-full-access
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus-full-access
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/metrics
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups:
  - extensions
  - apps
  resources:
  - deployments
  - replicasets
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources:
  - configmaps
  verbs: ["get"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
---
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: default
  labels:
    prometheus: default
spec:
  logLevel: debug
  podMetadata:
    annotations:
      traffic.sidecar.istio.io/includeInboundPorts: ""
      traffic.sidecar.istio.io/includeOutboundIPRanges: ""
      proxy.istio.io/config: |  # configure an env variable `OUTPUT_CERTS` to write certificates to the given folder
        proxyMetadata:
          OUTPUT_CERTS: /etc/istio-output-certs
      sidecar.istio.io/userVolumeMount: '[{"name": "istio-certs", "mountPath": "/etc/istio-output-certs"}]'
  volumes:
  - emptyDir:
      medium: Memory
    name: istio-certs
  volumeMounts:
  - mountPath: /etc/prom-certs/
    name: istio-certs
  replicas: 2
  version: v2.26.0
  serviceAccountName: prometheus-full-access
  serviceMonitorSelector:
    matchLabels:
      team: bookinfo
  ruleSelector:
    matchLabels:
      role: alert-rules
      prometheus: example

Run the following command to apply the prometheus.yaml file to the cluster.
```
kubectl apply -f prometheus.yaml
```

Run the following command to check if the Prometheus instance started correctly.

kubectl get pods

Expected output:

NAME                                   READY   STATUS    RESTARTS
prometheus-default-0                   3/3     Running   0 
prometheus-default-1                   3/3     Running   0  
prometheus-operator-58dd988c9c-qhrrp   2/2     Running   0

The output shows that the prometheus-default-0 and prometheus-default-1 pods are running.

Step 3: Define scraping rules by using ServiceMonitor

Save the following YAML to a local file named service-monitor.yaml.
The YAML file contains the declaration of the ServiceMonitor API, which describes how to collect metrics from the workload. The YAML file shows the certificate path and the scheme setting mentioned in Configure a Prometheus instance to collect TLS metrics.
Note
The Prometheus-related CRs provided in this topic are for demonstration purposes only. Adjust them based on your production environment.
```
iVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: productpage
  labels:
    app: productpage
    team: bookinfo
spec:
  selector:
    matchLabels:
      app: productpage
  endpoints:
  - port: http-9080
    interval: 30s
    path: /metrics
    scheme: https
    tlsConfig:
      caFile: /etc/prom-certs/root-cert.pem
      certFile: /etc/prom-certs/cert-chain.pem
      keyFile: /etc/prom-certs/key.pem
      insecureSkipVerify: true
```
Run the following command to apply the service-monitor.yaml file to the cluster.
```
kubectl apply -f service-monitor.yaml
```

Step 4: Access the Prometheus UI and verify metrics

Run the following command to set up port forwarding from your local port 9090 to port 9090 of the prometheus-operated service.
```
kubectl port-forward svc/prometheus-operated 9090
```
Enter localhost:9090 in a browser to open the Prometheus web UI.
The Prometheus query interface opens. You can enter a query expression in the Expression input box, click Execute to run the query, and view the results in the Table or Graph tab.
In the top menu bar, select Status > Target to view the status of the monitoring target.
Note
If a target is scraped over TLS, it might initially show a status of "Unavailable". This can occur if TLS is configured but the authentication settings are incorrect.
Confirm that the State for the target is Up, which indicates that metrics are being scraped successfully.
In the top menu bar, click Graph, enter python_gc_objects_collected_total in the query text box, and click Execute on the right.
After the query completes, the reported metric data is displayed.
The query results are displayed in the Table view, grouped by the generation tag into three records (generation 0, 1, and 2). These records correspond to the total number of objects collected by each of Python's three garbage collection (GC) generations.