Create service discovery using a ServiceMonitor - Managed Service for Prometheus

Managed Service for Prometheus supports service discovery using the ServiceMonitor custom resource definition (CRD). You can use a ServiceMonitor to define the namespace for pod discovery and select Services to monitor with matchLabel. This topic describes how to create a service discovery configuration using a ServiceMonitor for an application built on the Spring Boot framework.

Demo

Download the demo project for a complete example of how to create a service discovery configuration using a ServiceMonitor.

Step 1: Add dependencies

Create a Maven application and add the following dependencies to the pom.xml file.

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-registry-prometheus</artifactId>
            <version>1.6.6</version>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-configuration-processor</artifactId>
            <optional>true</optional>
        </dependency>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <optional>true</optional>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>

Add the following configuration to the src/resources/applications.properties file in the project.
```
management.endpoints.web.exposure.include=prometheus
```
Start the project and access http://{host}:{port}/actuator/prometheus in a browser.
The monitoring data for the corresponding Java Virtual Machine (JVM) is displayed. The following figure shows an example.

Step 2: Deploy a Kubernetes cluster

Build an image and upload the image to an image repository. For more information, see Bind a source code hosting platform.

Create a deployment using the following YAML configuration.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: micrometer-prometheus
  namespace: default
  labels:
    app: demo-prometheus
spec:
  replicas: 3
  selector:
    matchLabels:
      app: demo-prometheus
  template:
    metadata:
      labels:
        app: demo-prometheus
    spec:
      containers:
        - name: micrometer-prometheus
          image: manjusakalza/micrometer-prometheus:latest
          ports:
            - containerPort: 8080

Create a Service using the following YAML configuration.

apiVersion: v1
kind: Service
metadata:
  name: prometheus-metrics-demo
  namespace: default
  labels:
    micrometer-prometheus-discovery: 'true'
spec:
  selector:
    app: demo-prometheus
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
      name: metrics

Step 3: Create a ServiceMonitor

Save the YAML file to your computer and run kubectl apply -f {path/to/the/YAML/file} to apply the file.

The following is a sample YAML file for a ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: micrometer-demo
  namespace: default
spec:
  endpoints:
    - interval: 15s
      path: /actuator/prometheus
      port: metrics    # Note: Configure the port name, not the port number.
  namespaceSelector:
    any: true
  selector:
    matchLabels:
      micrometer-prometheus-discovery: 'true'

The fields in this YAML file are described as follows:

The metadata fields name and namespace specify key metadata for the ServiceMonitor.
The spec field endpoints specifies the server endpoints from which Prometheus scrapes metrics. endpoints is an array that lets you create multiple endpoints. Each endpoint contains the following three fields:
- interval: Specifies the interval at which Prometheus scrapes the current endpoint. The unit is seconds. In this example, the value is set to 15s.
- path: Specifies the scrape path for Prometheus. In this example, the value is set to /actuator/prometheus.
- port: The name of the service port from which to collect metrics. Set this parameter to the value of the name parameter specified for the Service that you created in Step 2. In this example, this parameter is set to metrics.
  Important
  Configure the port name, not the port number.
The spec field namespaceSelector specifies the scope for Service discovery. namespaceSelector contains two mutually exclusive fields. The fields are described as follows:
- any: Has only one value, true. If this field is set to `true`, the system monitors all Services that match the selector filter conditions in all namespaces.
- matchNames: An array that specifies the namespaces to monitor. For example, to monitor Services only in the default and arms-prom namespaces, set matchNames as follows:
```
namespaceSelector:
  matchNames:
  - default
  - arms-prom
```
The spec field selector is used to select Services.
The Service used in this example has the micrometer-prometheus-discovery: 'true' label. Therefore, set the selector as follows:
```
selector:
  matchLabels:
    micrometer-prometheus-discovery: 'true'
```

To use the basic auth feature, see the following sample YAML file.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: cloud-open-api-monitor     #  Set a unique name for the ServiceMonitor.
  namespace: default   #  Set the namespace where the ServiceMonitor resides.
spec:
  endpoints:
  - interval: 30s
    # Specify the interval at which Prometheus scrapes the current endpoint.
    port: tcp-8080
    # Enter the path exposed in the Prometheus exporter code.
    path: /api/actuator/prometheus
    basicAuth:
      password:
        name: basic-auth
        key: <password>
      username:
        name: basic-auth
        key: <userName>
    scheme: http
  namespaceSelector:
    any: true
  selector:
    matchLabels:
      # Match Services with the following label.
      edas.oam.acname: cloud-open-api

If you do not have the required permissions to use the basic auth feature, add a ClusterRole with the required permissions to the cluster. Then, bind this ClusterRole to the ServiceAccount named arms-prom-operator in the arms-prom namespace using a ClusterRoleBinding. This grants the required permissions to the Prometheus Agent.

ClusterRole YAML file

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus-agent-role
  labels:
    app: prometheus-agent
rules:
  - apiGroups: [""]
    resources: ["pods", "services", "endpoints", "nodes"]
    verbs: ["get", "list", "watch"]  # Adjust as needed.
  - apiGroups: ["monitoring.coreos.com"]  # Adjust the apiGroups as needed.
    resources: ["*"]
    verbs: ["get", "list", "watch", "create", "update", "delete"]  # Adjust as needed.

ClusterRoleBinding YAML file

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus-agent-binding
subjects:
  - kind: ServiceAccount
    name: arms-prom-operator  # Service account name
    namespace: arms-prom      # The namespace where the service account resides.
roleRef:
  kind: ClusterRole
  name: prometheus-agent-role  # The name of the ClusterRole defined above.
  apiGroup: rbac.authorization.k8s.io

Step 4: Verify the ServiceMonitor

Perform the following steps to verify that Prometheus successfully discovers the service.

Log on to the Managed Service for Prometheus console. In the left-side navigation pane, click Integration Management.
On the Integrated Environments tab, click the name of the target container environment in the Container Service list.
Click the Self-Monitoring tab.
On the Targets tab, check whether a target named {namespace}/{serviceMonitorName}/x exists.
Click the row for the {namespace}/{serviceMonitorName}/x target to expand it, and click the endpoint link.
Verify that the metrics are displayed correctly.