Application Real-Time Monitoring Service:Use Managed Service for Prometheus to monitor Tencent Cloud services

Procedure

The following figure describes how to use Managed Service for Prometheus to monitor a Tencent Cloud service:

Step 1: Deploy qcloud exporter

1. Build an image for qcloud exporter.

   git clone https://github.com/tencentyun/tencentcloud-exporter.git
   make build

2. Configure the service and the instances.

   • Configure the credentials of TencentCloud API.

   • Configure the service and the instances whose metrics you want to export.

   If you want to export all metrics of all CVM instances, you can use the following code:

   credential:
     access_key: "access_key"  #The SecretId of TencentCloud API.
     secret_key: "secret_key"  #The SecretKey of TencentCloud API.
     region: "ap-nanjing" #The region where the instances reside.
   
   rate_limit: 15              #The maximum number of API calls that CM can make per second to pull data. Example: 20. For more information, see https://cloud.tencent.com/document/product/248/31014
   
   products:
     - namespace: QCE/CVM      #For more information about the metrics, see https://cloud.tencent.com/document/product/248/6843
       all_metrics: true       #Specifies that all supported metrics are exported.
       all_instances: true     #Specifies that the metrics of all instances in the specified region are exported.
       #only_include_metrics: []
       #only_include_instances: [ins-xxxxxxxx]
       extra_labels: [InstanceId, InstanceName] #The metric labels to which the fields of instances are exported.
       #statistics_types: [last]
       #period_seconds: 60
       #metric_name_type: 2

   Note

   For more information about how to configure the qcloud.yaml file, see tencentcloud-exporter.

3. Deploy qcloud exporter.

   Create a Docker image by using the preceding YAML file and upload the image to an image repository, such as DockerHub or Alibaba Cloud Container Registry.

   1. Log on to the ACK console.

   2. In the left-side navigation pane, click Clusters.

   3. On the Clusters page, find the cluster that you want to manage and click Applications in the Actions column of the cluster.

   4. Create a container group.

      1. In the left-side navigation pane, choose Workloads > Deployments.

      2. On the Deployments page, click Create from YAML in the upper-right corner.

      3. On the Create page, enter the following code in the Template code editor and click Create:

         apiVersion: apps/v1
         kind: Deployment
         metadata:
           generation: 5
           labels:
             app: qcloud-exporter-demo
           name: qcloud-exporter-demo
           namespace: default
         spec:
           replicas: 1
           selector:
             matchLabels:
               app: qcloud-exporter-demo
           template:
             metadata:
               labels:
                 app: qcloud-exporter-demo
             spec:
               containers:
                 - args:
                     - '--config.file=/usr/local/etc/qcloud-cvm-product.yml'
                   image: 'registry.cn-hangzhou.aliyuncs.com/fuling/qcloud-exporter:v0.1'
                   imagePullPolicy: Always
                   name: qcloud-exporter
                   ports:
                     - containerPort: 9123
                       name: web-normal
                       protocol: TCP

         The newly created container group is displayed on the Deployments page.

   5. Create a Service.

      1. In the left-side navigation pane, choose Network > Services.

      2. On the Services page, click Create Resources in YAML.

      3. On the Create page, enter the following code in the Template code editor and click Create:

         apiVersion: v1
         kind: Service
         metadata:
           labels:
             app: qcloud-exporter-demo
           name: qcloud-exporter-demo-svc
           namespace: default
         spec:
           ports:
             - name: qcloud-exporter-metrics
               port: 9123
               protocol: TCP
               targetPort: 9123
           selector:
             app: qcloud-exporter-demo

         The newly created Service is displayed on the Services page.

Step 2: Configure service discovery

1. Log on to the ARMS console.

2. In the left-side navigation pane, choose Prometheus Monitoring > Prometheus Instances.

3. In the top navigation bar of the Prometheus Monitoring page, select the region where the monitored ACK cluster resides. Then, click the name of the Prometheus instance that monitors the ACK cluster.

4. In the left-side navigation pane, click Service Discovery. Then, click the Configure tab.

5. On the Configure page, click the ServiceMonitor tab, and then click Add ServiceMonitor. In the Add ServiceMonitor dialog box, enter the following code and click OK.

   apiVersion: monitoring.coreos.com/v1
   kind: ServiceMonitor
   metadata:
     name: qcloud-exporter-sm
     namespace: default
   spec:
     endpoints:
     - interval: 60s
       path: /metrics
       port: qcloud-exporter-metrics
       scrapeTimeout: 60s
     namespaceSelector:
       any: true
     selector:
       matchLabels:
         app: qcloud-exporter-demo

   The newly configured service discovery task is displayed on the ServiceMonitor tab.

Step 3: Create an alert rule

1. Log on to the ARMS console.

2. In the left-side navigation pane, choose Prometheus Monitoring > Prometheus Instances.

3. In the top navigation bar of the Prometheus Monitoring page, select the region where the monitored ACK cluster resides. Then, click the name of the ACK cluster.

4. In the left-side navigation pane, click Alert Configurations.

5. On the Alert Configurations page, click Create Alert in the upper-right corner.

6. In the Create Alert panel, set the following parameters:

   1. Optional:Select a template from the Alert Template drop-down list.

   2. Enter a rule name in the Rule Name field. Example: Alert for inbound traffic.

   3. Enter a PromQL statement as the expression in the Alert Expression field. Example: qce_cvm_cpuusage_avg >= 80.

   4. Enter a number in the Duration field. For example, if you enter 1, an alert notification is sent only when the alert condition persists for one minute.

   5. Enter the notification content in the Alert Message field.

   6. Optional:Click Create Tag next to Tags of the Advanced Settings section to add one or more tags to the alert rule. The specified tags can be used as options for a dispatch rule in a notification policy.

   7. Optional:Click Create Annotation next to Annotations of the Advanced Settings section. Then, enter message in the Key field and {{variable name}} alert message in the Value field. The specified annotation is in the format of message:{{variable name}} alert message. Example: message:{{$labels.pod_name}} restart.

      You can use a custom variable name or select an existing tag as the variable name. Existing tags include:

      • The tags that are included in an alert rule expression.

      • The tags that are created in an alert rule.

   8. Select a notification policy from the Notification Policy drop-down list.

      For more information about how to create a notification policy, see Create and manage a notification policy.

   9. Click OK.

   The newly created alert rule is displayed on the Alert Configurations page.