This topic describes how to configure service discovery for Prometheus instance for ECS. The service discovery feature helps you collect the metrics exposed on an Elastic Compute Service (ECS) instance and monitor the instance based on the metrics.

Prerequisites

A Prometheus instance for ECS is created. For more information, see Create a Prometheus instance to monitor an ECS instance.

Step 1: Enable ECS service discovery

  1. Log on to the ARMS console.
  2. In the left-side navigation pane, choose Prometheus Service > Prometheus Instances.
  3. Click the name of the Prometheus instance to which you want to connect.
  4. In the left-side navigation pane, click Service Discovery. Then, click the Configurations tab.
  5. Find vpc-ecs-service-discovery in the Name column and click er in the Actions column. In the message that appears, click Enable to enable ECS service discovery.
    After ECS service discovery is enabled, Managed Service for Prometheus can collect data from the monitored ECS instance.

Step 2: Modify the configurations of ECS service discovery

  1. Find vpc-ecs-service-discovery in the Name column and click Details in the Actions column.
  2. In the YAML Configurations dialog box, change the value of the metrics_path or port parameter. If your service port is 9090/metrics, you only need to change the value of the port parameter to 9090, as shown in the following figure.
    r

(Optional) Step 3: Check the security group

If a Deny rule is configured in the security group of an ECS instance, you are unable to collect the monitoring data of the ECS instance. You can perform the following steps to check the security group rules:

  1. Log on to the ARMS console.
  2. In the left-side navigation pane, choose Prometheus Service > Prometheus Instances.
  3. Click Create Prometheus Instance. On the page that appears, click Prometheus Instance for ECS.
  4. In the panel that appears, view the ID of the security group in the Security Group column and view the ID of the vSwitch in the vSwitch column. The security group and vSwitch are the same as the security group and vSwitch that you selected when you created the Prometheus instance.
  5. Log on to the ECS console and add an Allow rule to the security group that is obtained in the previous step.
    Note When you add an Allow rule, set the Authorization Object parameter to the CIDR block of the vSwitch that you selected when you installed the Prometheus agent. The ID of the vSwitch is obtained in the previous step.

(Optional) Step 4: Filter ECS instances

If only some ECS instances in your VPC provide the metric query service and you only need to collect data from some ECS instances, you can filter ECS instances.

  1. Log on to the ARMS console.
  2. In the left-side navigation pane, choose Prometheus Service > Prometheus Instances.
  3. Click the name of the Prometheus instance to which you want to connect.
  4. In the left-side navigation pane, click Service Discovery. Then, click the Configurations tab.
  5. Find vpc-ecs-service-discovery in the Name column and click Details in the Actions column.
  6. In the YAML Configurations dialog box, edit the Prometheus.yaml file and click Confirm. The following code segment provides an example to show how to edit the Prometheus.yaml file for ECS service discovery. For more information, see Sample Prometheus.yaml. 
    global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 30s
scrape_configs:
- job_name: _aliyun-prom/ecs-sd
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: https
  aliyun_sd_configs:
    - port:  8888                                 # The port number that is used by Managed Service for Prometheus for service discovery.
      user_id: <aliyun userId>                    # The ID of the Alibaba Cloud account. If you specify this parameter, the __meta_ecs_user_id tag is included in the discovery target.
      refresh_interval: 30s
      region_id: cn-hangzhou                      # The ID of the region where the ECS instances reside.
      access_key: <aliyun ak>                     # The AccessKey ID of the Alibaba Cloud account.
      access_key_secret: <aliyun sk>              # The AccessKey secret of the Alibaba Cloud account.
      tag_filters:                                # The tags that are used to filter ECS instances. Specify tags in key-value pairs.
       - key: 'testK'
         values: ['*', 'test1*']
       - key: 'testM'
         values: ['test2*']
#      limit: 40                                  # The maximum number of instances that can be obtained. If you do not specify this parameter, all ECS instances are obtained.

  relabel_configs:

#   1. Specify which type of IP address is used to filter ECS instances.
#   By default, ECS searches and assigns the collection IP address in the following order: public IP address for the classic network > private IP address for the classic network > public IP address for a VPC > private IP address for a VPC. The port number used for data collection is specified by the aliyun_sd_configs.port parameter.
#   Set the IP addresses used to filter ECS instances.
    - source_labels: [__meta_ecs_public_ip]       # Set __meta_ecs_public_ip to the public IP address for the classic network.
#    - source_labels: [__meta_ecs_inner_ip]       # Set __meta_ecs_inner_ip to the private IP address for the classic network.
#    - source_labels: [__meta_ecs_eip]            # Set __meta_ecs_eip to the public IP address for a VPC.
#    - source_labels: [__meta_ecs_private_ip]     # Set __meta_ecs_private_ip to the private IP address for a VPC.
      regex: (.*)
      target_label: __address__
      replacement: $1:<port>                      # Set the port number used to filter ECS instances.

#   2. Filter ECS instances based on attributes. To include the monitored objects filtered by a condition, set the action parameter to keep. To exclude the monitored objects filtered by a condition, set the action parameter to drop.
#   __meta_ecs_instance_id          The ID of the ECS instance.
#   __meta_ecs_region_id            The region ID of the ECS instance. Note that the region ID of the ECS instance is specified by the aliyun_sd_configs.region_id parameter.
#   __meta_ecs_status               The status of the ECS instance. Valid values: Running, Starting, Stopping, and Stopped.
#   __meta_ecs_zone_id              The zone ID of the ECS instance.
#   __meta_ecs_network_type         The network type of the ECS instance. Valid values: classic and vpc.
#   __meta_ecs_tag_<TagKey>         The tag of the ECS instance. Set TagKey to the tag key.
    - source_labels:  ["__meta_ecs_instance_id"]
      regex: ".+"       # or other value regex
      action: keep      # keep / drop

    For example, if you only need to collect ECS instances whose tag key is app and tag value is myNginx, you can configure the tag_filters parameter as shown in the following script. 

    tag_filters:
   - key: 'app'
     values: ['myNginx']

(Optional) Step 5: Configure the relabel parameter

To obtain the hostname and tag information from the data collected by Managed Service for Prometheus, perform the following operations:

  1. Log on to the ECS console and view the tags of the ECS instance.
  2. Log on to the ARMS console.
  3. Click the name of the Prometheus instance to which you want to connect.
  4. In the left-side navigation pane, click Service Discovery. Then, click the Configurations tab.
  5. Find vpc-ecs-service-discovery in the Name column and click Details in the Actions column.
  6. In the YAML Configurations dialog box, add the following configurations to the Prometheus.yaml file, specify the tag and hostname to be collected, and then click Confirm. 
      relabel_configs:
  - regex: (.*)
    action: replace
    source_labels:
    - __meta_ecs_private_ip
    replacement: $1:88889
    separator: ;
    target_label: __address__
  - regex: (.*)
    action: replace
    source_labels:
    - __meta_ecs_tag_app
    replacement: $1
    separator: ;
    target_label: tag_app
  - regex: (.*)
    action: replace
    source_labels:
    - __meta_ecs_instance_name
    replacement: $1
    separator: ;
    target_label: instance_name
    vrp
  7. View the collected monitoring data.
    1. Go to the homepage of Grafana dashboards.
    2. In the left-side navigation pane, click the Explore icon.
    3. In the upper part of the Explore page, select the monitored cluster from the drop-down list. Then, enter a PromQL statement in the Metrics field. The monitoring data collected by Managed Service for Prometheus is displayed. vri