All Products
Search
Document Center

Application Real-Time Monitoring Service:Why does the ServiceMonitor or PodMonitor that I configured not take effect?

Last Updated:Feb 20, 2024

Issue

After ServiceMonitor or PodMonitor is added to a Prometheus instance in Managed Service for Prometheus, the Prometheus instance does not collect the monitoring data as expected.

Troubleshooting

Check whether the target is detected.

  1. Log on to the ARMS console.

  2. In the left-side navigation pane, choose Managed Service for Prometheus > Instances.

  3. Click the name of the Prometheus instance you want to manage. In the left-side navigation pane, click Service Discovery. On the Service Discovery page, click the Targets tab and check whether the target is detected.

    • If the target is not detected, perform the following steps:

      1. Check whether ServiceMonitor or PodMonitor is enabled.

        On the Service Discovery page, click the Configurations tab and then click the ServiceMonitor or PodMonitor tab. A status in the following figure (Icon 1) indicates that ServiceMonitor or PodMonitor is enabled. Proceed to the next step.vr

      2. Check whether the ServiceMonitor or PodMonitor resource that you checked in the previous step is delivered to the Container Service for Kubernetes (ACK) cluster.

        1. Log on to the ACK console.

        2. In the left-side navigation pane, click Clusters. On the Clusters page, find the cluster that you want to check and click the name of the cluster.

        3. In the left-side navigation pane, choose Workloads > Custom Resources.

        4. Click the CRDs tab and click ServiceMonitor or PodMonitor in the Name column, as indicated by Icon 1 in the following figure. You are redirected to the Resource Objects tab. Select the namespace to check whether ServiceMonitor or PodMonitor that you enabled is in the list.de

          • If ServiceMonitor or PodMonitor is not in the list, you can log on to the Managed Service for Prometheus console and enable ServiceMonitor or PodMonitor again in the Actions column on the Configurations tab of the Service Discovery page.

          • If ServiceMonitor or PodMonitor is displayed, proceed to the next step.

      3. Check whether you can select the pod or service as expected by using the label selector.

        Run the following command. If no result is returned, you must configure the label selector or namespace selector to select the pod or service as expected.

        # Assume that "any" is set to "true" in the namespace selector. 
        kubectl get pod/service -l label_key=label_value -A
        # If a namespace is specified, the specified namespace is used. 
        kubectl get pod/service -l label_key=label_value -n xx_namespace_xx
      4. Check the port settings.

        Set the value of the port field to the spec.containers[i].ports[i].name value in the pod definition or to the spec.ports[i].name value in the services definition.

    • If the target is detected but is displayed in red, perform the following step:

      Troubleshoot the issue based on the information that is displayed in the Error column. A detected target displayed in red indicates that the Prometheus instance failed to capture monitoring data due to a target error.

    • If the target is detected and its status is UP (Icon 1) but no monitoring data is collected as expected, perform the following step:

      Click the endpoint of the target, as indicated by Icon 2 in the following preceding figure. The system directly pulls raw data from the target. You can check whether the raw data contains the monitoring metrics that you want to monitor. If the raw data contains the monitoring metrics that you want to monitor, you can click Configure Discarded Metrics in the upper-right corner on the Metrics tab of the Service Discovery page. In the dialog box that appears, check whether the discarded metrics contain the metrics that you want to monitor. You can delete the metrics and re-enable them. For more information, see Configure metrics.

References

  • For information about how to manually configure ServiceMonitor or PodMonitor, see Manage service discoveries.

  • After ServiceMonitor or PodMonitor is configured in the cluster, you may need to view the metrics collected by service discovery. For more information, see Configure metrics.

  • After ServiceMonitor or PodMonitor is synchronized in the cluster, you may be charged for the metrics that are collected. For more information, see Billing (new) and Billing (old).