Application Real-Time Monitoring Service:Why is my ServiceMonitor or PodMonitor configuration not working?

Check whether the target is discovered.

1. Log on to the ARMS console.

2. In the left navigation pane, choose Managed Service for Prometheus > Instances.

3. Click the name of the target Prometheus instance. In the navigation pane on the left, click Service Discovery, and then click the Targets tab to check whether the related target is discovered.

   • If the target is not discovered:

     1. Check whether the ServiceMonitor or PodMonitor is enabled.

        On the Service Discovery page, click the Configurations tab. Then, click the ServiceMonitor or PodMonitor tab and check whether the status is Enabled.

     2. Check whether the defined ServiceMonitor or PodMonitor custom resource exists in the ACK cluster.

        1. Log on to the ACK console.

        2. In the navigation pane on the left, click Cluster. On the Clusters page, click the name of the target cluster.

        3. In the navigation pane on the left, choose Workload > Custom Resources.

        4. On the CRDs tab, click the ServiceMonitor or PodMonitor hyperlink in the Name column. The view switches to the Resource Objects tab. Select the relevant namespace and verify that the ServiceMonitor or PodMonitor you enabled appears in the list.

           • If it is not in the list, go to Service Discovery > Configurations in the Managed Service for Prometheus console, and re-enable the ServiceMonitor or PodMonitor in the Operation column.

           • If it is in the list, proceed to the next step.

     3. Verify that the label selector selects the desired pod or service.

        Run the following command and check the output. If no results are returned, adjust the label selector or namespace selector to select the desired pod or service.

        # If 'any' is set to true in the namespace selector.
        kubectl get pod/service -l label_key=label_value -A
        # If a namespace is specified, use the specified namespace.
        kubectl get pod/service -l label_key=label_value -n <your_namespace>

     4. Check the port settings.

        • If the port is specified by using the port field, set it to the value of spec.containers[i].ports[i].name in the pod definition, or to the value of spec.ports[i].name in the services definition.

        • If the port is specified by using the targetPort field, set it to the value of spec.containers[i].ports[i].containerPort in the pod definition, or to the value of spec.ports[i].targetPort in the services definition.

   • If the target is discovered but displayed in red:

     This indicates a scrape failure, which is typically caused by an issue with the target. Troubleshoot the issue based on the error message in the Error column.

   • If the target is discovered and the State column shows UP, but expected metrics are still missing:

     Click the target's endpoint hyperlink (icon ②) to view its raw data pulled from the target. Check this data for the expected metrics. If the metrics are present, go to Service Discovery > Metric and click Configure Discarded Metrics in the upper-right corner. In the dialog box that appears, check if the metric is on the discarded list. If it is, that metric was manually discarded. Delete it from the list to resume its collection. For more information, see Enable a discarded metric.