Obtain container service component metrics through custom collection tasks - Application Real-Time Monitoring Service

In the basic collection jobs provided by default in container cluster monitoring, apart from container cluster basic metrics, some metrics are not collected by default in Managed Service for Prometheus monitoring. This topic describes how to collect the required metrics by adding custom collection configurations.

Prerequisites

You have integrated Container Observability.

Procedure

You can collect container service component metrics through custom collection configurations. The _kube-state-metrics Job is used as an example.

Obtain the reference configuration.
1. Log on to the ARMS console. In the left-side navigation pane, click Integration Management.
2. On the Access Management page, click the target access environment, and then click the Metric Collection tab.
3. In the left-side navigation pane, click Custom Collection, and then find cs-default-monitor > _kube-state-metrics > View Configuration. After clicking View Configuration, copy the collection task configuration of _kube-state-metrics.

Create a custom Job.

Click Custom Collection on the right side of the page, and then click Add.

On the Add Custom Collection Task page, configure the parameters according to the following table.

Parameter

Description

Example

Task Name

The name of the custom task.

Note

Do not use uppercase letters in the custom task name.

ksm_demo

Collection Configuration

Use the content copied in 1. Obtain the reference configuration as the reference configuration for the new task.
Add metric_relabel_configs: at the same level as relabel_configs.

scrape_configs:
- job_name: ksm_job
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 30s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  
  ## Add the following content
  metric_relabel_configs:
  - source_labels:
    - __name__
    regex: (kube_cronjob_next_schedule_time|kube_job_status_start_time|kube_job_spec_completions)
    action: keep
  ## kube_cronjob_next_schedule_time|kube_job_status_start_time|kube_job_spec_completions are used as examples. Replace them with other metrics as needed
  
  relabel_configs:
  - action: labelmap
    regex: .*label_o11y_aliyun_dev_(.*)
    replacement: $1
  - source_labels:
    - __meta_kubernetes_service_label_app
    - __meta_kubernetes_service_label_k8s_app
    separator: ;
    regex: (kube-state-metrics);.*|.*;(kube-state-metrics)
    replacement: $1
    action: keep
  - source_labels:
    - __meta_kubernetes_pod_container_port_number
    separator: ;
    regex: 8080
    replacement: $1
    action: keep
  - source_labels:
    - __address__
    - __meta_kubernetes_pod_container_port_number
    separator: ;
    regex: ([^:]+)(?::\d+)?;(\d+)
    target_label: __address__
    replacement: $1:$2
    action: replace
  kubernetes_sd_configs:
  - api_server: https://kubernetes.default:8443
    role: endpoints
    authorization:
      type: Bearer
      credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    tls_config:
      insecure_skip_verify: true
    follow_redirects: true
    namespaces:
      names:
      - cs-ask-c8034dddd2cf646318b63faf1********

Click Check And Create.

Click the Self-monitoring tab to check the metric status.
If State is UP, the status is normal.
Click the Metric Explorer tab to query the required metrics.
Note
The filter value for the metrics is the newly added Job name.