Cloud Monitor:Modeling MetricSets for Prometheus

Definition and purpose of MetricSet

As a core component of metric modeling, a MetricSet serves several key functions:

• Metric organization: Groups related metrics into logical collections for simplified management and querying.

• Label management: Defines common labels and filter rules to enable consistent metric dimensions.

• Query optimization: Offers a unified interface for efficient and optimized querying.

• Semantic expression: Includes rich metadata and supports multiple languages for enhanced semantic clarity.

MetricSet structure overview

Observable Data System
├── EntitySet                     # Entity set
├── TelemetryDataSet              # Telemetry dataset (metrics, logs, traces)
    └── MetricSet                 # Metric set
        ├── labels                # Label definitions
        │   ├── keys: Field[]     # Label field list
        │   ├── dynamic: boolean  # Dynamic label generation
        │   └── filter: string    # Label filter
        └── metrics: Metric[]     # Metric list
            ├── name              # Metric name
            ├── generator         # Query generator
            ├── aggregator        # Aggregation method
            └── data_format       # Format method

Label design principles

Labels define the dimensional properties of metrics. To ensure effective metric modeling, follow these principles:

• Generality: MetricSet-level labels should represent dimensions shared across all metrics in the set.

• Dynamic generation: Use dynamic methods to automatically generate labels and avoid hard coding.

• Efficient filtering: Design labels to support efficient indexing and filtering for fast queries.

• Cardinality control: Avoid using high-cardinality labels because they can introduce performance issues.

Label property configuration

Label field definition

Each label field inherits all properties of a general Field. Pay special attention to the following properties:

Label configuration examples

Kubernetes scenario

labels:
  dynamic: true
  filter: 'kube_deployment_spec_replicas'
  keys:
    - name: namespace
      display_name:
        zh_cn: 命名空间
        en_us: Namespace
      type: string
      filterable: true
      analysable: true
      pattern: ".*"
    - name: deployment  
      display_name:
        zh_cn: 部署名称
        en_us: Deployment
      type: string
      filterable: true
      analysable: true
      pattern: ".*"

APM scenario

labels:
  dynamic: true
  filter: 'arms_app_requests_count_ign_destid_endpoint_parent_ppid_prpc{callKind=~"http|rpc|custom_entry|server|consumer|schedule"}'
  keys:
    - name: service
      display_name:
        zh_cn: 服务名称
        en_us: Service
      type: string
      filterable: true
      analysable: true
    - name: rpc
      display_name:
        zh_cn: 接口名称  
        en_us: Operation
      type: string
      filterable: true
      analysable: true

Metric design principles

As core components of a MetricSet, each metric represents a distinct, queryable monitoring dimension. When designing metrics, consider the following principles:

• Business semantics: Metric names should clearly reflect their business meaning.

• Complete calculation: The generator must contain the complete calculation logic, not just reference a raw metric.

• Clear units: Configure the data_format and unit properties correctly.

• Appropriate aggregation: Select the correct aggregator based on the metric's properties.

Core metric properties

A Metric extends the basic field system with monitoring-specific properties:

Data format specification

Metrics are often visualized in charts, making the data_format field important. Whenever possible, use the enumeration values defined in the schema and avoid using custom formats.

For a list of available format options, see Field system.

Aggregator selection guide

Choosing the correct aggregator ensures query accuracy:

Note: For ratio metrics (for example, usage/limit × 100) where the generator already calculates the ratio, do not set the aggregator. UModel automatically selects the aggregation policy when needed.

The following are common Generator and Aggregator combinations:

- name: request_count
  generator: 'sum_over_time_lcro(arms_app_requests_count_ign_destid_endpoint_parent_ppid_prpc{callKind=~"http|rpc|custom_entry|server|consumer|schedule"}[60s])'
  aggregator: sum
  data_format: KMB

2. Replica count metric: Desired number of deployment replicas.

- name: deployment_desired_replicas
  generator: kube_deployment_spec_replicas{}
  aggregator: sum
  data_format: KMB

3. Ratio metric (no aggregator): Deployment availability.

- name: deployment_availability_rate
  generator: (sum by (namespace, deployment) (kube_deployment_status_replicas_ready{}) / (sum by (namespace, deployment) (kube_deployment_spec_replicas{}))!=0) * 100
  # Do not set an aggregator
  data_format: percent

4. Average latency metric: APM average response time.

- name: avg_request_latency_seconds
  generator: 'sum(sum_over_time_lcro(arms_app_requests_seconds_ign_destid_endpoint_parent_ppid_prpc{callKind=~"http|rpc|custom_entry|server|consumer|schedule"}[60s])) / sum(sum_over_time_lcro(arms_app_requests_count_ign_destid_endpoint_parent_ppid_prpc{callKind=~"http|rpc|custom_entry|server|consumer|schedule"}[60s]))'
  # Do not set an aggregator
  data_format: s

Golden metric identification

Golden Metrics are the most critical metrics in a MetricSet:

• Quantity limit: 3–5 per MetricSet; do not exceed 8.

• Selection criteria: Core metrics that directly indicate system health.

• Scenarios: Alert rules, dashboards, and automated O&M tasks.

Metrics configuration examples

• CPU usage metric.

  - name: deployment_cpu_usage_vs_requests
    display_name:
      zh_cn: CPU使用率相对于请求值
      en_us: CPU Usage vs. Requests
    description:
      zh_cn: Deployment 的 CPU 使用率相对于请求值的百分比
      en_us: The CPU usage of the deployment as a percentage of its requested CPU.
    generator: |
      sum by (namespace, deployment) (
        rate(container_cpu_usage_seconds_total{pod!~"POD"}[5m])
      ) / sum by (namespace, deployment) (
        kube_pod_container_resource_requests_cpu_cores{pod!~"POD"}
      ) * 100
    data_format: percent
    unit: ''
    golden_metric: true
    interval_us: [15000000]
    type: gauge

• Request count metric.

  - name: request_count
    display_name:
      zh_cn: 请求次数
      en_us: Request Count
    description:
      zh_cn: 请求次数指的是对一个特定应用或接口发起调用的总次数
      en_us: The total number of calls to a specific application or API.
    generator: 'sum_over_time_lcro(arms_app_requests_count_ign_destid_endpoint_parent_ppid_prpc{callKind=~"http|rpc|custom_entry|server|consumer|schedule"}[60s])'
    aggregator: sum
    data_format: KMB
    golden_metric: true
    interval_us: 15000000

Variable substitution mechanism

• In the UModel system, MetricSets that use Prometheus syntax support built-in label filtering and injection. When you filter label values in Labels, the system automatically injects the filter's conditional expression into the generator.

• The generator in a Metric supports join operations on multiple metrics. However, not all metrics support all label injections. For example, some advanced Kubernetes metrics require multi-metric joins for data enrichment. To address these differences in label fields, you can use the variable substitution mechanism.

# Example: The node label exists only in kube_pod_info
kube_pod_info{node=~"${{node|.*}}"}

Label association example

# Memory resource-related metrics that associate pods and deployments through a ReplicaSet
- aggregator: sum
  data_format: byte
  description:
      en_us: Total memory requested by all pods in the deployment.
      zh_cn: Deployment 所有 Pod 的内存请求总量
  display_name:
      en_us: Deployment Memory Requests Total
      zh_cn: Deployment 内存请求总量
  generator: sum by (namespace, deployment) (kube_pod_container_resource_requests_memory_bytes{pod!~"POD"} * on (pod, namespace) group_left (deployment) (max by (pod, namespace, deployment) (label_replace(kube_pod_info{created_by_kind="ReplicaSet"}, "replicaset", "$1", "created_by_name", "(.*)") * on (namespace, replicaset) group_left (deployment) label_replace(kube_replicaset_owner{owner_kind="Deployment", owner_name=~"${{deployment|.*}}"}, "deployment", "$1", "owner_name", "(.*)"))))
  golden_metric: false
  interval_us:
  - 15000000
  launch_stage: ga
  name: deployment_memory_requests_total
  type: gauge
  unit: ''

Design principles

1. Clear semantics: The metric name should clearly express its business meaning.

2. Reasonable labels: Avoid high-cardinality labels to maintain query performance.

3. Complete calculation: The generator should contain the complete business logic.

4. Standard format: Use only the data_format enumeration values defined in the schema and avoid using custom formats.

5. Correct aggregation: Select the appropriate aggregator based on the metric's properties.

Performance optimization

1. Label filtering: Apply label filters early in the query process.

2. Aggregation optimization: Use functions such as sum by to reduce the number of time series.

3. Dynamic labels: Use dynamic: true to improve label management efficiency.