Compute Nest:Configure business monitoring and alerting for a fully managed service deployed in an ACK cluster

How it works

Compute Nest uses the multi-tenant monitoring feature of Managed Service for Prometheus, which is part of Application Real-Time Monitoring Service (ARMS).

• The Prometheus agent in the cluster tags metrics at the pod or namespace level using tenant tags. This routes each tenant's metrics to the corresponding service instance.

• The storage backend provides native multi-tenant isolation. As a service provider, you only need to add tenant tags to each tenant's workloads.

• Multi-tenant fully managed services use Kubernetes namespaces to isolate tenant resources. After you add tenant tags to a namespace, the backend automatically forwards that tenant's metrics to the correct service instance.

• The remote write feature lets you receive all tenant monitoring data in your service provider account, while each tenant can view only their own data.

Data visibility summary:

Prerequisites

Before you begin, ensure that you have:

• An ACK cluster where the fully managed service is deployed

• An AccessKey ID and AccessKey secret for your service provider account, with at least the arms:GetPrometheusApiToken permission (see the least-privilege policy below)

• A product identifier for your service — use cn-mariadb for testing, or contact technical support to get a unique identifier for production (DingTalk group ID: 31045016300)

• (Step 3) A Grafana dashboard created in Alibaba Cloud Managed Grafana

Least-privilege AccessKey policy:

Grant only the arms:GetPrometheusApiToken permission to the AccessKey used for metric forwarding. This follows the principle of least privilege and limits exposure if the key is compromised.

{
  "Version": "1",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "arms:GetPrometheusApiToken"
      ],
      "Resource": "*"
    }
  ]
}

Step 1: Configure Managed Service for Prometheus in the ACK cluster

The configuration method depends on whether your service is single-tenant or multi-tenant.

Single-tenant fully managed service

Compute Nest encapsulates the Managed Service for Prometheus configuration into a Resource Orchestration Service (ROS) module. Add the following block to your ROS template when you create the service.

Sample ROS template snippet:

ClusterArmsConfig:
  Type: 'MODULE::ACS::ComputeNest::AckArmsConfig'
  Version: v1
  Properties:
    ClusterId:
      Fn::If:
        - Condition: CreateACKCondition
        - Ref: ManagedKubernetesCluster
        - Ref: ClusterId
    WhetherSupplierNeedMetric: true
    AccessKeyID: LTAI****************
    AccessKeySecret: yourAccessKeySecret
    SupplierAliuid: 15634578xxxxxx

Parameters:

Parameter	Required	Description
ClusterId	Yes	ID of the ACK cluster. References ManagedKubernetesCluster if CreateACKCondition is true, otherwise references ClusterId.
WhetherSupplierNeedMetric	Yes	Set to true to receive tenant metrics in the service provider account. When true, also specify AccessKeyID, AccessKeySecret, and SupplierAliuid.
AccessKeyID	Conditional	AccessKey ID of the service provider account. Required when WhetherSupplierNeedMetric is true.
AccessKeySecret	Conditional	AccessKey secret of the service provider account. Required when WhetherSupplierNeedMetric is true.
SupplierAliuid	Conditional	Alibaba Cloud account UID of the service provider. Required when WhetherSupplierNeedMetric is true.
SyncServiceMonitor	No	Set to true to automatically sync ServiceMonitor resources in the cluster. Use this to select which Services to monitor. For details, see Use ServiceMonitors to discover and monitor Services.
SyncPodMonitor	No	Set to true to automatically sync PodMonitor resources in the cluster.

If the ACK cluster is created by the ROS template (rather than an existing cluster), add the Addons parameter to the ACK resource block in the template.

Multi-tenant fully managed service

Managed Service for Prometheus does not support metric forwarding by default. You can configure it to forward metrics related to user applications to specific users. For multi-tenant services, create a Managed Service for Prometheus instance in the ACK cluster using the container infrastructure service on the Service Catalog page in the Compute Nest console.

Step 2: Configure the ROS template

ClusterNameSpaceApplication:
    Type: ALIYUN::CS::ClusterApplication
    Properties:
      YamlContent:
        Fn::Sub:
          - |
            apiVersion: v1
            kind: Namespace
            metadata:
              name: '${Name}'
              labels:
                tenant_userid: '{{ aliUid }}'
                tenant_clusterid: '{{ tenantClusterId }}'
                tenant_token: '{{ tenantToken }}'
                tenant_cloudproductcode: '{{ tenantCloudProductCode }}'
          - Name: '{{ serviceInstanceId }}'
      ClusterId:
        Fn::If:
          - Condition: CreateACKCondition
          - Ref: ManagedKubernetesCluster
          - Ref: ClusterId

apiVersion: v1
kind: Service
metadata:
  name: {{ serviceInstanceId }}
  labels:
    io.mysql.service: {{ serviceInstanceId }}
spec:
  selector:
    app: mysql
  ports:
    - protocol: TCP
      port: 3306
      targetPort: 3306
      name: mysql
    - protocol: TCP
      port: 9104
      targetPort: 9104
      name: mysql-exporter
  type: LoadBalancer

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: prometheus-service-monitor
  annotations:
    arms.prometheus.io/discovery: 'true'
  labels:
    prometheus-service-monitor: prometheus-service-monitor
spec:
  selector:
    matchLabels:
      io.mysql.service: {{ serviceInstanceId }}
  namespaceSelector:
    matchNames:
      - {{ serviceInstanceId }}
  endpoints:
    - port: mysql-exporter
      scheme: http
      path: /metrics
      interval: 10s
      scrapeTimeout: 10s

Step 3: Create the Grafana dashboard

Each service instance supports exactly one dashboard. If you have multiple dashboards, combine them into a single dashboard before proceeding.

The dashboard must use namespace as a fixed global variable to filter metrics per application. Create the dashboard in Alibaba Cloud Managed Grafana and copy the resulting dashboard URL — you will need it in the next step.

Step 4: Configure the product identifier and dashboard link

Before a multi-tenant fully managed service can use Compute Nest's Managed Service for Prometheus capabilities, specify the product identifier and dashboard settings.

1. Set the product identifier. The product identifier uniquely identifies a service in the monitoring system. For testing, use the shared identifier cn-mariadb, which applies to all services in the China (Hangzhou) and China (Hong Kong) regions.

   To get a unique product identifier before publishing a service, contact technical support via the DingTalk group (ID: 31045016300).

2. Configure dashboard settings. Provide the title and URL of the Grafana dashboard you created in Step 3.

   Dashboard settings must be configured manually with support from the DingTalk group.

The following screenshot shows the configuration for a sample Prometheus service using the cn-mariadb product identifier.

Step 5: Create and apply a Prometheus alert rule template

1. Log on to the ARMS console. On the Prometheus Alert Rule Templates page, create an alert rule template. For details, see Create and manage an alert rule template.

Step 6: View monitoring data

After the Prometheus service instance is deployed, both service providers and tenants can view the dashboard on the service instance details page.

• Tenant view: Tenants can see metrics for their own service instances only.

  

• Service provider view: Service providers can see metrics across all tenant service instances.