Use Prometheus to monitor disks and create a custom monitoring dashboard for disks in Grafana - Elastic Container Instance

After you create Elastic Container Instance pods in a Kubernetes cluster, the pods run on the virtual node. Since no real nodes exist, disks are associated with pods instead of nodes. You must configure pod-level disk monitoring before you can monitor disk performance metrics. This topic describes how to use Prometheus Service of Application Real-Time Monitoring Service (ARMS) to monitor disks that are attached to Elastic Container Instance pods. This topic also describes how to create a custom monitoring dashboard for disks in Grafana.

Prerequisites

A Container Service for Kubernetes (ACK) Serverless cluster is created, and ARMS Prometheus Service is enabled for the cluster. For more information, see Enable Managed Service for Prometheus for an ACK Serverless cluster.
A Grafana workspace that is not of the Shared Edition is created. Dashboards and data sources are migrated to the workspace. For more information, see Create a Grafana workspace and Integrate cloud services.
Important
By default, a Shared Edition Grafana workspace is used after you enable ARMS Prometheus for an ACK Serverless cluster. You cannot create custom dashboards in Shared Edition Grafana workspaces.

Procedure

Log on to the ARMS console.
Go to the Grafana logon page. In the following example, Grafana 8.2.x is used.
1. In the left-side navigation pane, choose Managed Service for Grafana > Workspace Management.
2. Find the workspace in which you want to create a custom monitoring dashboard for disks and click the URL in the URL column that corresponds to the workspace. The Grafana logon page appears.
Enter the account information to log on to Grafana.
You can log on to Grafana by using the admin account and the password that you set when you created the workspace. You can also use another account.
In the left-side navigation pane, click the icon and select Import.
Click Upload JSON file to upload a JSON file.
The JSON file contains dashboard configurations. For information about an example file, see Dashboard template for monitoring disks for pods.
Set the folder and data source of the dashboard to the cluster that you want to monitor.
Click Import.
After the JSON file is imported, you can view monitoring data about pod disks in the new disk monitoring dashboard.
(Optional) To view Deployment- or StatefulSet-level monitoring data, you can copy the corresponding panel from the current disk monitoring dashboard to the monitoring dashboard of the Deployment or StatefulSet within the same cluster or modify the setting of the disk monitoring dashboard.
When you modify the settings of the disk monitoring dashboard, perform the following steps to filter variables:
1. In the disk monitoring dashboard of the Grafana page, click the icon in the upper-right corner.
2. In the left-side navigation pane of the Settings page, click Variables.
3. Click New in the upper-right corner, configure parameters in the General and Query Options sections, and then click Update.
  When you add variables, set Query based on the variable types. The following table describes example Query settings for different variable types.
  Type
  Query Type
  Label *
  Metric
  deployment
  Label_values
  deployment
  kube_deployment_created{namespace=~"$namespace"}
  statefulset
  Label_values
  statefulset
  kube_statefulset_created{namespace=~"$namespace"}
  The following figure shows an example on how to add a variable named deployment.
4. Click the default variable named pod, modify Query on the Edit page, and then click Update.
  Modify Query based on the type of the variable added in the previous step. The following table describes example Query settings for different variable types.
  Type
  Query Type
  Query
  deployment
  Query_result
  kube_pod_info{namespace=~'$namespace',pod=~"$deployment.*"} >= 1
  statefulset
  Query_result
  kube_pod_info{namespace=~'$namespace',pod=~"$statefulset.*"} >= 1
5. Adjust the order of variables until the deployment variable or a variable named statefulset is in a higher position than the pod variable. The following figure shows that the deployment variable is in a higher position than the pod variable.
6. Click Save dashboard to save the configurations.
7. Go back to the disk monitoring dashboard to check whether the filtering feature has taken effect.
  The following figure shows an example of the disk monitoring dashboard where the Deployment-level filtering feature has taken effect.

Type	Query Type	Label *	Metric
deployment	Label_values	deployment	kube_deployment_created{namespace=~"$namespace"}
statefulset	Label_values	statefulset	kube_statefulset_created{namespace=~"$namespace"}

Type	Query Type	Query
deployment	Query_result	kube_pod_info{namespace=~'$namespace',pod=~"$deployment.*"} >= 1
statefulset	Query_result	kube_pod_info{namespace=~'$namespace',pod=~"$statefulset.*"} >= 1