Documentation
Home Page Application Real-Time Monitoring Service Application Monitoring Use Cases General tutorials Customize an observability dashboard

Customize an observability dashboard

Updated at:
Copy as MD
Product

The ARMS console fully supports open-source solutions. For scenarios that require a high degree of customization, such as creating custom observability dashboards, this topic explains how to use Grafana and Prometheus. It also includes examples of how to write custom query statements to retrieve data.

Background information

Managed Service for Prometheus integrates the Application Monitoring data source by default. You can use Managed Service for Prometheus to obtain application monitoring data, view pre-built dashboards, and customize them as needed.

Prerequisites

Application Monitoring automatically creates a free Alibaba Cloud Prometheus instance in each region. All metrics for Application Monitoring in the China (Hangzhou) region are stored in the arms_metrics_cn-hangzhou_cloud_hangzhou instance of Managed Service for Prometheus in the same region.

image

Scenarios

  • Create dashboards based on metrics from Application Monitoring.

  • For daily O&M, create dedicated dashboards by combining metrics from Application Monitoring with metrics from specific components in other data sources.

  • Before a website traffic peak, create scenario-specific dashboards by combining metrics from Application Monitoring with metrics from upstream and downstream components in other data sources.

  • Display Application Monitoring data, which is obtained via Prometheus monitoring, in your self-managed Grafana.

  • Perform further development on Application Monitoring data obtained from Prometheus monitoring.

Prerequisites

Customize a visualization dashboard

Step 1: Add the Application Monitoring data source

Managed Service for Prometheus integrates the Application Monitoring data source by default, so you only need to add the corresponding Managed Service for Prometheus instance as a data source in Grafana.

Self-managed Grafana

  1. Obtain the Prometheus HTTP API endpoint.

    1. In the ARMS console, navigate to the Managed Service for Prometheus > Instances page. In the top menu bar, select the region where your application is deployed.

      An instance of the Prometheus for Application Monitoring type is the default integrated instance for Application Monitoring.

    2. Click the name of the Application Monitoring instance and click Settings in the navigation pane on the left to view the HTTP API endpoint.

      At the bottom of the Settings page, in the HTTP API Address (Grafana Read Address) section, you can find the public and internal URLs.

    3. (Optional) To secure data reads from Grafana, you can click Generate Token to get an authentication token for the Prometheus instance.

      Important

      After you generate a token, you must configure it when you add the data source in Grafana. Otherwise, Grafana cannot read monitoring data from Prometheus.

  2. Add the Prometheus data source in Grafana.

    This section uses Grafana 10.x as an example. For information about how to add a data source in other Grafana versions, see the official Grafana documentation.

    1. Log on to your self-managed Grafana as an administrator.

    2. Click the image icon in the upper-left corner of the page, and then choose Manage > Data Source.

    3. Click + Add new data source, and then select Prometheus as the data source type.

    4. On the Settings tab, enter a custom name in the Name field, and paste the HTTP API endpoint from the previous step into the Prometheus server URL field.

    5. (Optional) In the Custom HTTP Headers section, click +Add header, set the Header to Authorization, and set the Value to your authentication token.

    6. At the bottom of the tab, click Save & Test.

Managed Service for Grafana

Alibaba Cloud Managed Service for Grafana allows you to integrate Application Monitoring data sources and dashboards with a single click.

  1. In the ARMS console, navigate to Managed Service for Grafana > Workspace Management and click the ID of your workspace.

  2. On the Workspace Information page, in the Cloud Service Integration section, click ARMS Application Monitoring Service, and then click Integrations to the right of the instance in the corresponding region.

  3. In the dialog box, click Confirm.

  4. After synchronization is complete, click the Folder icon on the right.

  5. In Grafana, on the Dashboards tab, click a dashboard name to view the pre-built default dashboards for Application Monitoring.

    The pre-built default dashboards for Application Monitoring include four dashboards: DB, Application, API, and Host.

  6. Click the image icon in the upper-left corner of the page, and then choose Manage > Data Source to view the added Application Monitoring data source.

    After the data source is successfully added, a Prometheus data source appears in the data source list. The name is similar to arms_metrics_cn-hangzhou_cloud_hangzhou_xxx, and the corresponding URL is http://cn-hangzhou.arms.aliyuncs.com:9090/api/v1/prometheus/<InstanceID>.

Step 2: Configure a custom dashboard

  1. In Grafana, click the image icon in the upper-left corner of the page, and then click Dashboards.

  2. On the Dashboards page, choose Create > New dashboard. On the New dashboard page, click Add visualization to create a panel.

  3. In the Select data source dialog box that appears, select the Application Monitoring data source. The data source name is arms_metrics_{regionid}_cloud by default.

    Important

    Make sure that data exists in the current data source.

  4. In the query editor on tab A, set the query metric.

    PromQL query

    In Code mode, enter a PromQL query statement to the right of Metrics browser.

    In the ARMS console, on the monitoring details page of Application Monitoring, you can click the image icon to view the PromQL statement corresponding to a metric. For more information, see Query statements.

    For example, after you expand the Request Count metric panel, you can view the corresponding query statement in the PromQL query editor, such as sum by (callType,)(sum_over_time_lorc(arms_app_requests_count_ign_destid_endpoint_ppid_prpc{pid="...",callKind=~"http|rpc|custom_entry|server|consumer|schedule",source="apm"}[1m])).

    For more information about how to use PromQL query statements, see the official Prometheus documentation.

    Application Monitoring metrics

    In Builder mode, select a specific metric from the Metric drop-down list, such as arms_jvm_threads_count, to query the number of JVM threads. For details about metrics supported by Application Monitoring, see Application Monitoring metrics.

    In the Label filters section, set the label filtering conditions, for example, host = target IP. The corresponding PromQL query statement is automatically generated, such as arms_jvm_threads_count{host="xxx.xxx.xx.67"}.

  5. On the right, set the panel title and select a chart type.

    Note

    For the same metric, you can select different chart types to visualize data at different granularities.

    In the panel editor on the right, set the visualization type to Time series. In the Panel options section, enter JVM Threads Count in the Title field.

  6. After you complete the settings, click Apply in the upper-right corner.

Customize a query statement

You can use the Prometheus HTTP API and PromQL statements from the ARMS console to make HTTP requests by using Shell scripts or Java code to customize the retrieved data.

For example, if the Prometheus HTTP API endpoint is https://cn-hangzhou.arms.aliyuncs.com:9443/api/v1/prometheus/7a8f8e783c66a44a0f6a1dc9a0b8f1/1277589232893727/arms-metrics-99214ff7de7d0b2d/cn-hangzhou, combine it with a PromQL statement in the following format:

{{Prometheus HTTP API}}/api/v1/query_range?query={{PromQL}}&start={{startTime}}&end={{endTime}}&step={{step}}.

The resulting query statement is as follows:

https://cn-hangzhou.arms.aliyuncs.com:9443/api/v1/prometheus/7a8f8e783c66a44a0f6a1dc9a0b8f1/1277589232893727/arms-metrics-99214ff7de7d0b2d/cn-hangzhou/api/v1/query_range?query=sum by (callType,)(sum_over_time_lorc(arms_app_requests_count_ign_destid_endpoint_ppid_prpc{pid="ckv8e2vzfj@7e393063f3fd6ad",serverIp=~".*",callKind=~"http|rpc|custom_entry|server|consumer|schedule",source="apm", }[1m]))&start=1728719399&end=1728722999&step=60s

Sample response data:

{"status":"success","data":{"resultType":"matrix","result":[{"metric":{"callType":"http"},"values":[[1728719399,"1575"],[1728719459,"1599"],[1728719519,"1593"],[1728719579,"1584"],[1728719639,"1578"],[1728719699,"1568"],[1728719759,"1561"],[1728719819,"1555"],[1728719879,"1547"],[1728719939,"1540"],[1728719999,"1533"],[1728720059,"1524"],[1728720119,"1518"],[1728720179,"1512"],[1728720239,"1503"],[1728720299,"1497"],[1728720359,"1491"],[1728720419,"1482"],[1728720479,"1476"],[1728720539,"1467"],[1728720599,"1464"],[1728720659,"1453"],[1728720719,"1448"],[1728720779,"1443"],[1728720839,"1434"],[1728720899,"1428"],[1728720959,"1419"],[1728721019,"1416"],[1728721079,"1407"],[1728721139,"1401"],[1728721199,"1395"],[1728721259,"1389"],[1728721319,"1381"],[1728721379,"1374"],[1728721439,"1366"],[1728721499,"1367"],[1728721559,"1350"],[1728721619,"1350"],[1728721679,"1344"],[1728721739,"1335"],[1728721799,"1331"],[1728721859,"1324"],[1728721919,"1320"],[1728721979,"1314"],[1728722039,"1305"],[1728722099,"1302"],[1728722159,"1296"],[1728722219,"1287"],[1728722279,"1284"],[1728722339,"1233"],[1728722399,"1272"],[1728722459,"1155"],[1728722519,"1116"],[1728722579,"1110"],[1728722639,"1107"],[1728722699,"1098"],[1728722759,"1095"],[1728722819,"1086"],[1728722879,"1083"],[1728722939,"1077"],[1728722999,"1184"]]}]}}

FAQ

Query charges

Yes.

PromQL query throttling

Throttling is active, but the limits are not based on a simple queries-per-second (QPS) count at the entry point, as different queries consume different amounts of resources. Avoid high-consumption queries, such as querying all APIs or querying data over a time range longer than 30 days, and keep the query rate below 50 QPS. If you need to run sustained, high-consumption queries, submit a ticket in advance to request a quota increase.

Previous：Real-time trace analysisNext：Add and use tags