How to use Prometheus to monitor TiDB - Application Real-Time Monitoring Service

This topic describes how to monitor TiDB using Prometheus.

Prerequisites

You have connected a VPC-connected instance to Managed Service for Prometheus. For more information, see Prometheus for ECS.

Pain points of using a self-managed Prometheus to monitor TiDB

When you monitor a self-managed TiDB cluster on an ECS instance in a VPC using a self-managed Prometheus, you may encounter the following issues:

For security or organizational management reasons, your services are usually deployed in multiple isolated VPCs. This requires you to repeatedly and independently deploy Prometheus in multiple VPCs, which results in high deployment and operations and maintenance (O&M) costs.
Each complete self-managed monitoring system requires you to install and configure components such as Prometheus, Grafana, and AlertManager. This process is complex and time-consuming.
A service discovery mechanism that can be quickly integrated with Alibaba Cloud ECS is not available. As a result, you cannot flexibly define scrape targets based on ECS tags. To implement a similar feature, you must use the Go language to develop code that calls the Alibaba Cloud ECS POP API and integrates it with the open source Prometheus code. Then, you must compile, package, and deploy the code. This method has a high implementation threshold, is complex, and makes version upgrades difficult.
Common open source Grafana dashboards cannot be customized in depth based on the principles and best practices of monitoring components.
TiDB alert templates are not available. You must configure alert rules yourself, which is time-consuming and technically challenging.

Comparison between a self-managed Prometheus and Alibaba Cloud Managed Service for Prometheus

Managed Service for Prometheus is seamlessly integrated with ECS. By default, it provides core metric collection for various TiDB components, such as TiDB, TiPD, TiKV, and TiConsole. It also provides corresponding dashboards and alert rules, which delivers an out-of-the-box TiDB monitoring solution.

The following table compares a self-managed Prometheus with Managed Service for Prometheus in a TiDB monitoring scenario.

Item	Self-managed Prometheus	Managed Service for Prometheus
Deployment and O&M costs	You must purchase ECS instances to deploy Prometheus, Grafana, and AlertManager in multiple VPCs. This results in high O&M costs.	Managed Service for Prometheus provides an integrated, fully managed, and out-of-the-box solution that includes Grafana and alerting capabilities.
Availability, performance, and data capacity	A self-managed setup has limitations in high availability, performance, and data capacity.	Provides high availability, high performance, and large data capacity.
Service discovery	In an ECS environment, service discovery is implemented using methods such as open source static_configs and third-party service registries. This approach is inconvenient and has high maintenance costs.	The built-in aliyun_sd_configs lets you use ECS tags to match target ECS instances. This provides an experience consistent with labelSelector in Kubernetes scenarios and greatly simplifies the configuration and maintenance of service discovery.
Grafana dashboards	Open source Grafana dashboards for TiDB are often basic. Most of them only display the collected metrics and are not optimized in depth based on TiDB principles and best practices.	Specialized TiDB dashboard templates are provided to help you quickly and accurately understand the end-to-end operational status of TiDB and troubleshoot issues.
Alert rules	TiDB alert rule templates are not available. You need to research and configure alert rules yourself.	Based on TiDB monitoring best practices, specialized and flexible alert rule templates are provided. You can configure alert rules using a GUI.

The preceding comparison shows that using Managed Service for Prometheus to monitor TiDB can greatly simplify O&M and reduce costs. For more information about how to use Managed Service for Prometheus to monitor TiDB, see Monitor TiDB using Prometheus.

Use Prometheus to monitor TiDB

Step 1: Integrate TiDB

Log in to the ARMS console.
In the left navigation pane, click Integration Center. Then, in the Database section, click TIDB. On the page that appears, follow the on-screen instructions to integrate the component.
Select the environment type, which can be a Kubernetes Environment or ECS (VPC). Then, Select a Kubernetes cluster or Select VPC.

In the Configuration Information section, configure the parameters and click OK.

Parameter	Description
Access Name	A custom integration name.
TIDB Cluster Name	The name of the TiDB cluster. You must use a different cluster name for each integration to prevent metric collection conflicts that may cause dashboard display errors.
Namespace	A custom namespace.
PD Server Container Name	If there are multiple container names, you can use a vertical bar (\|) to separate them for an "OR" match. Example: "pd\|pd1".
PD Metrics Port	The listener ports for the metrics of the PD, TiDB, and TiKV components, which allow Managed Service for Prometheus to access these ports to obtain monitoring data. It also includes the HTTP scrape path for the metrics, which is usually `/metrics`.
PD Metrics Collection Path
TiDB Server Container Name
TiDB Metrics Port
TiDB Metrics Collection Path
TiKV Server Container Name
TiKV Metrics Port
TiKV Metrics Collection Path
Metric collection Interval (units/second)	Managed Service for Prometheus scrapes TiDB monitoring data. The default value is 30 seconds.

Note

You can view the metrics on the Collect Metrics tab.

The installed component is displayed on the Integration Management page in the ARMS console. The Integration Management page consists of the Integrated Environments, Integrated Addons, and Query Dashboards tabs, where you can view information such as targets, metrics, dashboards, and alerts.

Step 2: View TiDB dashboard data

Managed Service for Prometheus provides more than 20 built-in Grafana dashboards that are commonly used for TiDB. You can view the monitoring dashboards of various TiDB components without having to install Grafana separately.

On the Integration Management page, select a target cluster and click the Dashboards tab to view the component's dashboards. Click a hyperlink to open the corresponding monitoring dashboard in Alibaba Cloud Grafana.

Step 3: Configure TiDB monitoring alerts

On the Integration Management page, select the target cluster and click the Alert Rules tab to view all Prometheus alert rules for the TiDB component.

Managed Service for Prometheus automatically creates more than 30 key alert rules for TiDB components. You only need to modify the thresholds of these alert rules and enable them. You can also add new alert rules as needed. For more information, see Create a Prometheus alert rule.