Use Alibaba Cloud Managed Service for Prometheus to monitor TiDB databases

This topic describes how to use Alibaba Cloud Managed Service for Prometheus to monitor TiDB databases.

Prerequisites

Elastic Compute Service (ECS) instances on which the services are deployed in different virtual private clouds (VPCs) are monitored by Managed Service for Prometheus. For more information, see Create a Prometheus instance to monitor an ECS instance.

Disadvantages of using a self-managed Prometheus service to monitor TiDB databases

When you use a self-managed Prometheus service to monitor TiDB databases that are hosted on ECS instances in different VPCs, the following issues may occur:

To ensure security and facilitate organization management, we recommend that you deploy business in separate VPCs. If you want to use a self-managed Prometheus service to monitor your business, you must deploy the service in each VPC. This increases deployment and O&M costs.
You must deploy and configure Prometheus, Grafana, and Alertmanager services to build a self-managed monitoring system. The process is complex and requires a long time to complete.
The self-managed Prometheus service does not have a service discovery mechanism that can be quickly implemented for Alibaba Cloud ECS. The targets that are deployed on ECS instances cannot be monitored based on the ECS tags. If you want to implement a similar mechanism, you must write code in Golang to call the POP API of Alibaba Cloud ECS and integrate the open source Prometheus service. Then, you must compile and package the code, and deploy the open source Prometheus service. This process is complex and causes difficulties in version upgrades.
You cannot create custom dashboards in open source Grafana based on the working mechanisms of monitoring components and best practices.
Self-managed Prometheus services do not have alerting templates for TiDB. You must configure alert rules. This causes heavy workloads and technological challenges.

Comparison between self-managed Prometheus services and Alibaba Cloud Managed Service for Prometheus

Alibaba Cloud Managed Service for Prometheus that is integrated with Alibaba Cloud ECS provides out-of-the-box TiDB monitoring capabilities. You can monitor key metrics of TiDB components, such as TiDB, TiPD, TiKV, and TiConsole, on dashboards and configure alerting based on your business requirements.

The following table describes the differences between using self-managed Prometheus services and Alibaba Cloud Managed Service for Prometheus in TiDB monitoring.


Item	Self-managed Prometheus service	Alibaba Cloud Managed Service for Prometheus
Deployment costs and O&M costs	You must purchase ECS instances to deploy Prometheus, Grafana, and Alertmanager services in multiple VPCs. This results in high O&M costs.	Alibaba Cloud Managed Service for Prometheus integrates Prometheus, Grafana, and Alertmanager services to provide fully managed, out-of-the-box, and O&M-free capabilities.
Availability, performance, and storage capacity	The overall performance and availability are poor, and the storage capacity is small.	The overall performance and availability are high, and the storage capacity is large.
Service discovery	Service discovery of ECS instances is implemented by using the open source Static_Configs or third-party service registries. The service discovery process is complex and requires high maintenance costs.	Similar to the label selector in Kubernetes, the built-in aliyun_sd_configs allows you to use ECS tags to match ECS instances. This greatly simplifies service configuration and O&M tasks.
Grafana dashboards	The open source Grafana dashboards show only the collected TiDB metrics. You cannot add custom metrics based on the working mechanism of TiDB and best practices for TiDB monitoring.	Alibaba Cloud Managed Service for Prometheus provides professional dashboard templates for monitoring TiDB databases. The dashboards allow you to quickly obtain an accurate overview of the TiDB status and helps you troubleshoot issues.
Alert rules	No alerting template is available for monitoring TiDB databases. You must select metrics and configure the alert rules based on your business requirements.	Alibaba Cloud Managed Service for Prometheus provides professional and flexible alerting templates based on the best practices for monitoring TiDB databases. You can configure alert rules on the GUI.

Based on the preceding table, you can use Alibaba Cloud Managed Service for Prometheus for TiDB monitoring to simplify the O&M process and reduce O&M costs. For information about how to use Alibaba Cloud Managed Service for Prometheus, see Use Alibaba Cloud Managed Service for Prometheus to monitor TiDB databases.

Procedures

Procedure 1: Integration Center of Managed Service for Prometheus

Log on to the ARMS console.
In the left-side navigation pane, choose Prometheus Service > Prometheus Instances.
Click the name of the Prometheus instance instance that you want to manage to go to the Integration Center page.

Procedure 2: Integration Center in the ARMS console

Log on to the ARMS console.
In the left-side navigation pane, click Integration Center. On the Integration Center page, click TiDB in the Application Components section. In the Monitor TiDB panel that appears, follow the on-screen instructions to add the component.

Step 1: Install and configure a TiDB exporter

This section describes how to install and configure a TiDB exporter on the Integration Center page of Managed Service for Prometheus.

Install a TiDB exporter.
- If you install a TiDB exporter for the first time, perform the following operation:
  In the uninstalled section of the Integration Center page, find the TiDB exporter and click Install.
- If you have installed a TiDB exporter, you need to add the exporter again:
  In the Installed section of the Integration Center page, click Add on the TiDB exporter.

On the configuration tab in the STEP2 section, configure the parameters and click OK. The following table describes the parameters.


Parameter	Description
Exporter Name	The name of the TiDB exporter. We recommend that you use the name of the TiDB database.
TiDBServer metrics listening port	The listening ports of TiDB components. Alibaba Cloud Managed Service for Prometheus can obtain metric data over the ports.
TiKVServer metrics listening port
TiPDServer metrics listening port
TiControlServer metrics listening port
TiDBServer Metrics path	The HTTP path from which Alibaba Cloud Managed Service for Prometheus collects metric data of TiDB components. Default value: `/metrics`.
TiKVServer Metrics path
TiPDServer Metrics path
TiControlServer Metrics path
TiDBServer Pod/ECS Label Key(Service Discovery)	The tag keys of the TiDB components. The tags are attached to ECS instances in VPCs. This way, Alibaba Cloud Managed Service for Prometheus can identify the required ECS instances based on tags.
TiKVServer Pod/ECS Label Key(Service Discovery)
TiPDServer Pod/ECS Label Key(Service Discovery)
TiControlServer Pod/ECS Label Key(Service Discovery)
Label value of TiDBServer monitor	The tag values of the TiDB components. The tags are attached to ECS instances in VPCs.
Label value of TiKVServer monitor
Label value of TiPDServer monitor
Label value of TiPDServer monitor
Metrics scrape interval (seconds)	The interval at which Alibaba Cloud Managed Service for Prometheus collects TiDB monitoring data. Default value: 30. Unit: seconds.

Note You can view the monitoring metrics on the Metrics tab in the STEP2 section.

The installed components are displayed in the Installed section of the Integration Center page. Click the component. In the panel that appears, you can view information such as targets, metrics, dashboard, alerts, service discovery configurations, and exporters. For more information, see Integration center.

Step 2: View the TiDB monitoring data on the dashboards

By default, Alibaba Cloud Managed Service for Prometheus provides the 20 common Grafana dashboards for TiDB monitoring. You can view the monitoring data of TiDB components on the dashboards without the need to deploy a Grafana service.

On the Integration Center page, click the TiDB exporter in the Installed section. In the TiDB panel that appears, click the Dashboards tab to view the thumbnails and hyperlinks of TiDB dashboards. Click a hyperlink to go to the Grafana page and view the dashboard.

Step 3: Configure alert rules for monitoring TiDB databases

On the Integration Center page, click the TiDB exporter in the Installed section. In the TiDB panel that appears, click the Alerts tab and select All Alert Contact Group from the drop-down list to view all alert rules that are configured in Managed Service for Prometheus for the TiDB databases.

Alibaba Cloud Managed Service for Prometheus creates more than 30 key alert rules for TiDB components. You can change the thresholds of the alert rules and then enable the alert rules. You can also create alert rules based on your business requirements. For more information, see Create an alert rule for a Prometheus instance.