This topic describes the current landscape of container cluster observability, the challenges of monitoring multi-cloud container clusters, and potential solutions. It also provides a case study to show you how to use Alibaba Cloud Managed Service for Prometheus and ACK One registered clusters for unified management and monitoring of your multi-cloud container clusters.
Container cluster observability
As Kubernetes (K8s) has become the de facto industry standard for container orchestration, Prometheus has emerged as the standard for container cluster monitoring.
Prometheus can effectively monitor metrics at the system, application, and business layers. It collects and stores these metrics, and when combined with Grafana, provides visualization and alerting capabilities. The combination of Prometheus and Grafana offers a complete solution for metric collection, storage, visualization, and alerting. It helps businesses identify and resolve issues, ensures the reliability of cloud-native applications, and has become the standard monitoring solution for container clusters.
You have the following two options for managing and operating container clusters:
Option 1: Build a self-managed monitoring system
This approach requires you to build and maintain a production-ready monitoring system using Prometheus and Grafana. It involves a significant upfront investment in human resources to configure metric collection, storage, and visualization, create effective dashboards, and set up alerting rules, including false-positive alert filtering. This approach also leads to high ongoing operational costs.
Option 2: Use a cloud monitoring product
Alibaba Cloud offers Managed Service for Prometheus, which is available in both subscription and pay-as-you-go billing models. This service reduces the upfront costs of building your own alerting system and provides ongoing technical support, significantly lowering operational costs.
Challenges of multi-cloud container cluster observability
As businesses expand and diversify their cloud services, they often use container clusters across multiple regions and cloud providers. This shift from single-cluster to multi-cloud operations introduces new monitoring challenges.
You can choose one of the following two options to monitor your multi-cloud and hybrid container clusters:
Option 1: Use a self-managed Prometheus and Grafana system
If you choose this option, you face the following challenges:
-
Building a complete monitoring system requires integrating collection, storage, visualization, and alerting components. This integration demands continuous maintenance from dedicated personnel and increases operational costs.
-
The Time Series Database (TSDB) in open source Prometheus stores data on a single node using local SSDs. This creates a single point of failure and risks data loss.
-
Open source Prometheus has collection limits and runs as a single process that lacks auto scaling capabilities. This can lead to performance bottlenecks in metric collection during peak traffic.
Option 2: Use Prometheus products from cloud providers
If you choose this option, you face the following challenges:
-
Prometheus services from different cloud providers vary in features and usage, requiring a significant learning investment.
-
Using multiple monitoring products leads to fragmented management, duplicated operational tasks, and delays in identifying business-critical issues.
Regardless of the option you choose, you face a common problem: scattered metrics. This makes it difficult to perform unified queries, consolidated analysis and visualization, and centralized alerting.
Benefits of Managed Service for Prometheus
To address these challenges, ACK One registered clusters enable central management of non-Alibaba Cloud container clusters. Alibaba Cloud Managed Service for Prometheus provides a complete container monitoring system that includes metric collection, Grafana visualization, and alerting. With both pay-as-you-go and subscription billing options, it improves monitoring efficiency and dramatically reduces the operational costs associated with self-managed systems.
Combining ACK One registered clusters with Managed Service for Prometheus makes multi-cloud container monitoring simple and efficient, and offers the following key benefits:
-
Powerful capabilities: This solution solves common multi-cloud monitoring challenges, such as fragmented management, complex system setup, low operational efficiency, and the inability to perform unified queries and alerting. By centralizing management, configuration, queries, and alerting for scattered container clusters, it significantly improves efficiency, reduces operational costs and repetitive tasks, and allows your team to focus on core business objectives.
-
Cost-effective: Alibaba Cloud Managed Service for Prometheus provides free collection of basic metrics, which meets the basic monitoring requirements for container clusters. For small-scale container clusters, you can use the pay-as-you-go billing method. This ensures that your services are effectively monitored while minimizing your monitoring costs. For detailed billing information about the pay-as-you-go billing method for Alibaba Cloud Managed Service for Prometheus, see Pay-as-you-go. For large-scale clusters, you can use the subscription billing method. Compared with the pay-as-you-go method, the subscription billing method can reduce costs by about 67% and significantly lowers monitoring costs for large-scale clusters.
-
Low resource footprint: Managed Service for Prometheus only requires you to deploy a lightweight agent in your cluster. The agent supports auto scaling, and a single instance with 2 vCPUs and 4 GB of memory can collect 6 million metric series. The service discovery module in open source Prometheus can place a heavy load on the API server of a cluster. Managed Service for Prometheus is optimized to reduce the load on the API server. This ensures minimal resource consumption while maximizing metric collection to safeguard your business.
Benefit 1: Improved performance
|
Item |
Managed Service for Prometheus |
Self-managed Prometheus |
|
High availability |
Supports multiple replicas for collection and storage components with horizontal scaling for high availability. |
Runs as a single process, cannot be horizontally scaled, and provides low availability. |
|
Data storage |
Unlimited capacity with cloud-based storage. |
Limited by local storage capacity. |
|
Data visualization |
Includes a built-in Grafana instance with out-of-the-box templates for common monitoring scenarios. |
Requires you to deploy Grafana and configure dashboards separately. |
|
Alert management |
Integrates with the Application Real-Time Monitoring Service (ARMS) alerting center for improved efficiency and accuracy. |
Requires manual integration of the AlertManager plug-in. |
|
Single-replica collection performance (2 vCPUs, 4 GB) |
6 million metric series |
1 million metric series |
|
Data query performance (600 million samples) |
8 to 10 seconds |
180 seconds |
|
Other capabilities |
Supports features such as pre-aggregation, downsampling, and Global View. |
Not supported |
Benefit 2: Multi-cluster aggregated queries
The service provides a virtual aggregated instance that consolidates data from multiple Alibaba Cloud Prometheus instances or self-managed Prometheus clusters. This instance lets you run unified queries on Prometheus metrics, use a single Grafana data source, and configure unified alerting.
-
This solves the problem of scattered data in open source Prometheus. You no longer need to configure multiple data source addresses in Grafana, and you can efficiently consolidate data from different sources to obtain a holistic view of your application's health across different regions.
-
You do not need to deploy a Prometheus server in each region. You can simply use the Remote Write feature to send data to Alibaba Cloud Managed Service for Prometheus. You can also use the Global View feature without deploying the numerous components of Thanos.
-
Global query is based on a distributed and performance-optimized query engine. It supports horizontal and vertical scaling for large query volumes and delivers excellent query performance and stability.
-
This feature is available out of the box. With Alibaba Cloud Managed Service for Prometheus, you do not need to deploy any additional components, which significantly reduces operational costs.
Benefit 3: Lightweight installation
Unlike open source Prometheus, Managed Service for Prometheus only requires you to install a lightweight agent in your container cluster. The backend storage is fully managed, which saves resources in your business container clusters.
Benefit 4: Integrated Grafana service
Alibaba Cloud Grafana service is a cloud-native platform for visualizing operational data. It provides a maintenance-free, quick-start Grafana environment with the following benefits:
-
By default, it integrates with various Alibaba Cloud data sources, such as Managed Service for Prometheus and Simple Log Service (SLS). It also supports third-party or self-managed data sources, allowing you to quickly create unified operational dashboards.
-
It provides dedicated instances with a high service level agreement (SLA) to ensure a highly available and elastic monitoring system, making operations more reliable and lowering maintenance costs.
-
It integrates with Alibaba Cloud account single sign-on (SSO) and self-managed account systems. This integration ensures data security while enabling fine-grained management of data sources and dashboards.
-
It helps solve the following problems:
-
Aggregating and unifying monitoring data from various cloud services is difficult and increases operational complexity.
-
Core metric charts for various cloud services need to be repeatedly configured.
-
Alert rules for various cloud services are scattered and difficult to manage in a unified way.
-
-
It provides the following capabilities:
-
Default integration: It integrates with core Alibaba Cloud services such as elastic computing and database services by default.
-
Unified dashboards: It establishes a unified dashboard system across data sources for more fine-grained visualized operations.
-
Unified alerting: You can easily build an integrated alerting system to improve alert management efficiency.
-
Benefit 5: Integrated alerting system
By default, Alibaba Cloud Managed Service for Prometheus is integrated with the unified alerting system of Alibaba Cloud. This system has the following features:
-
Globalization
-
Global alert rule templates allow you to configure alerts for global events from a single place.
-
Global contacts and notification policies can be configured once to take effect worldwide.
-
-
More efficient management with event integration
-
Alert Management supports one-click integration with common Alibaba Cloud monitoring tools by default and allows you to manually connect more tools for centralized maintenance.
-
The stable event integration module provides 24/7 uninterrupted event processing services.
-
The module ensures low latency when processing massive amounts of event data.
-
-
Timely and accurate alert notifications for contacts
-
You can configure notification rules to consolidate events before sending alert notifications, which reduces notification fatigue for your operations team.
-
Based on the alert severity, you can select different notification methods such as email, SMS, phone call, or DingTalk to notify contacts to handle alerts.
-
You can use escalation policies to send repeated reminders for unhandled alerts to ensure timely resolution.
-
-
Rapid alert management
-
Contacts can handle alerts at any time through DingTalk.
-
A universal alert format helps contacts better analyze alerts.
-
Multiple contacts can collaborate to handle alerts through DingTalk.
-
-
Alert event reprocessing
-
You can use event processing flows to create simple workflows to reprocess alert events from any source to meet various data processing requirements.
-
Event management supports deduplication, compression, noise reduction, and silencing for alert events from any source. This helps reduce alert storms.
-
-
Alert configuration management
-
The system provides monitoring templates for common core metrics of container clusters and an alert template feature that allows you to generate and distribute alert templates for rapid, batch configuration of alerts.
-
The system provides a GUI-based configuration wizard and a preview feature. You can view events that match alert conditions in real time and perform fine-grained configurations.
-
-
You can view alert statistics and analyze handling status in real time to improve alert resolution efficiency and monitor business operations.
Example: Connecting a multi-cloud container cluster to Managed Service for Prometheus
Prerequisites
You can connect your cluster by using the Internet or an internal network. For more information, see FAQ about registered clusters.
Step 1: Create an ACK One registered cluster
Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click Create Kubernetes Cluster.
-
On the ACK One Registered Cluster tab, configure the parameters. For more information about the parameters, see Create an ACK One registered cluster.
-
In the right part of the page, click Create Cluster to create the cluster. You can view the created registCluster cluster on the Clusters page. After the cluster is created, the newly registered cluster, such as test-external-cluster1, appears in the Clusters list. The cluster type is external Kubernetes cluster and its status is Waiting for Connection.
Step 2: Add a multi-cloud cluster
This section uses a Tencent Kubernetes Engine (TKE) cluster as an example to show you how to add the TKE cluster to an ACK One registered cluster and use Managed Service for Prometheus to scrape and display metrics.
-
On the Clusters page of the ACK console, find the registCluster cluster that you created in Step 1: Create an ACK One registered cluster and click Details in the Operation column.
-
Click the Connection Information tab. Click Obtain Temporary kubeconfig or Obtain Long-term kubeconfig. In the dialog box that appears, select the Internal Access or Public Access tab, and then click Copy to copy the cluster access credential.
-
Log on to the Tencent Cloud Container Service console. On the Cluster page, click the name of the target cluster. In the upper-right corner, click Create Resource from YAML. Paste the copied public access cluster connection credential into the editor, and then click Finish to install the agent. Then, on the Tencent Cloud Cluster page, check the status of the ack-cluster-agent deployment. If the deployment is running, the agent is installed successfully.
-
Log on to the Alibaba Cloud ACK console. In the left-side navigation pane, choose Clusters. On the Clusters page, check the status of the registCluster registered cluster that you created in Step 1: Create an ACK One registered cluster. If the status is Running, the cluster is added.
Step 3: Install the Prometheus component
For more information, see Connect an Alibaba Cloud Prometheus instance to a registered cluster.
Step 4: View monitoring data
Managed Service for Prometheus provides default Grafana dashboards for you to view monitoring data, including dashboards for Deployments and DaemonSets. To view monitoring data from a dashboard, perform the following steps:
Log on to the ARMS console. In the left-side navigation pane, choose .
-
Click the Prometheus instance that corresponds to the registCluster cluster that you created in Step 1: Create an ACK One registered cluster.
-
In the left-side navigation pane, click Dashboards. On the Dashboards page, click a dashboard name to view detailed metrics.
Step 5: View alerts
Managed Service for Prometheus enables monitoring for key container cluster metrics by default, ensuring you do not miss important metrics. It also includes various alert templates for core metrics that you can enable as needed, without having to write PromQL queries. To view alerts in Managed Service for Prometheus, perform the following steps:
Log on to the ARMS console. In the left-side navigation pane, choose .
-
Click the Prometheus instance that corresponds to the registCluster cluster that you created in Step 1: Create an ACK One registered cluster.
-
In the left-side navigation pane, click Alarm Rules. On the Alarm Rules page, you can view alert details.
Activate services
-
ACK One registered cluster: For information about how to activate a registered cluster, see Activate a registered cluster.
-
Managed Service for Prometheus: This service offers a subscription billing model, which can save you at least 67% on costs compared to the pay-as-you-go model.