Container Service for Kubernetes:Monitor backup center and configure alerts with Prometheus

Prerequisites

• Version v1.7.10 or later of the migrate-controller component is installed. For more information, see Install the migrate-controller component and configure permissions and Upgrade components.

• Clusters that run a Kubernetes version earlier than 1.20 cannot be upgraded to the latest version of the migrate-controller component. To use the backup center's monitoring feature, you must first upgrade your cluster. For more information, see Manually upgrade a cluster.

• Managed Service for Prometheus is enabled for the cluster.

Billing

The migrate-controller component sends monitoring metrics to Managed Service for Prometheus. These metrics are considered custom metrics. Using custom metrics incurs additional fees.

Before you proceed, read the Billing overview to understand the billing policy for custom metrics and prevent unexpected charges. Fees may vary based on factors such as your cluster size and the number of applications. Use resource usage statistics to monitor and manage your resource consumption.

Integrate the backup center with Prometheus

You can use Managed Service for Prometheus to monitor the status of backup vaults and backup tasks in the current cluster.

1. Log on to the ARMS console.

2. In the left-side navigation pane, click Integration Center. On the Infrastructure tab, search for ACK BackupCenter and click ACK BackupCenter to go to the integration page.

3. On the Start Access tab, select the target Container Service for Kubernetes (ACK) cluster where the backup center is installed and click OK.

   After the integration status check is complete, you can view the dashboard data in the ACK console or the ARMS console.

View the backup center monitoring dashboards

Access the dashboards

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Operations > Prometheus Monitoring.

3. On the Prometheus Monitoring page, click the Others tab and view the backup center dashboards under ACK BackupCenter.

Dashboard overview

The backup center monitoring dashboard has three sections: Backup Locations (backup vault status), Backup Operation Status (backup task status), and Addon Status (working component status).

Backup Locations

This dashboard monitors the basic information of backup vaults and displays the details of the backup vaults associated with the current cluster (Backuplocation Detail).

A backup vault stores backups and links the cluster's backup center to an OSS bucket. You can create backups, snapshots, and restorations only when the backup vault is in the Available state.

Backup Operation Status

This dashboard monitors the status of backup tasks and includes two sections: a backup task overview (Backup Overview) and details of failed backup tasks (Failed Backup Detail).

• Backup Overview: A bar chart that shows the number of backup tasks created in each backup vault of the cluster. Ad-hoc backups or scheduled backup plans create backup tasks. The x-axis represents the names of backup vaults and the y-axis represents the number of backup tasks. The following table describes the monitoring metrics for Backup Overview.

  Metric	Description
  Backup (Failed)	The number of failed backup tasks, shown as a red bar.
  Backup (Completed)	The number of successful backup tasks, shown as a green bar.

• Failed Backup Detail: A table that shows basic information about backup tasks that are in the Failed state in the cluster. The following table describes the monitoring metrics for Failed Backup Detail.

  Metric	Description
  Backup	The name of the backup task.
  Backuplocation	The name of the backup vault where the backup task resides.
  BackupType	The backup type of the backup task. Valid values: AppBackup: Only the application is backed up (YAML backup). AppAndPvBackup: Both the application and its data are backed up (YAML and PV data backup).
  DataType	The data backup type. Valid values: snapshot: The backed-up PVs use only disk storage. hbr: The backed-up PVs use file storage, such as HostPath local storage, NAS, or OSS. all: The backed-up PVs include both disk storage and file system storage. none: Data backup is enabled, but no PV storage is used in the selected namespace.
  FromSchedule	The source of the backup task. Empty: The task was created by an ad-hoc backup. Not empty: The task was created by a scheduled backup plan. The value is the name of the corresponding backup plan.

Addon Status

This dashboard monitors the status of the csdr-controller and csdr-velero components. These components must be running correctly to perform operations such as backups, snapshots, and restorations in the backup center.

After the migrate-controller component of the backup center is installed, it performs a precheck on the cluster. After the check is complete, the migrate-controller component deploys the csdr-controller and csdr-velero working components in the csdr namespace where the backup center runs.

The working components of the backup center are displayed as Deployments (Pods) and include the csdr-controller and csdr-velero components. The following table describes the monitoring metrics for Addon Status.

Configure alerting for backup task failures

Alerts for backup task failures are event-based. The CustomResourceDefinition (CRD) for backup tasks is applicationbackups in the csdr.alibabacloud.com resource group. When a backup task fails, the resource of the same name generates a Warn event.

Query Warn events for failed tasks

Run the following command to query the Warn event for a failed backup task.

kubectl -n csdr get events --field-selector='type!=Normal' 

Example output:

VaultError: backup vault is unavailable: oss: service returned error: StatusCode=403, ErrorCode=AccessDenied, ErrorMessage="The bucket you access does not belong to you.", RequestId=668516BC35F915******

In the output, VaultError indicates the cause of the backup task failure.

Configure alert rules for task failures

Configure these alerts by using the cluster alert configuration feature. For more information, see Alert management for ACK.

Troubleshoot monitoring issues

Troubleshoot unhealthy or missing components

• After you install the backup center, working components are missing or are repeatedly deployed.

  Run the following command to query the running status of the migrate-controller component.

  kubectl -n kube-system get pod -l app=migrate-controller

  If the component is in the CrashLoopBackOff state or keeps restarting, the cluster did not pass the precheck. This is typically because the cluster uses the FlexVolume storage plug-in or the registered cluster is not configured with the required permissions. For troubleshooting information, see FAQ about the backup center and registered clusters.

• The working component remains in the UnHealth state for a long period, and the Pods dashboard shows no data or an abnormal state.

  The Pod of the working component fails to start. For more information about troubleshooting, see Troubleshoot pod exceptions.

• The working component is in the Health state, but the Restarts count on the Pods dashboard is not 0.

  The memory usage of csdr-velero can peak during a backup, which can cause out-of-memory (OOM) issues and lead to component restarts. Increase the memory resource limit to resolve this.

  Note

  During the backup process, if the Pod of a working component unexpectedly exits, the task will fail or remain in the InProgress state for an extended period.

Troubleshoot Unavailable vaults

Run the following command to query the error message.

Replace <unavailable-backuplocation-name> with the name of the unavailable backup vault.

kubectl -n csdr describe backuplocation <unavailabe-backuplocation-name> 

For more information about how to handle backup vault exceptions, see FAQ about the backup center.

Troubleshoot failed backup tasks

kubectl -n csdr describe applicationbackup <failed-applicationbackup-name> 

Appendix: Metrics

By default, the csdr-controller component, which runs in the csdr working namespace, listens on port 8190 at the /csdr-metrics path to expose Prometheus metrics data.

The exposed metrics are categorized into two main groups: backup vault (BackupLocation) status and application backup (ApplicationBackup) status. If you use a self-managed Prometheus instance, use the following information to configure metric scraping.