Resolve failed cluster checks - Container Service for Kubernetes

ACK provides cluster checks for cluster upgrades, cluster migrations, components, and node pools. You can run a check before these operations to verify that your cluster meets the requirements and get recommended fixes for failed check items.

Cluster checks

Cluster upgrade checks

Due to the complexity of Kubernetes, upgrading a cluster involves significant risks. To ensure a smooth upgrade, ACK performs a series of pre-upgrade checks. You can upgrade only after your cluster passes all checks. The specific check items vary based on the cluster type, Kubernetes version, and container runtime. For the definitive list of checks for your cluster, see the ACK console.

Cluster upgrade checks include three categories:

Cluster Resources: Checks the cloud resources associated with the ACK cluster, such as SLB, ECS, and VPC.
Cluster Components: Checks the configurations of the ACK cluster, its components, and applications, such as whether component versions meet requirements and whether applications use deprecated APIs.
Cluster Configuration: Checks the configurations of the nodes in the ACK cluster. This check creates a Pod on the nodes to collect information.

Category	Check item	Description
Cluster Resources	API Server SLB	Verifies that the SLB instance exists.
		Verifies that the SLB instance is healthy.
		Verifies that the SLB listener is configured correctly.
		Verifies that the backend server group of the SLB instance is configured correctly.
		Verifies that the SLB access control settings are correct. If no access control is configured, this check passes.
	VPC	Verifies that the VPC instance exists.
	VPC	Verifies that the VPC instance is healthy.
	vSwitch	Verifies that the vSwitch exists.
		Verifies that the vSwitch is healthy.
		Verifies that the vSwitch has at least two available IP addresses.
	ECS	Verifies that the ECS instance exists.
		Verifies that the ECS instance is healthy.
		Verifies that the ECS security group is healthy.
		Verifies that the ECS service period is valid.
		Verifies that the ECS instance type meets requirements.
		Verifies that the Cloud Assistant client is healthy.
Cluster Components	Kube Proxy master	Verifies that the component exists.
	Kube Proxy worker	Verifies that the component exists.
	API Service	Checks for unavailable API services.
	Cluster instance	Verifies that the cluster has 3 or 5 master nodes.
	Cluster components	Verifies that the Terway component version meets requirements.
		Verifies that the CoreDNS component version meets requirements.
		Verifies that the cloud-controller-manager component version meets requirements.
		Verifies that the Nginx Ingress Controller component version meets requirements.
		Verifies that the ACK Virtual Node component version meets requirements.
		Verifies that the Metrics Server component version meets requirements.
	Node	Verifies that the node has an IP address.
		Verifies that the node is schedulable.
		Verifies that the node is in the Ready state.
		Verifies that the node operating system supports the upgrade.
		Verifies that the node has more than two available Pods.
	Deprecated API	Checks for usage of deprecated APIs in the cluster.
Cluster Configuration	iptables configuration	Verifies that the iptables configuration is correct.
	Operating system	Verifies that the operating system supports the upgrade.
	yum	Verifies that yum works correctly.
	Disk	Verifies that the node file system is healthy.
	Disk	Verifies that the node has more than 5% free disk space.
	Swap	Checks if Swap is enabled on the node.
	NTP	Verifies that NTP is functioning correctly on the node.
	Systemd	Verifies that the Systemd version is later than systemd-219-67.
	kubelet	Verifies that the kubelet configuration matches the expected settings.
	Container runtime	Verifies that the Docker or containerd runtime is healthy.
	Kernel configuration	Verifies that the node kernel configuration is correct.
	Manifest configuration	Verifies that the Manifest file matches the expected settings.

Cluster migration checks

Before a cluster migration, ACK runs pre-migration checks. Migration can proceed only after all checks pass. This applies to the following scenarios:

Migrating from an ACK dedicated cluster to an ACK Pro managed cluster.
Migrating from a basic ACK managed cluster to an ACK Pro managed cluster.

Cluster migration checks include four categories:

Cluster Resources: Checks the cloud resources associated with the ACK cluster, such as SLB, ECS, and VPC.
Cluster Components: Checks the component configurations in the ACK cluster, such as whether any unavailable API services exist.
Cluster Configuration: Checks the configurations of the nodes in the ACK cluster. This check creates a Pod on the nodes to collect information.
Component Usage: When you migrate an ACK dedicated cluster, some of its components transition to being managed by ACK. This check verifies that these components are healthy before the migration.

Category	Check item	Description
Cluster Resources	API Server SLB	Verifies that the SLB instance exists.
		Verifies that the SLB instance is healthy.
		Verifies that the SLB listener is configured correctly.
		Verifies that the backend server group of the SLB instance is configured correctly.
		Verifies that the SLB access control settings are correct. If no access control is configured, this check passes.
	VPC	Verifies that the VPC instance exists.
	VPC	Verifies that the VPC instance is healthy.
	vSwitch	Verifies that the vSwitch exists.
		Verifies that the vSwitch is healthy.
		Verifies that the vSwitch has at least two available IP addresses.
	ECS	Verifies that the ECS instance exists.
		Verifies that the ECS instance is healthy.
		Verifies that the ECS security group is healthy.
		Verifies that the Cloud Assistant client is healthy.
Cluster Components	Kube Proxy master	Verifies that the component exists.
	Kube Proxy worker	Verifies that the component exists.
	API Service	Checks for unavailable API services.
	Cluster instance	Verifies that the cluster has 3 or 5 master nodes.
	Node	Verifies that the node has an IP address.
		Verifies that the node is schedulable.
		Verifies that the node is in the Ready state.
		Verifies that the node operating system supports the upgrade.
		Verifies that the node has more than two available Pods.
Cluster configuration	Operating system	Verifies that the operating system supports the upgrade.
Cluster configuration	yum	Verifies that yum works correctly.
Component usage	cloud-controller-manager	Verifies that the cloud-controller-manager component is healthy.

Component checks

Component checks apply to component upgrade scenarios. Before a component is upgraded, ACK runs pre-upgrade checks. The upgrade can proceed only after the component passes all checks.

Category	Check item	Description
cloud-controller-manager	Addon_CCM	Checks if upgrading this component affects the SLB.
cloud-controller-manager	Component_Block_Version	Verifies that the CCM version can be upgraded.
csi-plugin	DaemonSet_Annotation	Verifies that the DaemonSet annotations match the expected settings.
	Csi_Driver_Attributes	Verifies that the CSI driver attributes meet requirements.
	Node_Status_Ready	Verifies that the cluster nodes are in the Ready state.
csi-provisioner	Stateful_Set_Exist	Verifies that the component is deployed as a StatefulSet.
	Deployment_Annotation	Verifies that the Deployment annotations match the expected settings.
	Storage_Class_Attributes	Verifies that the StorageClass attributes meet requirements.
	Csi_Provisioner_Node_Count	Verifies that there are at least two Ready nodes.
terway-eniip	Systemd	Verifies that the Systemd version on the node is later than systemd-219-67.
nginx-ingress-controller	Deployment_Healthy	Verifies that the Nginx Ingress Controller Deployment is healthy.
	Deployment_Not_Under_HPA	Checks if a HorizontalPodAutoscaler (HPA) is configured for the Deployment.
	Deployment_Not_Modified	Checks if the Deployment has been modified.
	Nginx_Ingress_Pod_Error_Log	Checks for error logs in Nginx.
	LoadBalancer_Service_Healthy	Verifies that the Nginx Service is healthy.
	Nginx_Ingress_Configuration	Checks for incompatible Ingress configurations.
aliyun-acr-credential-helper	RamRole_Exist	Verifies that the component is granted the AliyunCSManagedAcrRole.
ack-cost-exporter	RamRole_Exist	Verifies that the component is granted the AliyunCSManagedCostRole.

Node pool checks

Node pool checks apply to node pool upgrade scenarios. When you upgrade a node pool, ACK runs pre-upgrade checks. You can upgrade the node pool only after it passes all checks.

Node pool checks include three categories:

Cluster Resources: Checks the cloud resources associated with the ACK cluster, such as SLB and VPC.
Cluster Components: Checks the configurations of the ACK cluster, nodes, and applications.
Cluster Configuration: Checks the configurations of the nodes in the ACK cluster. This check creates a Pod on the nodes to collect information.

Category	Check item	Description
Cluster Resources	API Server SLB	Verifies that the SLB instance exists.
		Verifies that the SLB instance is healthy.
		Verifies that the SLB listener is configured correctly.
		Verifies that the backend server group of the SLB instance is configured correctly.
		Verifies that the SLB access control settings are correct. If no access control is configured, this check passes.
	VPC	Verifies that the VPC instance exists.
	VPC	Verifies that the VPC instance is healthy.
	vSwitch	Verifies that the vSwitch exists.
		Verifies that the vSwitch is healthy.
		Verifies that the vSwitch has at least two available IP addresses.
Cluster Components	API Service	Checks for unavailable API services.
	Cluster instance	Verifies that the cluster has 3 or 5 master nodes.
	Node	Verifies that the node is in the Ready state.
	Node	Verifies that the node has more than two available Pods.
	HostPath	Checks if any Pods on the node use HostPath volumes.
Cluster Configuration	iptables configuration	Verifies that the iptables configuration is correct.
	Operating system	Verifies that the operating system supports the upgrade.
	yum	Verifies that yum works correctly.
	Disk	Verifies that the node file system is healthy.
	Node remaining disk space	Verifies that the node has more than 5% free disk space.
	Swap	Checks if Swap is enabled on the node.
	NTP	Verifies that NTP is functioning correctly on the node.
	Systemd	Verifies that the Systemd version is later than systemd-219-67.
	kubelet	Verifies that the kubelet configuration matches the expected settings.
	Container runtime	Verifies that the Docker or containerd runtime is healthy.
	Kernel configuration	Verifies that the node kernel configuration is correct.
	Manifest configuration	Verifies that the Manifest file matches the expected settings.

Solutions for failed checks

Failed check	Solution
systemd version is too low	Upgrade the systemd version.
component version is too low	Upgrade the component version. For more information, see Manage components.
yum check timed out	Run the following command to check for yum timeouts. The default timeout is 10 seconds. `time if type yum&>/dev/null; then yum list yum; fi`
API Service is unavailable	Run the following command to identify unavailable API Services. `kubectl -n kube-system get apiservices \|grep -i false` Confirm the purpose of the unavailable API Service. If the service is not required, run the following command to delete it. Important Deleting a required API Service can cause the cluster to malfunction. If you are unsure about the purpose of the unavailable API Service, contact technical support. `kubectl -n kube-system delete apiservices ${your-abnormal-apiservice-name}`
A node contains pods that use HostPath	Upgrading a node by replacing its system disk can cause data loss for pods that use HostPath to mount a host directory. Check the mounted directory of each affected pod. If there is no impact, you can proceed with the upgrade. This check is informational only and does not block the upgrade.
The cluster uses deprecated APIs	Identify the sources of the deprecated APIs and take appropriate action. For more information, see Deprecated APIs.

Deprecated APIs

When you run a pre-upgrade check on a cluster that runs Kubernetes 1.20 or later, the check detects any deprecated API usage. The check results list these APIs.

For example, when you upgrade a cluster from Kubernetes 1.20 to Kubernetes 1.22, the system scans the audit logs from the previous day to find any deprecated API usage.

If the cluster uses a deprecated API, the check result is only a notification and does not block the upgrade.
If you continue to use a deprecated API in Kubernetes 1.22, this may introduce security risks. You must assess the impact on your business.

Deprecated APIs are categorized into the following four types based on the request source (user agent). Before you upgrade the cluster, identify the source of each deprecated API based on its Category in the following table and take the recommended action.

Type	Recommended action	Example
core	Kubernetes core components: ACK automatically upgrades these components during a cluster upgrade. They are not displayed on the check page, and no action is required.	kube-apiserver, kube-scheduler, kube-controller-manager
ack	ACK components: These components are provided by ACK and are not displayed on the deprecated API page. Instead, ACK guides you to upgrade these components on the Add-ons page. Note In the ACK console, go to Operations > Add-ons to upgrade the components. The deprecated API information disappears the day after you upgrade the components. In some cases, the CoreDNS component may use a deprecated API in a Kubernetes cluster of version 1.24 or later. If the check result includes coredns, see Why is CoreDNS using a deprecated API? for instructions. The deprecated API information is only a notification and does not affect the upgrade. After the upgrade, the system replaces deprecated API resources with new ones. We recommend that you avoid creating resources with deprecated APIs to prevent security risks.	metrics-server, nginx-ingress-controller, CoreDNS
opensource	Open source components: This type includes components from the open source community. Decide whether to upgrade them based on your requirements. Note The deprecated API notification is informational only and does not affect the upgrade. Upgrade the components based on your requirements to avoid affecting their functionality.	rancher, elasticsearch-operator, etc.
unknown	Unknown source: Sources that do not match the preceding types are marked as unknown. You must decide whether to upgrade the component and perform the upgrade yourself. Note The deprecated API notification is informational only and does not affect the upgrade. Upgrade the components based on your requirements to avoid affecting their functionality.	kubectl, agent, Go-http-client, okhttp

To view deprecated API information:

On the Upgrade Cluster page, click Precheck, and then click View Details.
On the Report page, click View Details.
The details page displays the deprecated API, user agent, type, deprecated Kubernetes version, last access time, and the source IP address.

The Report page contains three tabs: Cluster Resources, Cluster Components, and Node Configuration. The Node Configuration tab includes a warning that reads: "The cluster is still using a deprecated Kubernetes API."