Container Service for Kubernetes:Important notes and high-risk operations

Category

High-risk operation

Impact

Recovery solution

API Server

Reuse the CLB used by the API Server for other purposes, such as using a LoadBalancer-type Service with the same CLB.

Cluster becomes unavailable, affecting service traffic.

Restore the original configuration or contact customer support.

Modify CLB configurations that control forwarding, such as listeners, server groups, ACLs, or CLB tags used by the API Server.

Cluster malfunctions.

Restore the original configuration.

Delete the CLB used by the API Server.

Cluster becomes inoperable.

Irreversible. Recreate the cluster. For steps, see Create an ACK managed cluster.

Worker nodes

Modify the security group of cluster nodes.

Nodes can become unavailable.

Add the nodes back to the automatically created node security group. For more information, see Associate a security group with an instance (primary NIC).

Node expiration or deletion.

The node becomes unavailable.

Irreversible.

Reinstall the operating system.

Components on the node are deleted.

Remove and re-add the node to the cluster. For steps, see Remove a node and Add existing nodes.

Manually upgrade node component versions.

Nodes can become unusable.

Roll back to the original version.

Change the node IP address.

Node becomes unavailable.

Restore the original IP address.

Manually modify parameters of core components (such as kubelet, Docker, or containerd).

Nodes can become unavailable.

Use the recommended configuration parameters from the official documentation.

Modify operating system configurations.

Nodes can become unavailable.

Attempt to restore the configuration or delete and recreate the node.

Modify node time.

Components on the node can malfunction.

Restore the original node time.

Add node computing resources to the cluster using unsupported methods.

ACK supports adding node computing resources through the console, OpenAPI, or CLI. For more information, see Add existing nodes. Nodes added through other methods cannot be recognized by ACK, so lifecycle management, automated O&M, or technical support are unavailable. For details, see Why does the console show the node pool source as "Other nodes"?.

Manage computing resources through node pools. If you continue using unsupported methods, ensure compatibility between the node and cluster components (such as Kubernetes components, networking, storage, and security).

Master nodes (ACK dedicated clusters)

Modify the security group of cluster nodes.

Master nodes can become unavailable.

Add the nodes back to the automatically created node security group. For more information, see Associate a security group with an instance (primary NIC).

Node expiration or deletion.

The master node becomes unavailable.

Irreversible.

Reinstall the operating system.

Components on the master node are deleted.

Irreversible.

Manually upgrade master or etcd component versions.

The cluster can become unusable.

Roll back to the original version.

Delete or format core directories such as /etc/kubernetes on the node.

The master node becomes unavailable.

Irreversible.

Change the node IP address.

The master node becomes unavailable.

Restore the original IP address.

Manually modify parameters of core components (such as etcd, kube-apiserver, or Docker).

Master nodes can become unavailable.

Use the recommended configuration parameters from the official documentation.

Manually replace master or etcd certificates.

The cluster can become unusable.

Irreversible.

Manually add or remove master nodes.

The cluster can become unusable.

Irreversible.

Modify node time.

Components on the node can malfunction.

Restore the original node time.

Other

Modify permissions or configurations through RAM.

Cluster resources, such as SLB instances, can fail to be created.

Restore the original permissions.

Modify or delete preset PodSecurityPolicy resources in the cluster, including the PodSecurityPolicy named ack.privileged, or ClusterRole, ClusterRoleBinding, Role, and RoleBinding resources with names starting with ack:podsecuritypolicy:.

Core cluster components can malfunction. Pod creation and updates can fail.

Restore the related resources. For steps, see Configure or restore the default ACK Pod security policy.

High-risk operation

Impact

Recovery solution

Delete a scaling group.

Node pool malfunctions.

Irreversible. Recreate the node pool. For steps, see Create a node pool.

Remove a node using kubectl.

Node count displayed in the node pool does not match the actual count.

Remove the node through the ACK console or node pool APIs (see Remove a node) or adjust the desired node count to scale in (see Create and manage node pools).

Directly release an ECS instance.

The node pool details page can display incorrectly. For node pools with a desired node count enabled, the system automatically scales out to maintain the desired count.

Irreversible. Correctly scale in by adjusting the desired node count through the ACK console or node pool APIs (see Create and manage node pools) or remove specific nodes (see Remove a node).

Manually scale out or in a node pool with auto scaling enabled.

The auto scaling component adjusts the node count based on policies, leading to unexpected results.

Irreversible. Do not manually intervene in auto scaling node pools.

Modify the maximum or minimum instance count of an ESS scaling group.

Scaling can malfunction.

• For node pools without auto scaling, reset the ESS scaling group's max and min instance counts to the defaults (2000 and 0).

• For node pools with auto scaling enabled, align the ESS scaling group's max and min instance counts with the node pool's max and min node counts.

Add existing nodes without backing up data.

Data on the instance is lost.

Irreversible.

• Back up all data you want to retain before manually adding existing nodes.

• When adding nodes automatically, the system replaces the system disk. Back up any useful data stored on the system disk beforehand.

Store important data on the node's system disk.

Node self-healing operations can reset node configurations, causing data loss on the system disk.

Irreversible. Store important data on additional data disks, cloud disks, NAS, or OSS.

High-risk operation

Impact

Recovery solution

Uninstall the virtual node component.

Serverless Pod management fails: existing ECI and ACS pods cannot be deleted, and new ones cannot be created.

Reinstall the virtual node component.

High-risk operation

Impact

Recovery solution

Set the kernel parameter net.ipv4.ip_forward=0.

Network connectivity fails.

Set the kernel parameter to net.ipv4.ip_forward=1.

Modify kernel parameters:

• net.ipv4.conf.all.rp_filter = 1|2

• net.ipv4.conf.[ethX].rp_filter = 1|2

  Note

  ethX represents all NICs starting with eth.

Network connectivity fails.

Set kernel parameters to:

• net.ipv4.conf.all.rp_filter = 0

• net.ipv4.conf.[ethX].rp_filter = 0

Set the kernel parameter net.ipv4.tcp_tw_reuse = 1.

Pod health checks fail.

Set the kernel parameter to net.ipv4.tcp_tw_reuse = 0.

Set the kernel parameter net.ipv4.tcp_tw_recycle = 1.

NAT malfunctions.

Set the kernel parameter to net.ipv4.tcp_tw_recycle = 0.

Modify the kernel parameter net.ipv4.ip_local_port_range.

Intermittent network connectivity issues occur.

Restore the kernel parameter to its default value: net.ipv4.ip_local_port_range="32768 60999".

Install firewall software such as Firewalld or ufw.

Container networking fails.

Uninstall the firewall software and restart the node.

Do not allow UDP port 53 for the container CIDR in the node security group.

Cluster DNS fails.

Configure the security group according to official recommendations.

Modify or delete tags added by ACK to an SLB instance.

SLB malfunctions.

Restore the SLB tags.

Modify configurations of ACK-managed SLB instances through the SLB console, including SLB instances, listeners, or vServer groups.

SLB malfunctions.

Restore the SLB configuration.

Remove the annotation for reusing an existing SLB from a Service: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id: ${YOUR_LB_ID}.

SLB malfunctions.

Add the annotation for reusing an existing SLB back to the Service.

Delete an ACK-created SLB through the SLB console.

Cluster networking can fail.

Delete the SLB by deleting the associated Service. For steps, see Delete a Service.

Manually delete the nginx-ingress-lb Service in the kube-system namespace when the Nginx Ingress Controller is installed.

Ingress Controller malfunctions or crashes.

Create a new Service with the same name using the following YAML.

apiVersion: v1
kind: Service
metadata:
  annotations:
  labels:
    app: nginx-ingress-lb
  name: nginx-ingress-lb
  namespace: kube-system
spec:
  externalTrafficPolicy: Local
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: ingress-nginx
  type: LoadBalancer

Add or modify the nameserver option in the DNS configuration file /etc/resolv.conf on ECS nodes.

If the configured DNS server is misconfigured, DNS resolution can fail, affecting cluster operations.

If you want to use a self-managed DNS server as an upstream server, configure it in CoreDNS instead. For steps, see Unmanaged CoreDNS configuration.

Modify or delete elastic network interfaces (ENIs) or Lingjun ENIs created by ACK.

Pod networking fails.

Irreversible.

Modify or delete network-related CRDs.

podnetworkings.network.alibabacloud.com
podenis.network.alibabacloud.com
networkinterfaces.network.alibabacloud.com
nodes.network.alibabacloud.com
noderuntimes.network.alibabacloud.com
*.cilium.io
*.crd.projectcalico.org

Terway fails. Severe cases can cause network or Pod failures.

Irreversible.

Create, modify, or delete network-related system CRs.

podenis.network.alibabacloud.com
networkinterfaces.network.alibabacloud.com
nodes.network.alibabacloud.com
noderuntimes.network.alibabacloud.com
*.cilium.io
*.crd.projectcalico.org

Terway fails. Severe cases can cause network or Pod failures.

Delete custom CR definitions and recreate associated Pods.

Modify fields in Terway network configurations that are not allowed to be changed. For configurable parameters, see Custom Terway configuration parameters.

Terway fails. Severe cases can cause network or Pod failures.

Restore the original configuration and restart the node.

High-risk operation

Impact

Recovery solution

Manually detach a cloud disk through the console.

Pods report IO errors on writes.

Restart the Pod and manually clean up residual mounts on the node.

Run umount on the disk mount path on the node.

Pods write to the local disk.

Restart the Pod.

Directly operate cloud disks on the node.

Pods write to the local disk.

Irreversible.

Mount the same cloud disk to multiple Pods.

Pods write to the local disk or report IO errors.

Ensure one cloud disk is used by only one Pod.

Manually delete the NAS mount directory.

Pods report IO errors on writes.

Restart the Pod.

Delete a NAS file system or mount target that is in use.

Pods experience IO hangs.

Restart the ECS node. For steps, see Restart an ECS instance.

High-risk operation

Impact

Recovery solution

Delete the /tmp/ccs-log-collector/pos directory on the host.

Logs are collected repeatedly.

Irreversible. This directory records log collection positions.

Delete the /tmp/ccs-log-collector/buffer directory on the host.

Logs are lost.

Irreversible. This directory stores cached logs awaiting processing.

Delete aliyunlogconfig CRD resources.

Log collection stops.

Recreate the deleted CRD and its resources, but logs collected during the outage cannot be recovered.

Deleting the CRD also deletes all associated instances. Even after restoring the CRD, you must manually recreate the deleted instances.

Delete log components.

Log collection stops.

Reinstall the log components and manually restore aliyunlogconfig CRD instances. Logs collected during the outage cannot be recovered.

Deleting log components removes both the aliyunlogconfig CRD and the Logtail collector, disabling all log collection capabilities during that period.