Container Service for Kubernetes:Usage notes and risky operations

Item

References

Usage notes

• Data plane components

• Cluster upgrades

• Native Kubernetes configurations

• ACK serverless clusters

• Registered clusters

• App Catalog

Risky operations

• Risky operations related to clusters

• Risky operations related to node pools

• Risky operations related to networking and load balancing

• Risky operations related to storage

• Risky operations related to logs

Category

Risky operation

Impact

Recovery plan

API Server

Delete the SLB instance that is used by the API Server.

The cluster becomes inoperable.

Unrecoverable. You must recreate the cluster. For more information about how to recreate a cluster, see Create an ACK managed cluster.

Worker node

Modify the security group of a node in the cluster.

The node may become unavailable.

You can add the node back to the security group that was automatically created for the cluster. For more information, see Associate a security group with an instance (primary ENI).

The node expires or is destroyed.

The node becomes unavailable.

Unrecoverable.

Reinstall the operating system.

Components on the node are deleted.

You can remove the node from the cluster and then add it back. For more information, see Remove a node and Add existing nodes.

Upgrade node components by yourself.

The node may become unavailable.

You can roll back to the original version.

Change the IP address of the node.

The node becomes unavailable.

You can change the IP address back to the original one.

Modify the parameters of core components such as kubelet, docker, and containerd by yourself.

The node may become unavailable.

You can configure the parameters as recommended in the official documentation.

Modify the operating system configuration.

The node may become unavailable.

You can try to restore the configuration items or delete the node and purchase a new one.

Modify the node time.

Components on the node may become abnormal.

You can restore the node time.

Add computing resources to the cluster using a method that is not supported by ACK.

ACK provides multiple methods to add computing resources to a cluster, such as using the console, OpenAPI, and command-line interface (CLI). For more information, see Add existing nodes. If you add a node to a cluster using other methods, ACK cannot identify the source of the node. As a result, ACK cannot provide product capabilities such as node lifecycle management, automated O&M, and technical support. For more information about the risks, see Why does the console display that the source of the node pool to which a node belongs is "Other Nodes"?.

You can use node pools to manage computing resources. If you want to continue using the node, you must ensure the compatibility between the node and cluster components, such as Kubernetes, networking, storage, and security components.

Master node (for ACK dedicated clusters)

Modify the security group of a node in the cluster.

The master node may become unavailable.

You can add the node back to the security group that was automatically created for the cluster. For more information, see Associate a security group with an instance (primary ENI).

The node expires or is destroyed.

The master node becomes unavailable.

Unrecoverable.

Reinstall the operating system.

Components on the master node are deleted.

Unrecoverable.

Upgrade the Master or etcd component version by yourself.

The cluster may become unusable.

You can roll back to the original version.

Delete or format the data in core directories such as /etc/kubernetes on the node.

The master node becomes unavailable.

Unrecoverable.

Change the IP address of the node.

The master node becomes unavailable.

You can change the IP address back to the original one.

Modify the parameters of core components such as etcd, kube-apiserver, and docker by yourself.

The master node may become unavailable.

You can configure the parameters as recommended in the official documentation.

Replace the Master or etcd certificate by yourself.

The cluster may become unusable.

Unrecoverable.

Add or remove Master nodes by yourself.

The cluster may become unusable.

Unrecoverable.

Modify the node time.

Components on the node may become abnormal.

You can restore the node time.

Other

Change or modify permissions using RAM.

Some cluster resources, such as SLB instances, may fail to be created.

You can restore the original permissions.

Modify or delete the preset PodSecurityPolicy-related resources in the cluster. These resources include the PodSecurityPolicy resource named ack.privileged and the ClusterRole, ClusterRoleBinding, Role, and RoleBinding resources whose names start with ack:podsecuritypolicy:.

Core components of the cluster may become abnormal. You may be unable to create or update pods in the cluster.

You can restore the related resources. For more information, see Configure or restore the default pod security policies of ACK.

Risky operation

Impact

Recovery plan

Delete the scaling group.

The node pool becomes abnormal.

Unrecoverable. You can only recreate the node pool. For more information about how to recreate a node pool, see Create a node pool.

Remove a node using kubectl.

The number of nodes displayed for the node pool is inconsistent with the actual number.

You can remove the specified node using the ACK console or by calling the API operations related to node pools (see Remove a node) or scale in the node pool by changing the number of expected nodes (see Create and manage a node pool).

Directly release the ECS instance.

The node pool product page may be displayed abnormally. A node pool for which you specified the number of expected nodes automatically scales out to the expected number of nodes based on the node pool configuration.

Unrecoverable. The correct practice is to scale in the node pool by changing the number of expected nodes in the ACK console or by calling the API operations related to node pools (see Create and manage a node pool) or remove the specified node (see Remove a node).

Manually scale out or scale in a node pool for which auto scaling is enabled.

The auto scaling component automatically adjusts the number of nodes based on the policy. The result may not be what you expect.

Unrecoverable. Do not manually intervene with an auto scaling node pool.

Modify the maximum or minimum number of instances in the ESS scaling group.

Scaling may become abnormal.

• For a node pool for which auto scaling is disabled, you can set the maximum and minimum number of instances in the ESS scaling group to the default values of 2000 and 0.

• For a node pool for which auto scaling is enabled, you can set the maximum and minimum number of instances in the ESS scaling group to be the same as the maximum and minimum number of nodes in the node pool.

Do not back up data before you add an existing node.

Data on the instance is lost after the instance is added.

Unrecoverable.

• Before you manually add an existing node, you must back up all data that you want to retain.

• When a node is automatically added, its system disk is replaced. You must back up useful data from the system disk in advance.

Save important data on the system disk of a node.

The self-healing feature of a node pool may reset the node configuration to repair the node. This may cause data loss on the system disk.

Unrecoverable. The correct practice is to store important data on a data disk, cloud disk, NAS volume, or OSS bucket.

Risky operation

Impact

Recovery plan

Modify the kernel parameter net.ipv4.ip_forward=0.

Network connection fails.

You can modify the kernel parameter to net.ipv4.ip_forward=1.

Modify the kernel parameter:

• net.ipv4.conf.all.rp_filter = 1|2

• net.ipv4.conf.[ethX].rp_filter = 1|2

  Note

  ethX represents all network interface cards whose names start with eth.

Network connection fails.

You can modify the kernel parameter to:

• net.ipv4.conf.all.rp_filter = 0

• net.ipv4.conf.[ethX].rp_filter = 0

Modify the kernel parameter net.ipv4.tcp_tw_reuse = 1.

The health check of pods becomes abnormal.

You can modify the kernel parameter to net.ipv4.tcp_tw_reuse = 0.

Modify the kernel parameter net.ipv4.tcp_tw_recycle = 1.

NAT becomes abnormal.

You can modify the kernel parameter to net.ipv4.tcp_tw_recycle = 0.

Modify the kernel parameter net.ipv4.ip_local_port_range.

The network connection intermittently fails.

You can modify the kernel parameter to the default value net.ipv4.ip_local_port_range="32768 60999".

Install firewall software, such as Firewalld or ufw.

The container network connection fails.

You can uninstall the firewall software and restart the node.

The security group of the node does not allow access to UDP port 53 of the container CIDR block.

The DNS service in the cluster cannot work as expected.

You can configure the security group to allow access as recommended in the official documentation.

Modify or delete the tags of an SLB instance added by ACK.

The SLB instance becomes abnormal.

You can restore the tags of the SLB instance.

Modify the configurations of an SLB instance managed by ACK in the SLB console. The configurations include the SLB instance, listeners, and vServer groups.

The SLB instance becomes abnormal.

You can restore the configurations of the SLB instance.

Remove the annotation for reusing an existing SLB instance from the Service. The annotation is service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id: ${YOUR_LB_ID}.

The SLB instance becomes abnormal.

You can add the annotation for reusing an existing SLB instance to the Service.

Delete an SLB instance created by ACK in the SLB console.

The cluster network may become abnormal.

You can delete the SLB instance by deleting the Service. For more information about how to delete a Service, see Delete a Service.

Manually delete the nginx-ingress-lb Service from the kube-system namespace when the Nginx Ingress Controller component is installed.

The Ingress controller does not work as expected and may even crash.

You can create a Service with the same name using the following YAML template.

apiVersion: v1
kind: Service
metadata:
  annotations:
  labels:
    app: nginx-ingress-lb
  name: nginx-ingress-lb
  namespace: kube-system
spec:
  externalTrafficPolicy: Local
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: ingress-nginx
  type: LoadBalancer

Add or modify the nameserver option in the /etc/resolv.conf DNS configuration file on an ECS node.

If the DNS server is not properly configured, DNS resolution may fail. This affects the normal operation of the cluster.

If you want to use a self-managed DNS server as the upstream server, you can configure it on the CoreDNS side. For more information, see Instructions on how to configure unmanaged CoreDNS components.

Modify or delete ENIs or Lingjun ENIs created by ACK.

The pod network is interrupted.

Unrecoverable.

Modify or delete network-related CRDs.

podnetworkings.network.alibabacloud.com
podenis.network.alibabacloud.com
networkinterfaces.network.alibabacloud.com
nodes.network.alibabacloud.com
noderuntimes.network.alibabacloud.com
*.cilium.io
*.crd.projectcalico.org

The Terway component will stop working. In severe cases, this may cause network interruptions and abnormal pods.

Unrecoverable.

Create, modify, or delete network-related system CRs.

podenis.network.alibabacloud.com
networkinterfaces.network.alibabacloud.com
nodes.network.alibabacloud.com
noderuntimes.network.alibabacloud.com
*.cilium.io
*.crd.projectcalico.org

The Terway component will stop working. In severe cases, this may cause network interruptions and abnormal pods.

You can delete the custom CR definitions and recreate the associated pods.

Modify or delete fields that are not allowed to be modified in the Terway network configuration. For more information about the configuration parameters, see Customize Terway parameters.

The Terway component will stop working. In severe cases, this may cause network interruptions and abnormal pods.

You can restore the original configuration and restart the node.

Risky operation

Impact

Recovery plan

Manually detach a cloud disk in the console.

An I/O error is reported when a pod writes data.

You can restart the pod and manually clear the residual mount information on the node.

Run the umount command on the disk mount path on the node.

The pod writes data to the local disk.

You can restart the pod.

Directly operate on the cloud disk on the node.

The pod writes data to the local disk.

Unrecoverable.

Mount the same cloud disk to multiple pods.

The pod writes data to the local disk or an I/O error is reported.

You must make sure that one cloud disk is used by only one pod.

Manually delete the NAS mount directory.

An I/O error is reported when a pod writes data.

You can restart the pod.

Delete a NAS volume or mount target that is in use.

The pod experiences an I/O hang.

You can restart the ECS node. For more information about how to restart an ECS instance, see Restart an ECS instance.

Risky operation

Impact

Recovery plan

Delete the /tmp/ccs-log-collector/pos directory on the host.

Logs are repeatedly collected.

Unrecoverable. The files in this directory record the log collection position.

Delete the /tmp/ccs-log-collector/buffer directory on the host.

Logs are lost.

Unrecoverable. This directory contains cache files of logs to be consumed.

Delete the aliyunlogconfig CRD resource.

Log collection fails.

You can recreate the deleted CRD and its corresponding resources. However, logs collected during the failure period cannot be recovered.

Deleting a CRD also deletes all its instances. Even if you restore the CRD, you still need to manually create the deleted instances.

Delete the log component.

Log collection fails.

You can reinstall the log component and manually restore the aliyunlogconfig CRD instances. Logs collected during the deletion period cannot be recovered.

Deleting the log component is equivalent to deleting the aliyunlogconfig CRD and the Logtail agent. All log collection capabilities are lost during this period.