All Products
Search

This Product

    Container Service for Kubernetes:Important notes and high-risk operations

    Document Center

    Container Service for Kubernetes:Important notes and high-risk operations

    Last Updated:Aug 04, 2026

    Container Service for Kubernetes (ACK) provides managed services for core container infrastructure. For non-managed components and applications running in ACK clusters, improper operations can cause service failures or data loss. Read the usage notes and high-risk operation guidance in this topic before you use ACK.

    Index

    Information Item

    References

    Usage notes

    High-risk operations

    Usage notes

    Data plane components

    Data plane components such as CoreDNS, Ingress, kube-proxy, Terway, and kubelet run on your ECS instances, requiring joint maintenance from you and Alibaba Cloud.

    ACK provides the following support for data plane components:

    • Parameterized configuration, feature updates, bug fixes, CVE patches, and documentation.

    • Observability such as monitoring and alerts. Some core component logs are delivered through SLS.

    • Best practices and recommendations tailored to your cluster size.

    • Regular inspections and alerts for component versions, configurations, loads, deployment topologies, instance counts, and other metrics.

    Follow these recommendations when using data plane components:

    • Use the latest component versions for bug fixes and new features. Upgrade at an appropriate time following the instructions in Components.

    • Configure contact information and alert notifications in the ACK alert center to receive alerts and service notifications. See ACK alert management.

    • Address stability risk reports promptly per the provided instructions to eliminate security risks.

    • Configure custom component parameters only through the ACK console by navigating to Operations management > Component management, or using OpenAPI. Other methods can cause malfunctions. See Manage components.

    • Do not modify the component runtime environment through IaaS-layer OpenAPI, such as changing ECS instance states, security group or network settings, or SLB configurations. Unauthorized IaaS changes can cause data plane malfunctions.

    • Some data plane components use upstream open-source versions that may contain bugs or vulnerabilities. Upgrade promptly to avoid disruptions.

    Cluster upgrades

    Always use the ACK cluster upgrade feature. Manual upgrades can cause stability and compatibility issues. See Upgrade clusters and independently upgrade control planes and node pools.

    ACK provides the following support for cluster upgrades:

    • Kubernetes version upgrade features.

    • Pre-upgrade checks to verify cluster readiness.

    • Release notes for new Kubernetes versions, including changes from previous versions.

    • Risk notifications about potential issues due to resource changes during upgrades.

    Follow these recommendations when using the cluster upgrade feature:

    • Run pre-upgrade checks and resolve all blocking issues before proceeding.

    • Evaluate upgrade risks based on your cluster and workload status. See [Deprecated] Kubernetes version release overview.

    • Cluster upgrades cannot be rolled back. Create a thorough upgrade plan and back up beforehand.

    • Upgrade within the current version's support period per the ACK Version Guide.

    Kubernetes native configurations

    • Do not modify the paths, links, or contents of the following critical directories:

      • /var/lib/kubelet

      • /var/lib/docker

      • /etc/kubernetes

      • /etc/kubeadm

      • /var/lib/containerd

    • Do not use Kubernetes-reserved annotations in YAML templates. This can cause resource unavailability, creation failures, or abnormal behavior. Annotations prefixed with kubernetes.io/ or k8s.io/ are reserved. Example: pv.kubernetes.io/bind-completed: "yes".

    ACK serverless clusters

    ACK serverless clusters do not provide compensation in the following scenarios:

    • To simplify operations, ACK Serverless clusters manage some system components when component management is enabled. No compensation is provided if your business is affected by unintentional deletion of Kubernetes resources that managed components depend on.

    Registered clusters

    • When registering external clusters through the ACK console, ensure stable network connectivity between the external cluster and Alibaba Cloud.

    • ACK supports registering external Kubernetes clusters but cannot control their stability. Exercise caution when configuring labels, annotations, or tags on external cluster nodes, as this can cause application failures.

    App Catalog

    ACK Marketplace provides an App Catalog with applications adapted from open-source software. ACK cannot control open-source defects. See Marketplace.

    High-risk operations

    The following operations can significantly impact ACK service stability.

    Cluster-related high-risk operations

    Category

    High-risk operation

    Impact

    Recovery solution

    API Server

    Reuse the API Server's CLB for other purposes, such as a LoadBalancer-type Service.

    Cluster becomes unavailable, affecting service traffic.

    Restore the original configuration or contact customer support.

    Modify API Server CLB forwarding configurations, such as listeners, server groups, ACLs, or tags.

    Cluster malfunctions.

    Restore the original configuration.

    Delete the CLB used by the API Server.

    Cluster becomes inoperable.

    Irreversible. Recreate the cluster. See Create an ACK managed cluster.

    Worker nodes

    Modify the security group of cluster nodes.

    Nodes can become unavailable.

    Add nodes back to the auto-created security group. See Associate a security group with an instance (primary NIC).

    Node expiration or deletion.

    The node becomes unavailable.

    Irreversible.

    Reinstall the operating system.

    Components on the node are deleted.

    Remove and re-add the node. See Remove a node and Add existing nodes.

    Manually upgrade node component versions.

    Nodes can become unusable.

    Roll back to the original version.

    Change the node IP address.

    Node becomes unavailable.

    Restore the original IP address.

    Manually modify parameters of core components (such as kubelet, Docker, or containerd).

    Nodes can become unavailable.

    Use recommended parameters from official documentation.

    Modify operating system configurations.

    Nodes can become unavailable.

    Attempt to restore the configuration or delete and recreate the node.

    Modify node time.

    Components on the node can malfunction.

    Restore the original node time.

    Add node computing resources using unsupported methods.

    ACK supports adding nodes through the console, OpenAPI, or CLI. See Add existing nodes. Nodes added by other methods are unrecognized—no lifecycle management, automated O&M, or technical support. See Why does the console show the node pool source as "Other nodes"?.

    Manage computing resources through node pools. If using unsupported methods, ensure compatibility between the node and cluster components (such as Kubernetes, networking, storage, and security).

    Master nodes (ACK dedicated clusters)

    Modify the security group of cluster nodes.

    Master nodes can become unavailable.

    Add nodes back to the auto-created security group. See Associate a security group with an instance (primary NIC).

    Node expiration or deletion.

    The master node becomes unavailable.

    Irreversible.

    Reinstall the operating system.

    Components on the master node are deleted.

    Irreversible.

    Manually upgrade master or etcd component versions.

    The cluster can become unusable.

    Roll back to the original version.

    Delete or format core directories such as /etc/kubernetes on the node.

    The master node becomes unavailable.

    Irreversible.

    Change the node IP address.

    The master node becomes unavailable.

    Restore the original IP address.

    Manually modify parameters of core components (such as etcd, kube-apiserver, or Docker).

    Master nodes can become unavailable.

    Use recommended parameters from official documentation.

    Manually replace master or etcd certificates.

    The cluster can become unusable.

    Irreversible.

    Manually add or remove master nodes.

    The cluster can become unusable.

    Irreversible.

    Modify node time.

    Components on the node can malfunction.

    Restore the original node time.

    Other

    Modify permissions or configurations through RAM.

    Cluster resources, such as SLB instances, can fail to be created.

    Restore the original permissions.
    Note

    Applies only to clusters earlier than version 1.26.

    Modify or delete preset PodSecurityPolicy resources, including the PodSecurityPolicy named ack.privileged, or ClusterRole, ClusterRoleBinding, Role, and RoleBinding resources prefixed with ack:podsecuritypolicy:.

    Core components can malfunction. Pod creation and updates can fail.

    Restore the related resources. See Configure or restore the default ACK Pod security policy.

    Node pool-related high-risk operations

    High-risk operation

    Impact

    Recovery solution

    Delete a scaling group.

    Node pool malfunctions.

    Irreversible. Recreate the node pool. See Create a node pool.

    Remove a node using kubectl.

    Node pool node count does not match the actual count.

    Remove the node through the ACK console or node pool APIs (see Remove a node) or adjust the desired node count to scale in (see Create and manage node pools).

    Directly release an ECS instance.

    The node pool details page may display incorrectly. Node pools with desired count enabled automatically scale out to restore it.

    Irreversible. Scale in by adjusting the desired node count through the ACK console or node pool APIs (see Create and manage node pools) or remove specific nodes (see Remove a node).

    Manually scale out or in a node pool with auto scaling enabled.

    The auto scaling component adjusts node count based on policies, causing unexpected results.

    Irreversible. Do not manually intervene in auto scaling node pools.

    Modify the max or min instance count of an ESS scaling group.

    Scaling can malfunction.

    • For node pools without auto scaling, reset the ESS scaling group's max and min instance counts to the defaults (2000 and 0).

    • For node pools with auto scaling, align the ESS scaling group's max and min instance counts with the node pool settings.

    Add existing nodes without backing up data.

    Data on the instance is lost.

    Irreversible.

    • Back up data before manually adding existing nodes.

    • Automatic node addition replaces the system disk. Back up system disk data beforehand.

    Store important data on the node's system disk.

    Node self-healing can reset configurations, causing system disk data loss.

    Irreversible. Store important data on additional data disks, cloud disks, NAS, or OSS.

    Virtual node-related high-risk operations

    High-risk operation

    Impact

    Recovery solution

    Uninstall the virtual node component.

    Serverless Pod management fails: existing ECI and ACS pods cannot be deleted, and new ones cannot be created.

    Reinstall the virtual node component.

    Network and Server Load Balancer-related high-risk operations

    High-risk operation

    Impact

    Recovery solution

    Set the kernel parameter net.ipv4.ip_forward=0.

    Network connectivity fails.

    Set the kernel parameter to net.ipv4.ip_forward=1.

    Modify kernel parameters:

    • net.ipv4.conf.all.rp_filter = 1|2

    • net.ipv4.conf.[ethX].rp_filter = 1|2

      Note

      ethX represents all NICs starting with eth.

    Network connectivity fails.

    Set kernel parameters to:

    • net.ipv4.conf.all.rp_filter = 0

    • net.ipv4.conf.[ethX].rp_filter = 0

    Set the kernel parameter net.ipv4.tcp_tw_reuse = 1.

    Pod health checks fail.

    Set the kernel parameter to net.ipv4.tcp_tw_reuse = 0.

    Set the kernel parameter net.ipv4.tcp_tw_recycle = 1.

    NAT malfunctions.

    Set the kernel parameter to net.ipv4.tcp_tw_recycle = 0.

    Modify the kernel parameter net.ipv4.ip_local_port_range.

    Intermittent network connectivity issues occur.

    Restore the kernel parameter to its default value: net.ipv4.ip_local_port_range="32768 60999".

    Install firewall software such as Firewalld or ufw.

    Container networking fails.

    Uninstall the firewall software and restart the node.

    Do not allow UDP port 53 for the container CIDR in the node security group.

    Cluster DNS fails.

    Configure the security group according to official recommendations.

    Modify or delete tags added by ACK to an SLB instance.

    SLB malfunctions.

    Restore the SLB tags.

    Modify ACK-managed SLB instances, listeners, or vServer groups through the SLB console.

    SLB malfunctions.

    Restore the SLB configuration.

    Remove the annotation for reusing an existing SLB from a Service: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id: ${YOUR_LB_ID}.

    SLB malfunctions.

    Add the SLB reuse annotation back to the Service.

    Note

    A Service reusing an existing SLB cannot switch to an auto-created SLB. Recreate the Service.

    Delete an ACK-created SLB through the SLB console.

    Cluster networking can fail.

    Delete the SLB by deleting the associated Service. For steps, see Delete a Service.

    Delete the nginx-ingress-lb Service in kube-system when Nginx Ingress Controller is installed.

    Ingress Controller malfunctions or crashes.

    Create a new Service with the same name using the following YAML.

    apiVersion: v1
kind: Service
metadata:
  annotations:
  labels:
    app: nginx-ingress-lb
  name: nginx-ingress-lb
  namespace: kube-system
spec:
  externalTrafficPolicy: Local
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: ingress-nginx
  type: LoadBalancer

    Add or modify the nameserver option in the DNS configuration file /etc/resolv.conf on ECS nodes.

    Misconfigured DNS servers can cause resolution failures, affecting cluster operations.

    To use a self-managed DNS server as upstream, configure it in CoreDNS. See Unmanaged CoreDNS configuration.

    Modify or delete elastic network interfaces (ENIs) or Lingjun ENIs created by ACK.

    Pod networking fails.

    Irreversible.

    Modify or delete network-related CRDs.

    podnetworkings.network.alibabacloud.com
podenis.network.alibabacloud.com
networkinterfaces.network.alibabacloud.com
nodes.network.alibabacloud.com
noderuntimes.network.alibabacloud.com
*.cilium.io
*.crd.projectcalico.org

    Terway fails. Severe cases can cause network or Pod failures.

    Irreversible.

    Create, modify, or delete network-related system CRs.

    podenis.network.alibabacloud.com
networkinterfaces.network.alibabacloud.com
nodes.network.alibabacloud.com
noderuntimes.network.alibabacloud.com
*.cilium.io
*.crd.projectcalico.org

    Terway fails. Severe cases can cause network or Pod failures.

    Delete custom CR definitions and recreate associated Pods.

    Modify non-configurable Terway network fields. See Custom Terway configuration parameters.

    Terway fails. Severe cases can cause network or Pod failures.

    Restore the original configuration and restart the node.

    Storage-related high-risk operations

    High-risk operation

    Impact

    Recovery solution

    Detach a cloud disk through the console.

    Pods report IO errors on writes.

    Restart the Pod and clean up residual mounts on the node.

    Run umount on the disk mount path on the node.

    Pods write to the local disk.

    Restart the Pod.

    Directly operate cloud disks on the node.

    Pods write to the local disk.

    Irreversible.

    Mount the same cloud disk to multiple Pods.

    Pods write to the local disk or report IO errors.

    Ensure one cloud disk is used by only one Pod.

    Important

    Cloud disks are non-shared storage, mountable to only one Pod at a time.

    Delete the NAS mount directory.

    Pods report IO errors on writes.

    Restart the Pod.

    Delete an in-use NAS file system or mount target.

    Pods experience IO hangs.

    Restart the ECS node. See Restart an ECS instance.

    Log-related high-risk operations

    High-risk operation

    Impact

    Recovery solution

    Delete the /tmp/ccs-log-collector/pos directory on the host.

    Logs are collected repeatedly.

    Irreversible. This directory records log collection positions.

    Delete the /tmp/ccs-log-collector/buffer directory on the host.

    Logs are lost.

    Irreversible. This directory stores cached logs awaiting processing.

    Delete aliyunlogconfig CRD resources.

    Log collection stops.

    Recreate the CRD and its resources. Logs during the outage are unrecoverable.

    Deleting the CRD removes all associated instances, which must be manually recreated after restoration.

    Delete log components.

    Log collection stops.

    Reinstall log components and manually restore aliyunlogconfig CRD instances. Logs during the outage are unrecoverable.

    Deleting log components removes the aliyunlogconfig CRD and Logtail collector, disabling log collection.