Container Service for Kubernetes:Automatic upgrade for unmanaged CoreDNS

How the upgrade works

ACK upgrades CoreDNS using a rolling update. New Pods must be running before legacy Pods are deleted, so the replica count stays the same throughout. The upgrade uses graceful termination — legacy replicas are not stopped immediately. The process takes about 2 minutes, though the actual time varies based on the number of CoreDNS replicas. If the upgrade fails, ACK automatically rolls back within 10 minutes.

During the rolling update, legacy Pods may still be handling DNS resolution requests when new Pods start. To protect DNS availability throughout the upgrade, use the NodeLocal DNSCache component. For more information, see Use the NodeLocal DNSCache component.

Back up custom configurations

Automatic upgrades overwrite YAML template customizations, including tolerations, memory and CPU resource requests, and limits. Before upgrading, back up your current CoreDNS Deployment configuration:

kubectl get deployment coredns -n kube-system -o yaml > coredns-backup.yaml

After the upgrade, manually re-apply your customizations. For more information, see Manually update unmanaged CoreDNS.

Enable the ready plugin

If you previously manually upgraded CoreDNS to a version later than 1.5.0, verify that the ready plugin is enabled in the Corefile. Without it, CoreDNS fails to start during an automatic upgrade.

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Clusters page, click the cluster name. In the left navigation pane, choose Configurations > ConfigMaps.

3. On the ConfigMaps page, set Namespace to kube-system. Find coredns and click Edit YAML in the Actions column.

4. In the Edit YAML panel, check for the ready field. If it is missing, add ready and click OK. Make sure the indentation is consistent with the kubernetes block.

   apiVersion: v1
   data:
    Corefile: |
     .:53 {
       errors
       health {
         lameduck 15s
       }
       ready # If this line does not exist, add it. Make sure the indent is consistent with kubernetes.
       kubernetes cluster.local in-addr.arpa ip6.arpa {
         pods verified
         fallthrough in-addr.arpa ip6.arpa
       }
       prometheus :9153
       forward . /etc/resolv.conf {
         max_concurrent 1000
       }
       cache 30
       loop
       log
       reload
       loadbalance
     }

5. Verify that the new configuration is loaded. Run the following command and check the output for plugin/reload, which confirms the configuration was hot-reloaded (takes about 30 seconds):

   kubectl logs coredns-78d4b8bd88-n6wjm -n kube-system

Mitigate IPVS-related DNS failures

If your cluster uses kube-proxy in IPVS mode, the IPVS session persistence policy may cause cluster-wide DNS resolution timeouts or failures for up to five minutes after the upgrade. Use one of the following options to mitigate this:

• Modify the IPVS UDP session persistence timeout in kube-proxy. For more information, see How do I modify the IPVS UDP session persistence timeout in kube-proxy?

• Use NodeLocal DNSCache. For more information, see Improve stability with NodeLocal DNSCache.

• Upgrade the node kernel (Alibaba Cloud Linux 2 only). Upgrade to kernel version 4.19.91-25.1.al7.x86_64 or later. For more information, see Alibaba Cloud Linux 2 image release notes.

• Configure the IPVS UDP timeout on other operating systems. See Configure the UDP timeout for an IPVS cluster in this topic.

• Connect all application containers to NodeLocal DNSCache before upgrading. For more information, see Use the NodeLocal DNSCache component.

To check whether your cluster uses IPVS mode, see View cluster information.

Kubernetes 1.18 or later

Using the console

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Clusters page, click the cluster name. In the left navigation pane, choose Configurations > ConfigMaps.

3. On the ConfigMaps page, select the kube-system namespace. Find kube-proxy-worker and click Edit YAML in the Actions column.

4. In the Edit YAML panel, add udpTimeout: 10s under the ipvs field and click OK.

   apiVersion: v1
   data:
     config.conf: |
       apiVersion: kubeproxy.config.k8s.io/v1alpha1
       kind: KubeProxyConfiguration
       # Other irrelevant fields are omitted.
       mode: ipvs
       # If the ipvs key does not exist, add it.
       ipvs:
         udpTimeout: 10s

5. Recreate all kube-proxy-worker Pods.

   1. On the cluster details page, in the left navigation pane, choose Workloads > DaemonSets.

   2. In the DaemonSet list, click kube-proxy-worker.

   3. On the kube-proxy-worker page, click the Pods tab. In the row of a Pod, choose More > Delete, and then click OK. Repeat this step for all Pods. The system recreates them automatically.

6. Verify that the UDP timeout is configured.

   1. Install ipvsadm:

      sudo yum install -y ipvsadm

   2. Run the following command on any ECS node in the cluster:

      sudo ipvsadm -L --timeout

      If the third number in the output is 10, the UDP timeout is set correctly.

      Important: After the timeout is configured, wait at least five minutes before proceeding with the CoreDNS upgrade.

Using the command line

1. Edit the kube-proxy-worker ConfigMap:

   kubectl -n kube-system edit configmap kube-proxy-worker

2. Add udpTimeout: 10s under the ipvs field. Save and exit.

   apiVersion: v1
   data:
     config.conf: |
       apiVersion: kubeproxy.config.k8s.io/v1alpha1
       kind: KubeProxyConfiguration
       # Other irrelevant fields are omitted.
       mode: ipvs
       # If the ipvs key does not exist, add it.
       ipvs:
         udpTimeout: 10s

3. Recreate all kube-proxy-worker Pods.

   1. List the current Pods:

      kubectl -n kube-system get pod -o wide | grep kube-proxy-worker

   2. Delete each Pod. The system recreates them automatically.

      kubectl -n kube-system delete pod <kube-proxy-worker-****>

      Replace <kube-proxy-worker-****> with the actual Pod names from the previous step.

4. Verify that the UDP timeout is configured.

   1. Install ipvsadm:

      sudo yum install -y ipvsadm

   2. Run the following command on any ECS node in the cluster:

      sudo ipvsadm -L --timeout

      If the third number in the output is 10, the UDP timeout is set correctly.

      Important: After the timeout is configured, wait at least five minutes before proceeding with the CoreDNS upgrade.

Kubernetes 1.16 or earlier

kube-proxy in clusters running Kubernetes 1.16 or earlier does not support the udpTimeout parameter. Use Operation Orchestration Service (OOS) to run the following ipvsadm commands in batches on all cluster nodes:

sudo yum install -y ipvsadm
sudo ipvsadm -L --timeout > /tmp/ipvsadm_timeout_old
sudo ipvsadm --set 900 120 10
sudo ipvsadm -L --timeout > /tmp/ipvsadm_timeout_new
diff /tmp/ipvsadm_timeout_old /tmp/ipvsadm_timeout_new

For more information about running batch operations in OOS, see Batch operation instances.