Container Service for Kubernetes:Automatically upgrade unmanaged CoreDNS

How it works

ACK upgrades CoreDNS with a rolling update. New Pods start before legacy Pods are deleted, so replica count stays constant. Legacy replicas terminate gracefully instead of stopping immediately. The process takes about 2 minutes, depending on CoreDNS replica count. If the upgrade fails, ACK automatically rolls back within 10 minutes.

During the rolling update, legacy Pods may still handle DNS requests when new Pods start. To protect DNS availability, use the NodeLocal DNSCache component.

Back up custom configurations

Automatic upgrades overwrite YAML template customizations, including tolerations and CPU and memory requests and limits. Back up your CoreDNS Deployment configuration before upgrading:

kubectl get deployment coredns -n kube-system -o yaml > coredns-backup.yaml

After the upgrade, manually re-apply your customizations.

Enable the ready plugin

If you manually upgraded CoreDNS to a version later than 1.5.0, verify the ready plugin is enabled in the Corefile. Without it, CoreDNS fails to start during an automatic upgrade.

1. Log on to the ACK console and click Clusters in the left navigation pane.

2. On the Clusters page, click the cluster name. Choose Configurations > ConfigMaps in the left navigation pane.

3. On the ConfigMaps page, set Namespace to kube-system, find coredns, and click Edit YAML in the Actions column.

4. In the Edit YAML panel, check for the ready field. If it is missing, add ready and click OK. Match the indentation of the kubernetes block.

   apiVersion: v1
   data:
    Corefile: |
     .:53 {
       errors
       health {
         lameduck 15s
       }
       ready # If this line does not exist, add it. Make sure the indent is consistent with kubernetes.
       kubernetes cluster.local in-addr.arpa ip6.arpa {
         pods verified
         fallthrough in-addr.arpa ip6.arpa
       }
       prometheus :9153
       forward . /etc/resolv.conf {
         max_concurrent 1000
       }
       cache 30
       loop
       log
       reload
       loadbalance
     }

5. Verify the new configuration is loaded. Check the output for plugin/reload, which confirms hot reload (takes about 30 seconds):

   kubectl logs coredns-78d4b8bd88-n6wjm -n kube-system

Mitigate IPVS-related DNS failures

If your cluster uses kube-proxy in IPVS mode, IPVS session persistence may cause cluster-wide DNS timeouts or failures for up to five minutes after the upgrade. Use one of the following options:

• Modify the IPVS UDP session persistence timeout in kube-proxy. See How do I modify the IPVS UDP session persistence timeout in kube-proxy?

• Use NodeLocal DNSCache. See Improve stability with NodeLocal DNSCache.

• Upgrade the node kernel (Alibaba Cloud Linux 2 only). Upgrade to kernel version 4.19.91-25.1.al7.x86_64 or later. See Alibaba Cloud Linux 2 image release notes.

• Configure the IPVS UDP timeout on other operating systems. See Configure the UDP timeout for an IPVS cluster below.

• Connect all application containers to NodeLocal DNSCache before upgrading. See Use the NodeLocal DNSCache component.

See View cluster information to check whether your cluster uses IPVS mode.

Kubernetes 1.18 or later

Using the console

1. Log on to the ACK console and click Clusters in the left navigation pane.

2. On the Clusters page, click the cluster name. Choose Configurations > ConfigMaps in the left navigation pane.

3. On the ConfigMaps page, select the kube-system namespace, find kube-proxy-worker, and click Edit YAML in the Actions column.

4. In the Edit YAML panel, add udpTimeout: 10s under the ipvs field and click OK.

   apiVersion: v1
   data:
     config.conf: |
       apiVersion: kubeproxy.config.k8s.io/v1alpha1
       kind: KubeProxyConfiguration
       # Other irrelevant fields are omitted.
       mode: ipvs
       # If the ipvs key does not exist, add it.
       ipvs:
         udpTimeout: 10s

5. Recreate all kube-proxy-worker Pods.

   1. On the cluster details page, choose Workloads > DaemonSets in the left navigation pane.

   2. In the DaemonSet list, click kube-proxy-worker.

   3. On the kube-proxy-worker page, click the Pods tab. For each Pod, choose More > Delete and click OK. The system recreates them automatically.

6. Verify the UDP timeout configuration.

   1. Install ipvsadm:

      sudo yum install -y ipvsadm

   2. On any ECS node in the cluster, run:

      sudo ipvsadm -L --timeout

      If the third number in the output is 10, the UDP timeout is set correctly.

      Important: After you configure the timeout, wait at least five minutes before upgrading CoreDNS.

Using the command line

1. Edit the kube-proxy-worker ConfigMap:

   kubectl -n kube-system edit configmap kube-proxy-worker

2. Add udpTimeout: 10s under the ipvs field, then save and exit.

   apiVersion: v1
   data:
     config.conf: |
       apiVersion: kubeproxy.config.k8s.io/v1alpha1
       kind: KubeProxyConfiguration
       # Other irrelevant fields are omitted.
       mode: ipvs
       # If the ipvs key does not exist, add it.
       ipvs:
         udpTimeout: 10s

3. Recreate all kube-proxy-worker Pods.

   1. List the kube-proxy-worker Pods:

      kubectl -n kube-system get pod -o wide | grep kube-proxy-worker

   2. Delete each Pod. The system recreates them automatically.

      kubectl -n kube-system delete pod <kube-proxy-worker-****>

      Replace <kube-proxy-worker-****> with the Pod names from the previous step.

4. Verify the UDP timeout configuration.

   1. Install ipvsadm:

      sudo yum install -y ipvsadm

   2. On any ECS node in the cluster, run:

      sudo ipvsadm -L --timeout

      If the third number in the output is 10, the UDP timeout is set correctly.

      Important: After you configure the timeout, wait at least five minutes before upgrading CoreDNS.

Kubernetes 1.16 or earlier

kube-proxy in clusters running Kubernetes 1.16 or earlier does not support the udpTimeout parameter. Use Operation Orchestration Service (OOS) to run these ipvsadm commands in batches on all cluster nodes:

sudo yum install -y ipvsadm
sudo ipvsadm -L --timeout > /tmp/ipvsadm_timeout_old
sudo ipvsadm --set 900 120 10
sudo ipvsadm -L --timeout > /tmp/ipvsadm_timeout_new
diff /tmp/ipvsadm_timeout_old /tmp/ipvsadm_timeout_new

See Batch operation instances for running batch operations in OOS.