If the current CoreDNS version is outdated but the Upgrade button for CoreDNS is not displayed on the Add-ons page of the ACK console, you cannot configure the Container Service for Kubernetes (ACK) cluster to automatically update CoreDNS. In this scenario, you can manually update CoreDNS. This topic describes how to manually update CoreDNS.

Prerequisites

A kubectl client is connected to your cluster. For more information, see Connect to ACK clusters by using kubectl.

Precautions for updating CoreDNS

  • When ACK updates CoreDNS, ACK overwrites the YAML file of CoreDNS. ACK updates the coredns ConfigMap but does not change the number of CoreDNS pods.
  • If you modified the toleration, CPU and memory requests, or CPU and memory limits in the YAML file of CoreDNS, ACK overwrites the changes when it updates CoreDNS. To retain these changes in the YAML file of CoreDNS, you must manually update CoreDNS, or apply these changes to the YAML file of CoreDNS again after CoreDNS is automatically updated. For more information about how to manually update CoreDNS, see Manually update CoreDNS.
  • If the load balancing mode of kube-proxy is set to IP Virtual Server (IPVS), all DNS queries within the cluster may fail or time out after CoreDNS is updated. This situation lasts about 5 minutes. You can use one of the following methods to avoid this issue:
  • ACK requires about 2 minutes to update CoreDNS. The time consumption varies based on the number of CoreDNS pods. If the newly provisioned CoreDNS pods cannot be scheduled or launched, Submit a ticket. When ACK updates CoreDNS, ACK does not stop the CoreDNS pods that are provisioned before CoreDNS is updated. Therefore, the update process does not interrupt DNS resolution services in the cluster. The update can be rolled back within 10 minutes after the update starts.

Manual update

  1. Check whether the Kubernetes version is compatible with the CoreDNS version.
    Check the Kubernetes version of the ACK cluster. Make sure that the Kubernetes version is compatible with CoreDNS 1.6.2. The following table describes the Kubernetes versions that are compatible with CoreDNS 1.6.2. Kubernetes 1.11, 1.12, 1.14, and 1.16 are compatible with CoreDNS 1.6.2.
    Item Compatible version
    Kubernetes 1.11 1.12 1.14 1.16
    CoreDNS 1.6.2 1.6.2 1.6.2 1.6.2

    To check the Kubernetes version of an ACK cluster, perform the following steps:

    1. Log on to the ACK console.
    2. In the left-side navigation pane of the ACK console, click Clusters.
    3. On the Clusters page, find the cluster that you want to manage and check the Kubernetes version in the Version column. version
  2. Check the CoreDNS version.
    • You can check the CoreDNS version in the ACK console.
      1. Log on to the Container Service for Kubernetes (ACK) console.

      2. In the left-side navigation pane of the ACK console, click Clusters.
      3. On the Clusters page, find the cluster that you want to manage. Then, click the name of the cluster or click Details in the Actions column.
      4. In the left-side navigation pane of the cluster details page, choose Workloads > Deployments.
      5. On the Deployments page, set Namespace to kube-system and check the CoreDNS version in the Image column. dns
    • You can also run the following kubectl command to query the CoreDNS version:
      kubectl get deployment coredns -n kube-system -o jsonpath="{.spec.template.spec.containers[0].image}"

      Expected output:

      registry-vpc.cn-hangzhou.aliyuncs.com/acs/coredns:1.3.1
  3. Modify the coredns ConfigMap.
    In CoreDNS 1.6.2, the proxy plug-in is replaced with the forward plug-in. You must replace proxy with forward in the coredns ConfigMap that belongs to the kube-system namespace.
    • You can modify the coredns ConfigMap in the ACK console.
      1. Log on to the Container Service for Kubernetes (ACK) console.

      2. In the left-side navigation pane of the ACK console, click Clusters.
      3. On the Clusters page, find the cluster that you want to manage. Then, click the name of the cluster or click Details in the Actions column.
      4. In the left-side navigation pane of the cluster details page, choose Configurations > ConfigMaps.
      5. In the upper part of the ConfigMap page, set Namespace to kube-system. Then, find the coredns ConfigMap and click Edit YAML in the Actions column.
      6. In the View in YAML panel, replace proxy with forward and click OK. forward
    • You can also run the following kubectl command to modify the coredns ConfigMap:
      # Modify the coredns ConfigMap. 
      kubectl edit configmap/coredns -n kube-system
      
      # Replace proxy with forward. 
      # Save the change and exit. 
  4. Print the log of a CoreDNS pod to check whether the new configuration is loaded. The pod requires about 30 seconds to complete hot loading.
    1. Run the following command to check whether the CoreDNS pods are running:
      kubectl get pods -n kube-system | grep coredns

      Expected output:

      coredns-78d4b8bd88-6****                           1/1     Running   0          9d
      coredns-78d4b8bd88-n****                           1/1     Running   0          9d
    2. Run the following command to print the log of a CoreDNS pod:
      kubectl logs coredns-78d4b8bd88-n**** -n kube-system

      Expected output:

      .:53
      [INFO] plugin/reload: Running configuration MD5 = 71c5f1ff539d304c630521f315dc****
      CoreDNS-1.6.7
      linux/amd64, go1.13.6, da7f65b
      [INFO] 127.0.0.1:48329 - 42313 "HINFO IN 1108347002237365533.450654176893960****. udp 57 false 512" NXDOMAIN qr,rd,ra 132 0.008874794s

      If the output contains plugin/reload, the new CoreDNS configuration is loaded.

  5. Change the image version of CoreDNS to V1.6.2.
    • Change the image version of CoreDNS in the ACK console.
      1. Log on to the Container Service for Kubernetes (ACK) console.

      2. In the left-side navigation pane of the ACK console, click Clusters.
      3. On the Clusters page, find the cluster that you want to manage. Then, click the name of the cluster or click Details in the Actions column.
      4. In the left-side navigation pane of the cluster details page, choose Workloads > Deployments.
      5. In the upper part of the Deployments page, set Namespace to kube-system. Then, find coredns and choose More > View in YAML in the Actions column.
      6. In the Edit YAML dialog box, change the version number in the image field to 1.6.2. Click Update. image
    • Change the image version by using kubectl.
      # Modify the coredns ConfigMap. 
      kubectl edit deployment/coredns -n kube-system
      
      # Change the image version to 1.6.2. 
      # Save the change and exit. 
  6. Verify the result.
    Run the following command to check whether all CoreDNS pods in the cluster are in the Running state:
    kubectl get pods -n kube-system | grep coredns
    Expected output:
    coredns-78d4b8bd88-6****                          1/1     Running   0          9d
    coredns-78d4b8bd88-n****                          1/1     Running   0          9d

Change the UDP timeout period in IPVS mode

If kube-proxy runs in IPVS mode, DNS resolution may fail in the first 5 minutes after CoreDNS is updated due to the session persistence policy of IPVS. You can use one of the following methods to reduce the timeout period of UDP sessions in IPVS mode to 10 seconds. This way, less DNS resolution errors occur after CoreDNS is updated. If applications that use UDP are deployed in your cluster, evaluate the impact on these applications before you update CoreDNS. You can also Submit a ticket to request technical support.
Note If kube-proxy does not run in IPVS mode, you do not need to change the timeout period of UDP sessions. For more information about how to check the load balancing mode of kube-proxy, see View basic information.

ACK clusters that run Kubernetes 1.18 or later

Use the ACK console

  1. Log on to the ACK console.
  2. In the left-side navigation pane of the ACK console, click Clusters.
  3. On the Clusters page, find the cluster that you want to manage and click the name of the cluster or click Details in the Actions column. The details page of the cluster appears.
  4. In the left-side navigation pane of the details page, choose Configurations > ConfigMaps.
  5. On the ConfigMap page, select the kube-system namespace, find the kube-proxy-worker ConfigMap, and then click Edit YAML in the Actions column.
  6. In the View in YAML panel, add udpTimeout: 10s to the ipvs field and click OK.
    apiVersion: v1
    data:
      config.conf: |
        apiVersion: kubeproxy.config.k8s.io/v1alpha1
        kind: KubeProxyConfiguration
        # Irrelevant fields are not shown. 
        mode: ipvs
        // If the ipvs field does not exist, you must add the field. 
        ipvs:
          udpTimeout: 10s
  7. Recreate all of the pods named kube-proxy-worker.
    1. In the left-side navigation pane of the details page, choose Workloads > DaemonSets.
    2. On the DaemonSets page, find and click kube-proxy-worker.
    3. On the Pods tab of the kube-proxy-worker page, select a pod and choose More > Delete in the Actions column. In the message that appears, click Confirm.
      Repeat the preceding steps to delete all of the pods. After you delete the pods, the system automatically recreates the pods.
  8. Check whether the timeout period of UDP sessions is changed.
    1. Run the following command to install ipvsadm.

      ipvsadm is a tool that you can use to manage IPVS. For more information, see ipvsadm.

      yum install -y ipvsadm
    2. Log on to an Elastic Compute Service (ECS) instance in your cluster and then run the following command to check the third value in the output:
      ipvsadm -L --timeout
      If the third value in the output is 10, the timeout period of UDP sessions is changed.
      Note After the timeout period of UDP sessions is changed, wait at least 5 minutes before you proceed.

Use the CLI

  1. Run the following command to modify the kube-proxy-worker ConfigMap:
    kubectl -n kube-system edit configmap kube-proxy-worker
  2. Add udpTimeout: 10s to the ipvs field of the kube-proxy-worker ConfigMap. Then, save the modification and exit.
    apiVersion: v1
    data:
      config.conf: |
        apiVersion: kubeproxy.config.k8s.io/v1alpha1
        kind: KubeProxyConfiguration
        # Irrelevant fields are not shown. 
        mode: ipvs
        // If the ipvs field does not exist, you must add the field. 
        ipvs:
          udpTimeout: 10s
  3. Run the following command to recreate all of the pods named kube-proxy-worker.
    1. Run the following command to query the pods:
      kubectl -n kube-system get pod -o wide | grep kube-proxy-worker
    2. Run the following command to delete all of the pods that are returned in the preceding step. Then, the system recreates the pods named kube-proxy-worker.
      kubectl -n kube-system delete pod <kube-proxy-worker-****>
      Note Replace <kube-proxy-worker-****> with the name of a pod that is returned in the preceding step.
  4. Check whether the timeout period of UDP sessions is changed.
    1. Run the following command to install ipvsadm.

      ipvsadm is a tool that you can use to manage IPVS. For more information, see ipvsadm.

      yum install -y ipvsadm
    2. Log on to an Elastic Compute Service (ECS) instance in your cluster and then run the following command to check the third value in the output:
      ipvsadm -L --timeout
      If the third value in the output is 10, the timeout period of UDP sessions is changed.
      Note After the timeout period of UDP sessions is changed, wait at least 5 minutes before you proceed.

ACK clusters that run Kubernetes 1.16 or earlier

kube-proxy in an ACK cluster that runs Kubernetes 1.16 or an earlier version does not support the udpTimeout parameter. To change the timeout period of UDP sessions, we recommend that you use Operation Orchestration Service (OOS) to run the following ipvsadm commands on all ECS instances in the cluster at the same time:
yum install -y ipvsadm
ipvsadm -L --timeout > /tmp/ipvsadm_timeout_old
ipvsadm --set 900 120 10
ipvsadm -L --timeout > /tmp/ipvsadm_timeout_new
diff /tmp/ipvsadm_timeout_old /tmp/ipvsadm_timeout_new

For more information about how to use OOS to manage multiple ECS instances at the same time, see Manage multiple instances.

What to do next

After CoreDNS is updated, you can optimize the configurations of CoreDNS as needed. For more information, see Properly configure CoreDNS.