Container Service for Kubernetes:DNS best practices

Monitor metrics

CoreDNS exposes health metrics, such as DNS resolution results, through a standard Prometheus interface to help detect errors on the CoreDNS server and even on upstream DNS servers.

Managed Service for Prometheus provides built-in metrics monitoring and alerting rules for CoreDNS. You can enable Prometheus and its dashboard features in the ACK console. For more information, see Monitor the CoreDNS component.

If you use a self-managed Prometheus instance to monitor your Kubernetes cluster, you can observe relevant metrics in Prometheus and set alerts for key metrics. For more information, see the official CoreDNS Prometheus documentation.

Logs

In the event of a DNS error, CoreDNS logs help you quickly diagnose the root cause. We recommend that you enable CoreDNS domain name resolution logging and Log Service collection. For more information, see Analyze and monitor CoreDNS logs.

Kubernetes event delivery

In CoreDNS v1.9.3.6-32932850-aliyun and later, you can enable the k8s_event plugin to deliver critical CoreDNS logs to the event center as Kubernetes events. For more information about the k8s_event plugin, see k8s_event.

This feature is enabled by default in new CoreDNS deployments. If you upgrade CoreDNS from an earlier version to v1.9.3.6-32932850-aliyun or later, you need to manually modify the configuration file to enable this feature.

1. Run the following command to open the CoreDNS configuration file:

   kubectl -n kube-system edit configmap/coredns

2. Add the kubeAPI and k8s_event plugins.

   apiVersion: v1
   data:
     Corefile: |
       .:53 {
           errors
           health {
               lameduck 15s
           }
   
           // Start of addition (ignore other differences)
           kubeapi
           k8s_event {
             level info error warning // Delivers critical logs of the info, error, and warning levels.
           }
           // End of addition
   
           kubernetes cluster.local in-addr.arpa ip6.arpa {
               pods verified
               fallthrough in-addr.arpa ip6.arpa
           }
           // ... (remaining configuration omitted)
       }

3. Check the status and logs of the CoreDNS pod. If the logs contain the word reload, the modification is successful.

Assess the load on CoreDNS

You can run a DNS stress test in your cluster to assess the load. Many open source tools, such as DNSPerf, can help you achieve this. If you cannot accurately assess the DNS load in your cluster, refer to the following recommendations.

• We recommend that you set the number of CoreDNS pods to at least 2 in all scenarios. The resource limit for a single pod must be at least 1 core and 1 GiB of memory.

• The DNS resolution QPS that CoreDNS can provide is positively correlated with CPU consumption. With NodeLocal DNSCache, each CPU core can typically support over 10,000 QPS. The QPS requirements for DNS requests vary significantly among different types of services. You can observe the peak CPU usage of each CoreDNS pod. If CPU consumption exceeds one core during peak hours, scale out CoreDNS replicas. If you cannot determine peak CPU usage, use a conservative 1:8 replica-to-node ratio.

Adjust CoreDNS replicas

The number of CoreDNS replicas directly determines the computing resources that CoreDNS can use. You can adjust the number of CoreDNS replicas based on your assessment.

• Automatically adjust based on the recommended policy

  You can deploy the following dns-autoscaler. It automatically adjusts the number of CoreDNS replicas in real time based on the recommended 1:8 replica-to-node ratio. The number of replicas is calculated by using the following formula: replicas = max(ceil(cores × 1/coresPerReplica), ceil(nodes × 1/nodesPerReplica)). The result is also constrained by the max and min values.

  dns-autoscaler

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: dns-autoscaler
    namespace: kube-system
    labels:
      k8s-app: dns-autoscaler
  spec:
    selector:
      matchLabels:
        k8s-app: dns-autoscaler
    template:
      metadata:
        labels:
          k8s-app: dns-autoscaler
      spec:
        serviceAccountName: admin
        containers:
        - name: autoscaler
          image: registry.cn-hangzhou.aliyuncs.com/acs/cluster-proportional-autoscaler:1.8.4
          resources:
            requests:
              cpu: "200m"
              memory: "150Mi"
          command:
          - /cluster-proportional-autoscaler
          - --namespace=kube-system
          - --configmap=dns-autoscaler
          - --nodelabels=type!=virtual-kubelet
          - --target=Deployment/coredns
          - --default-params={"linear":{"coresPerReplica":64,"nodesPerReplica":8,"min":2,"max":100,"preventSinglePointFailure":true}}
          - --logtostderr=true
          - --v=9

• Manually adjust

  You can run the following command to manually adjust the number of CoreDNS replicas:

  kubectl scale --replicas={target} deployment/coredns -n kube-system # Replace {target} with the desired number of replicas.

• Do not use workload auto-scaling

  While workload auto-scaling mechanisms, such as Horizontal Pod Autoscaler (HPA) and CronHPA, can also adjust the number of replicas, they perform frequent scaling. Because scaling in can cause resolution errors, do not use workload auto-scaling for CoreDNS.

Adjust CoreDNS pod specifications

You can also adjust CoreDNS resources by modifying pod specifications. In an ACK managed Pro cluster, the default memory limit for CoreDNS pods is 2 GiB, with no CPU limit. For consistent performance, we recommend a CPU limit of at least 1024m. You can adjust these resource requests and limits in the console.

Schedule CoreDNS pods

We recommend that you deploy CoreDNS pods in different availability zones and on different cluster nodes to prevent single points of failure. CoreDNS add-on versions before v1.8.4.3 have a default preferred node anti-affinity, which may cause pods to be deployed on the same node if resources are insufficient. If this occurs, delete the pods to trigger rescheduling or upgrade the add-on to the latest version. CoreDNS add-on versions earlier than v1.8 are no longer maintained. Upgrade to the latest version as soon as possible.

Ensure that the nodes running CoreDNS are not saturated with high CPU or memory usage, as this affects the QPS and response latency of DNS resolution. If cluster node resources permit, consider using custom parameters to schedule CoreDNS to independent cluster nodes to provide stable DNS resolution services.

Disable affinity for the kube-dns service

Disable the autopath plugin

The autopath plugin, enabled in some early CoreDNS versions, can cause resolution errors in edge cases. Check if the plugin is enabled and disable it by editing the configuration file. For more information, see Autopath.

1. Run the kubectl -n kube-system edit configmap coredns command to open the CoreDNS configuration file.

2. Delete the autopath @kubernetes line, and then save and exit.

3. Check the status and logs of the CoreDNS pod. If the logs contain the word reload, the modification is successful.

Configure graceful shutdown for CoreDNS

lameduck is a graceful shutdown mechanism in CoreDNS. It ensures that when CoreDNS is stopped or restarted, ongoing requests are completed properly and are not abruptly interrupted. The lameduck mechanism works as follows:

• When the CoreDNS process is about to terminate, it enters lameduck mode.

• In lameduck mode, CoreDNS stops accepting new requests but continues to process requests it has already received until all requests are completed or the lameduck timeout period is exceeded.

Configure the forward plugin protocol

When using NodeLocal DNSCache, the communication chain is: Application → NodeLocal DNSCache (TCP) → CoreDNS (TCP). By default, CoreDNS then communicates with upstream DNS servers by using the same protocol as the source request. This means external domain name resolution requests from your workloads pass through NodeLocal DNSCache and CoreDNS, and are then sent to the VPC DNS servers (the two IP addresses 100.100.2.136 and 100.100.2.138 that are configured by default on ECS instances) over TCP.

# Before modification
forward . /etc/resolv.conf
# After modification
forward . /etc/resolv.conf {
  prefer_udp
}

Configure the ready plugin

CoreDNS versions 1.5.0 and later require the ready plugin to be configured to enable readiness probes.

1. Run the following command to open the CoreDNS configuration file:

   kubectl -n kube-system edit configmap/coredns

2. Check whether the file contains a line with ready. If not, add the line ready, press the Esc key, enter :wq!, and then press Enter to save the modified configuration file and exit edit mode.

   apiVersion: v1
   data:
    Corefile: |
     .:53 {
       errors
       health {
         lameduck 15s
       }
       ready # If this line does not exist, add it. Ensure that the indentation is consistent with Kubernetes.
       kubernetes cluster.local in-addr.arpa ip6.arpa {
         pods verified
         fallthrough in-addr.arpa ip6.arpa
       }
       prometheus :9153
       forward . /etc/resolv.conf {
         max_concurrent 1000
               prefer_udp
       }
       cache 30
       loop
       log
       reload
       loadbalance
     }

3. Check the status and logs of the CoreDNS pod. If the logs contain the word reload, the modification is successful.

Configure the multisocket plugin

CoreDNS v1.12.1 introduced the multisocket plugin. You can enable this plugin to allow CoreDNS to listen on the same port by using multiple sockets, which improves CoreDNS performance in high-CPU scenarios. For more information about the plugin, see the community documentation.

You need to enable multisocket in the coredns ConfigMap:

.:53 {
        ...
        prometheus :9153
        multisocket [NUM_SOCKETS]
        forward . /etc/resolv.conf
        ...
}

NUM_SOCKETS determines the number of sockets that listen on the same port.

Configuration recommendation: Align NUM_SOCKETS with the estimated CPU utilization, CPU resource limits, and available cluster resources, for example:

• If CoreDNS consumes 4 cores during peak hours and the available resources are 8 cores, set NUM_SOCKETS to 2.

• If CoreDNS consumes 8 cores during peak hours and 64 cores are available, set NUM_SOCKETS to 8.

To determine the optimal configuration, we recommend that you test the QPS and load with different configurations.

If you do not specify NUM_SOCKETS, the default value is GOMAXPROCS, which is equal to the CPU limit of the CoreDNS pod. If the CPU limit of the pod is not set, the value is equal to the number of CPU cores on the node where the pod is running.