Container Service for Kubernetes:Best practices for DNS in Kubernetes clusters

Monitoring metrics

CoreDNS exposes health metrics such as resolution results through a standard Prometheus interface to help detect anomalies in CoreDNS or upstream DNS servers.

Prometheus for ACK includes built-in CoreDNS monitoring metrics and alerting rules. Enable Prometheus and Dashboard in the Container Service for Kubernetes console. See CoreDNS component monitoring.

With a self-managed Prometheus instance, observe relevant metrics and set alerts for critical ones. See the official CoreDNS Prometheus documentation.

Operational logs

Use CoreDNS logs to diagnose root causes when DNS anomalies occur. Enable domain resolution logs and SLS log collection. See Analyze and monitor CoreDNS logs.

Kubernetes event delivery

In CoreDNS v1.9.3.6-32932850-aliyun and later, enable the k8s_event plugin to deliver critical CoreDNS logs as Kubernetes events to the Event Hub. See the k8s_event plugin documentation.

Newly deployed CoreDNS instances have this feature enabled by default. When upgrading from an older version, manually modify the configuration file to enable it.

1. Open the CoreDNS configuration file:

   kubectl -n kube-system edit configmap/coredns

2. Add the kubeAPI and k8s_event plugins.

   apiVersion: v1
   data:
     Corefile: |
       .:53 {
           errors
           health {
               lameduck 15s
           }
           // Begin addition (ignore other differences).
           kubeapi
           k8s_event {
             level info error warning // Deliver critical logs with info, error, or warning levels.
           }
           // End addition.
           kubernetes cluster.local in-addr.arpa ip6.arpa {
               pods verified
               fallthrough in-addr.arpa ip6.arpa
           }
           // Omitted below.
       }

3. Check the CoreDNS pod status and logs. If the logs contain the word reload, the modification is successful.

Assess CoreDNS component pressure

Run DNS stress tests to evaluate CoreDNS add-on pressure. Open-source tools such as DNSPerf can help. If you cannot accurately assess DNS pressure, follow these recommendations:

• Deploy at least two CoreDNS pods, each with resource limits of at least 1 CPU core and 1 GB of memory.

• CoreDNS QPS capacity correlates with CPU usage. With NodeLocal DNSCache, each CPU core supports more than 10,000 QPS. Business workload DNS QPS varies significantly. Monitor peak CPU usage per CoreDNS pod and scale out replicas when usage exceeds one core during peak periods. If peak usage is unknown, deploy one CoreDNS pod per eight cluster nodes as a conservative measure.

Adjust the number of CoreDNS pods

The number of CoreDNS pods directly determines available compute resources. Adjust based on your assessment.

• Automatically adjust the number of pods based on recommended policies

  Deploy the following dns-autoscaler. It adjusts CoreDNS pod count in real time at a ratio of one pod per eight cluster nodes. Replica count: `replicas = max(ceil(cores × 1/coresPerReplica), ceil(nodes × 1/nodesPerReplica))`, constrained by max and min limits.

  dns-autoscaler

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: dns-autoscaler
    namespace: kube-system
    labels:
      k8s-app: dns-autoscaler
  spec:
    selector:
      matchLabels:
        k8s-app: dns-autoscaler
    template:
      metadata:
        labels:
          k8s-app: dns-autoscaler
      spec:
        serviceAccountName: admin
        containers:
        - name: autoscaler
          image: registry.cn-hangzhou.aliyuncs.com/acs/cluster-proportional-autoscaler:1.8.4
          resources:
            requests:
              cpu: "200m"
              memory: "150Mi"
          command:
          - /cluster-proportional-autoscaler
          - --namespace=kube-system
          - --configmap=dns-autoscaler
          - --nodelabels=type!=virtual-kubelet
          - --target=Deployment/coredns
          - --default-params={"linear":{"coresPerReplica":64,"nodesPerReplica":8,"min":2,"max":100,"preventSinglePointFailure":true}}
          - --logtostderr=true
          - --v=9

• Manual adjustments

  Manually adjust CoreDNS pod count:

  kubectl scale --replicas={target} deployment/coredns -n kube-system # Replace {target} with the desired pod count

• Do not use workload autoscaling

  Workload autoscaling mechanisms such as horizontal pod autoscaling (HPA) or CronHPA trigger frequent scaling operations. Due to resolution anomalies during pod scale-in, do not use workload autoscaling for CoreDNS pods.

Adjust CoreDNS pod specifications

Modify pod specifications to adjust CoreDNS resources. In ACK managed clusters Pro, CoreDNS pods have a default memory limit of 2Gi and no CPU limit. Set the CPU limit to 4096m, with a minimum of 1024m. Adjust CoreDNS pod configuration in the console.

Schedule CoreDNS pods

Deploy CoreDNS pods across zones and nodes to avoid single-node or single-zone failures. CoreDNS versions earlier than v1.8.4.3 use weak node anti-affinity by default, which may place all pods on the same node when resources are insufficient. Delete pods to trigger rescheduling or upgrade to the latest component version. CoreDNS versions earlier than v1.8 are no longer maintained. Upgrade as soon as possible.

Avoid deploying CoreDNS on nodes with full CPU or memory utilization, which affects DNS QPS and response latency. Use custom parameters to schedule CoreDNS on dedicated nodes for stable domain name resolution.

Disable affinity settings for kube-dns service

Disable the Autopath plugin

Some older CoreDNS versions have the Autopath plugin enabled, which may produce incorrect resolution results in extreme scenarios. Verify whether the plugin is enabled and disable it by editing the configuration file. See Autopath.

1. Open the CoreDNS configuration file with kubectl -n kube-system edit configmap coredns.

2. Delete the autopath @kubernetes line and save the change.

3. Check the CoreDNS pod status and logs. If the logs contain the word reload, the modification is successful.

Configure graceful shutdown for CoreDNS

The lameduck mechanism in CoreDNS enables graceful shutdown. When CoreDNS stops or restarts, ongoing requests complete normally instead of being abruptly interrupted. The lameduck mechanism works as follows:

• When CoreDNS terminates, it enters Lameduck mode.

• In lameduck mode, CoreDNS stops accepting new requests but continues processing existing requests until they complete or the lameduck timeout expires.

Set the default protocol for the Forward plugin to communicate with upstream VPC DNS servers

NodeLocal DNSCache communicates with CoreDNS using TCP. CoreDNS uses the same protocol as incoming requests when communicating with upstream DNS servers. By default, external domain resolution requests from application containers pass through NodeLocal DNSCache and CoreDNS, then reach VPC DNS servers (100.100.2.136 and 100.100.2.138) over TCP.

# Before modification
forward . /etc/resolv.conf
# After modification
forward . /etc/resolv.conf {
  prefer_udp
}

Configure the Ready readiness probe plugin

CoreDNS versions later than 1.5.0 require the ready plugin to enable readiness probes.

1. Open the CoreDNS configuration file:

   kubectl -n kube-system edit configmap/coredns

2. Check for the ready line. If it is missing, add ready. Press Esc, type :wq!, and then press Enter to save the change and exit.

   apiVersion: v1
   data:
    Corefile: |
     .:53 {
       errors
       health {
         lameduck 15s
       }
       ready # Add this line if missing, ensuring consistent indentation with Kubernetes.
       kubernetes cluster.local in-addr.arpa ip6.arpa {
         pods verified
         fallthrough in-addr.arpa ip6.arpa
       }
       prometheus :9153
       forward . /etc/resolv.conf {
         max_concurrent 1000
               prefer_udp
       }
       cache 30
       loop
       log
       reload
       loadbalance
     }

3. Check the CoreDNS pod status and logs. If the logs contain the word reload, the modification is successful.

Configure the multisocket plugin to enhance CoreDNS resolution performance

CoreDNS v1.12.1 introduced the multisocket plugin. When enabled, CoreDNS uses multiple sockets to listen on the same port, enhancing performance in high-CPU scenarios. See the community documentation.

Enable multisocket in the coredns ConfigMap:

.:53 {
        ...
        prometheus :9153
        multisocket [NUM_SOCKETS]
        forward . /etc/resolv.conf
        ...
}

NUM_SOCKETS specifies the number of sockets that listen on the same port.

Align NUM_SOCKETS with estimated CPU usage, CPU resource limits, and available cluster resources. For example:

• If CoreDNS consumes 4 cores at its peak and 8 cores are available, set NUM_SOCKETS to 2.

• If CoreDNS consumes 8 cores at its peak and 64 cores are available, set NUM_SOCKETS to 8.

Test different settings and measure QPS and load to determine the optimal configuration.

If you do not specify NUM_SOCKETS, CoreDNS uses GOMAXPROCS by default, which equals the CoreDNS pod CPU limit or the node CPU core count when no limit is set.