Container Service for Kubernetes:DNS best practices

Metrics

CoreDNS exposes health metrics, such as resolution results, through a standard Prometheus interface to detect exceptions on the CoreDNS server and even upstream DNS servers.

Prometheus for Alibaba Cloud provides built-in metrics monitoring and alert rules for CoreDNS. You can enable the Prometheus and dashboard features in the Container Service for Kubernetes (ACK) console. For more information, see Monitor the CoreDNS component.

If you use a self-managed Prometheus instance to monitor your Kubernetes cluster, you can observe relevant metrics in Prometheus and set alerts for key metrics. For more information, see the official CoreDNS Prometheus documentation.

Running logs

In the event of a DNS exception, CoreDNS logs can help you quickly diagnose the root cause. We recommend that you enable CoreDNS domain name resolution logs and their SLS log collection. For more information, see Analyze and monitor CoreDNS logs.

Kubernetes event delivery

In CoreDNS v1.9.3.6-32932850-aliyun and later, you can enable the k8s_event plugin to deliver key CoreDNS logs to Event Hub as Kubernetes events. For more information about the k8s_event plugin, see k8s_event.

This feature is enabled by default for newly deployed CoreDNS. If you upgrade from an earlier version of CoreDNS to v1.9.3.6-32932850-aliyun or later, you must manually modify the configuration file to enable this feature.

1. Run the following command to open the CoreDNS configuration file.

   kubectl -n kube-system edit configmap/coredns

2. Add the kubeAPI and k8s_event plugins.

   apiVersion: v1
   data:
     Corefile: |
       .:53 {
           errors
           health {
               lameduck 15s
           }
   
           // Add the following content (ignore other differences).
           kubeapi
           k8s_event {
             level info error warning // Deliver key logs with the info, error, and warning statuses.
           }
           // End of added content.
   
           kubernetes cluster.local in-addr.arpa ip6.arpa {
               pods verified
               fallthrough in-addr.arpa ip6.arpa
           }
           // The following content is omitted.
       }

3. Check the running status and logs of the CoreDNS pod. If the logs contain the word reload, the modification is successful.

Evaluate the pressure on the CoreDNS component

You can perform a DNS stress test in the cluster to evaluate the load on the component. Many open source tools, such as DNSPerf, can help you achieve this. If you cannot accurately evaluate the DNS load in the cluster, you can refer to the following recommended standards.

• We recommend that you set the number of CoreDNS pods to at least 2. The resource limit of a single pod should not be less than 1 core and 1 GB.

• The domain name resolution queries per second (QPS) that CoreDNS can provide is positively correlated with CPU consumption. If NodeLocal DNSCache is used, each CPU core can support more than 10,000 QPS for domain name resolution requests. The QPS requirements for domain name requests vary greatly among different types of services. You can observe the peak CPU usage of each CoreDNS pod. If it occupies more than one CPU core during peak business hours, we recommend that you scale out the CoreDNS replicas. If you cannot determine the peak CPU usage, you can conservatively deploy pods at a ratio of 1 pod to 8 cluster nodes. That is, for every 8 cluster nodes added, one CoreDNS pod is added.

Adjust the number of CoreDNS pods

The number of CoreDNS pods directly determines the computing resources that CoreDNS can use. You can adjust the number of CoreDNS pods based on the evaluation results.

• Automatically adjust based on the recommended policy

  You can deploy the dns-autoscaler. It automatically adjusts the number of CoreDNS pods based on a recommended policy, such as one pod for every eight cluster nodes. The number of replicas is calculated using the formula: replicas = max(ceil(cores × 1/coresPerReplica), ceil(nodes × 1/nodesPerReplica)), and is constrained by the max and min values.

  dns-autoscaler

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: dns-autoscaler
    namespace: kube-system
    labels:
      k8s-app: dns-autoscaler
  spec:
    selector:
      matchLabels:
        k8s-app: dns-autoscaler
    template:
      metadata:
        labels:
          k8s-app: dns-autoscaler
      spec:
        serviceAccountName: admin
        containers:
        - name: autoscaler
          image: registry.cn-hangzhou.aliyuncs.com/acs/cluster-proportional-autoscaler:1.8.4
          resources:
            requests:
              cpu: "200m"
              memory: "150Mi"
          command:
          - /cluster-proportional-autoscaler
          - --namespace=kube-system
          - --configmap=dns-autoscaler
          - --nodelabels=type!=virtual-kubelet
          - --target=Deployment/coredns
          - --default-params={"linear":{"coresPerReplica":64,"nodesPerReplica":8,"min":2,"max":100,"preventSinglePointFailure":true}}
          - --logtostderr=true
          - --v=9

• Manually adjust

  You can run the following command to manually adjust the number of CoreDNS pods.

  kubectl scale --replicas={target} deployment/coredns -n kube-system # Replace target with the target number of pods.

• Do not use workload autoscaling

  Although workload autoscaling, such as horizontal pod autoscaling (HPA) and CronHPA, can also automatically adjust the number of pods, they perform frequent scaling operations. Because of the resolution exceptions caused by pod scale-in mentioned earlier, do not use workload autoscaling to control the number of CoreDNS pods.

Adjust CoreDNS pod specifications

Another way to adjust CoreDNS resources is to adjust pod specifications. In an ACK Pro cluster, the default memory limit for a CoreDNS pod is 2 GiB, and there is no CPU limit. We recommend that you set the CPU limit to 4096m, and the minimum value should not be less than 1024m. You can adjust the CoreDNS pod configuration in the console.

Schedule CoreDNS pods

We recommend that you deploy CoreDNS pods on different cluster nodes in different zones to avoid single-node and single-zone failures. CoreDNS components earlier than v1.8.4.3 are configured with weak anti-affinity by node by default. This may cause some or all pods to be deployed on the same node because of insufficient node resources. If this occurs, you can delete the pods to trigger a scheduling adjustment, or upgrade the component to the latest version. CoreDNS component versions earlier than v1.8 are no longer maintained. We recommend that you upgrade them as soon as possible.

The cluster nodes where CoreDNS runs should not have full CPU and memory usage. Otherwise, the QPS and response latency of domain name resolution will be affected. If cluster node conditions permit, you can consider using custom parameters to schedule CoreDNS to independent cluster nodes to provide stable domain name resolution services.

Disable the affinity configuration of the kube-dns service

Disable the Autopath plugin

Some earlier versions of CoreDNS have the Autopath plugin enabled. This plugin may cause incorrect resolution results in some extreme scenarios. You should check whether it is enabled and edit the configuration file to disable it. For more information, see Autopath.

1. Run the kubectl -n kube-system edit configmap coredns command to open the CoreDNS configuration file.

2. Delete the autopath @kubernetes line and save and exit.

3. Check the running status and logs of the CoreDNS pod. If the logs contain the word reload, the modification is successful.

Configure graceful shutdown for CoreDNS

lameduck is a mechanism in CoreDNS that implements graceful shutdown. If CoreDNS needs to be stopped or restarted, this mechanism ensures that requests being processed can be completed normally without sudden interruption. The lameduck mechanism works as follows:

• If the CoreDNS process is about to terminate, it enters lameduck mode.

• In lameduck mode, CoreDNS stops accepting new requests but continues to process received requests until all requests are completed or the lameduck timeout period is exceeded.

Configure the default protocol for the forward plugin and upstream VPC DNS servers

NodeLocal DNSCache uses the TCP protocol to communicate with CoreDNS. CoreDNS communicates with upstream DNS servers using the protocol that is used by the source of the request. Therefore, by default, requests to resolve domain names outside the cluster from application containers pass through NodeLocal DNSCache and CoreDNS, and are finally sent over TCP to the VPC DNS servers. The VPC DNS servers are the two IP addresses, 100.100.2.136 and 100.100.2.138, that are configured by default on ECS instances.

# Before modification
forward . /etc/resolv.conf
# After modification
forward . /etc/resolv.conf {
  prefer_udp
}

Configure the ready plugin for readiness probes

For CoreDNS versions 1.5.0 and later, you must configure the ready plugin to enable readiness probes.

1. Run the following command to open the CoreDNS configuration file.

   kubectl -n kube-system edit configmap/coredns

2. Check whether the file contains the ready line. If not, add the ready line. Press the Esc key, enter :wq!, and then press Enter to save the modified configuration file and exit edit mode.

   apiVersion: v1
   data:
    Corefile: |
     .:53 {
       errors
       health {
         lameduck 15s
       }
       ready # If this line does not exist, add it. Note that the indent must be consistent with Kubernetes.
       kubernetes cluster.local in-addr.arpa ip6.arpa {
         pods verified
         fallthrough in-addr.arpa ip6.arpa
       }
       prometheus :9153
       forward . /etc/resolv.conf {
         max_concurrent 1000
               prefer_udp
       }
       cache 30
       loop
       log
       reload
       loadbalance
     }

3. Check the running status and logs of the CoreDNS pod. If the logs contain the word reload, the modification is successful.

Configure the multisocket plugin to enhance CoreDNS parsing performance

CoreDNS v1.12.1 introduced the multisocket plugin. Enabling this plugin allows CoreDNS to listen on the same port using multiple sockets, which enhances CoreDNS performance in scenarios with high CPU usage. For a detailed introduction to the plugin, see the community documentation.

You need to enable multisocket through the coredns ConfigMap:

.:53 {
        ...
        prometheus :9153
        multisocket [NUM_SOCKETS]
        forward . /etc/resolv.conf
        ...
}

NUM_SOCKETS determines the number of sockets listening on the same port.

Configuration recommendation: Align the value of NUM_SOCKETS with the estimated CPU utilization, CPU resource limits, and available cluster resources. For example:

• If CoreDNS consumes 4 cores at its peak and the available resources are 8 cores, set NUM_SOCKETS to 2.

• If CoreDNS consumes 8 cores at its peak and the available resources are 64 cores, set NUM_SOCKETS to 8.

To determine the optimal configuration, we recommend that you test the QPS and load with different configurations.

If you do not specify NUM_SOCKETS, GOMAXPROCS is used by default, which is equal to the CPU limit of the CoreDNS pod. If the pod CPU limit is not set, it is equal to the number of CPU cores on the node where the pod resides.