Container Service for Kubernetes:Best Practices for DNS services

Monitoring metrics

CoreDNS provides health metrics such as resolution results through standard Prometheus interfaces, which are essential for detecting exceptions on CoreDNS and even upstream DNS servers.

Alibaba Cloud Prometheus monitoring offers built-in metrics and alert rules for CoreDNS. You can activate the Prometheus and gauge features through the Container Service Management Console or the . For more information, see Monitor CoreDNS components.

If you use open-source Prometheus to monitor your Kubernetes cluster, you can access the related metrics in Prometheus and establish alert rules based on key metrics. For more information, see the official documentation of CoreDNS Prometheus.

Running logs

In the event of DNS resolution errors, you can examine the CoreDNS log to pinpoint the issues. It is advisable to activate the DNS resolution log and SLS log collection for CoreDNS. For more information, see Analyze and monitor CoreDNS logs.

Sink Kubernetes events

From CoreDNS v1.9.3.6-32932850-aliyun onwards, the k8s_event plug-in is available to sink Kubernetes events, including Info, Error, and Warning logs from CoreDNS, to the event center. For details about the k8s_event plug-in, see k8s_event.

By default, the k8s_event plug-in is activated upon deploying CoreDNS v1.9.3.6-32932850-aliyun or later. If you have updated from an earlier version to v1.9.3.6-32932850-aliyun or beyond, you will need to modify the CoreDNS ConfigMap to enable the k8s_event plug-in.

1. Execute the command below to access the CoreDNS ConfigMap.

   kubectl -n kube-system edit configmap/coredns

2. Incorporate the kubeAPI and k8s_event plug-ins.

   apiVersion: v1
   data:
     Corefile: |
       .:53 {
           errors
           health {
               lameduck 15s
           }
   
           // The start of the plug-in configuration (ignore other differences).
           kubeapi
           k8s_event {
             level info error warning // Report Info, Error, and Warning logs to Kubernetes events.
           }
           // The end of the plug-in configuration.
   
           kubernetes cluster.local in-addr.arpa ip6.arpa {
               pods verified
               fallthrough in-addr.arpa ip6.arpa
           }
           // Details are not shown.
       }

3. Verify the CoreDNS pod's running status and log. The presence of the term reload in the log indicates a successful modification.

Evaluate the load on CoreDNS components

Perform DNS stress testing within the cluster to assess component load. Open source tools, such as DNSPerf, can facilitate this process. If precise DNS load evaluation within the cluster is challenging, consider adhering to the following recommended standards:

• It is advisable to maintain a minimum of two CoreDNS pods under all circumstances, with each pod having at least 1 CPU core and 1 GB of memory allocated.

• The DNS queries per second (QPS) that CoreDNS can handle is directly proportional to CPU consumption. With NodeLocal DNSCache, each CPU core can support over 10,000 DNS QPS. Service types can greatly influence DNS QPS requirements. Monitor the peak CPU usage of CoreDNS pods; if usage exceeds one core during peak times, consider scaling out CoreDNS pods. If peak CPU usage is unknown, a conservative approach is to deploy CoreDNS pods at a ratio of one pod per eight cluster nodes, adding one CoreDNS pod for every eight additional nodes.

Adjust the number of CoreDNS pods

The number of CoreDNS pods is a direct indicator of the available computing resources for CoreDNS. Adjust the pod count based on load evaluation results.

• Automatically adjust the number of CoreDNS pods based on the recommended policy

  Deploy the dns-autoscaler to automatically adjust the number of CoreDNS pods in real-time according to the earlier mentioned policy of one pod per eight cluster nodes. The pod count is calculated using the formula: replicas = max(ceil(cores × 1/coresPerReplica), ceil(nodes × 1/nodesPerReplica)). The number of pods is constrained by the max and min values.

  dns-autoscaler

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: dns-autoscaler
    namespace: kube-system
    labels:
      k8s-app: dns-autoscaler
  spec:
    selector:
      matchLabels:
        k8s-app: dns-autoscaler
    template:
      metadata:
        labels:
          k8s-app: dns-autoscaler
      spec:
        serviceAccountName: admin
        containers:
        - name: autoscaler
          image: registry.cn-hangzhou.aliyuncs.com/acs/cluster-proportional-autoscaler:1.8.4
          resources:
            requests:
              cpu: "200m"
              memory: "150Mi"
          command:
          - /cluster-proportional-autoscaler
          - --namespace=kube-system
          - --configmap=dns-autoscaler
          - --nodelabels=type!=virtual-kubelet
          - --target=Deployment/coredns
          - --default-params={"linear":{"coresPerReplica":64,"nodesPerReplica":8,"min":2,"max":100,"preventSinglePointFailure":true}}
          - --logtostderr=true
          - --v=9

• Manually adjust the number of CoreDNS pods

  To manually change the number of CoreDNS pods, execute the following command:

  kubectl scale --replicas={target} deployment/coredns -n kube-system # Replace target with the number of target pods

• Avoid using workload auto scaling

  Workload auto scaling, such as horizontal pod autoscaling (HPA) and scheduled horizontal pod autoscaling (CronHPA), can lead to frequent scaling in and out of pods. As previously mentioned, this can cause DNS resolution issues during pod scale-in, so it's best to avoid using workload auto scaling for CoreDNS pod management.

Adjust the specifications of CoreDNS pods

An alternative way to modify CoreDNS resources involves altering pod specifications. In ACK Pro clusters, CoreDNS pods come with a default memory limit of 2Gi, while the CPU is unlimited. It is advisable to establish a CPU limit of 4096m, with the minimum threshold being 1024m. You can modify CoreDNS pod settings through the console.

Schedule CoreDNS pods

To mitigate the risk of single-node or single-zone failures, it is recommended to deploy CoreDNS pods across different zones and nodes. Versions of CoreDNS prior to v1.8.4.3 have default weak anti-affinity rules based on node allocation. If nodes are resource-constrained, pods may cluster on a single node. In such cases, delete the pods to trigger a rescheduling or upgrade the component to the latest version. Versions earlier than v1.8 are no longer supported, and an update is strongly recommended.

Avoid deploying CoreDNS pods on nodes with fully utilized CPU and memory resources to prevent adverse effects on DNS QPS and response times. If the cluster has enough nodes, you can schedule CoreDNS pods to dedicated nodes by setting custom parameters, ensuring stable domain name resolution services.

Disable the affinity settings of the kube-dns service

Disable the autopath plug-in

The autopath plug-in, enabled in earlier CoreDNS versions, can lead to DNS resolution failures in certain scenarios. To disable the autopath plug-in, update the coredns ConfigMap. For more information, see Autopath.

1. To open the coredns ConfigMap, run the kubectl -n kube-system edit configmap coredns command.

2. Remove the line containing autopath @kubernetes, save the changes, and exit.

3. Verify the CoreDNS pod's running status and examine the logs. If the term reload is present in the logs, this signifies a successful modification.

Configure graceful shutdown for CoreDNS

lameduck is a feature in CoreDNS designed to facilitate a graceful shutdown. It allows ongoing requests to complete without interruption during a CoreDNS stop or restart. The lameduck feature operates in the following way:

• Before termination, the CoreDNS process enters lameduck mode.

• In lameduck mode, CoreDNS ceases to accept new requests while still processing any received requests until they are all completed or the lameduck timeout period expires.

Configure the default protocol of the forward plug-in and upstream VPC DNS servers

NodeLocal DNSCache communicates with CoreDNS using TCP, and CoreDNS in turn communicates with upstream DNS servers based on the originating protocol of DNS queries. Thus, DNS queries from client pods for external domain names traverse NodeLocal DNSCache and CoreDNS before reaching VPC DNS servers over TCP. The VPC DNS servers, with IP addresses 100.100.2.136 and 100.100.2.138, are automatically configured on Elastic Compute Service (ECS) instances.

# The original setting
forward . /etc/resolv.conf
# The modified setting
forward . /etc/resolv.conf {
  prefer_udp
}

Configure the ready plug-in

The ready plug-in must be configured for CoreDNS versions 1.5.0 and above.

1. To open the coredns ConfigMap, run the following command.

   kubectl -n kube-system edit configmap/coredns

2. Verify if the line containing ready is present. If it is not, insert the line with ready. To save the changes and exit edit mode, press the Esc key, type :wq!, and press Enter.

   apiVersion: v1
   data:
    Corefile: |
     .:53 {
       errors
       health {
         lameduck 15s
       }
       ready # Add this line and make sure that the word "ready" is aligned with the word "kubernetes".
       kubernetes cluster.local in-addr.arpa ip6.arpa {
         pods verified
         fallthrough in-addr.arpa ip6.arpa
       }
       prometheus :9153
       forward . /etc/resolv.conf {
         max_concurrent 1000
               prefer_udp
       }
       cache 30
       loop
       log
       reload
       loadbalance
     }

3. Verify the CoreDNS pod's running status and examine the logs. If the term reload is present in the logs, this signifies a successful modification.