Container Service for Kubernetes:Troubleshoot DNS resolution errors

Common error messages

Match your error message to identify the failure category:

Set up a debug pod

Before running any diagnostic commands, establish a clean observation environment. This eliminates ambiguity between application bugs and DNS bugs.

kubectl run -it --rm debug --image=nicolaka/netshoot -- dig

You can also run the following command:

nslookup <Domain name>

Step 1: Check the DNS configuration of the application pod

# Get the pod's YAML to check its dnsPolicy field
kubectl get pod <pod-name> -o yaml

# If dnsPolicy looks correct, inspect the in-container DNS config
kubectl exec -it <pod-name> -- cat /etc/resolv.conf

Check whether nameserver points to CoreDNS or NodeLocal DNSCache:

• CoreDNS (kube-dns Service IP, e.g., 172.21.0.10): continue to Step 2.

• NodeLocal DNSCache (169.254.20.10): see NodeLocal DNSCache issues.

• Neither (no cluster DNS configured): the pod may be running on an overloaded node, or the conntrack table may be full. See Client overload and Conntrack table full.

apiVersion: v1
kind: Pod
metadata:
  name: <pod-name>
  namespace: <pod-namespace>
spec:
  containers:
  - image: <container-image>
    name: <container-name>
  dnsPolicy: None
  dnsConfig:
    nameservers:
    - 169.254.20.10
    - 172.21.0.10
    options:
    - name: ndots
      value: "3"
    - name: timeout
      value: "1"
    - name: attempts
      value: "2"
    searches:
    - default.svc.cluster.local
    - svc.cluster.local
    - cluster.local

Step 2: Check CoreDNS pod status

kubectl -n kube-system get pod -o wide -l k8s-app=kube-dns

Expected output (all pods in Running state):

NAME                      READY   STATUS    RESTARTS   AGE   IP            NODE
coredns-xxxxxxxxx-xxxxx   1/1     Running   0          25h   172.20.6.53   cn-hangzhou.192.168.0.198

Check resource usage:

kubectl -n kube-system top pod -l k8s-app=kube-dns

Expected output:

NAME                      CPU(cores)   MEMORY(bytes)
coredns-xxxxxxxxx-xxxxx   3m           18Mi

If a pod is not in Running state, describe it to find the cause:

kubectl -n kube-system describe pod <coredns-pod-name>

See CoreDNS pods not running as expected for common error logs and fixes.

Step 3: Check CoreDNS operational logs

kubectl -n kube-system logs -f --tail=500 --timestamps <coredns-pod-name>

Look for NXDOMAIN, SERVFAIL, or REFUSED responses in the logs. If any of these appear for external domain names, the upstream DNS server is returning errors. See External domain name cannot be resolved.

Step 4: Check whether the issue is reproducible

• Always fails: proceed with Diagnose the DNS query log and Diagnose network connectivity between application pods and CoreDNS.

• Intermittent: capture packets as described in Capture packets.

DNS query log format

[INFO] 172.20.2.25:44525 - 36259 "A IN redis-master.default.svc.cluster.local. udp 56 false 512" NOERROR qr,aa,rd 110 0.000116946s

DNS response codes

For the full specification, see RFC 1035.

If the log shows NXDOMAIN, SERVFAIL, or REFUSED for external domain names, the upstream DNS server is the root cause. By default, CoreDNS uses VPC DNS servers 100.100.2.136 and 100.100.2.138 as upstream resolvers. Submit a ticket to the ECS team and include:

What to do if an external domain name cannot be resolved

Internal domain names resolve correctly, but external domain names fail.

Check the CoreDNS query log for NXDOMAIN, SERVFAIL, or REFUSED on the failing domain name. These codes indicate that the upstream DNS server (by default, 100.100.2.136 and 100.100.2.138) is returning an error. Submit a ticket to the ECS team with the domain name, response code, timestamp, and ECS instance IDs of the nodes running CoreDNS.

What to do if headless Service domain names cannot be resolved

CoreDNS cannot resolve headless Service domain names.

This typically happens with CoreDNS versions earlier than 1.7.0, which can exit unexpectedly during Kubernetes API server network jitters, leaving headless Service DNS records stale. Update CoreDNS to 1.7.0 or later. See [Component Updates] Update CoreDNS.

If dig shows the tc flag in the response, the headless Service has too many backing IP addresses and the DNS response exceeds the UDP packet size limit. Configure the client to send DNS queries over TCP:

• For glibc-based resolvers, add use-vc to dnsConfig:

  dnsConfig:
    options:
    - name: use-vc

  This maps to the options directive in /etc/resolv.conf. For details, see the Linux man page for resolv.conf.

• For Go applications, configure the resolver to use TCP:

  package main
  
  import (
    "fmt"
    "net"
    "context"
  )
  
  func main() {
    resolver := &net.Resolver{
      PreferGo: true,
      Dial: func(ctx context.Context, network, address string) (net.Conn, error) {
        return net.Dial("tcp", address)
      },
    }
  
    addrs, err := resolver.LookupHost(context.TODO(), "example.com")
    if err != nil {
      fmt.Println("Error:", err)
      return
    }
    fmt.Println("Addresses:", addrs)
  }

What to do if headless Service domain names cannot be resolved after updating CoreDNS

After upgrading to Kubernetes 1.20 or later with CoreDNS 1.8.4 or later, open-source components such as etcd, Nacos, and Kafka fail to discover services.

CoreDNS 1.8.4 switched to the EndpointSlice API. Some open-source components rely on the service.alpha.kubernetes.io/tolerate-unready-endpoints annotation from the older Endpoint API to publish not-ready Services during initialization. This annotation is not supported in EndpointSlice — it was replaced by publishNotReadyAddresses.

Check whether the YAML or Helm chart for the affected component uses service.alpha.kubernetes.io/tolerate-unready-endpoints. If it does, upgrade the component or consult the component's community.

What to do if StatefulSet pod domain names cannot be resolved

Domain names of StatefulSet pods (e.g., pod.headless-svc.ns.svc.cluster.local) cannot be resolved, but the headless Service's own domain name resolves correctly.

The StatefulSet pod YAML template has serviceName set to the wrong value or left blank. Set serviceName in the StatefulSet spec to the exact name of the headless Service that exposes the StatefulSet.

What to do if security group rules block DNS queries

DNS resolution fails on some or all nodes, persistently.

The security group rules or network access control lists (ACLs) on the ECS instances are blocking UDP port 53. Modify the security group rules or network ACLs to allow UDP port 53.

What to do if container network connectivity errors occur

DNS resolution fails on some or all nodes, persistently.

Container network connectivity errors or other issues are blocking UDP port 53. Use the network diagnostics feature to diagnose network connectivity between the application pods and the CoreDNS pod. If the issue persists, submit a ticket.

What to do if CoreDNS pods are overloaded

DNS resolution latency is high, or failures occur persistently or intermittently. CoreDNS pod CPU or memory usage is near the upper limit.

The number of CoreDNS replicas is too low to handle the DNS query volume. Take one or both of these steps:

• Enable NodeLocal DNSCache to reduce the load on CoreDNS. See Configure NodeLocal DNSCache.

• Scale out CoreDNS pods so that peak CPU utilization per pod stays below the node's available CPU headroom.

What to do if DNS query load is unbalanced

DNS resolution latency is high or failures are intermittent on some (not all) nodes. CoreDNS pod CPU utilization differs significantly between pods. Fewer than two CoreDNS replicas are running, or multiple pods are scheduled on the same node.

DNS queries are unevenly distributed due to imbalanced pod scheduling or SessionAffinity settings on the kube-dns Service. To fix this:

• Scale out CoreDNS pods and schedule them on separate nodes.

• Remove the SessionAffinity setting from the kube-dns Service. See Configure ACK to automatically update CoreDNS.

What to do if CoreDNS pods are not running as expected

DNS resolution latency is high, failures occur on some nodes, CoreDNS pods are not in Running state, the restart count keeps increasing, or the CoreDNS log shows errors.

Misconfigured YAML or a misconfigured CoreDNS ConfigMap is preventing pods from running. Check pod status and logs.

Common error messages:

What to do if DNS fails because the client is overloaded

DNS errors occur intermittently or during peak hours. ECS instance monitoring shows abnormal NIC retransmission rates or high CPU utilization.

The ECS instance hosting the pod that sends DNS queries is fully loaded, causing UDP packet loss. Submit a ticket. In parallel, enable NodeLocal DNSCache to reduce per-node DNS load. See Configure NodeLocal DNSCache.

What to do if the conntrack table is full

DNS resolution fails frequently on some or all nodes during peak hours but works during off-peak hours. Running dmesg -H on the instance shows conntrack full in the logs during the failure window.

The Linux kernel conntrack table is full, so UDP and TCP packets cannot be processed. Increase the maximum number of entries in the conntrack table. See How do I increase the maximum number of tracked connections in the conntrack table of the Linux kernel?

What to do if the autopath plugin does not work as expected

External domain names occasionally fail to resolve or resolve to a wrong IP address, while internal domain names resolve correctly. When the cluster creates containers at a high rate, internal domain names are occasionally resolved to wrong IP addresses.

The autopath plugin in CoreDNS has a known defect. Disable it:

1. Edit the CoreDNS ConfigMap:

   kubectl -n kube-system edit configmap coredns

2. Delete the autopath @kubernetes line, save the file, and exit.

3. Verify that CoreDNS loaded the new configuration by checking the logs for the reload keyword.

What to do if DNS fails due to concurrent A and AAAA record queries

CoreDNS DNS resolution fails intermittently. Packet captures or DNS query logs show A record and AAAA record queries sent simultaneously over the same source port.

Concurrent A and AAAA record queries cause conntrack table errors, which lead to UDP packet loss. On ARM machines, libc versions earlier than 2.33 have this issue (see GLIBC#26600).

Apply one or more of these fixes based on your environment:

• NodeLocal DNSCache: reduces the impact of packet loss. See Configure NodeLocal DNSCache.

• CentOS or Ubuntu with libc: update libc to 2.33 or later, or add resolver options: options timeout:2 attempts:3 rotate single-request-reopen.

• PHP cURL: add CURL_IPRESOLVE_V4 to specify that domain names can be resolved only to IPv4 addresses. See curl_setopt.

• Alpine Linux: replace with a non-Alpine base image. See Alpine Linux caveats.

What to do if DNS fails due to IPVS errors

DNS resolution fails intermittently when nodes are added or removed from the cluster, when nodes shut down, or when CoreDNS is scaled in. Failures typically last about 5 minutes.

kube-proxy is running in IPVS mode. When UDP backend pods are removed from CentOS or Alibaba Cloud Linux 2 nodes with kernel versions earlier than 4.19.91-25.1.al7.x86_64, source port conflicts cause UDP packets to be dropped.

To fix this:

• Enable NodeLocal DNSCache to bypass IPVS for DNS traffic. See Configure NodeLocal DNSCache.

• Shorten the UDP session timeout in IPVS mode. See Change the UDP timeout period in IPVS mode.

NodeLocal DNSCache issues