Container Service for Kubernetes:Troubleshoot DNS resolution errors

Test network connectivity between application pods and CoreDNS

Connect to the application pod's network namespace using one of the following methods:

• Method 1 (recommended): Run kubectl exec -it <pod-name> -- bash to enter the pod directly.

• Method 2: Log in to the node, find the process ID with ps aux | grep <application-process-name>, then enter the network namespace with nsenter -t <pid> -n bash.

• Method 3 (for frequently restarting pods):

  1. Log in to the node.

  2. Run docker ps -a | grep <application-container-name> to find sandboxed container IDs (names start with k8s_POD_).

  3. Run docker inspect <sandboxed-container-ID> | grep netns to find the network namespace path in /var/run/docker/netns/xxxx.

  4. Run nsenter -n<netns-path> -n bash to enter the namespace. > Note: Do not add a space between -n and <netns-path>.

Once inside the pod's network namespace, run these connectivity tests:

# Test connectivity to the kube-dns Service
dig <domain> @<kube-dns-svc-ip>

# Test Internet Control Message Protocol (ICMP) connectivity to the CoreDNS pod
ping <coredns-pod-ip>

# Test DNS query directly to the CoreDNS pod
dig <domain> @<coredns-pod-ip>

Replace <kube-dns-svc-ip> with the kube-dns Service IP in the kube-system namespace, and <coredns-pod-ip> with the IP of a CoreDNS pod.

Test network connectivity of CoreDNS

1. Log in to the node where the CoreDNS pod runs.

2. Run ps aux | grep coredns to get the CoreDNS process ID.

3. Run nsenter -t <pid> -n bash to enter the CoreDNS network namespace.

4. Test connectivity:

   # Test connectivity to the Kubernetes API server
   telnet <apiserver-slb-ip> 443
   
   # Test connectivity to upstream DNS servers
   dig <domain> @100.100.2.136
   dig <domain> @100.100.2.138

Diagnose the container network

1. Log in to the ACK consoleACK console.

2. On the Clusters page, click the name of your cluster or click Details in the Actions column.

3. In the left-side navigation pane, choose Operations > Cluster Check.

4. On the Container Intelligence Service page, choose Cluster Check > Diagnosis.

5. On the Diagnosis page, click the Network Diagnosis tab.

6. Set Source address to the application pod IP, Destination address to the kube-dns Service IP, and Destination port to 53. Select Enable packet tracing and I know and agree, then click Create diagnosis.

7. In the diagnosis list, click Diagnosis details for your record.

The results show the Diagnosis result, Packet paths, and All possible paths sections, along with identified error causes. For details, see Use the cluster diagnostics feature to troubleshoot cluster issues.

Capture packets

Use packet capture when errors are intermittent and hard to reproduce.

1. Log in to the nodes where the application pods and the CoreDNS pod run.

2. Run the following command on each ECS instance:

   tcpdump -i any port 53 -C 20 -W 200 -w /tmp/client_dns.pcap

   This captures all traffic on port 53, rotating across up to 200 files of 20 MB each.

3. Reproduce the error, then analyze the packets from the time window when DNS failures occurred. Check your application log for the exact timestamps.

Packet capture does not affect your service. It causes only a slight increase in CPU utilization and disk I/O.

External domain name cannot be resolved

Check the CoreDNS DNS query log for the response code. Enable the CoreDNS log plugin if it is not already on (see Configure DNS resolution), then look for entries for the failing domain. A NXDOMAIN response means the domain does not exist on the upstream server—often because a search domain suffix was appended to a short name, creating an invalid fully qualified domain name (FQDN). A SERVFAIL or REFUSED response means the upstream DNS server itself has a problem; check the CoreDNS configuration and its connectivity to 100.100.2.136 and 100.100.2.138.

Domain names of headless Services cannot be resolved

In CoreDNS versions earlier than 1.7.0, a network jitter on the Kubernetes API server can cause CoreDNS to exit unexpectedly, which stops it from updating headless Service domain records while it is down. Update CoreDNS to 1.7.0 or later to fix this. See \[Component Updates\] Update CoreDNS.

Domain names of StatefulSet pods cannot be resolved

The pod YAML template must set serviceName to the name of the headless Service that exposes the StatefulSet. Without this, the per-pod DNS names (for example, pod.headless-svc.ns.svc.cluster.local) cannot be resolved, even though the Service-level name (for example, headless-svc.ns.svc.cluster.local) works fine. Set serviceName in the StatefulSet spec to the headless Service name.

DNS queries are blocked by security group rules or network ACLs

Security group rules or network access control lists (ACLs) that control the network communications of the ECS instance are blocking UDP port 53, causing CoreDNS resolution failures on some or all nodes. Modify the security group rules or network ACLs to allow inbound and outbound traffic on UDP port 53.

Container network connectivity errors cause DNS failures

UDP port 53 is blocked due to container network errors. Use the Network Diagnosis feature to identify the broken network path and the root cause.

CoreDNS pods are overloaded

When DNS query volume exceeds what the current number of CoreDNS replicas can handle, resolution latency rises and failures occur. Check whether CPU and memory usage on the CoreDNS pods is near the limit (kubectl -n kube-system top pod -l k8s-app=kube-dns).

Two fixes:

• Deploy NodeLocal DNSCache to absorb queries locally and reduce the load on CoreDNS. See Configure NodeLocal DNSCache.

• Scale out CoreDNS replicas so that peak CPU utilization per pod stays well below the node's available CPU.

DNS queries are not evenly distributed among CoreDNS pods

Imbalanced pod scheduling or a sessionAffinity setting on the kube-dns Service can cause some CoreDNS pods to handle far more queries than others. You'll see noticeably different CPU utilization across CoreDNS pods.

Two fixes:

• Scale out CoreDNS pods and spread them across different nodes.

• Remove the sessionAffinity setting from the kube-dns Service. See Configure the kube-dns Service.

CoreDNS pods do not run as normal

Misconfigured YAML or ConfigMap settings can prevent CoreDNS from starting or cause it to crash. Symptoms include pods not in Running state, a continuously increasing restart count, or errors in the operational log.

Check the CoreDNS log for these common errors:

DNS resolutions fail because the client is overloaded

When the ECS instance running the pod is fully loaded, UDP packets can be dropped before they reach CoreDNS. Look for an abnormal network interface controller (NIC) retransmission rate and high CPU utilization on the instance in your monitoring data.

Two options:

• Submit a ticketSubmit a ticketSubmit a ticketSubmit a ticketSubmit a ticketSubmit a ticketSubmit a ticketSubmit a ticketSubmit a ticketSubmit a ticket for detailed investigation.

• Deploy NodeLocal DNSCache to reduce inter-node DNS traffic. See Configure NodeLocal DNSCache.

Conntrack table is full

When the Linux kernel's conntrack table is full, new UDP and TCP connections are dropped. This typically causes DNS failures during peak hours that recover during off-peak times. To confirm, run dmesg -H on the affected node and look for the keyword conntrack full during the failure window.

Increase the maximum number of entries in the conntrack table. See How do I increase the maximum number of tracked connections in the conntrack table of the Linux kernel?

autopath plugin does not work as normal

A known defect in CoreDNS's autopath plugin causes occasional resolution failures or wrong IP addresses for external domain names. Internal domain names continue to resolve correctly. The issue becomes more visible in clusters that create containers at a high frequency.

Disable the autopath plugin:

1. Run kubectl -n kube-system edit configmap coredns to open the CoreDNS ConfigMap.

2. Delete the autopath @kubernetes line. Save and exit.

3. Verify the new configuration loaded by checking the CoreDNS log for the keyword reload.

DNS resolutions fail due to concurrent A and AAAA record queries

Some Linux distributions send A and AAAA record queries simultaneously over the same port. This can trigger a conntrack table conflict that drops UDP packets.

Symptoms: intermittent CoreDNS resolution failures, with packet capture or DNS query logs showing simultaneous A and AAAA queries from the same source port.

Fixes depend on your image base:

• CentOS or Ubuntu: Add options timeout:2 attempts:3 rotate single-request-reopen to the DNS resolver configuration.

• Alpine Linux: Replace the Alpine-based image with one based on another OS. See Alpine caveats.

• PHP with cURL: Add CURL_IPRESOLVE_V4 to force IPv4-only resolution. See cURL functions.

• All environments: Deploy NodeLocal DNSCache, which mitigates the race condition. See Configure NodeLocal DNSCache.

DNS resolutions fail due to IPVS errors

If kube-proxy runs in IPVS mode and the cluster nodes run CentOS or Alibaba Cloud Linux 2 with a kernel earlier than 4.19.91-25.1.al7.x86_64, removing IPVS UDP backend pods causes source port conflicts that drop UDP packets. This appears as DNS failures lasting approximately 5 minutes during node scaling, node shutdown, or CoreDNS scaling events.

Two fixes:

• Deploy NodeLocal DNSCache to bypass the IPVS path for DNS queries. See Configure NodeLocal DNSCache.

• Shorten the UDP session timeout in IPVS mode. See Change the UDP timeout period in IPVS mode.

NodeLocal DNSCache does not work

All DNS queries go to CoreDNS instead of NodeLocal DNSCache when either of these conditions applies:

• dnsConfig was not injected into the application pods, so they still point to the kube-dns Service IP.

• The pods use an Alpine Linux base image, which sends DNS queries to all nameservers concurrently, including CoreDNS pods directly.

To fix the first case, enable automatic dnsConfig injection. See Configure NodeLocal DNSCache. For Alpine-based images, replace them with images built on another OS. See Alpine caveats.

Alibaba Cloud DNS PrivateZone names cannot be resolved

Alibaba Cloud DNS PrivateZone does not support TCP—it requires UDP. When NodeLocal DNSCache is used, domain names added to Alibaba Cloud DNS PrivateZone cannot be resolved, the endpoints of Alibaba Cloud service APIs that contain vpc-proxy cannot be resolved, or domain names are resolved to wrong IP addresses.

Add prefer_udp to the CoreDNS configuration to force UDP for upstream queries. See Configure CoreDNS.