Container Service for Kubernetes (ACK) provides the ACK CoreDNS DNSTAP Analyser component. The component can identify and classify DNS messages that are exchanged between a client and CoreDNS, or between CoreDNS and an upstream DNS server. You can use this component to identify the causes of DNS resolution errors in an efficient manner. This topic describes how to use ACK CoreDNS DNSTAP Analyser to diagnose CoreDNS resolution errors.

Background information

dnstap is a log format for DNS messages. You can use dnstap to diagnose DNS resolution errors on DNS servers. For more information, see dnstap.

Step 1: Install ACK CoreDNS DNSTAP Analyser

  1. Log on to the ACK console.
  2. In the left-side navigation pane of the ACK console, choose Marketplace > App Catalog.
  3. On the App Catalog page, enter ack-coredns-dnstap-analyser in the search box and click the search icon. In the search result, find and click ack-coredns-dnstap-analyser.
  4. In the Deploy panel on the App Catalog - ack-coredns-dnstap-analyser page, select the cluster in which you want to install the component. Click the Description tab and read the parameter descriptions. Then, click the Parameters tab and set the parameters based on your requirements.
  5. On the right side of the page, click Create.

Step 2: Configure the DNSTAP plug-in in CoreDNS

CoreDNS is pre-installed with the DNSTAP plug-in, which sends DNSTAP messages. The DNS messages that are sent and received by the DNSTAP plug-in can be forwarded to ACK CoreDNS DNSTAP Analyser. Before you can use ACK CoreDNS DNSTAP Analyser to diagnose CoreDNS resolution errors, you must configure the DNSTAP plug-in in CoreDNS.

  1. Run the following command to query the cluster IP address of ACK CoreDNS DNSTAP Analyser in the kube-system namespace. This IP address is used by the DNSTAP plug-in. In this example, the IP address is 172.21.0.10.
    kubectl -n kube-system get svc dnstap-analyser
  2. Run the following command to modify the ConfigMap of CoreDNS:
    kubectl -n kube-system edit cm coredns -o yaml
  3. Modify the default settings of CoreDNS based on the comments in the following code block.

    In the ConfigMap, you must enable the DNSTAP plug-in and set its destination address to the cluster IP address of ACK CoreDNS DNSTAP Analyser. This way, the DNS messages that are sent and received by the DNSTAP plug-in can be forwarded to ACK CoreDNS DNSTAP Analyser.

    json      Corefile: |
            .:53 {
                // Details not shown.
                ready
    
                // Add the following line and replace 172.21.0.10 with the actual cluster IP address of CoreDNS DNSTAP Analyser. 
                dnstap tcp://172.21.0.10:6000 full
    
                kubernetes cluster.local in-addr.arpa ip6.arpa {
                  // Details not shown.
                }
                // Details not shown.
                reload
                loadbalance
            }
  4. Run the following command to query all pods that run CoreDNS:
    kubectl -n kube-system get pod | grep coredns
    Expected output:
    coredns-7d56l         1/1     Running   0          30m
    coredns-s7m2t         1/1     Running   0          30m
  5. Run the following command to print the CoreDNS log:
    kubectl -n kube-system logs -f --tail=500 coredns-7d56l
    Note Replace coredns-7d56l with the name of a pod that is returned in Step 3.
    Wait for one minute. If the following output is returned and no exceptions occur, the configuration is modified.
    [INFO] Reloading complete

After you complete the preceding steps, the DNSTAP plug-in is enabled in CoreDNS. CoreDNS can automatically forward the DNS messages that are sent and received by the DNSTAP plug-in to ACK CoreDNS DNSTAP Analyser for diagnosis.

Step 3: View the log of ACK CoreDNS DNSTAP Analyser

  1. Run the following command to query all pods that run ACK CoreDNS DNSTAP Analyser:
    kubectl -n kube-system get pod | grep dnstap-analyser
    Expected output:
    dnstap-analyser-bbdf879-gpfkm         1/1     Running   0          30m
  2. Run the following command to print the log of ACK CoreDNS DNSTAP Analyser:
    kubectl -n kube-system logs -f dnstap-analyser-bbdf879-gpfkm
    Note Replace dnstap-analyser-bbdf879-gpfkm with the name of a pod that is returned in Step 1.

Step 4: Analyze the log of ACK CoreDNS DNSTAP Analyser

In the log of ACK CoreDNS DNSTAP Analyser, each line includes the aggregated information about a DNS session. A DNS session contains all requests and responses that are exchanged among a client, CoreDNS, and an upstream DNS server. A DNS session includes the following fields:
  • Status: The value of the Status field varies based on the scenario.
    Status Description
    Succeeded The domain name was resolved. The response carries the NXDOMAIN or NOERROR status code.
    Failed The domain name failed to be resolved. The response carries an error status code, or the connection to the upstream DNS server timed out.
    SampleLoss CoreDNS or ACK CoreDNS DNSTAP Analyser is overloaded and drops DNSTAP messages. In this case, you must increase the number of pods that run CoreDNS or ACK CoreDNS DNSTAP Analyser to reduce the load on each pod. This improves the accuracy of diagnostic results.
  • BitMap: indicates the type of the DNSTAP message that is captured from the session. You can identify possible causes of DNS resolution errors based on message types.
    BitMap Description
    1 CoreDNS received a DNS query from the client but did not return a response to the client.
    3 The internal domain name was resolved. The DNS query sent from a client hit the local DNS cache of CoreDNS or a DNS record in the Kubernetes cluster. The result was returned to the client.
    5 The upstream DNS server failed to be connected. CoreDNS forwarded the DNS query sent from a client to the upstream DNS server. The upstream DNS server did not return a result.
    15 The external domain name was resolved. CoreDNS forwarded the DNS query sent from a client to the upstream DNS server. The upstream DNS server resolved the domain name and returned the result to CoreDNS. Then, CoreDNS returned the result to the client.
  • Messages: This field records the content of the original DNS query and response in JSON format. You can analyze the message content based on dns-parameters-6.

Step 5 (optional): View information about error DNS messages on Log Service dashboards

If you configured dashboards in Log Service and used the recommended env setting for log collection when you installed ACK CoreDNS DNSTAP Analyser, you can view information about error DNS messages on the Kubernetes CoreDNS DNSTAP dashboard of your cluster in Log Service. For more information, see the status descriptions in Step 4. For more information, see Monitor CoreDNS and analyze the CoreDNS log.