Container Service for Kubernetes (ACK) provides the ACK CoreDNS DNSTAP Analyser component. The component can identify and classify DNS messages that are exchanged between a client and CoreDNS, or between CoreDNS and an upstream DNS server. You can use this component to identify the causes of DNS resolution errors in an efficient manner. This topic describes how to use ACK CoreDNS DNSTAP Analyser to diagnose CoreDNS resolution errors.

Background information

dnstap is a log format for DNS messages. You can use dnstap to diagnose DNS resolution errors on DNS servers. For more information, see dnstap.

Step 1: Install ACK CoreDNS DNSTAP Analyser

  1. Log on to the ACK console.
  2. In the left-side navigation pane of the ACK console, choose Marketplace > App Catalog.
  3. On the App Catalog page, enter ack-coredns-dnstap-analyser in the search box and click the search icon. In the search result, find and click ack-coredns-dnstap-analyser.
  4. On the ack-coredns-dnstap-analyser page, click Deploy.
  5. In the Deploy wizard, select a cluster and namespace, and then click Next. On the Parameters wizard page, configure the parameters based on your business requirements and click OK.

Step 2: Configure the DNSTAP plug-in in CoreDNS

CoreDNS is pre-installed with the DNSTAP plug-in, which sends DNSTAP messages. The DNS messages that are sent and received by the DNSTAP plug-in can be forwarded to ACK CoreDNS DNSTAP Analyser. Before you can use ACK CoreDNS DNSTAP Analyser to diagnose CoreDNS resolution errors, you must configure the DNSTAP plug-in in CoreDNS.

  1. Run the following command to query the cluster IP address of ACK CoreDNS DNSTAP Analyser in the kube-system namespace. This IP address is used by the DNSTAP plug-in. In this example, the IP address is 172.21.0.10.
    kubectl -n kube-system get svc dnstap-analyser
  2. Run the following command to modify the ConfigMap of CoreDNS:
    kubectl -n kube-system edit cm coredns -o yaml
  3. Modify the default settings of CoreDNS based on the comments in the following code block.

    In the ConfigMap, you must enable the DNSTAP plug-in and set its destination address to the cluster IP address of ACK CoreDNS DNSTAP Analyser. This way, the DNS messages that are sent and received by the DNSTAP plug-in can be forwarded to ACK CoreDNS DNSTAP Analyser.

    Corefile: 
            .:53 {
                #Details are not shown.
                ready
    
                // Add the following line and replace 172.21.0.10 with the actual cluster IP address of CoreDNS DNSTAP Analyser. 
                dnstap tcp://172.21.0.10:6000 full
    
                kubernetes cluster.local in-addr.arpa ip6.arpa {
                #Details are not shown.
                }
                #Details are not shown.
                reload
                loadbalance
            }
  4. Run the following command to query all pods that run CoreDNS:
    kubectl -n kube-system get pod  grep coredns
    Expected output:
    coredns-7d56l         1/1     Running   0          30m
    coredns-s7m2t         1/1     Running   0          30m
  5. Run the following command to print the CoreDNS log:
    kubectl -n kube-system logs -f --tail=500 coredns-7d56l
    Note Replace coredns-7d56l with the name of a pod that is returned in Step 3.
    Wait for 1 minute. If the following output is returned and no exceptions occur, the configuration is modified.
    [INFO] Reloading complete

After you complete the preceding steps, the DNSTAP plug-in is enabled in CoreDNS. CoreDNS can automatically forward the DNS messages that are sent and received by the DNSTAP plug-in to ACK CoreDNS DNSTAP Analyser for diagnosis.

Step 3: View the log of ACK CoreDNS DNSTAP Analyser

  1. Run the following command to query all pods that run ACK CoreDNS DNSTAP Analyser:
    kubectl -n kube-system get pod  grep dnstap-analyser
    Expected output:
    dnstap-analyser-bbdf879-g****         1/1     Running   0          30m
  2. Run the following command to print the log of ACK CoreDNS DNSTAP Analyser:
    kubectl -n kube-system logs -f dnstap-analyser-bbdf879-g****
    Note Replace dnstap-analyser-bbdf879-gpfkm with the name of a pod that is returned in Step 1.

Step 4: Analyze the log of ACK CoreDNS DNSTAP Analyser

In the log of ACK CoreDNS DNSTAP Analyser, each line includes the aggregated information about a DNS session. A DNS session contains all requests and responses that are exchanged among a client, CoreDNS, and an upstream DNS server. A DNS session includes the following fields:
  • Status: The value of the Status field varies based on the scenario.
    Status Description
    Succeeded The domain name was resolved. The response carries the NXDOMAIN or NOERROR status code.
    Failed The domain name failed to be resolved. The response carries an error status code, or the connection to the upstream DNS server timed out.
    SampleLoss CoreDNS or ACK CoreDNS DNSTAP Analyser is overloaded and drops DNSTAP messages. In this case, you must increase the number of pods that run CoreDNS or ACK CoreDNS DNSTAP Analyser to reduce the load on each pod. This improves the accuracy of diagnostic results.
  • BitMap: indicates the type of the DNSTAP message that is captured from the session. You can identify possible causes of DNS resolution errors based on message types.
    BitMap Description
    1 CoreDNS received a DNS query from the client but did not return a response to the client.
    3 The internal domain name was resolved. The DNS query sent from a client hit the local DNS cache of CoreDNS or a DNS record in the Kubernetes cluster. The result was returned to the client.
    5 The upstream DNS server failed to be connected. CoreDNS forwarded the DNS query sent from a client to the upstream DNS server. The upstream DNS server did not return a result.
    15 The external domain name was resolved. CoreDNS forwarded the DNS query sent from a client to the upstream DNS server. The upstream DNS server resolved the domain name and returned the result to CoreDNS. Then, CoreDNS returned the result to the client.
  • Messages: This field records the content of the original DNS query and response in JSON format. You can analyze the message content based on dns-parameters-6.

Step 5 (optional): View information about error DNS messages on Log Service dashboards

  1. Log on to the ACK console.
  2. In the left-side navigation pane of the ACK console, click Clusters.
  3. On the Clusters page, find the cluster that you want to manage and click the name of the cluster or click Details in the Actions column. The details page of the cluster appears.
  4. On the Cluster Information page, click the Cluster Resources tab and click the hyperlink to the right side of Log Service Project.
  5. In the left-side navigation pane of the Logstores page, click the 图标 icon. In the dashboards list, find and click Kubernetes CoreDNS DNSTAP Message Monitoring.
    On the Kubernetes CoreDNS DNSTAP Message Monitoring page, you can view the number of DNS messages and information about error messages.
  6. Optional:To view information about error messages, perform the following steps:
    1. Record the message ID.
    2. In the left-side navigation pane of the Logstore page, click the 123 icon.
    3. In the Logstores list, click the Logstore that is configured when you install ACK CoreDNS DNSTAP Analyser. If the default Logstore k8s-coredns-dnstap is used, click k8s-coredns-dnstap.
    4. On the details page of the Logstore, search for the message ID in the log.
    5. If you need assistance, copy the log entry and then Submit a ticket. To copy the log entry, click the log entry and then click the 456 icon.

Step 6: (Optional) Uninstall ACK CoreDNS DNSTAP Analyser

After you identify the cause, we recommend that you uninstall ACK CoreDNS DNSTAP Analyser to avoid unnecessary costs. To uninstall ACK CoreDNS DNSTAP Analyser, perform the following steps:

  1. Delete the line that contains dnstap from the ConfigMap of CoreDNS:
    1. Run the following command to modify the ConfigMap of CoreDNS:
      kubectl -n kube-system edit cm coredns -o yaml
    2. Delete the line that contains dnstap from the ConfigMap of CoreDNS based on the following content:
      Corefile: 
              .:53 {
                  #Details are not shown.
                  ready
      
                  # Delete the following line
                  dnstap tcp://10.10.10.10:6000 full
      
                  kubernetes cluster.local in-addr.arpa ip6.arpa {
                  #Details are not shown.
                  }
                  #Details are not shown.
                  reload
                  loadbalance
              }
    3. Run the following command to query all pods that run CoreDNS:
      kubectl -n kube-system get pod  grep coredns

      Expected output:

      coredns-7d56l         1/1     Running   0          30m
      coredns-s7m2t         1/1     Running   0          30m
    4. Run the following command to print the CoreDNS log:
      kubectl -n kube-system logs -f --tail=500 coredns-7d56l
      Note Replace coredns-7d56l with the name of a pod that is returned in Step iii.
      Wait for 1 minute. If the following output is returned and no exceptions occur, the configuration is modified.
      [INFO] Reloading complete
  2. Uninstall the ack-coredns-dnstap-analyser component.
    1. Log on to the ACK console.
    2. In the left-side navigation pane of the ACK console, click Clusters.
    3. On the Clusters page, find the cluster that you want to manage and click the name of the cluster or click Details in the Actions column. The details page of the cluster appears.
    4. In the left-side navigation pane of cluster details page, choose Applications > Helm.
    5. On the Helm page, find ack-coredns-dnstap-analyser and click Delete in the Actions column.
    6. In the Delete message, click OK.