Container Service for Kubernetes:Collect and analyze CoreDNS logs

Step 1: Enable logtail-ds for CoreDNS

ACK creates a ConfigMap named coredns in the kube-system namespace of the cluster. You can modify the coredns ConfigMap by specifying the logging component in the log field of the Corefile configuration. This enables logtail-ds for CoreDNS. For more information about how to modify a ConfigMap, see Modify a ConfigMap.

Corefile: |
    .:53 {
        errors
        log // Specify the logging component. 
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          upstream
          fallthrough in-addr.arpa ip6.arpa
          ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
    // If you want to log DNS queries of containers in other domains, you must specify the logging component for these domains by using the same configuration format. 
    demo.com:53 {
        ... 
        log // Specify the logging component. 
    }

Step 2: Enable logging for CoreDNS

Method 1: Use the ACK console

1. Log on to the ACK console and click Clusters in the left-side navigation pane.
2. On the Clusters page, click the name of a cluster and choose Operations > Log Center in the left-side navigation pane.
3. On the Log Center page, click the Network Component Logs tab and click Install.
   The system then automatically installs the relevant components and enables logging for CoreDNS.

Method 2: Use the CLI

You can use AliyunLogConfig CustomResourceDefinitions (CRDs) to describe logging configurations. alibaba-log-controller automatically configures Log Service settings and creates log reports based on the logging configurations. For more information about how to create an AliyunLogConfig CRD, see Manage custom resources.

1. Create a YAML file named k8s-coredns-log.yaml. The following code block shows an example:

   apiVersion: log.alibabacloud.com/v1alpha1
   kind: AliyunLogConfig
   metadata:
     #     Your config name, must be unique in you k8s cluster.
     name: k8s-coredns-log
     namespace: kube-system
   spec:
     # logstore name to upload log
     logstore: coredns-log
     # logtail config detail
     productCode: k8s-coredns
     logtailConfig:
       inputType: plugin
       # logtail config name, should be same with [metadata.name]
       configName: k8s-coredns-log
       inputDetail:
         plugin:
           inputs:
           - type: service_docker_stdout
             detail:
               IncludeLabel:
                 io.kubernetes.container.name: coredns
               Stderr: true
               Stdout: true
           processors:
           - type: processor_regex
             detail:
               KeepSource: false
               KeepSourceIfParseError: true
               Keys:
               - level
               - remote
               - port
               - id
               - type
               - class
               - name
               - proto
               - size
               - do
               - bufsize
               - rcode
               - rflags
               - rsize
               - duration
               NoKeyError: true
               NoMatchError: false
               FullMatch: false
               Regex: \[([^]]+)]\s([^:]+):(\S+)\s+-\s+(\S+)\s+"(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+([^"]+)"\s+(\S+)\s+(\S+)\s+(\S+)\s+([\d\.]+).*
               SourceKey: content
           - type: processor_regex
             detail:
               KeepSource: false
               KeepSourceIfParseError: true
               Keys:
               - error
               - rcode
               - name
               - type
               - errorMsg
               NoKeyError: false
               NoMatchError: false
               FullMatch: false
               Regex: \[ERROR]\s+(plugin/errors):\s+(\S)+\s+(\S+)\s+([^:]*):\s+(.*)
               SourceKey: content

2. Run the following command to enable logging for CoreDNS:

   kubectl apply -f k8s-coredns-log.yaml

For more information about how to create AliyunLogConfig CRDs to define storage, network, and scaling resources, see Step 1: Create configuration files for collecting log files of system components.

Step 3: View CoreDNS logs

1. Log on to the ACK console and click Clusters in the left-side navigation pane.
2. On the Clusters page, click the name of a cluster and choose Operations > Log Center in the left-side navigation pane.
3. On the Log Center page, click the Network Component Logs tab and click Dashboards to view the Kubernetes CoreDNS Log Analysis page.
   On the Kubernetes CoreDNS Log Analysis page, you can view aggregated information about the number of queries to CoreDNS, the success rate of DNS queries, and the response latencies. You can also view the list of most frequently accessed domain names, the list of invalid domain names, the list of slow resolutions, and the list of queries for high-risk domain names.

Step 4: Configure alert rules based on CoreDNS logs

On the Kubernetes CoreDNS Log Analysis page, you can configure alert rules based on each chart or list. You can perform this operation only in the Log Service console.

1. Log on to the ACK console and click Clusters in the left-side navigation pane.
2. On the Clusters page, click the name of a cluster and click Cluster Information in the left-side navigation pane.
3. On the Cluster Information page, click the Cluster Resources tab. Then, click the hyperlink to the right of Log Service Project to log on to the Log Service console.
4. In the left-side navigation page of the Logstores page, click . Then, find and click Kubernetes CoreDNS Log Analysis in the Dashboard list.
5. In the upper-right corner of the card that you want to manage on the Kubernetes CoreDNS Log Analysis page, choose > Save as Alert.
   For more information about how to configure an alert rule, see Create an alert monitoring rule for logs.

   After an alert rule is created, you can view, modify, and disable the alert rule. For more information see Manage an alert monitoring rule.

Disable logging

kubectl -n kube-system delete AliyunLogConfig k8s-coredns-log