how to monitor CoreDNS by collecting and analyzing CoreDNS logs - Container Service for Kubernetes

CoreDNS is deployed in ACK clusters and serves as a DNS server. You can examine CoreDNS logs to identify causes of slow DNS resolution or analyze DNS queries for high-risk domain names. This topic describes how to monitor CoreDNS by collecting and analyzing its logs.

Prerequisites

Install the log collection component.

If you use the Logtail component, ensure that the version of alibaba-log-controller is 0.2.0.0-76648ee-aliyun or later.
If an earlier version of alibaba-log-controller is used, update the Logtail component. For more information about how to update components, see Components.
If you use the LoongCollector component, there are no version restrictions.

Step 1: Enable the CoreDNS log plugin

Precautions

Ensure that CoreDNS is updated to the latest version. For more information about how to update components, see Manage components.
The log plugin is enabled for the default domain in the new version of CoreDNS. If you need to enable it for other domains, follow the steps below.

Procedure

Note

After you enable logtail-ds for CoreDNS, the CPU usage increases by about 10% and data transfer also increases. If the replicated pods of CoreDNS are running with high CPU usage, you can add more CoreDNS pods. For more information about how to add CoreDNS pods, see Manually scale pods for an application.

In the kube-system namespace of your ACK cluster, choose Configuration Management > ConfigMaps, click the coredns ConfigMap, and check whether the log plugin exists in the Corefile field of the coredns ConfigMap. If not, add it according to the following example to enable logging for CoreDNS domain name resolution. For more information about how to modify a ConfigMap, see Modify a ConfigMap.

The following content is an example of the coredns ConfigMap that uses the default log format:

Corefile: |
    .:53 {
        errors
        log # Specify the logging component.
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          upstream
          fallthrough in-addr.arpa ip6.arpa
          ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
    # If you want to log DNS queries of containers in other domains, you must specify the logging component for these domains by using the same configuration format.
    demo.com:53 {
        ... 
        log # Specify the logging component.
    }

Step 2: Enable logging for CoreDNS

Method 1: Use the console

Log on to the ACK console. In the navigation pane on the left, click Clusters.
On the Clusters page, find the cluster you want and click its name. In the left-side pane, choose Operations > Log Center.
On the Log Center page, click the Network Component Logs tab and click Install.
The system then automatically installs the relevant components and enables logging for CoreDNS.

Method 2: Use kubectl

You can use AliyunLogConfig CustomResourceDefinitions (CRDs) to describe logging configurations. alibaba-log-controller automatically configures Log Service settings and creates log reports based on the logging configurations. For more information about how to create an AliyunLogConfig CRD, see Manage custom resources.

Important

The following configurations take effect only when the default log format of CoreDNS is used. If CoreDNS uses a custom log format, you need to modify the regular expression in the Regex field.

For more information about how to customize the log format of CoreDNS, see log.

For more information about the log collection configuration and procedure, see Use CRDs to collect container logs in DaemonSet mode.

Create a YAML file named k8s-coredns-log.yaml. The following code block shows an example:

apiVersion: log.alibabacloud.com/v1alpha1
kind: AliyunLogConfig
metadata:
  #     Your config name, must be unique in you k8s cluster.
  name: k8s-coredns-log
  namespace: kube-system
spec:
  # logstore name to upload log
  logstore: coredns-log
  # logtail config detail
  productCode: k8s-coredns
  logtailConfig:
    inputType: plugin
    # logtail config name, should be same with [metadata.name]
    configName: k8s-coredns-log
    inputDetail:
      plugin:
        inputs:
        - type: service_docker_stdout
          detail:
            IncludeLabel:
              io.kubernetes.container.name: coredns
            Stderr: true
            Stdout: true
        processors:
        - type: processor_regex
          detail:
            KeepSource: false
            KeepSourceIfParseError: true
            Keys:
            - level
            - remote
            - port
            - id
            - type
            - class
            - name
            - proto
            - size
            - do
            - bufsize
            - rcode
            - rflags
            - rsize
            - duration
            NoKeyError: true
            NoMatchError: false
            FullMatch: false
            Regex: \[([^]]+)]\s([^:]+):(\S+)\s+-\s+(\S+)\s+"(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+([^"]+)"\s+(\S+)\s+(\S+)\s+(\S+)\s+([\d\.]+).*
            SourceKey: content
        - type: processor_regex
          detail:
            KeepSource: false
            KeepSourceIfParseError: true
            Keys:
            - error
            - rcode
            - name
            - type
            - errorMsg
            NoKeyError: false
            NoMatchError: false
            FullMatch: false
            Regex: \[ERROR]\s+(plugin/errors):\s+(\S)+\s+(\S+)\s+([^:]*):\s+(.*)
            SourceKey: content

Run the following command to enable logging for CoreDNS:
```
kubectl apply -f k8s-coredns-log.yaml
```

For more information about how to configure AliyunLogConfig CRDs for storage, network, and Auto Scaling resources, see Step 1: Create configuration files for collecting log files of system components.

Step 3: Query and analyze CoreDNS logs (Logstores)

Log on to the ACK console. In the navigation pane on the left, click Clusters.
On the Clusters page, find the cluster you want and click its name. In the left-side pane, choose Operations > Log Center.
On the Log Center page, click the Network Component Logs tab. On the CoreDNS tab, click Logstores to query and analyze DNS logs.
Enter a query and analyze statement into the Search & Analyze box and select 1 Week to specify a time range.

Click Search & Analyze to view the query and analysis results.

Log Service allows you to view the distribution of query and analysis results in a histogram, analyze the raw logs, and check the query and analysis results in different types of charts. For more information about log query and analysis, see Guide to log query and analysis.

Log histogram
The log distribution histogram displays the distribution of query and analysis results in different time ranges.

Raw Logs

The current query result is displayed on the Raw Logs tab. You can click Table or Raw Data to check whether the current domain name can be resolved as expected by analyzing the logs. Pay close attention to the returned response codes. For more information about the definitions of the response codes, see DNS troubleshooting.

image..png

The following table describes the parameters.

Parameter	Description	Example
{level}	Log level	INFO
{remote}	The IP address of the client	172.16.0.10
{port}	The port of the client	58008
{id}	The ID of a query	34518
{type}	Request type	A
{class}	The request class	IN
{name}	The domain name that is queried	kube-dns.kube-system.svc.cluster.local.
{proto}	The protocol that is used	tcp
{size}	The request size. Unit: bytes	56
{do}	Query whether the EDNS0 DO (DNSSEC OK) bit is set	false
{bufsize}	The size of the EDNS0 buffer defined in the request. Unit: bytes	65535
{rcode}	The response code	NOERROR
{rflags}	The response flags. All specified flags are displayed	qr,aa,rd
{rsize}	The size of the response before compression. Unit: bytes	110
{duration}	The response time. Unit: seconds	0.00011

View charts
After you execute a query and analyze statement, you can view the query and analysis results on the Graph tab.
LogReduce
On the LogReduce tab, you can click Enable LogReduce to aggregate similar logs. For more information, see LogReduce.

Step 4: View the collected CoreDNS logs in dashboards

Log on to the ACK console. In the navigation pane on the left, click Clusters.
On the Clusters page, find the cluster you want and click its name. In the left-side pane, choose Operations > Log Center.
On the Log Center page, click the Network Component Logs tab. On the CoreDNS tab, click Dashboards to view the Kubernetes CoreDNS Log Analysis page.
On the Kubernetes CoreDNS Log Analysis page, you can view aggregated information about the number of queries to CoreDNS, the success rate of DNS queries, and the response latencies. You can also view the list of most frequently accessed domain names, the list of invalid domain names, the list of slow resolutions, and the list of queries for high-risk domain names.

Step 5: Configure alert rules based on CoreDNS logs

On the Kubernetes CoreDNS Log Analysis page, you can configure alert rules based on each chart or list. You can perform this operation only in the Log Service console.

Log on to the ACK console. In the navigation pane on the left, click Clusters.
On the Clusters page, find the target cluster and click its name. In the navigation pane on the left, click Cluster Information.
On the Cluster Information page, click the Basic Information tab. Then, click the hyperlink to the right of Log Service Project to log on to the Log Service console.
In the left-side navigation page of the Logstores page, click icon. Then, find and click Kubernetes CoreDNS Log Analysis in the Dashboard list.
In the upper-right corner of the card that you want to manage on the Kubernetes CoreDNS Log Analysis page, choose > Save As Alert.
For more information about the alerting parameters, see Create a log alert rule.
After an alert rule is created, you can view, modify, and disable the alert rule. For more information, see Manage an alert rule.

Related operations

Disable log collection

If you want to disable log collection for CoreDNS, run the following command to delete the relevant CRD:

kubectl -n kube-system delete AliyunLogConfig k8s-coredns-log

After the CRD is deleted, CoreDNS logs are no longer delivered to Log Service.