How to enable log analysis and monitoring for CoreDNS - Container Service for Kubernetes

Container Service for Kubernetes (ACK) uses CoreDNS as the in-cluster DNS server. You can view CoreDNS logs to analyze issues such as slow DNS resolution and access to high-risk domains. This topic describes how to enable log analysis and monitoring for CoreDNS.

Prerequisites

You have installed the log collection component.

If you use the Logtail component, make sure that the alibaba-log-controller application is version 0.2.0.0-76648ee-aliyun or later.
If the alibaba-log-controller version is outdated, you can upgrade the Logtail component. For more information about how to upgrade components, see Components.
If you use the LoongCollector component, no version requirements apply.

Step 1: Enable the CoreDNS log plugin

Precautions

Ensure that the CoreDNS component is the latest version. For more information about how to upgrade, see Manage components.
The log plugin is enabled by default for the default domain in the latest version of CoreDNS. To enable the plugin for other domains, follow these steps.

Procedure

Note

After you enable CoreDNS logs, CPU usage increases by about 10% (depending on the actual requests) and network traffic also increases. If the CoreDNS replica pods are running with high CPU usage, you can add more CoreDNS pods. For more information, see Manually scale pods.

In the kube-system namespace of the ACK cluster, navigate to Configuration Management > Configuration Items, click the coredns configuration item, and check the Corefile field of the coredns configuration item for the log plugin. If the plugin is not found, add it as shown in the following example to enable logging for CoreDNS domain name resolution. For more information, see Modify a Configuration Item.

The following code shows the coredns configuration that uses the default log format.

Corefile: |
    .:53 {
        errors
        log # Add the log plugin here.
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          upstream
          fallthrough in-addr.arpa ip6.arpa
          ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
    # If you have other domains and need to collect logs for their containers, add the log plugin in the same format.
    demo.com:53 {
        ... 
        log # Add the log plugin here.
    }

Step 2: Enable the CoreDNS log service

Method 1: Enable the service in the console

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, find the cluster you want and click its name. In the left-side pane, choose Operations > Log Center.
On the Log Center page, on the Network Component Logs > CoreDNS tab, click Install and then Enable CoreDNS Logs.

Method 2: Enable the service using kubectl

The log collection feature for ACK is extended using CustomResourceDefinitions (CRDs). You can deploy an AliyunLogConfig CRD, and the Log Controller automatically creates the corresponding collection configurations and reports in Simple Log Service. For more information about how to create the AliyunLogConfig custom resource, see Manage custom resources.

Important

The following configuration applies only to the default log format in CoreDNS. If you use a custom log format for CoreDNS, you must modify the `Regex` section in the CRD configuration.

For more information about custom log formats for CoreDNS, see log.

For more information about the log collection flow and collection configurations, see Collect container logs using a DaemonSet-CRD.

Create a YAML file named k8s-coredns-log.yaml. The following code is a complete sample YAML file.

apiVersion: log.alibabacloud.com/v1alpha1
kind: AliyunLogConfig
metadata:
  #     Your config name, must be unique in you k8s cluster.
  name: k8s-coredns-log
  namespace: kube-system
spec:
  # logstore name to upload log
  logstore: coredns-log
  # logtail config detail
  productCode: k8s-coredns
  logtailConfig:
    inputType: plugin
    # logtail config name, should be same with [metadata.name]
    configName: k8s-coredns-log
    inputDetail:
      plugin:
        inputs:
        - type: service_docker_stdout
          detail:
            IncludeLabel:
              io.kubernetes.container.name: coredns
            Stderr: true
            Stdout: true
        processors:
        - type: processor_regex
          detail:
            KeepSource: false
            KeepSourceIfParseError: true
            Keys:
            - level
            - remote
            - port
            - id
            - type
            - class
            - name
            - proto
            - size
            - do
            - bufsize
            - rcode
            - rflags
            - rsize
            - duration
            NoKeyError: true
            NoMatchError: false
            FullMatch: false
            Regex: \[([^]]+)]\s([^:]+):(\S+)\s+-\s+(\S+)\s+"(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+([^"]+)"\s+(\S+)\s+(\S+)\s+(\S+)\s+([\d\.]+).*
            SourceKey: content
        - type: processor_regex
          detail:
            KeepSource: false
            KeepSourceIfParseError: true
            Keys:
            - error
            - rcode
            - name
            - type
            - errorMsg
            NoKeyError: false
            NoMatchError: false
            FullMatch: false
            Regex: \[ERROR]\s+(plugin/errors):\s+(\S)+\s+(\S+)\s+([^:]*):\s+(.*)
            SourceKey: content

Run the following command to enable the CoreDNS log service.
```
kubectl apply -f k8s-coredns-log.yaml
```

For more information about how to configure the AliyunLogConfig CRD for resources such as storage, networking, and Auto Scaling, see Step 1: Create the log configuration file for data plane components.

Step 3: Query and analyze logs in the Logstore

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, find the cluster you want and click its name. In the left-side pane, choose Operations > Log Center.
On the Log Center page, click the Network Component Logs tab. Then, click Logstore on the CoreDNS tab to query and analyze DNS logs.
In the search box, enter a query statement and select Last 1 Week for the time range.

Click Query/Analysis to view the results.

Simple Log Service lets you view the distribution of query and analysis results in a histogram, analyze raw logs, and check the query and analysis results in different types of charts. For more information about how to query and analyze logs, see Quick Guide to Query and Analysis.

Log Histogram
The log distribution histogram shows the distribution of logs over time.

Raw Logs

The query results are displayed on the Raw Logs tab. You can click Table or Raw to view the logs and analyze them to determine if the domain name resolution is abnormal. Pay close attention to the value of the return code (RCODE). For an explanation of the RCODE values, see Troubleshoot DNS resolution errors.

image..png

The following table describes the fields.

Field	Description	Example
{level}	The log level.	INFO
{remote}	The client IP address.	172.16.0.10
{port}	The client port.	58008
{id}	The query ID.	34518
{type}	The request type.	A
{class}	The request class.	IN
{name}	The name of the requested domain.	kube-dns.kube-system.svc.cluster.local.
{proto}	The protocol used.	tcp
{size}	The request size (bytes).	56
{do}	Specifies whether the EDNS0 DO (DNSSEC OK) bit is set in the query.	false
{bufsize}	The EDNS0 buffer size (bytes) that is defined in the query.	65535
{rcode}	The response code.	NOERROR
{rflags}	The response flags. Each flag that is set is displayed.	qr,aa,rd
{rsize}	The uncompressed response size (bytes).	110
{duration}	The response time (seconds).	0.00011

Statistical Chart
After you run a query statement, you can view the visualized results on the Statistical Chart tab.
LogReduce
On the LogReduce tab, click Enable LogReduce to aggregate highly similar logs during log collection. For more information, see LogReduce.

Step 4: View CoreDNS log collection data on the dashboard

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, find the cluster you want and click its name. In the left-side pane, choose Operations > Log Center.
On the Log Center page, click the Network Component Logs tab, and then the CoreDNS tab. Click Dashboard to view the Kubernetes CoreDNS Log Analysis dashboard.
On the Kubernetes CoreDNS Log Analysis page, you can view key metrics such as the total number of CoreDNS requests, request success rate, and request latency. This dashboard also displays the top requested domains, domains with errors, domains with slow resolution, and an analysis of access to high-risk domains.

Step 5: Create alerts for CoreDNS logs

You can create an alert rule from a chart on the Kubernetes CoreDNS Log Analysis dashboard. This operation is available only in the Simple Log Service (SLS) console.

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, find the target cluster and click its name. In the navigation pane on the left, click Cluster Information.
On the Basic Information tab of the Cluster Information page, click the link next to Simple Log Service Project to go to the SLS console.
In the navigation pane on the left of the Logstore page, click the Dashboard icon. In the dashboard list, find and click Kubernetes CoreDNS Log Analysis.
On the Kubernetes CoreDNS Log Analysis page, hover over the upper-right corner of the target chart and select > Save As Alert.
For more information about alert configuration parameters, see Create a log alerting and monitoring rule.
After an alert rule is created, you can view, modify, and disable the alert rule. For more information, see Manage alert monitoring rules.

Related operations

Disable log collection

If you no longer need to collect CoreDNS logs, run the following command to uninstall the log collection CRD and stop log collection.

kubectl -n kube-system delete AliyunLogConfig k8s-coredns-log

After you delete the CRD, CoreDNS logs are no longer sent to SLS.