How to enable CoreDNS log analysis and monitoring - Container Service for Kubernetes

ACK deploys CoreDNS as the in-cluster DNS server. You can view CoreDNS logs to analyze issues such as slow DNS resolution and requests to high-risk domains. This topic describes how to enable CoreDNS log analysis and monitoring.

Prerequisites

The log collection add-on is installed.

If you use the Logtail add-on, make sure that the version of alibaba-log-controller is 0.2.0.0-76648ee-aliyun or later. In the ACK console, go to the Stateless page, find the alibaba-log-controller Deployment, and check its version in the Image column (for example, 0.2.8.0-245ff7b-aliyun).

If the alibaba-log-controller version is outdated, upgrade the Logtail add-on. For more information about how to upgrade add-ons, see Manage components.
For the LoongCollector component, there is no version requirement.

Step 1: Enable the CoreDNS log plugin

Notes

Ensure that the CoreDNS component is updated to the latest version. For more information about how to upgrade the component, see Manage components.
In the latest version of CoreDNS, the log plugin is enabled by default for the default domain. If you need to enable the log plugin for other domains, follow the procedure in this section.

Procedure

Note

Enabling CoreDNS logs consumes additional CPU resources (approximately 10%, depending on the request volume) and network traffic. If the CPU utilization of your existing CoreDNS replicas is high, you can scale out the CoreDNS pods. For more information about scaling CoreDNS pods, see Manual scaling for pods.

In the kube-system namespace of the ACK cluster, select Configurations > ConfigMaps, click the CoreDNS ConfigMap, and check the Corefile field of the CoreDNS ConfigMap to verify that the log plugin is included. If it is not included, add the plugin as shown in the following example to enable logging for CoreDNS DNS resolution. For more information about how to modify a ConfigMap, see Modify a ConfigMap.

The following example shows the CoreDNS configuration that uses the default log format.

Corefile: |
    .:53 {
        errors
        log # Add the log plugin here.
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          upstream
          fallthrough in-addr.arpa ip6.arpa
          ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
    # If you have other domains for which you need to collect logs, add the log plugin to their blocks in the same format.
    demo.com:53 {
        ... 
        log # Add the log plugin here.
    }

Step 2: Enable CoreDNS log collection

Use the console

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, click Operations > Log Center.
On the Log Center page, go to the Network Component Logs > CoreDNS tab. Click Install, and then click Enable Log Collection.

Use kubectl

When you deploy an AliyunLogConfig resource, the Log Controller automatically creates the required log collection configurations and dashboards in Simple Log Service (SLS). For more information about how to create an AliyunLogConfig custom resource, see Manage custom resources.

Important

The following configuration applies only to the default CoreDNS log format. If you use a custom log format, modify the Regex section in the CRD configuration.

For more information about custom log formats in CoreDNS, see log.

For more information about the log collection process and configurations, see Collect container logs by using a DaemonSet and a CRD.

Create a file named k8s-coredns-log.yaml with the following content.

apiVersion: log.alibabacloud.com/v1alpha1
kind: AliyunLogConfig
metadata:
  # Your config name, must be unique in your k8s cluster.
  name: k8s-coredns-log
  namespace: kube-system
spec:
  # The name of the Logstore to which logs are uploaded.
  logstore: coredns-log
  # Details of the Logtail configuration.
  productCode: k8s-coredns
  logtailConfig:
    inputType: plugin
    # The name of the Logtail configuration. This must be the same as metadata.name.
    configName: k8s-coredns-log
    inputDetail:
      plugin:
        inputs:
        - type: service_docker_stdout
          detail:
            IncludeLabel:
              io.kubernetes.container.name: coredns
            Stderr: true
            Stdout: true
        processors:
        - type: processor_regex
          detail:
            KeepSource: false
            KeepSourceIfParseError: true
            Keys:
            - level
            - remote
            - port
            - id
            - type
            - class
            - name
            - proto
            - size
            - do
            - bufsize
            - rcode
            - rflags
            - rsize
            - duration
            NoKeyError: true
            NoMatchError: false
            FullMatch: false
            Regex: \[([^]]+)]\s([^:]+):(\S+)\s+-\s+(\S+)\s+"(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+([^"]+)"\s+(\S+)\s+(\S+)\s+(\S+)\s+([\d\.]+).*
            SourceKey: content
        - type: processor_regex
          detail:
            KeepSource: false
            KeepSourceIfParseError: true
            Keys:
            - error
            - rcode
            - name
            - type
            - errorMsg
            NoKeyError: false
            NoMatchError: false
            FullMatch: false
            Regex: \[ERROR]\s+(plugin/errors):\s+(\S)+\s+(\S+)\s+([^:]*):\s+(.*)
            SourceKey: content

Run the following command to enable CoreDNS log collection:
```
kubectl apply -f k8s-coredns-log.yaml
```

For more information about configuring an AliyunLogConfig CRD for resources such as storage, networking, and auto scaling, see Step 1: Create a log configuration file for data plane components.

Step 3: Query and analyze logs

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, click Operations > Log Center.
On the Log Center page, click the Network Component Logs tab. Then, on the CoreDNS tab, click Logstores to query and analyze DNS logs.
Enter a query and analysis statement in the search box, and then select Last 1 Week to set the time range for the query.

Click Search & Analyze to view the results.

Simple Log Service (SLS) displays the query and analysis results as a log histogram, raw logs, and graphs. For more information about how to query and analyze logs, see Quick start for log query and analysis.

Log Histogram

The log histogram shows the time-based distribution of the queried logs.

Raw Logs

The query results are displayed on the Raw Logs tab. You can click Table or Origin to view the logs and check for domain resolution anomalies. Pay close attention to the RCODE value. For more information about the meanings of RCODEs, see Troubleshoot DNS resolution issues.

The following table describes the fields in the logs.

Parameter	Description	Example
{level}	The log level.	INFO
{remote}	The IP address of the client.	172.16.0.10
{port}	The port of the client.	58008
{id}	The query ID.	34518
{type}	The request type.	A
{class}	The request class.	IN
{name}	The requested domain name.	kube-dns.kube-system.svc.cluster.local.
{proto}	The protocol that is used.	tcp
{size}	The request size in bytes.	56
{do}	Indicates whether the EDNS0 DO (DNSSEC OK) bit is set in the query.	false
{bufsize}	The EDNS0 buffer size in bytes that is defined in the query.	65535
{rcode}	The response code.	NOERROR
{rflags}	The response flags. Each flag that is set is displayed.	qr,aa,rd
{rsize}	The uncompressed response size in bytes.	110
{duration}	The response time in seconds.	0.00011

Graph

After you run a query and analysis statement, you can view the visualized results on the Graph tab.
LogReduce

On the LogReduce tab, click Enable LogReduce to automatically group collected logs with similar patterns. For more information, see LogReduce.

Step 4: View the CoreDNS log dashboard

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, click Operations > Log Center.
On the Log Center page, click the Network Component Logs tab. On the CoreDNS tab, click Dashboards to view the Kubernetes CoreDNS Logs Analysis dashboard.

On the Kubernetes CoreDNS Logs Analysis page, you can view aggregated metrics such as the CoreDNS request count, success rate, and latency. The dashboard also provides lists of top requested domains, error domains, slow-resolving domains, and an analysis of requests to high-risk domains.

Step 5: Create an alert for CoreDNS logs

On the Kubernetes CoreDNS Logs Analysis dashboard, you can configure an alert based on a single chart. You can perform this operation only in the SLS console.

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, click Cluster Information.
On the Cluster Information page, click the Basic Information tab. Click the link to the right of Project to go to the SLS console.
In the left-side navigation pane, click the dashboard icon. In the dashboard list, find and click Kubernetes CoreDNS Logs Analysis.
On the Kubernetes CoreDNS Logs Analysis page, in the upper-right corner of the chart for which you want to create an alert, choose > Save as Alert.

For more information about how to configure alert parameters, see Create a log-based alert rule.

After you create an alert rule, you can view, modify, or disable it. For more information, see Manage alert rules.

Related operations

Disable log collection

To stop collecting CoreDNS logs, run the following command to delete the custom resource:

kubectl -n kube-system delete AliyunLogConfig k8s-coredns-log

After the resource is deleted, CoreDNS logs are no longer reported to Simple Log Service (SLS).