How to enable log analysis and monitoring for CoreDNS - Container Service for Kubernetes

Enable CoreDNS log collection to troubleshoot DNS latency and detect high-risk domains in ACK clusters.

Prerequisites

The log collection add-on is installed.
- For Logtail: The alibaba-log-controller version is 0.2.0.0-76648ee-aliyun or later.
  If the alibaba-log-controller version is outdated, upgrade the Logtail add-on.
- For LoongCollector: All versions are supported.
CoreDNS is upgraded to the latest version. See Manage components.

Step 1 (optional): Enable the CoreDNS log plugin

Note

In the latest CoreDNS version, the log plugin is enabled by default for the default domain. Follow these steps only to configure logging for custom domains.

Usage notes

Enabling logs increases CoreDNS CPU usage by approximately 10%, depending on request volume. If CPU utilization is already high, scale out CoreDNS replicas before proceeding.

Procedure

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, click Operations > Log Center.
In the kube-system namespace, find the coredns ConfigMap and click Edit.
Check the Corefile for the log plugin (typically in the .:53 block). If missing, add it as shown below. See Modify a parameter.

The coredns configuration with the default log format:

Corefile: |
    .:53 {
        errors
        log # Add the log plugin here to enable DNS resolution logging.
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          upstream
          fallthrough in-addr.arpa ip6.arpa
          ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
    # If you have other domains, add the log plugin to their blocks as well.
    demo.com:53 {
        ... 
        log # Add the log plugin here.
    }

Step 2: Enable CoreDNS log collection

Method 1: Enable the service in the console

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, click Operations > Log Center.
On the Network Component Logs > CoreDNS tab, click Install (if prompted), then Enable Log Collection.

Method 2: Enable the service using kubectl

Deploy an AliyunLogConfig Custom Resource Definition (CRD) to automate logstore and dashboard creation. See Manage Custom Resources.

Important

This configuration applies only to the default CoreDNS log format. If you use a custom format, modify the Regex field in the CRD definition.
For custom log formats, see the CoreDNS log plugin.
See Use CRDs to collect container logs in DaemonSet mode.

Create a file named k8s-coredns-log.yaml with the following content:

apiVersion: log.alibabacloud.com/v1alpha1
kind: AliyunLogConfig
metadata:
  #     Your config name, must be unique in your ACK cluster.
  name: k8s-coredns-log
  namespace: kube-system
spec:
  # The logstore name to upload log
  logstore: coredns-log
  # Logtail config detail
  productCode: k8s-coredns
  logtailConfig:
    inputType: plugin
    # Logtail config name, should be the same as [metadata.name]
    configName: k8s-coredns-log
    inputDetail:
      plugin:
        inputs:
        - type: service_docker_stdout
          detail:
            IncludeLabel:
              io.kubernetes.container.name: coredns
            Stderr: true
            Stdout: true
        processors:
        - type: processor_regex
          detail:
            KeepSource: false
            KeepSourceIfParseError: true
            Keys:
            - level
            - remote
            - port
            - id
            - type
            - class
            - name
            - proto
            - size
            - do
            - bufsize
            - rcode
            - rflags
            - rsize
            - duration
            NoKeyError: true
            NoMatchError: false
            FullMatch: false
            Regex: \[([^]]+)]\s([^:]+):(\S+)\s+-\s+(\S+)\s+"(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+([^"]+)"\s+(\S+)\s+(\S+)\s+(\S+)\s+([\d\.]+).*
            SourceKey: content
        - type: processor_regex
          detail:
            KeepSource: false
            KeepSourceIfParseError: true
            Keys:
            - error
            - rcode
            - name
            - type
            - errorMsg
            NoKeyError: false
            NoMatchError: false
            FullMatch: false
            Regex: \[ERROR]\s+(plugin/errors):\s+(\S)+\s+(\S+)\s+([^:]*):\s+(.*)
            SourceKey: content

Apply the configuration.
```
kubectl apply -f k8s-coredns-log.yaml
```

To configure the AliyunLogConfig CRD for resources such as storage, networking, and auto scaling, see Step 1: Create log collection configurations.

Step 3: Query and analyze logs in Logstore

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, click Operations > Log Center.
On the Network Component Logs > CoreDNS tab, click Logstores to query and analyze DNS logs.
Enter a query statement and select Last 1 Week for the time range.

Click Search & Analyze to view the results.

Simple Log Service (SLS) displays results as histograms, raw logs, and statistical charts. See Query and analysis quick start.

Log histogram
Shows log distribution over time.

Raw logs

On the Raw Logs tab, click Table or Raw Data to check for DNS anomalies. Pay attention to the RCODE value. See Troubleshoot DNS resolution issues.

image..png

CoreDNS log field reference

Field	Description	Example
{level}	The log level.	INFO
{remote}	The client IP address.	172.16.0.10
{port}	The client port.	58008
{id}	The query ID.	34518
{type}	The request type.	A
{class}	The request class.	IN
{name}	The requested domain name.	kube-dns.kube-system.svc.cluster.local.
{proto}	The protocol used.	tcp
{size}	The request size (bytes).	56
{do}	Whether the EDNS0 DO (DNSSEC OK) bit is set in the query.	false
{bufsize}	The EDNS0 buffer size (bytes) in the query.	65535
{rcode}	The response code.	NOERROR
{rflags}	The response flags. Displays each set flag.	qr,aa,rd
{rsize}	The uncompressed response size (bytes).	110
{duration}	Resolution latency (seconds).	0.00011

Graph
View visualized query results on the Graph tab.
LogReduce
On the LogReduce tab, click Enable LogReduce to aggregate similar logs. See LogReduce.

Step 4: Visualize with dashboards

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, click Operations > Log Center.
On the Network Component Logs > CoreDNS tab, click Dashboards to view the Kubernetes CoreDNS Logs Analysis dashboard. This provides:
- Total Requests, Success Ratio, and Latency: Monitor DNS health at a glance.
- Top Requested Domains: Identify the most active services.
- Top Error Domains: Find domains causing NXDOMAIN or SERVFAIL errors.
- Timeout Domains, High-risk Domains, and Slow Logs: Identify performance bottlenecks.

Step 5: Configure alerts

On the Kubernetes CoreDNS Logs Analysis dashboard, set alerts based on individual charts. This is available only in the SLS console.

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, click Cluster Information.
On the Basic Information tab, click the link next to Log Service Project to go to the SLS console.
In the left navigation pane of the Logstores page, click > Dashboard. In the dashboard list, find and click Kubernetes CoreDNS Logs Analysis.
On the Kubernetes CoreDNS Logs Analysis page, hover over the upper-right corner of the target chart and select > Save as Alert.
See Create an alert rule.
After creating a rule, view, modify, or disable it in Manage alert monitoring rules.

Related operations

Disable log collection

To stop collecting CoreDNS logs, remove the configuration:

kubectl -n kube-system delete AliyunLogConfig k8s-coredns-log

After deletion, CoreDNS logs are no longer sent to SLS.