Container Service for Kubernetes (ACK) allows you to deploy NodeLocal DNSCache to improve the stability and performance of service discovery. NodeLocal DNSCache is implemented as a DaemonSet and runs a DNS caching agent on cluster nodes to improve the efficiency of DNS resolution for ACK clusters. This topic describes how to deploy and configure NodeLocal DNSCache for an application in an ACK cluster.

Prerequisites

Limits

  • NodeLocal DNSCache does not support pods that run in serverless Kubernetes (ASK) clusters or on elastic container instances in managed or dedicated Kubernetes clusters.
  • The Terway version must be 1.0.10.301 or later. If your cluster runs Terway in inclusive ENI mode based on IPVLAN, you must modify the configurations of Terway. For more information, see Modify the configurations of Terway.
  • To install NodeLocal DNSCache, you can install ACK NodeLocal DNSCache on the Add-ons page or install ack-node-local-dns on the App Catalog page.
  • NodeLocal DNSCache serves as a transparent caching proxy for CoreDNS and does not provide plug-ins such as hosts or rewrite. If you want to enable these plug-ins, modify CoreDNS configurations.
  • Before you use NodeLocal DNSCache, you must modify CoreDNS configurations. Otherwise, you cannot resolve domain names that are added to Alibaba Cloud DNS PrivateZone or the *.vpc-proxy.aliyuncs.com domain name provided by some cloud services. For more information, see Modify CoreDNS.

Introduction

ACK NodeLocal DNSCache is a local DNS caching solution developed based on the open source NodeLocal DNSCache project. The Helm chart name of ACK NodeLocal DNSCache is ack-node-local-dns. This solution consists of a DNS cache that runs as a DaemonSet and an admission controller that runs as a Deployment to dynamically inject DNSConfig.
  • The admission controller intercepts pod creation requests based on admission webhooks and dynamically injects cached DNSConfig to pod configurations.
    Note If you do not enable the admission controller to automatically inject DNSConfig, you must manually add DNS settings to pod configurations. For more information, see Method 2: Manually specify DNSConfig in Configure NodeLocal DNSCache for an application.
  • The DaemonSet that runs a DNS cache on each node can create a virtual network interface. By default, the virtual network interface listens for DNS queries that are sent to the IP address 169.254.20.10. To change the IP address, Submit a ticket. DNS queries that are generated in pods are proxied by the DaemonSet based on pod DNSConfig and node network settings.
    Notice The DaemonSets that run DNS caches are built on CoreDNS and provide only proxy and caching services. Do not enable other plug-ins such as hosts or rewrite.
Figure 1. How NodeLocal DNSCache works
How NodeLocal DNSCache works
Serial number Description
By default, a pod with the local DNSConfig injected uses NodeLocal DNSCache to listen for DNS queries that are sent to the IP address 169.254.20.10 on the node.
If NodeLocal DNSCache does not find a cache hit for the DNS query, the kube-dns Service is used to request CoreDNS to handle the query.
CoreDNS uses the DNS server deployed in the virtual private cloud (VPC) to resolve domain names that are not cluster-local.
If the pod with the local DNSConfig injected fails to connect to NodeLocal DNSCache, the pod uses the kube-dns Service to connect to CoreDNS for DNS resolution.
A pod without the local DNSConfig injected uses the kube-dns Service to connect to CoreDNS for DNS resolution.

Install NodeLocal DNSCache

You can install NodeLocal DNSCache on the Add-ons page or the App Catalog page. We recommend that you install NodeLocal DNSCache on the Add-ons page.

Install NodeLocal DNSCache on the Add-ons page

  1. Log on to the ACK console.
  2. In the left-side navigation pane of the ACK console, click Clusters.
  3. On the Clusters page, find the cluster that you want to manage and choose More > Manage System Components in the Actions column.
  4. On the Add-ons page, click the Networking tab and find ACK NodeLocal DNSCache.
  5. Click Install. In the message that appears, click OK.

Install NodeLocal DNSCache on the App Catalog page

  1. Log on to the ACK console.
  2. In the left-side navigation pane of the ACK console, choose Marketplace > App Catalog.
  3. On the App Catalog page, search for ack-node-local-dns. In the search result, click ack-node-local-dns.
  4. In the Deploy section, select the cluster where you want to deploy ack-node-local-dns.
  5. On the Description tab, read the introduction and parameter settings about ack-node-local-dns. Then, click the Parameters tab to set the parameters.
    The following table describes the parameters.
    Parameter Description How to obtain the value
    upstream_ip The cluster IP address of the kube-dns Service in the kube-system namespace. NodeLocal DNSCache uses the kube-dns Service to communicate with CoreDNS and resolve cluster-local domain names. You can modify the parameter value to specify other upstream DNS servers.
    clusterDomain The domain name of the cluster. Query the --cluster-domain parameter of the kubelet process on a cluster node. The default value of the parameter is cluster.local.
  6. In the Deploy section, click Create.

Configure NodeLocal DNSCache for an application

Note By default, NodeLocal DNSCache is not installed on master nodes. If you want to deploy business pods on master nodes and master nodes are added with taints, you must modify add matching tolerations to the configurations of node-local-dns in the kube-system namespace.
To redirect DNS requests that are previously forwarded to CoreDNS to the DaemonSet that runs a DNS cache, you must set the nameservers parameter of pod configurations to 169.254.20.10 and the IP address of kube-dns. To do this, use one of the following methods:
  • Method 1: Use the admission controller to automatically inject DNSConfig when pods are created. We recommend that you use this method.
  • Method 2: Manually specify DNSConfig when pods are created.
  • Method 3: Modify kubelet parameters and restart kubelet. We recommend that you do not use this method because your business may be interrupted.
Method 1: Automatically inject DNSConfig
You can use an admission controller to automatically inject DNSConfig to newly created pods. This way, you do not need to manually configure the YAML files of pods. By default, the application listens for pod creation requests from namespaces that have the node-local-dns-injection=enabled label. You can use the following command to add this label to a namespace.
kubectl label namespace default node-local-dns-injection=enabled
Note
  • The preceding command enables automatic DNSConfig injection only for the default namespace. To enable automatic DNSConfig injection for other namespaces, replace default based on your requirements.
  • If automatic DNSConfig injection is enabled for a namespace but you do not want to automatically inject DNSConfig to some pods, you can add the node-local-dns-injection=disabled label to the templates of these pods.
After automatic DNSConfig injection is enabled, the following parameters are added to newly created pods. To ensure the high availability of DNS services, the cluster IP address of kube-dns is added to the nameservers parameter as a backup DNS server.
dnsConfig:
    nameservers:
    - 169.254.20.10
    - 172.21.0.10
    options:
    - name: ndots
      value: "3"
    - name: attempts
      value: "2"
    - name: timeout
      value: "1"
    searches:
    - default.svc.cluster.local
    - svc.cluster.local
    - cluster.local
  dnsPolicy: None

To enable automatic DNSConfig injection, the following conditions must be met. If DNSConfig is not automatically injected, check whether the conditions are met:

  • Newly created pods do not belong to the kube-system or kube-public namespace.
  • The namespace to which newly created pods belong has the node-local-dns-injection=enabled label.
  • The namespace to which newly created pods belong does not have labels related to Elastic Container Instance-based pods, such as virtual-node-affinity-injection, eci, and alibabacloud.com/eci.
  • Newly created pods do not have labels related to elastic container instances, such as eci and alibabacloud.com/eci, or the node-local-dns-injection=disabled label.
  • Newly created pods use the hostNetwork network and the ClusterFirstWithHostNet DNS policy, or the pods do not use the hostNetwork network but use the ClusterFirst DNS policy.

Method 2: Manually specify DNSConfig

If you do not want to use admission webhooks to automatically inject DNSConfig, you can modify pod configurations to manually specify DNSConfig.
apiVersion: v1
kind: Pod
metadata:
  name: alpine
  namespace: default
spec:
  containers:
  - image: alpine
    command:
      - sleep
      - "10000"
    imagePullPolicy: Always
    name: alpine
  dnsPolicy: None
  dnsConfig:
    nameservers: ["169.254.20.10","172.21.0.10"]
    searches:
    - default.svc.cluster.local
    - svc.cluster.local
    - cluster.local
    options:
    - name: ndots
      value: "3"
    - name: attempts
      value: "2"
    - name: timeout 
      value: "1"
  • dnsPolicy: Set the value to None.
  • nameservers: Set the value to 169.254.20.10 and the cluster IP address of kube-dns.
  • searches: Set the DNS search domains. Make sure that internal domains can be resolved.
  • ndots: You can improve resolution efficiency by setting this parameter to a smaller value. Default value: 5. For more information, see resolv.conf.

Method 3: Configure kubelet startup parameters

The kubelet uses the --cluster-dns and --cluster-domain parameters to control pod DNSConfig. In the /etc/systemd/system/kubelet.service.d/10-kubeadm.conf file, add the --cluster-dns parameter and set the value to the local IP address 169.254.20.10. Then, run the systemctl daemon-reload and systemctl restart kubelet commands for the changes to take effect.
--cluster-dns=169.254.20.10 --cluster-dns=<kube-dns ip> --cluster-domain=<search domain>
  • cluster-dns: specifies the DNS servers that are used in the pod configurations. By default, only the IP address of `kube-dns` is specified. You must add the local IP address 169.254.20.10.
  • cluster-domain: specifies the DNS search domains that are used in the pod configurations. You can use the existing domain. In most cases, the existing domain is `cluster.local`.

Example on how to configure NodeLocal DNSCache for an application

The following example shows how to configure NodeLocal DNSCache for a Deployment that is created in the default namespace.

  1. Run the following command to add a label to the namespace to which the Deployment belongs. In this example, the Deployment is created in the default namespace.
    kubectl label namespace default node-local-dns-injection=enabled
    Notice The admission controller ignores applications in the kube-system and kube-public namespaces. Do not configure to automatically inject dnsConfig to applications in the two namespaces.
  2. Deploy a sample application in the default namespace.
    1. Use the following YAML template to create a sample application named ubuntu-deployment:
      apiVersion: apps/v1 # for versions before 1.8.0 use apps/v1beta1
      kind: Deployment
      metadata:
        name: ubuntu
        labels:
          app: ubuntu
      spec:
        replicas: 2
        selector:
          matchLabels:
            app: ubuntu
        template:
          metadata:
            labels:
              app: ubuntu
          spec:
            containers:
            - name: ubuntu
              image: ubuntu
              command: ["sh", "-c"]
              args: ["sleep 100000"]
    2. Run the following command to deploy the application in the cluster:
      kubectl apply -f ubuntu-deployment.yaml

      Expected output:

      deployment.apps/ubuntu created
    3. Run the following command to view information about the application:
      kubectl get deployment ubuntu

      Expected output:

      NAME     READY   UP-TO-DATE   AVAILABLE   AGE
      ubuntu   2/2     2            2           7s
  3. Check whether dnsConfig is injected.
    1. Run the following command to view the pods that are provisioned for the application:
      kubectl get pods

      Expected output:

      NAME                      READY   STATUS    RESTARTS   AGE
      ubuntu-766448f68c-mj8qk   1/1     Running   0          4m39s
      ubuntu-766448f68c-wf5hw   1/1     Running   0          4m39s
    2. Run the following command to check whether NodeLocal DNSCache is enabled in dnsConfig of a pod:
      kubectl get pod ubuntu-766448f68c-mj8qk -o=jsonpath='{.spec.dnsConfig}'

      Expected output:

      map[nameservers:[169.254.20.10 172.21.0.10] options:[map[name:ndots value:5]] searches:[default.svc.cluster.local svc.cluster.local cluster.local]]
      The preceding output indicates that NodeLocal DNSCache is enabled for the application.
      After NodeLocal DNSCache is enabled for an application, the following parameters are added to the pods that are provisioned for the application. To ensure the high availability of DNS services, the cluster IP address 172.21.0.10 of the kube-dns Service is specified as the IP address of the backup DNS server in the nameservers parameter.
        dnsConfig:
          nameservers:
          - 169.254.20.10
          - 172.21.0.10
          options:
          - name: ndots
            value: "3"
          - name: attempts
            value: "2"
          - name: timeout
            value: "1"
          searches:
          - default.svc.cluster.local
          - svc.cluster.local
          - cluster.local
        dnsPolicy: None

Upgrade NodeLocal DNSCache

If you installed NodeLocal DNSCache on the Add-ons page, you must upgrade NodeLocal DNSCache on the Add-ons page. If you installed NodeLocal DNSCache on the App Catalog page, you must uninstall the current version of NodeLocal DNSCache on the Helm page and then install the latest version.

Upgrade NodeLocal DNSCache on the Add-ons page

  1. Log on to the ACK console.
  2. In the left-side navigation pane of the ACK console, click Clusters.
  3. On the Clusters page, find the cluster that you want to manage and click the name of the cluster or click Details in the Actions column. The details page of the cluster appears.
  4. In the left-side navigation pane of the details page, choose Operations > Add-ons.
  5. On the Add-ons page, find NodeLocal DNSCache and click Upgrade. In the message that appears, click OK.
    Note If you modified tolerations for the node-local-dns DaemonSet, the modifications are overwritten during the upgrade process. You must configure the tolerations again after the upgrade is completed.

Upgrade NodeLocal DNSCache on the App Catalog page

If you installed NodeLocal DNSCache on the App Catalog page, you must uninstall the current version of NodeLocal DNSCache and then install the latest version. For more information, see Uninstall NodeLocal DNSCache and Install NodeLocal DNSCache.

Uninstall NodeLocal DNSCache

If you installed NodeLocal DNSCache on the Add-ons page, you must uninstall NodeLocal DNSCache on the Add-ons page. If you have NodeLocal DNSCache on the App Catalog page, you must uninstall NodeLocal DNSCache on the Helm page.

Uninstall NodeLocal DNSCache on the Add-ons page

  1. Log on to the ACK console.
  2. In the left-side navigation pane of the ACK console, click Clusters.
  3. On the Clusters page, find the cluster that you want to manage and click the name of the cluster or click Details in the Actions column. The details page of the cluster appears.
  4. In the left-side navigation pane of the details page, choose Operations > Add-ons.
  5. On the Add-ons page, find NodeLocal DNSCache and click Uninstall. In the message that appears, click OK.
    Note After NodeLocal DNSCache is uninstalled, all DNS queries are sent to CoreDNS. We recommend that you add scale out CoreDNS before you uninstall NodeLocal DNSCache.

Uninstall NodeLocal DNSCache on the App Catalog page

  1. Check the parameter settings.
    1. Log on to the ACK console.
    2. In the left-side navigation pane of the ACK console, click Clusters.
    3. On the Clusters page, find the cluster that you want to manage and click the name of the cluster or click Details in the Actions column. The details page of the cluster appears.
    4. In the left-side navigation pane of the details page, choose Applications > Helm.
    5. On the Helm page, find and click ack-node-local-dns-default.
    6. Click the Parameters tab and check the parameter settings.
      • If the local_dns_ip parameter is set to the cluster IP address of kube-dns, proceed to Step 2 to uninstall NodeLocal DNSCache.
      • If the high_availability parameter is set to enabled and the local_dns_ip parameter is set to 169.254.20.10, proceed to Step 2 to uninstall NodeLocal DNSCache.
      • If the preceding conditions are not met, perform the following steps to uninstall NodeLocal DNSCache:
        1. Run the kubectl get ns -o yaml command to find the namespaces that have the node-local-dns-injection=enabled label. Remove the label from the namespaces.
        2. Run the kubectl get pod -o yaml command to find the pods to which DNSConfig is injected. Delete the pods and recreate them.
        3. Perform Step 2 to uninstall NodeLocal DNSCache.
  2. Uninstall NodeLocal DNSCache.
    1. In the left-side navigation pane of the ACK console, choose Marketplace > App Catalog.
    2. In the left-side navigation pane of the ACK console, click Clusters.
    3. On the Clusters page, find the cluster that you want to manage and click the name of the cluster or click Details in the Actions column. The details page of the cluster appears.
    4. In the left-side navigation pane of the details page, choose Applications > Helm.
    5. On the Helm page, select ack-node-local-dns-default and click Delete in the Actions column. In the message that appears, click OK.

Modify CoreDNS

By default, NodeLocal DNSCache uses the TCP protocol to communicate with CoreDNS. CoreDNS communicates with the upstream servers based on the protocol used by the source of DNS queries. The DNS servers are deployed in the VPC where the cluster is deployed.

However, DNS servers in a VPC do not support TCP when they resolve domain names that are added to Alibaba Cloud DNS PrivateZone and the *.vpc-proxy.aliyuncs.com domain name provided by some cloud services. Therefore, NodeLocal DNSCache cannot resolve these domain names. To resolve these domain names, modify CoreDNS configurations based on the following modifications. This forces CoreDNS to use UDP as the default protocol when CoreDNS communicates with the upstream DNS servers in the VPC.

We recommend that you modify the configuration file of CoreDNS based on the following modifications. The configuration file is named as coredns in the kube-system namespace. Modify the setting of the forward plug-in and set the protocol that is used to request upstream servers to `perfer_udp`. This way, CoreDNS preferentially uses the UDP protocol to communicate with upstream servers. You can modify the setting based on the following modifications:
# The original setting
forward . /etc/resolv.conf
# The modified setting
forward . /etc/resolv.conf {
  prefer_udp
}

Modify the ConfigMap of Terway

  1. Run the following command to view the ConfigMap of Terway:
    kubectl -n kube-system edit cm eni-config -o yaml
  2. Check the ConfigMap of Terway.
    • Check whether IPVLAN is enabled in the eniip_virtual_type field. If this field does not exist in the ConfigMap or the value is not set to IPVLAN, you do not need to perform the following steps before you can install NodeLocal DNSCache. For more information, see Install NodeLocal DNSCache.
    • Check whether the host_stack_cidrs field is specified in the ConfigMap. If the host_stack_cidrs field is specified, you do not need to perform the following steps before you can install NodeLocal DNSCache. For more information, see Install NodeLocal DNSCache.
  3. Add the host_stack_cidrs field to the ConfigMap of Terway and set the value to 169.254.20.10/32. Then, save the ConfigMap and exit.
     10-terway.conf: |
      {
        "cniVersion": "0.3.0",
        "name": "terway",
        "eniip_virtual_type": "IPVlan",
        "host_stack_cidrs": ["169.254.20.10/32"], 
        "type": "terway"
      }
  4. Run the following command to query all of the DaemonSet pods provisioned for Terway:
    kubectl -n kube-system get pod | grep terway-eniip

    Expected output:

    terway-eniip-7d56l         2/2     Running   0          30m
    terway-eniip-s7m2t         2/2     Running   0          30m
  5. Run the following command to recreate the Terway pods:
     kubectl -n kube-system delete pod terway-eniip-7d56l terway-eniip-s7m2t
  6. Log on to a node of the cluster and run the following command to query the ConfigMap of Terway.
    If the ConfigMap contains the CIDR block that you have added, it indicates that the ConfigMap of Terway is modified.
    cat /etc/cni/net.d/*

    Expected output:

     {
       "cniVersion": "0.3.0",
       "name": "terway-chainer",
       "plugins": [
         {
           "eniip_virtual_type": "IPVlan",
           "host_stack_cidrs": [ 
             "169.254.20.10/32",
           ],
           "type": "terway"
         },
         {
           "type": "cilium-cni"
         }
       ]
     }

    After all the Terway pods are recreated and run as normal, you can install NodeLocal DNSCache.