Each ECS instance in a VPC has a DNS query limit of 5,000 queries per second. You can mitigate rate limiting by scaling out ECS instances or enabling the nscd caching service.
Each Elastic Compute Service (ECS) instance in a Virtual Private Cloud (VPC) has a DNS query limit of 5,000 queries per second. If the number of DNS queries from a single instance exceeds this threshold, the instance is subject to rate limiting and the availability SLA cannot be guaranteed. To mitigate this issue, consider the following approaches:
-
Scale out ECS instances so that DNS query requests are distributed across more instances, reducing the per-instance query volume.
-
Enable the nscd (Name Service Cache Daemon) service on the ECS host for cache acceleration.
NoteWhen the nscd service is enabled, it will take longer for changes to DNS records to take effect.
What is nscd?
nscd is a system cache service that caches name service information such as passwd, group, hosts, services, and netgroup. It reduces the number of name service queries and network traffic, resulting in faster service responses. This guide focuses on the hosts configuration to reduce DNS queries from ECS instances.
Common Linux commands for nscd
|
Command |
Description |
|
yum install -y nscd |
Install nscd. |
|
systemctl start nscd |
Start nscd. |
|
systemctl stop nscd |
Stop nscd. |
|
systemctl restart nscd |
Restart nscd. |
|
systemctl status nscd |
Query the operational status of nscd. |
|
nscd -g |
Outputs the effective configuration of all parameters and cache hit ratio statistics. |
|
nscd -i |
Invalidates the specified cache. You can specify passwd, group, hosts, services, netgroup, etc. For example: nscd -i hosts. |
|
cat /etc/nscd.conf |
Query the current nscd configuration details. |
|
vi /etc/nscd.conf |
Modify nscd configuration parameters. |
Installing nscd
nscd is generally installed by default on Linux. To check whether nscd is installed on your ECS instance, run the following command:
systemctl status nscd # Check the running status of nscd.If the output is as follows, nscd is not installed.
[root@iZbp1fd580caxxxxxxxxxxxxtZ ~]# systemctl status nscd
Unit nscd.service could not be found.Run the following command to install nscd:
yum install -y nscd[root@izbp1fd580ca5xxx ~]# yum install -y nscd
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Resolving Dependencies
--> Running transaction check
---> Package nscd.x86_64 0:2.17-326.el7_9 will be installed
--> Processing Dependency: glibc = 2.17-326.el7_9 for package: nscd-2.17-326.el7_9.x86_64
--> Running transaction check
---> Package glibc.x86_64 0:2.17-307.el7.1 will be updated
--> Processing Dependency: glibc = 2.17-307.el7.1 for package: glibc-common-2.17-307.el7.1.x86_64
---> Package glibc.x86_64 0:2.17-326.el7_9 will be an update
--> Running transaction check
---> Package glibc-common.x86_64 0:2.17-307.el7.1 will be updated
---> Package glibc-common.x86_64 0:2.17-326.el7_9 will be an update
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
nscd x86_64 2.17-326.el7_9 updates 289 k
Updating for dependencies:
glibc x86_64 2.17-326.el7_9 updates 3.6 M
glibc-common x86_64 2.17-326.el7_9 updates 12 M
Transaction Summary
================================================================================
Install 1 Package
Upgrade ( 2 Dependent packages)
Total download size: 15 M
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
(1/3): glibc-2.17-326.el7_9.x86_64.rpm | 3.6 MB 00:00:00
(2/3): nscd-2.17-326.el7_9.x86_64.rpm | 289 kB 00:00:00
(3/3): glibc-common-2.17-326.el7_9.x86_64.rpm | 12 MB 00:00:00
--------------------------------------------------------------------------------
Total 43 MB/s | 15 MB 00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Updating : glibc-common-2.17-326.el7_9.x86_64 1/5
Updating : glibc-2.17-326.el7_9.x86_64 2/5
Installing : nscd-2.17-326.el7_9.x86_64 3/5
Cleanup : glibc-common-2.17-307.el7.1.x86_64 4/5
Cleanup : glibc-2.17-307.el7.1.x86_64 5/5
Verifying : glibc-2.17-326.el7_9.x86_64 1/5
Verifying : glibc-common-2.17-326.el7_9.x86_64 2/5
Verifying : nscd-2.17-326.el7_9.x86_64 3/5
Verifying : glibc-common-2.17-307.el7.1.x86_64 4/5
Verifying : glibc-2.17-307.el7.1.x86_64 5/5
Installed:
nscd.x86_64 0:2.17-326.el7_9
Dependency Updated:
glibc.x86_64 0:2.17-326.el7_9 glibc-common.x86_64 0:2.17-326.el7_9
Complete!Run the status command again. nscd is now installed but not running.
[root@izbp1fxxx ~]# systemctl status nscd
● nscd.service - Name Service Cache Daemon
Loaded: loaded (/usr/lib/systemd/system/nscd.service; disabled; vendor preset: disabled)
Active: inactive (dead)Enabling the nscd service
Run the following command to start nscd:
systemctl start nscdRun the status command again to verify that nscd is running.
[root@izbp1fd58cxxx ~]# systemctl status nscd
● nscd.service - Name Service Cache Daemon
Loaded: loaded (/usr/lib/systemd/system/nscd.service; disabled; vendor preset: disabled)
Active: active (running) since Fri 2023-08-11 15:00:19 CST; 19s ago
Process: 12520 ExecStart=/usr/sbin/nscd $NSCD_OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 12521 (nscd)
CGroup: /system.slice/nscd.service
└─12521 /usr/sbin/nscdTo install nscd on a CentOS or Red Hat operating system, run the command: yum install -y nscd.
To install nscd on a Debian or Ubuntu operating system, run the command: apt-get install -y nscd.
Nscd configuration parameters
The default configuration file for nscd is /etc/nscd.conf. Run the following command to view the configuration:
cat /etc/nscd.conf[root@iZbp1iy9t5ctdys1... ~]# cat /etc/nscd.conf
#
# /etc/nscd.conf
#
# An example Name Service Cache config file. This file is needed by nscd.
#
# WARNING: Running nscd with a secondary caching service like sssd may lead to
# unexpected behaviour, especially with how long entries are cached.
#
# Legal entries are:
#
# logfile <file>
# debug-level <level>
# threads <initial #threads to use>
# max-threads <maximum #threads to use>
# server-user <user to run server as instead of root>
# server-user is ignored if nscd is started with -S parameters
# stat-user <user who is allowed to request statistics>
# reload-count unlimited|<number>
# paranoia <yes|no>
# restart-interval <time in seconds>
#
# enable-cache <service> <yes|no>
# positive-time-to-live <service> <time in seconds>
# negative-time-to-live <service> <time in seconds>
# suggested-size <service> <prime number>
# check-files <service> <yes|no>
# persistent <service> <yes|no>
#
# shared <service> <yes|no>
# NOTE: Setting 'shared' to a value of 'yes' will accelerate the lookup
# with the help of the client, but these lookups will not be
# counted as cache hits i.e. 'nscd -g' may show '0%'.
#
# max-db-size <service> <number bytes>
# auto-propagate <service> <yes|no>
#
# Currently supported cache names (services): passwd, group, hosts, services
#
logfile /var/log/nscd.log
# threads 4
# max-threads 32
server-user nscd
# stat-user somebody
debug-level 1
# reload-count 5
paranoia no
# restart-interval 3600
enable-cache passwd no
positive-time-to-live passwd 600
negative-time-to-live passwd 20
suggested-size passwd 211
check-files passwd yes
persistent passwd yes
shared passwd yes
max-db-size passwd 33554432
auto-propagate passwd yesKey configuration parameter descriptions
|
Configuration parameter |
Description |
|
debug-level |
|
|
reload-count |
Determines how many times a successfully cached entry is actively refreshed before expiring. |
|
paranoia |
Paranoia mode. If enabled, nscd will restart periodically. |
|
restart-interval |
The interval between automatic restarts when paranoia mode is enabled. |
|
enable-cache |
Enables the cache service. |
|
positive-time-to-live |
The TTL for successful response caches. |
|
negative-time-to-live |
The TTL for failed response caches. Set this to 0 to prevent failed lookups from affecting subsequent requests. |
|
check-files |
Periodically checks whether cache-related files (/etc/passwd, /etc/group, /etc/hosts) have been modified. If changes are detected, the corresponding cache is invalidated. |
|
persistent |
Retains cache content across nscd restarts. Recommended when paranoia mode is enabled. |
|
shared |
Shares the nscd database memory mapping with clients. Defaults to yes. To query the cache hit ratio with |
|
max-db-size |
The maximum size of the nscd cache database, in bytes. |
-
The positive-time-to-live setting has no practical effect on DNS caching. The actual TTL is determined by the value returned in the DNS response.
Testing nscd cache effectiveness
Testing with nscd disabled
-
Execute the following command on the ECS instance to capture UDP packets on port 53:
tcpdump -i any udp and port 53 -
Then, with nscd disabled, execute the following command on the ECS instance multiple times, testing three times consecutively.
ping -c 1 -n www.taobao.com # Send one ping command to the domain name www.taobao.com.[root@iZbp1iy9t5ctdysl6xxx ~]# ping -c 1 -n www.taobao.com PING www.taobao.com.danuoyi.tbcache.com (61.174.43.211) 56(84) bytes of data. 64 bytes from 61.174.43.211: icmp_seq=1 ttl=53 time=5.22 ms --- www.taobao.com.danuoyi.tbcache.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.225/5.225/5.225/0.000 ms [root@iZbp1iy9t5ctdysl6xxx ~]# ping -c 1 -n www.taobao.com PING www.taobao.com.danuoyi.tbcache.com (61.174.43.210) 56(84) bytes of data. 64 bytes from 61.174.43.210: icmp_seq=1 ttl=53 time=10.5 ms --- www.taobao.com.danuoyi.tbcache.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 10.534/10.534/10.534/0.000 ms [root@iZbp1iy9t5ctdysl6xxx ~]# ping -c 1 -n www.taobao.com PING www.taobao.com.danuoyi.tbcache.com (61.174.43.211) 56(84) bytes of data. 64 bytes from 61.174.43.211: icmp_seq=1 ttl=53 time=5.20 ms --- www.taobao.com.danuoyi.tbcache.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.205/5.205/5.205/0.000 ms -
Check the packet capture results. Three DNS query requests appear on port 53, each returning a resolution record. This confirms that DNS responses are not cached and that the ECS instance sends a new query through port 53 every time.
10:02:26.112177 IP 10.81.84.51.47500 > 10.143.22.116.domain: 18558+ A? www.taobao.com. (32) 10:02:26.112280 IP 10.143.22.116.domain > 10.81.84.51.47500: 18558 3/0/0 CNAME www.taobao.com.danuoyi.tbcache.com., A 61.174.43.211, A 61.174.43.210 (112) 10:02:35.447683 IP 10.81.84.51.60549 > 10.143.22.116.domain: 43179+ A? www.taobao.com. (32) 10:02:35.447784 IP 10.143.22.116.domain > 10.81.84.51.60549: 43179 3/0/0 CNAME www.taobao.com.danuoyi.tbcache.com., A 61.174.43.210, A 61.174.43.211 (112) 10:02:41.356464 IP 10.81.84.51.37941 > 10.143.22.116.domain: 33455+ A? www.taobao.com. (32) 10:02:41.356627 IP 10.143.22.116.domain > 10.81.84.51.37941: 33455 3/0/0 CNAME www.taobao.com.danuoyi.tbcache.com., A 61.174.43.211, A 61.174.43.210 (112)
Testing with nscd enabled
-
Execute the start command to enable the nscd service, and confirm that nscd is enabled using the status query command.
-
With nscd enabled, execute the following command on the ECS instance, testing six times consecutively. At the same time, capture packets on port 53 of the ECS instance.
ping -c 1 -n www.taobao.com[root@iZbp1iy9t5ctxxx ~]# ping -c 1 -n www.taobao.com PING www.taobao.com.danuoyi.tbcache.com (61.174.43.211) 56(84) bytes of data. 64 bytes from 61.174.43.211: icmp_seq=1 ttl=53 time=5.21 ms --- www.taobao.com.danuoyi.tbcache.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.213/5.213/5.213/0.000 ms [root@iZbp1iy9t5ctdysl6xxx ~]# ping -c 1 -n www.taobao.com PING www.taobao.com.danuoyi.tbcache.com (61.174.43.211) 56(84) bytes of data. 64 bytes from 61.174.43.211: icmp_seq=1 ttl=53 time=5.25 ms --- www.taobao.com.danuoyi.tbcache.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.254/5.254/5.254/0.000 ms [root@iZbp1iy9t5ctdysl6xxx ~]# ping -c 1 -n www.taobao.com PING www.taobao.com.danuoyi.tbcache.com (61.174.43.211) 56(84) bytes of data. 64 bytes from 61.174.43.211: icmp_seq=1 ttl=53 time=5.21 ms --- www.taobao.com.danuoyi.tbcache.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.212/5.212/5.212/0.000 ms [root@iZbp1iy9t5ctdysl6xxx ~]# ping -c 1 -n www.taobao.com PING www.taobao.com.danuoyi.tbcache.com (61.174.43.211) 56(84) bytes of data. 64 bytes from 61.174.43.211: icmp_seq=1 ttl=53 time=5.25 ms --- www.taobao.com.danuoyi.tbcache.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.253/5.253/5.253/0.000 ms [root@iZbp1iy9t5ctdysl6xxx ~]# ping -c 1 -n www.taobao.com PING www.taobao.com.danuoyi.tbcache.com (61.174.43.211) 56(84) bytes of data. 64 bytes from 61.174.43.211: icmp_seq=1 ttl=53 time=5.20 ms --- www.taobao.com.danuoyi.tbcache.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.206/5.206/5.206/0.000 ms [root@iZbp1iy9t5ctdysl6xxx ~]# ping -c 1 -n www.taobao.com PING www.taobao.com.danuoyi.tbcache.com (61.174.43.211) 56(84) bytes of data. 64 bytes from 61.174.43.211: icmp_seq=1 ttl=53 time=5.24 ms --- www.taobao.com.danuoyi.tbcache.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.247/5.247/5.247/0.000 ms -
Check the packet capture results. Only one DNS query request appears on port 53, confirming that subsequent queries hit the nscd cache instead of going through port 53.
In the packet capture results, the target domain for DNS A record queries is
www.taobao.com. The output is as follows.10:20:46.011924 IP 10.81.84.51.52131 > 10.143.22.118.domain: 12973+ A? www.taobao.com. (32) 10:20:46.012115 IP 10.143.22.118.domain > 10.81.84.51.52131: 12973 3/0/0 CNAME www.taobao.com.danuoyi.tbcache.com., A 61.174.43.211, A 61.174.43.210 (112) 10:20:46.012221 IP 10.81.84.51.56349 > 10.143.22.118.domain: 1209+ AAAA? www.taobao.com. (32) 10:20:46.013867 IP 10.143.22.118.domain > 10.81.84.51.56349: 1209 3/0/0 CNAME www.taobao.com.danuoyi.tbcache.com., AAAA 240e:f7:a093:101:3::3e8, AAAA 240e:f7:a093:101:3::3e7 (136)ImportantDuring packet capture, you may notice that tcpdump still captures DNS query packets after the ping command completes. This is caused by the nscd active refresh mechanism and is expected. To disable active refresh, set the reload-count parameter to 0.
-
You can also check cache hit statistics. Run the ping command multiple times, then run the following command to view the statistics:
nscd -g # Output the configuration of all active parameters and cache hit ratio statistics.In the hosts cache section of the output, the cache hits on positive entries value is 64, with a cache hit rate of 52%.
group cache: no cache is enabled yes cache is persistent yes cache is shared 0 suggested size 0 total data pool size 0 used data pool size 3600 seconds time to live for positive entries 60 seconds time to live for negative entries 0 cache hits on positive entries 0 cache hits on negative entries 0 cache misses on positive entries 0 cache misses on negative entries 0% cache hit rate 0 current number of cached values 0 maximum number of cached values 0 maximum chain length searched 0 number of delays on rdlock 0 number of delays on wrlock 0 memory allocations failed yes check /etc/group for changes hosts cache: yes cache is enabled yes cache is persistent no cache is shared 211 suggested size 216064 total data pool size 176 used data pool size 3600 seconds time to live for positive entries 20 seconds time to live for negative entries 64 cache hits on positive entries 0 cache hits on negative entries 50 cache misses on positive entries 7 cache misses on negative entries 52% cache hit rate 1 current number of cached values 6 maximum number of cached values 1 maximum chain length searched 0 number of delays on rdlock 0 number of delays on wrlock 0 memory allocations failed yes check /etc/hosts for changes