VPC内每台ECS实例每秒最高DNS峰值为5000次,单机每秒请求DNS峰值超过阈值后,将面临限速风险,可用性SLA将无法保证。为了缓解此类情况,建议您从如下两个方面考虑:
扩容ECS数量,使得DNS查询请求能够负载分摊到更多的ECS,从而降低每个ECS的DNS查询请求量;
ECS主机开启nscd(Name Service Cache Daemon)服务进行缓存加速;
说明nscd服务开启情况下,会导致变更解析记录的生效时间更长。
什么是nscd?
nscd是一个系统缓存服务,用于缓存名称服务信息,例如passwd、group、hosts、services以及netgroup等,它可以提高系统查询缓存命中率,从而减少对名称服务的查询次数和流量消耗,进而加快服务响应速度,本实践主要用于降低ECS的DNS查询请求数,所以重点介绍hosts配置。
nscd常用Linux指令
指令 | 说明 |
yum install -y nscd | 安装nscd。 |
systemctl start nscd | 启动nscd。 |
systemctl stop nscd | 停止nscd。 |
systemctl restart nscd | 重新启动nscd。 |
systemctl status nscd | 查询nscd的运行状态。 |
nscd -g | 输出生效后的各个参数选项配置情况和缓存命中率统计。 |
nscd -i | 使指定的缓存失效,可以指定passwd、group、hosts、services、netgroup等,例如:nscd -i hosts。 |
cat /etc/nscd.conf | 查询当前nscd配置详情。 |
vi /etc/nscd.conf | 修改nscd配置参数。 |
安装nscd
nscd一般在Linux操作系统中默认安装。如果不确定当前ECS是否已经安装了nscd,可以执行以下命令进行检测:
systemctl status nscd #检查nscd的运行状态。如果检测结果如下,则说明没有安装nscd。
[root@iZbp1fd580caxxxxxxxxxxxxtZ ~]# systemctl status nscd
Unit nscd.service could not be found.执行以下命令进行安装:
yum install -y nscd[root@izbp1fd580ca5xxx ~]# yum install -y nscd
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Resolving Dependencies
--> Running transaction check
---> Package nscd.x86_64 0:2.17-326.el7_9 will be installed
--> Processing Dependency: glibc = 2.17-326.el7_9 for package: nscd-2.17-326.el7_9.x86_64
--> Running transaction check
---> Package glibc.x86_64 0:2.17-307.el7.1 will be updated
--> Processing Dependency: glibc = 2.17-307.el7.1 for package: glibc-common-2.17-307.el7.1.x86_64
---> Package glibc.x86_64 0:2.17-326.el7_9 will be an update
--> Running transaction check
---> Package glibc-common.x86_64 0:2.17-307.el7.1 will be updated
---> Package glibc-common.x86_64 0:2.17-326.el7_9 will be an update
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
nscd x86_64 2.17-326.el7_9 updates 289 k
Updating for dependencies:
glibc x86_64 2.17-326.el7_9 updates 3.6 M
glibc-common x86_64 2.17-326.el7_9 updates 12 M
Transaction Summary
================================================================================
Install 1 Package
Upgrade ( 2 Dependent packages)
Total download size: 15 M
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
(1/3): glibc-2.17-326.el7_9.x86_64.rpm | 3.6 MB 00:00:00
(2/3): nscd-2.17-326.el7_9.x86_64.rpm | 289 kB 00:00:00
(3/3): glibc-common-2.17-326.el7_9.x86_64.rpm | 12 MB 00:00:00
--------------------------------------------------------------------------------
Total 43 MB/s | 15 MB 00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Updating : glibc-common-2.17-326.el7_9.x86_64 1/5
Updating : glibc-2.17-326.el7_9.x86_64 2/5
Installing : nscd-2.17-326.el7_9.x86_64 3/5
Cleanup : glibc-common-2.17-307.el7.1.x86_64 4/5
Cleanup : glibc-2.17-307.el7.1.x86_64 5/5
Verifying : glibc-2.17-326.el7_9.x86_64 1/5
Verifying : glibc-common-2.17-326.el7_9.x86_64 2/5
Verifying : nscd-2.17-326.el7_9.x86_64 3/5
Verifying : glibc-common-2.17-307.el7.1.x86_64 4/5
Verifying : glibc-2.17-307.el7.1.x86_64 5/5
Installed:
nscd.x86_64 0:2.17-326.el7_9
Dependency Updated:
glibc.x86_64 0:2.17-326.el7_9 glibc-common.x86_64 0:2.17-326.el7_9
Complete!此时再执行命令查看nscd的运行状态,nscd已经安装但是处于未启动状态。
[root@izbp1fxxx ~]# systemctl status nscd
● nscd.service - Name Service Cache Daemon
Loaded: loaded (/usr/lib/systemd/system/nscd.service; disabled; vendor preset: disabled)
Active: inactive (dead)开启nscd服务
输入如下指令开启nscd服务:
systemctl start nscd此时再执行命令查看nscd的运行状态,nscd已经处于运行状态。
[root@izbp1fd58cxxx ~]# systemctl status nscd
● nscd.service - Name Service Cache Daemon
Loaded: loaded (/usr/lib/systemd/system/nscd.service; disabled; vendor preset: disabled)
Active: active (running) since Fri 2023-08-11 15:00:19 CST; 19s ago
Process: 12520 ExecStart=/usr/sbin/nscd $NSCD_OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 12521 (nscd)
CGroup: /system.slice/nscd.service
└─12521 /usr/sbin/nscdCentOS/RedHat操作系统安装nscd指令:yum install -y nscd;
Debian/Ubuntu操作系统安装nscd指令:apt-get install -y nscd;
nscd配置参数
nscd的默认配置文件路径在/etc/nscd.conf,可以通过执行如下指令查看nscd配置:
cat /etc/nscd.conf[root@iZbp1iy9t5ctdys1... ~]# cat /etc/nscd.conf
#
# /etc/nscd.conf
#
# An example Name Service Cache config file. This file is needed by nscd.
#
# WARNING: Running nscd with a secondary caching service like sssd may lead to
# unexpected behaviour, especially with how long entries are cached.
#
# Legal entries are:
#
# logfile <file>
# debug-level <level>
# threads <initial #threads to use>
# max-threads <maximum #threads to use>
# server-user <user to run server as instead of root>
# server-user is ignored if nscd is started with -S parameters
# stat-user <user who is allowed to request statistics>
# reload-count unlimited|<number>
# paranoia <yes|no>
# restart-interval <time in seconds>
#
# enable-cache <service> <yes|no>
# positive-time-to-live <service> <time in seconds>
# negative-time-to-live <service> <time in seconds>
# suggested-size <service> <prime number>
# check-files <service> <yes|no>
# persistent <service> <yes|no>
#
# shared <service> <yes|no>
# NOTE: Setting 'shared' to a value of 'yes' will accelerate the lookup
# with the help of the client, but these lookups will not be
# counted as cache hits i.e. 'nscd -g' may show '0%'.
#
# max-db-size <service> <number bytes>
# auto-propagate <service> <yes|no>
#
# Currently supported cache names (services): passwd, group, hosts, services
#
logfile /var/log/nscd.log
# threads 4
# max-threads 32
server-user nscd
# stat-user somebody
debug-level 1
# reload-count 5
paranoia no
# restart-interval 3600
enable-cache passwd no
positive-time-to-live passwd 600
negative-time-to-live passwd 20
suggested-size passwd 211
check-files passwd yes
persistent passwd yes
shared passwd yes
max-db-size passwd 33554432
auto-propagate passwd yes关键配置参数说明
配置参数 | 说明 |
debug-level | |
reload-count | 跟缓存主动刷新有关,此配置参数决定了请求成功缓存主动去查询并更新缓存的次数。 |
paranoia | 偏执模式,如果开启则nscd会周期性地重启。 |
restart-interval | 如果paranoia开启,此参数表示重启间隔时间。 |
enable-cache | 开启缓存服务。 |
positive-time-to-live | 请求成功响应缓存的生存时间。 |
negative-time-to-live | 请求失败响应缓存的生存时间,建议设置为0,以免失败的缓存影响业务请求。 |
check-files | 定期检查/etc/passwd、/etc/group、/etc/hosts等缓存文件的修改时间,如果文件自上次检查以来有过更改,则缓存失效。 |
persistent | 开启时,nscd在重启之后会保留之前的缓存内容,如果paranoia开启,建议开启此功能。 |
shared | 用于服务nscd数据库的内存映射与客户端共享,默认为 yes,如果想利用 |
max-db-size | nscd 缓存数据库的最大大小,单位为Byte。 |
positive-time-to-live没有什么实际意义,TTL值以DNS查询请求返回的TTL为准。
nscd缓存效果检测
nscd关闭时拨测
在ECS上执行如下命令对53端口UDP报文进行抓包:
tcpdump -i any udp and port 53然后在nscd的状态为关闭情况下,在ECS上执行多次如下指令,连续拨测三次。
ping -c 1 -n www.taobao.com #对域名www.taobao.com发一次ping指令。[root@iZbp1iy9t5ctdysl6xxx ~]# ping -c 1 -n www.taobao.com PING www.taobao.com.danuoyi.tbcache.com (61.174.43.211) 56(84) bytes of data. 64 bytes from 61.174.43.211: icmp_seq=1 ttl=53 time=5.22 ms --- www.taobao.com.danuoyi.tbcache.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.225/5.225/5.225/0.000 ms [root@iZbp1iy9t5ctdysl6xxx ~]# ping -c 1 -n www.taobao.com PING www.taobao.com.danuoyi.tbcache.com (61.174.43.210) 56(84) bytes of data. 64 bytes from 61.174.43.210: icmp_seq=1 ttl=53 time=10.5 ms --- www.taobao.com.danuoyi.tbcache.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 10.534/10.534/10.534/0.000 ms [root@iZbp1iy9t5ctdysl6xxx ~]# ping -c 1 -n www.taobao.com PING www.taobao.com.danuoyi.tbcache.com (61.174.43.211) 56(84) bytes of data. 64 bytes from 61.174.43.211: icmp_seq=1 ttl=53 time=5.20 ms --- www.taobao.com.danuoyi.tbcache.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.205/5.205/5.205/0.000 ms查看相对应的抓包情况,能够发现53端口3次DNS请求查询,并且每次DNS请求查询都返回域名解析记录,证明DNS请求查询记录并未被缓存,ECS每次都要通过53端口发送DNS请求查询。
10:02:26.112177 IP 10.81.84.51.47500 > 10.143.22.116.domain: 18558+ A? www.taobao.com. (32) 10:02:26.112280 IP 10.143.22.116.domain > 10.81.84.51.47500: 18558 3/0/0 CNAME www.taobao.com.danuoyi.tbcache.com., A 61.174.43.211, A 61.174.43.210 (112) 10:02:35.447683 IP 10.81.84.51.60549 > 10.143.22.116.domain: 43179+ A? www.taobao.com. (32) 10:02:35.447784 IP 10.143.22.116.domain > 10.81.84.51.60549: 43179 3/0/0 CNAME www.taobao.com.danuoyi.tbcache.com., A 61.174.43.210, A 61.174.43.211 (112) 10:02:41.356464 IP 10.81.84.51.37941 > 10.143.22.116.domain: 33455+ A? www.taobao.com. (32) 10:02:41.356627 IP 10.143.22.116.domain > 10.81.84.51.37941: 33455 3/0/0 CNAME www.taobao.com.danuoyi.tbcache.com., A 61.174.43.211, A 61.174.43.210 (112)
nscd开启时拨测
执行启动指令将nscd服务开启,并通过状态查询指令确定nscd为开启状态。
在nscd的状态为开启情况下,在ECS上执行如下指令,连续拨测6次。同时需要对ECS的53端口进行抓包。
ping -c 1 -n www.taobao.com[root@iZbp1iy9t5ctxxx ~]# ping -c 1 -n www.taobao.com PING www.taobao.com.danuoyi.tbcache.com (61.174.43.211) 56(84) bytes of data. 64 bytes from 61.174.43.211: icmp_seq=1 ttl=53 time=5.21 ms --- www.taobao.com.danuoyi.tbcache.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.213/5.213/5.213/0.000 ms [root@iZbp1iy9t5ctdysl6xxx ~]# ping -c 1 -n www.taobao.com PING www.taobao.com.danuoyi.tbcache.com (61.174.43.211) 56(84) bytes of data. 64 bytes from 61.174.43.211: icmp_seq=1 ttl=53 time=5.25 ms --- www.taobao.com.danuoyi.tbcache.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.254/5.254/5.254/0.000 ms [root@iZbp1iy9t5ctdysl6xxx ~]# ping -c 1 -n www.taobao.com PING www.taobao.com.danuoyi.tbcache.com (61.174.43.211) 56(84) bytes of data. 64 bytes from 61.174.43.211: icmp_seq=1 ttl=53 time=5.21 ms --- www.taobao.com.danuoyi.tbcache.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.212/5.212/5.212/0.000 ms [root@iZbp1iy9t5ctdysl6xxx ~]# ping -c 1 -n www.taobao.com PING www.taobao.com.danuoyi.tbcache.com (61.174.43.211) 56(84) bytes of data. 64 bytes from 61.174.43.211: icmp_seq=1 ttl=53 time=5.25 ms --- www.taobao.com.danuoyi.tbcache.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.253/5.253/5.253/0.000 ms [root@iZbp1iy9t5ctdysl6xxx ~]# ping -c 1 -n www.taobao.com PING www.taobao.com.danuoyi.tbcache.com (61.174.43.211) 56(84) bytes of data. 64 bytes from 61.174.43.211: icmp_seq=1 ttl=53 time=5.20 ms --- www.taobao.com.danuoyi.tbcache.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.206/5.206/5.206/0.000 ms [root@iZbp1iy9t5ctdysl6xxx ~]# ping -c 1 -n www.taobao.com PING www.taobao.com.danuoyi.tbcache.com (61.174.43.211) 56(84) bytes of data. 64 bytes from 61.174.43.211: icmp_seq=1 ttl=53 time=5.24 ms --- www.taobao.com.danuoyi.tbcache.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.247/5.247/5.247/0.000 ms查看相对应的抓包情况,能够发现53端口只抓到1次DNS请求查询,证明DNS请求查询命中nscd缓存,没有通过53端口发送DNS请求查询。
抓包结果中,DNS A 记录查询的目标域名为
www.taobao.com,具体输出如下。10:20:46.011924 IP 10.81.84.51.52131 > 10.143.22.118.domain: 12973+ A? www.taobao.com. (32) 10:20:46.012115 IP 10.143.22.118.domain > 10.81.84.51.52131: 12973 3/0/0 CNAME www.taobao.com.danuoyi.tbcache.com., A 61.174.43.211, A 61.174.43.210 (112) 10:20:46.012221 IP 10.81.84.51.56349 > 10.143.22.118.domain: 1209+ AAAA? www.taobao.com. (32) 10:20:46.013867 IP 10.143.22.118.domain > 10.81.84.51.56349: 1209 3/0/0 CNAME www.taobao.com.danuoyi.tbcache.com., AAAA 240e:f7:a093:101:3::3e8, AAAA 240e:f7:a093:101:3::3e7 (136)重要您在抓包的过程中会发现即使ping指令已经执行完毕,但是每隔一段时间tcpdump依旧可以查到相关域名DNS查询请求报文,这是由于nscd的主动刷新机制导致的,属于正常现象。可以通过将reload-count 参数设置为0关闭主动刷新机制。
您也可以通过执行以下指令来查询缓存命中信息,在一段时间内多次执行ping指令后查看缓存命中情况。
nscd -g #输出生效后的各个参数选项配置情况和缓存命中率统计等。在输出结果的 hosts cache 区块中,cache hits on positive entries 值为 64,缓存命中率为 52%。
group cache: no cache is enabled yes cache is persistent yes cache is shared 0 suggested size 0 total data pool size 0 used data pool size 3600 seconds time to live for positive entries 60 seconds time to live for negative entries 0 cache hits on positive entries 0 cache hits on negative entries 0 cache misses on positive entries 0 cache misses on negative entries 0% cache hit rate 0 current number of cached values 0 maximum number of cached values 0 maximum chain length searched 0 number of delays on rdlock 0 number of delays on wrlock 0 memory allocations failed yes check /etc/group for changes hosts cache: yes cache is enabled yes cache is persistent no cache is shared 211 suggested size 216064 total data pool size 176 used data pool size 3600 seconds time to live for positive entries 20 seconds time to live for negative entries 64 cache hits on positive entries 0 cache hits on negative entries 50 cache misses on positive entries 7 cache misses on negative entries 52% cache hit rate 1 current number of cached values 6 maximum number of cached values 1 maximum chain length searched 0 number of delays on rdlock 0 number of delays on wrlock 0 memory allocations failed yes check /etc/hosts for changes