VPC內每台ECS執行個體每秒最高DNS峰值為5000次,單機每秒請求DNS峰值超過閾值後,將面臨限速風險,可用性SLA將無法保證。為了緩解此類情況,建議您從如下兩個方面考慮:
擴容ECS數量,使得DNS查詢請求能夠負載分攤到更多的ECS,從而降低每個ECS的DNS查詢請求量;
ECS主機開啟nscd(Name Service Cache Daemon)服務進行緩衝加速;
說明nscd服務開啟情況下,會導致變更解析記錄的生效時間更長。
什麼是nscd?
nscd是一個系統快取服務,用於緩衝名稱服務資訊,例如passwd、group、hosts、services以及netgroup等,它可以提高系統查詢快取命中率,從而減少對名稱服務的查詢次數和流量消耗,進而加快服務響應速度,本實踐主要用於降低ECS的DNS查詢請求數,所以重點介紹hosts配置。
nscd常用Linux指令
指令 | 說明 |
yum install -y nscd | 安裝nscd。 |
systemctl start nscd | 啟動nscd。 |
systemctl stop nscd | 停止nscd。 |
systemctl restart nscd | 重新啟動nscd。 |
systemctl status nscd | 查詢nscd的運行狀態。 |
nscd -g | 輸出生效後的各個參數選項配置情況和快取命中率統計。 |
nscd -i | 使指定的緩衝失效,可以指定passwd、group、hosts、services、netgroup等,例如:nscd -i hosts。 |
cat /etc/nscd.conf | 查詢當前nscd配置詳情。 |
vi /etc/nscd.conf | 修改nscd配置參數。 |
安裝nscd
nscd一般在Linux作業系統中預設安裝。如果不確定當前ECS是否已經安裝了nscd,可以執行以下命令進行檢測:
systemctl status nscd #檢查nscd的運行狀態。如果檢測結果如下,則說明沒有安裝nscd。
[root@iZbp1fd580caxxxxxxxxxxxxtZ ~]# systemctl status nscd
Unit nscd.service could not be found.執行以下命令進行安裝:
yum install -y nscd[root@izbp1fd580ca5xxx ~]# yum install -y nscd
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Resolving Dependencies
--> Running transaction check
---> Package nscd.x86_64 0:2.17-326.el7_9 will be installed
--> Processing Dependency: glibc = 2.17-326.el7_9 for package: nscd-2.17-326.el7_9.x86_64
--> Running transaction check
---> Package glibc.x86_64 0:2.17-307.el7.1 will be updated
--> Processing Dependency: glibc = 2.17-307.el7.1 for package: glibc-common-2.17-307.el7.1.x86_64
---> Package glibc.x86_64 0:2.17-326.el7_9 will be an update
--> Running transaction check
---> Package glibc-common.x86_64 0:2.17-307.el7.1 will be updated
---> Package glibc-common.x86_64 0:2.17-326.el7_9 will be an update
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
nscd x86_64 2.17-326.el7_9 updates 289 k
Updating for dependencies:
glibc x86_64 2.17-326.el7_9 updates 3.6 M
glibc-common x86_64 2.17-326.el7_9 updates 12 M
Transaction Summary
================================================================================
Install 1 Package
Upgrade ( 2 Dependent packages)
Total download size: 15 M
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
(1/3): glibc-2.17-326.el7_9.x86_64.rpm | 3.6 MB 00:00:00
(2/3): nscd-2.17-326.el7_9.x86_64.rpm | 289 kB 00:00:00
(3/3): glibc-common-2.17-326.el7_9.x86_64.rpm | 12 MB 00:00:00
--------------------------------------------------------------------------------
Total 43 MB/s | 15 MB 00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Updating : glibc-common-2.17-326.el7_9.x86_64 1/5
Updating : glibc-2.17-326.el7_9.x86_64 2/5
Installing : nscd-2.17-326.el7_9.x86_64 3/5
Cleanup : glibc-common-2.17-307.el7.1.x86_64 4/5
Cleanup : glibc-2.17-307.el7.1.x86_64 5/5
Verifying : glibc-2.17-326.el7_9.x86_64 1/5
Verifying : glibc-common-2.17-326.el7_9.x86_64 2/5
Verifying : nscd-2.17-326.el7_9.x86_64 3/5
Verifying : glibc-common-2.17-307.el7.1.x86_64 4/5
Verifying : glibc-2.17-307.el7.1.x86_64 5/5
Installed:
nscd.x86_64 0:2.17-326.el7_9
Dependency Updated:
glibc.x86_64 0:2.17-326.el7_9 glibc-common.x86_64 0:2.17-326.el7_9
Complete!此時再執行命令查看nscd的運行狀態,nscd已經安裝但是處於未啟動狀態。
[root@izbp1fxxx ~]# systemctl status nscd
● nscd.service - Name Service Cache Daemon
Loaded: loaded (/usr/lib/systemd/system/nscd.service; disabled; vendor preset: disabled)
Active: inactive (dead)開啟nscd服務
輸入如下指令開啟nscd服務:
systemctl start nscd此時再執行命令查看nscd的運行狀態,nscd已經處於運行狀態。
[root@izbp1fd58cxxx ~]# systemctl status nscd
● nscd.service - Name Service Cache Daemon
Loaded: loaded (/usr/lib/systemd/system/nscd.service; disabled; vendor preset: disabled)
Active: active (running) since Fri 2023-08-11 15:00:19 CST; 19s ago
Process: 12520 ExecStart=/usr/sbin/nscd $NSCD_OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 12521 (nscd)
CGroup: /system.slice/nscd.service
└─12521 /usr/sbin/nscdCentOS/RedHat作業系統安裝nscd指令:yum install -y nscd;
Debian/Ubuntu作業系統安裝nscd指令:apt-get install -y nscd;
nscd配置參數
nscd的預設設定檔路徑在/etc/nscd.conf,可以通過執行如下指令查看nscd配置:
cat /etc/nscd.conf[root@iZbp1iy9t5ctdys1... ~]# cat /etc/nscd.conf
#
# /etc/nscd.conf
#
# An example Name Service Cache config file. This file is needed by nscd.
#
# WARNING: Running nscd with a secondary caching service like sssd may lead to
# unexpected behaviour, especially with how long entries are cached.
#
# Legal entries are:
#
# logfile <file>
# debug-level <level>
# threads <initial #threads to use>
# max-threads <maximum #threads to use>
# server-user <user to run server as instead of root>
# server-user is ignored if nscd is started with -S parameters
# stat-user <user who is allowed to request statistics>
# reload-count unlimited|<number>
# paranoia <yes|no>
# restart-interval <time in seconds>
#
# enable-cache <service> <yes|no>
# positive-time-to-live <service> <time in seconds>
# negative-time-to-live <service> <time in seconds>
# suggested-size <service> <prime number>
# check-files <service> <yes|no>
# persistent <service> <yes|no>
#
# shared <service> <yes|no>
# NOTE: Setting 'shared' to a value of 'yes' will accelerate the lookup
# with the help of the client, but these lookups will not be
# counted as cache hits i.e. 'nscd -g' may show '0%'.
#
# max-db-size <service> <number bytes>
# auto-propagate <service> <yes|no>
#
# Currently supported cache names (services): passwd, group, hosts, services
#
logfile /var/log/nscd.log
# threads 4
# max-threads 32
server-user nscd
# stat-user somebody
debug-level 1
# reload-count 5
paranoia no
# restart-interval 3600
enable-cache passwd no
positive-time-to-live passwd 600
negative-time-to-live passwd 20
suggested-size passwd 211
check-files passwd yes
persistent passwd yes
shared passwd yes
max-db-size passwd 33554432
auto-propagate passwd yes關鍵配置參數說明
配置參數 | 說明 |
debug-level | |
reload-count | 跟緩衝主動重新整理有關,此配置參數決定了請求成功緩衝主動去查詢並更新緩衝的次數。 |
paranoia | 偏執模式,如果開啟則nscd會周期性地重啟。 |
restart-interval | 如果paranoia開啟,此參數表示重啟間隔時間。 |
enable-cache | 開啟快取服務。 |
positive-time-to-live | 請求成功響應緩衝的存留時間。 |
negative-time-to-live | 請求失敗響應緩衝的存留時間,建議設定為0,以免失敗的緩衝影響業務請求。 |
check-files | 定期檢查/etc/passwd、/etc/group、/etc/hosts等快取檔案的修改時間,如果檔案自上次檢查以來有過更改,則緩衝失效。 |
persistent | 開啟時,nscd在重啟之後會保留之前的緩衝內容,如果paranoia開啟,建議開啟此功能。 |
shared | 用於服務nscd資料庫的記憶體映射與用戶端共用,預設為 yes,如果想利用 |
max-db-size | nscd 快取資料庫的最大大小,單位為Byte。 |
positive-time-to-live沒有什麼實際意義,TTL值以DNS查詢請求返回的TTL為準。
nscd緩衝效果檢測
nscd關閉時撥測
在ECS上執行如下命令對53連接埠UDP報文進行抓包:
tcpdump -i any udp and port 53然後在nscd的狀態為關閉情況下,在ECS上執行多次如下指令,連續撥測三次。
ping -c 1 -n www.taobao.com #對網域名稱www.taobao.com發一次ping指令。[root@iZbp1iy9t5ctdysl6xxx ~]# ping -c 1 -n www.taobao.com PING www.taobao.com.danuoyi.tbcache.com (61.174.43.211) 56(84) bytes of data. 64 bytes from 61.174.43.211: icmp_seq=1 ttl=53 time=5.22 ms --- www.taobao.com.danuoyi.tbcache.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.225/5.225/5.225/0.000 ms [root@iZbp1iy9t5ctdysl6xxx ~]# ping -c 1 -n www.taobao.com PING www.taobao.com.danuoyi.tbcache.com (61.174.43.210) 56(84) bytes of data. 64 bytes from 61.174.43.210: icmp_seq=1 ttl=53 time=10.5 ms --- www.taobao.com.danuoyi.tbcache.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 10.534/10.534/10.534/0.000 ms [root@iZbp1iy9t5ctdysl6xxx ~]# ping -c 1 -n www.taobao.com PING www.taobao.com.danuoyi.tbcache.com (61.174.43.211) 56(84) bytes of data. 64 bytes from 61.174.43.211: icmp_seq=1 ttl=53 time=5.20 ms --- www.taobao.com.danuoyi.tbcache.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.205/5.205/5.205/0.000 ms查看相對應的抓包情況,能夠發現53連接埠3次DNS請求查詢,並且每次DNS請求查詢都返回網域名稱解析記錄,證明DNS請求查詢記錄並未被緩衝,ECS每次都要通過53連接埠發送DNS請求查詢。
10:02:26.112177 IP 10.81.84.51.47500 > 10.143.22.116.domain: 18558+ A? www.taobao.com. (32) 10:02:26.112280 IP 10.143.22.116.domain > 10.81.84.51.47500: 18558 3/0/0 CNAME www.taobao.com.danuoyi.tbcache.com., A 61.174.43.211, A 61.174.43.210 (112) 10:02:35.447683 IP 10.81.84.51.60549 > 10.143.22.116.domain: 43179+ A? www.taobao.com. (32) 10:02:35.447784 IP 10.143.22.116.domain > 10.81.84.51.60549: 43179 3/0/0 CNAME www.taobao.com.danuoyi.tbcache.com., A 61.174.43.210, A 61.174.43.211 (112) 10:02:41.356464 IP 10.81.84.51.37941 > 10.143.22.116.domain: 33455+ A? www.taobao.com. (32) 10:02:41.356627 IP 10.143.22.116.domain > 10.81.84.51.37941: 33455 3/0/0 CNAME www.taobao.com.danuoyi.tbcache.com., A 61.174.43.211, A 61.174.43.210 (112)
nscd開啟時撥測
執行啟動指令將nscd服務開啟,並通過狀態查詢指令確定nscd為開啟狀態。
在nscd的狀態為開啟情況下,在ECS上執行如下指令,連續撥測6次。同時需要對ECS的53連接埠進行抓包。
ping -c 1 -n www.taobao.com[root@iZbp1iy9t5ctxxx ~]# ping -c 1 -n www.taobao.com PING www.taobao.com.danuoyi.tbcache.com (61.174.43.211) 56(84) bytes of data. 64 bytes from 61.174.43.211: icmp_seq=1 ttl=53 time=5.21 ms --- www.taobao.com.danuoyi.tbcache.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.213/5.213/5.213/0.000 ms [root@iZbp1iy9t5ctdysl6xxx ~]# ping -c 1 -n www.taobao.com PING www.taobao.com.danuoyi.tbcache.com (61.174.43.211) 56(84) bytes of data. 64 bytes from 61.174.43.211: icmp_seq=1 ttl=53 time=5.25 ms --- www.taobao.com.danuoyi.tbcache.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.254/5.254/5.254/0.000 ms [root@iZbp1iy9t5ctdysl6xxx ~]# ping -c 1 -n www.taobao.com PING www.taobao.com.danuoyi.tbcache.com (61.174.43.211) 56(84) bytes of data. 64 bytes from 61.174.43.211: icmp_seq=1 ttl=53 time=5.21 ms --- www.taobao.com.danuoyi.tbcache.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.212/5.212/5.212/0.000 ms [root@iZbp1iy9t5ctdysl6xxx ~]# ping -c 1 -n www.taobao.com PING www.taobao.com.danuoyi.tbcache.com (61.174.43.211) 56(84) bytes of data. 64 bytes from 61.174.43.211: icmp_seq=1 ttl=53 time=5.25 ms --- www.taobao.com.danuoyi.tbcache.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.253/5.253/5.253/0.000 ms [root@iZbp1iy9t5ctdysl6xxx ~]# ping -c 1 -n www.taobao.com PING www.taobao.com.danuoyi.tbcache.com (61.174.43.211) 56(84) bytes of data. 64 bytes from 61.174.43.211: icmp_seq=1 ttl=53 time=5.20 ms --- www.taobao.com.danuoyi.tbcache.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.206/5.206/5.206/0.000 ms [root@iZbp1iy9t5ctdysl6xxx ~]# ping -c 1 -n www.taobao.com PING www.taobao.com.danuoyi.tbcache.com (61.174.43.211) 56(84) bytes of data. 64 bytes from 61.174.43.211: icmp_seq=1 ttl=53 time=5.24 ms --- www.taobao.com.danuoyi.tbcache.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.247/5.247/5.247/0.000 ms查看相對應的抓包情況,能夠發現53連接埠只抓到1次DNS請求查詢,證明DNS請求查詢命中nscd緩衝,沒有通過53連接埠發送DNS請求查詢。
抓包結果中,DNS A 記錄查詢的目標網域名稱為
www.taobao.com,具體輸出如下。10:20:46.011924 IP 10.81.84.51.52131 > 10.143.22.118.domain: 12973+ A? www.taobao.com. (32) 10:20:46.012115 IP 10.143.22.118.domain > 10.81.84.51.52131: 12973 3/0/0 CNAME www.taobao.com.danuoyi.tbcache.com., A 61.174.43.211, A 61.174.43.210 (112) 10:20:46.012221 IP 10.81.84.51.56349 > 10.143.22.118.domain: 1209+ AAAA? www.taobao.com. (32) 10:20:46.013867 IP 10.143.22.118.domain > 10.81.84.51.56349: 1209 3/0/0 CNAME www.taobao.com.danuoyi.tbcache.com., AAAA 240e:f7:a093:101:3::3e8, AAAA 240e:f7:a093:101:3::3e7 (136)重要您在抓包的過程中會發現即使ping指令已經執行完畢,但是每隔一段時間tcpdump依舊可以查到相關網域名稱DNS查詢請求報文,這是由於nscd的主動重新整理機制導致的,屬於正常現象。可以通過將reload-count 參數設定為0關閉主動重新整理機制。
您也可以通過執行以下指令來查詢快取命中資訊,在一段時間內多次執行ping指令後查看快取命中情況。
nscd -g #輸出生效後的各個參數選項配置情況和快取命中率統計等。在輸出結果的 hosts cache 區塊中,cache hits on positive entries 值為 64,快取命中率為 52%。
group cache: no cache is enabled yes cache is persistent yes cache is shared 0 suggested size 0 total data pool size 0 used data pool size 3600 seconds time to live for positive entries 60 seconds time to live for negative entries 0 cache hits on positive entries 0 cache hits on negative entries 0 cache misses on positive entries 0 cache misses on negative entries 0% cache hit rate 0 current number of cached values 0 maximum number of cached values 0 maximum chain length searched 0 number of delays on rdlock 0 number of delays on wrlock 0 memory allocations failed yes check /etc/group for changes hosts cache: yes cache is enabled yes cache is persistent no cache is shared 211 suggested size 216064 total data pool size 176 used data pool size 3600 seconds time to live for positive entries 20 seconds time to live for negative entries 64 cache hits on positive entries 0 cache hits on negative entries 50 cache misses on positive entries 7 cache misses on negative entries 52% cache hit rate 1 current number of cached values 6 maximum number of cached values 1 maximum chain length searched 0 number of delays on rdlock 0 number of delays on wrlock 0 memory allocations failed yes check /etc/hosts for changes