Overview
Search engines use web crawlers to automatically grab data in the World Wide Web according to specific rules. As programs or scrips, web crawlers are also known as spiders or spiderbots.
Scenarios
The traffic and bandwidth of website servers are consumed when the websites are accessed by web crawlers. You can add Domain Name System (DNS) records to add search engine lines. The DNS records point the domain names of the websites to specific server addresses. This way, DNS requests for the domain names from web crawlers can be redirected to the specified server addresses.
If you need to temporarily shut down your website and maintain the search engine optimization (SEO) ranking of your website, you can specify a search engine line to ensure that web crawlers can grab the information about the website after the website is shut down.
Search engine lines
Primary line (Line CODE) | Secondary line (Line CODE) | Tertiary line (Line CODE) |
Search Engine | Google (google) | Google_Chinese Mainland (cn_search_google) |
Google_Outside the Chinese Mainland (os_search_google) | ||
Baidu (baidu) | Baidu_Chinese Mainland (cn_search_baidu) | |
Baidu_Outside the Chinese Mainland (os_search_baidu) | ||
Bing (biying) | Bing_Chinese Mainland (cn_search_biying) | |
Bing_Outside the Chinese Mainland (os_search_biying) | ||
Sogou (sougou) | Sogou_Chinese Mainland (cn_search_sougou) | |
Sogou_Outside the Chinese Mainland (os_search_sougou) | ||
Qihoo (qihu) | Qihoo_Chinese Mainland (cn_search_qihu) | |
Qihoo_Outside the Chinese Mainland (os_search_qihu) | ||
Youdao (youdao) | Youdao_Chinese Mainland (cn_search_youdao) | |
Youdao_Outside the Chinese Mainland (os_search_youdao) | ||
Yahoo (yahoo) | Yahoo_Chinese Mainland (cn_search_yahoo) | |
Yahoo_Outside the Chinese Mainland (os_search_yahoo) |
FAQ
Why do DNS requests sent from Google Chrome not match search engine lines?
You can use search engine lines to specify the egress IP addresses of local DNS servers used by web crawlers. The egress IP addresses are not those of the local DNS servers used by browsers.
Why do DNS requests from web crawlers of a search engine fail to match the search engine line?
Take Baidu web crawlers as an example. If the local DNS servers used by Baidu web crawlers are not the local DNS servers that Baidu web crawlers are supposed to use, the DNS requests from Baidu web crawlers cannot match the Baidu search engine line.