Search engine lines let you direct DNS requests from search engine crawlers to a dedicated server address, helping you manage crawler traffic separately from regular user traffic.
Overview
Search engine crawlers, also known as web spiders or web robots, are programs or scripts that automatically crawl and index web content based on predefined rules.
Scenarios
-
Search engine crawlers consume server traffic and bandwidth. By configuring search engine lines to point to a dedicated server address, you can direct all crawler DNS requests to that address and control the crawling path.
-
During temporary site closures, you can use search engine lines to set up a dedicated line for search engines. This allows crawlers to continue indexing your site content, reducing the impact on your SEO ranking.
Search engine line enumeration
|
Primary line (line CODE) |
Secondary line (line CODE) |
Tertiary line (line CODE) |
|
Search engine (search) |
Google (google) |
Google_Mainland China (cn_search_google) |
|
Google_Overseas (os_search_google) |
||
|
Baidu (baidu) |
Baidu_Mainland China (cn_search_baidu) |
|
|
Baidu_Overseas (os_search_baidu) |
||
|
Bing (biying) |
Bing_Mainland China (cn_search_biying) |
|
|
Bing_Overseas (os_search_biying) |
||
|
Sogou (sougou) |
Sogou_Mainland China (cn_search_sougou) |
|
|
Sogou_Overseas (os_search_sougou) |
||
|
Qihu (qihu) |
Qihu_Mainland China (cn_search_qihu) |
|
|
Qihu_Overseas (os_search_qihu) |
||
|
Youdao (youdao) |
Youdao_Mainland China (cn_search_youdao) |
|
|
Youdao_Overseas (os_search_youdao) |
||
|
Yahoo (yahoo) |
Yahoo_Mainland China (cn_search_yahoo) |
|
|
Yahoo_Overseas (os_search_yahoo) |
FAQ
-
Why doesn't using the Google Chrome browser hit the search engine line?
Answer: Search engine lines match the LocalDNS exit IP of search engine crawlers, which differs from the LocalDNS exit IP used by browsers.
-
Why might search crawlers fail to capture the configured search engine line?
Answer: If the crawler server's local DNS does not use the search engine's own LocalDNS, the crawler cannot match the configured search engine line during DNS resolution. For example, if the Baidu crawler server does not use Baidu's LocalDNS, the Baidu search engine line will not take effect.