All Products
Search
Document Center

SearchEngine

Last Updated: Apr 13, 2020

Overview

Search engines use programs or scripts to collect content on the Internet based on specific rules in an automated manner. The programs or scripts are known as web crawlers, search engine spiders, or search engine robots.

Scenarios

  • A search engine robot consumes server traffic and bandwidth when accessing a website. To reduce server load from the search engine robot, you can configure a DNS record for the website domain and specify the search engine robot as the ISP line.

  • If you want to temporarily close your website for search engine optimization (SEO) without affecting the website ranking, you can specify the search engine robot as the ISP line. This allows the search engine robot to index your website from the specified IP address.

Procedure

For example, you can specify Baiduspider bot as the ISP line and configure an A record to direct Baiduspider bot to the IP address of your Web server, such as 2.2.2.2. Baiduspider bot will establish a connection with the Web server and collect the content of your website. Users can retrieve pictures and videos on your website by using the Baidu search engine.

1 . Log on to the Alibaba Cloud DNS console.

2 . In the left-side navigation pane, click Manage DNS. On the Domains tab, click a domain name to open the DNS Settings page.
搜索引擎线路-1

3 . On the DNS Settings page, click Add Record. In the Add Record dialog box, select Search Engine Robots from the ISP Line drop-down list and specify a search engine robot.
搜索引擎线路-2
搜索引擎线路-3

Q&A

1、 Do search engine lines need to be charged separately?

A: Free of charge.

2、search engine line support which search engine business?

A:search engine, Google, baidu, bing, youdao.

3、Why do search crawlers not crawl to the search engine configuration?

A:Take baidu crawler for example, if the localdns of baidu crawler server is not using baidu’s localdns, then when DNS query, baidu crawler will appear unable to grasp the search engine configuration of the line.Example of the following figure

搜索引擎原理