edit-icon download-icon

Search engine robots

Last Updated: Sep 30, 2017

Search engine robots are automated software agents (also known as spiders or crawlers) that automatically crawl the World Wide Web for information.

Using Alibaba Cloud DNS, you can add a DNS record to allow a search engine robot to index your website from the specified IP address.

Scenario

An Alibaba Cloud DNS record using a search engine robot line can be applied to the following scenarios:

  • Your enterprise has multiple servers and you want to specify one server to be crawled by various search engine crawlers. Using a search engine robot line allows you to effectively control the crawling paths used by web crawlers. This method can resolve issues generated by search engine robots such as website bandwidth occupation, traffic resources consumption, and slow website access speed.

  • You want to suspend your website without affecting its search engine indexing and weighting. You can change the default line to the search engine robots.

Use search engine robot lines

When adding a record, you can specify the ISP line to a search engine robot. The available search engine robots include Google robot, Baidu robot, and Bing robot.

  • Google robot: when specified as the ISP line, Google crawlers are directed to the address specified in this record. Crawlers of other search engines are directed to the default line.

  • Baidu robot: when specified as the ISP line, Baidu crawlers are directed to the address specified in this record. Crawlers of other search engines are directed to the default line.

  • Bing robot: when specified as the ISP line, Bing crawlers are directed to the address specified in this record. Crawlers of other search engines are directed to the default line.

Example

Add the following A records for a domain name:

Record Host ISP Line Value
1 www Search Engine Robots 1.1.1.1
2 www Google 2.2.2.2

By doing this,

  • Google crawlers are directed to 2.2.2.2 for indexing.
  • Baidu and Bing crawlers are directed to 1.1.1.1 for indexing.
  • Crawlers of search engines other than Google, Baidu, or Bing are directed to the default line for indexing.
Thank you! We've received your feedback.