The bot-detect plug-in identifies and blocks web crawlers from accessing your websites.
Plug-in type
Security protection.
Fields
|
Field |
Data type |
Required |
Default value |
Description |
|
allow |
array of string |
No |
- |
Regular expressions to match the User-Agent header. Matched requests are allowed. |
|
deny |
array of string |
No |
- |
Regular expressions to match the User-Agent header. Matched requests are blocked. |
|
blocked_code |
number |
No |
403 |
The HTTP status code returned when a request is blocked. |
|
blocked_message |
string |
No |
- |
The HTTP response body returned when a request is blocked. |
If neither the allow nor the deny field is configured, the default crawler identification logic is used. Use allow to exempt requests that match the default crawler rules, and deny to add custom crawler identification rules.
Examples
Allow requests that hit the default crawler identification logic
allow:
- ".*Go-http-client.*"Without this configuration, requests from the Golang HTTP library are identified as crawlers and blocked.
Add crawler identification logic
deny:
- "spd-tools.*"The following requests are blocked by this rule:
curl http://example.com -H 'User-Agent: spd-tools/1.1'
curl http://exmaple.com -H 'User-Agent: spd-tools'Enable security protection rules for specific routes or domain names
Save the route-a and route-b routes with empty configuration data.
Apply the following plug-in configuration to the *.example.com and test.com domain names:
allow:
- ".*Go-http-client.*"-
route-aandroute-bare routes specified when you create gateway routes. If a client request matches one of these routes, the built-in default rules are used to identify web crawlers. -
*.example.comandtest.commatch domain names in requests. If a client request matches one of these domain names, the corresponding rules take effect. -
Rules take effect in sequence. Once a rule is matched, subsequent rules are skipped.