...
| Code Block | ||
|---|---|---|
| ||
# Deny all robots that we do not specifically want to allow User-agent: * Disallow: / # Allow these robots only User-agent: slurp Allow: / User-agent: bingbot Allow: / User-agent: googlebot Allow: / |
But if robots.txt doesn't help, you may block bots in your .htaccess file. First of all, we need to find out how to identify a bot. You will need to check your raw access logs using appropriate option in your Cpanel. The "User Agent" string in the logs is the one we need. For example, in the line below you may see YandexBot string:
| Code Block |
|---|
HTTP/1.1" 200 927 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" |
| Code Block |
|---|
BrowserMatchNoCase YandexBot bad_bot
Order Deny,Allow
Deny from env=bad_bot |
The other bots can be blocked by adding BrowserMatchNoCase directive in the same way.
Related articles
| Content by Label | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
...