...
| Code Block | ||
|---|---|---|
| ||
User-agent: Googlebot Crawl-delay: 2 Allow: /project/*/downloads/pdf User-agent: * Crawl-delay: 10 Disallow: /project/*/downloads/pdf |
| Warning |
|---|
IMPORTANT: Completely blocking Googlebot and bingbot will negatively affect your search engine indexing and listing. |
Common Bot User-Agents - The most common user agents, or bots/crawlers, that will be checking your site are below. You can utilize these user-agents in your robots.txt formatting. Please see below:
- Baidu - baiduspider
- Bing - bingbot
- Bing - msnbot
- Bing (Images and Video) - msnbot-media
- Bing (Ads) - adidxbot
- Google - Googlebot
- Google (Images) - Googlebot-Image
- Google (Mobile) - Googlebot-Mobile
- Google (News) - Googlebot-News
- Google (Video) - Googlebot-Video
- Google (Commerce) - Storebot-Google
- Google (AdSense) - Mediapartners-Google
- Google (AdWords) - AdsBot-Google
- Yahoo! - slurp
- Yandex - yandex YandexBot
| Info |
|---|
You can check the bot activity on your site by clicking the "Awstats" icon inside your cPanel, selecting your domain, and scrolling to "Robots/Spiders visitors". |