A robots.txt file instructs automated search engine crawlers which URLs the crawler can access on your site. These files can be used to provide details about which site directories or files should or should not be crawled, how quickly they should be re-crawled, and which bots are welcome or not on your website.
| Warningnote |
|---|
The robots.txt file has a very strict formatting requirement, such as JPG is not the same as jpg, and other notes below. If you ever have any questions, feel free to contact GlowHost support. |
Important Details About robots.txt
...
| Code Block | ||
|---|---|---|
| ||
User-agent: Googlebot Crawl-delay: 2 Allow: /project/*/downloads/pdf User-agent: * Crawl-delay: 10 Disallow: /project/*/downloads/pdf |
| Warning |
|---|
IMPORTANT: Completely blocking Googlebot and bingbot will negatively affect your search engine indexing and listing. |
Common Bot User-Agents - The most common user agents, or bots/crawlers, that will be checking your site are below. You can utilize these user-agents in your robots.txt formatting. Please see below:
- Baidu - baiduspider
- Bing - bingbot
- Bing - msnbot
- Bing (Images and Video) - msnbot-media
- Bing (Ads) - adidxbot
- Google - Googlebot
- Google (Images) - Googlebot-Image
- Google (Mobile) - Googlebot-Mobile
- Google (News) - Googlebot-News
- Google (Video) - Googlebot-Video
- Google (Commerce) - Storebot-Google
- Google (AdSense) - Mediapartners-Google
- Google (AdWords) - AdsBot-Google
- Yahoo! - slurp
- Yandex - yandex YandexBot
| Info |
|---|
You can check the bot activity on your site by clicking the "Awstats" icon inside your cPanel, selecting your domain, and scrolling to "Robots/Spiders visitors". |