Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

A robots.txt file instructs automated search engine crawlers which URLs the crawler can access on your site. These files can be used to provide details about which site directories or files should or should not be crawled, how quickly they should be re-crawled, and which bots are welcome or not on your website.

Warningnote

The robots.txt file has a very strict formatting requirement, such as JPG is not the same as jpg, and other notes below. If you ever have any questions, feel free to contact GlowHost support.

Important Details About robots.txt

...

Code Block
languagexml
User-agent: Googlebot
Crawl-delay: 2
Allow: /project/*/downloads/pdf

User-agent: *
Crawl-delay: 10
Disallow: /project/*/downloads/pdf
Warning

IMPORTANT: Completely blocking Googlebot and bingbot will negatively affect your search engine indexing and listing.


Common Bot User-Agents -
The most common user agents, or bots/crawlers, that will be checking your site are below. You can utilize these user-agents in your robots.txt formatting. Please see below:

  • Baidu - baiduspider
  • Bing - bingbot
  • Bing - msnbot
  • Bing (Images and Video) - msnbot-media
  • Bing (Ads) - adidxbot
  • Google - Googlebot
  • Google (Images) - Googlebot-Image
  • Google (Mobile) - Googlebot-Mobile
  • Google (News) - Googlebot-News
  • Google (Video) - Googlebot-Video
  • Google (Commerce) - Storebot-Google
  • Google (AdSense) - Mediapartners-Google
  • Google (AdWords) - AdsBot-Google
  • Yahoo! - slurp
  • Yandex - yandex YandexBot
Info

You can check the bot activity on your site by clicking the "Awstats" icon inside your cPanel, selecting your domain, and scrolling to "Robots/Spiders visitors".