Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Code Block
languagexml
User-agent: Googlebot
Crawl-delay: 2
Allow: /project/*/downloads/pdf

User-agent: *
Crawl-delay: 10
Disallow: /project/*/downloads/pdf
Warning

IMPORTANT: Completely blocking Googlebot and bingbot will negatively affect your search engine indexing and listing.


Common Bot User-Agents -
The most common user agents, or bots/crawlers, that will be checking your site are below. You can utilize these user-agents in your robots.txt formatting. Please see below:

  • Baidu - baiduspider
  • Bing - bingbot
  • Bing - msnbot
  • Bing (Images and Video) - msnbot-media
  • Bing (Ads) - adidxbot
  • Google - Googlebot
  • Google (Images) - Googlebot-Image
  • Google (Mobile) - Googlebot-Mobile
  • Google (News) - Googlebot-News
  • Google (Video) - Googlebot-Video
  • Google (Commerce) - Storebot-Google
  • Google (AdSense) - Mediapartners-Google
  • Google (AdWords) - AdsBot-Google
  • Yahoo! - slurp
  • Yandex - yandex YandexBot
Info

You can check the bot activity on your site by clicking the "Awstats" icon inside your cPanel, selecting your domain, and scrolling to "Robots/Spiders visitors".