API tools faq
paste
Phylum

Blocking Bots with .htaccess

Phylum
May 23rd, 2013
300
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 1.91 KB | None | 0 0
raw download clone embed print report
  1. ErrorDocument 503 "Site disabled for crawling"
  2. RewriteEngine on
  3. Options +FollowSymlinks
  4.  
  5. RewriteCond %{HTTP_USER_AGENT} ^$ [NC,OR]
  6. RewriteCond %{HTTP_USER_AGENT} ^\Googlebot$ [NC,OR]
  7. RewriteCond %{HTTP_USER_AGENT} ^robot$ [NC,OR]
  8. RewriteCond %{HTTP_USER_AGENT} ^crawl$ [NC,OR]
  9. RewriteCond %{HTTP_USER_AGENT} ^spider$ [NC,OR]
  10. RewriteCond %{HTTP_USER_AGENT} ^Yahoo\ Slurp$ [NC,OR]
  11. RewriteCond %{HTTP_USER_AGENT} ^discovery$ [NC,OR]
  12. RewriteCond %{HTTP_USER_AGENT} ^MSNBot-Media$ [NC,OR]
  13. RewriteCond %{HTTP_USER_AGENT} ^bot\*$ [NC,OR]
  14. RewriteCond %{HTTP_USER_AGENT} ^\*bot$ [NC,OR]
  15. RewriteCond %{HTTP_USER_AGENT} ^\*bot\*$ [NC,OR]
  16. RewriteCond %{HTTP_USER_AGENT} ^.*(Google).*$
  17. # Not sure which is preferred/ideal
  18. #RewriteRule .* - [F,L]
  19. #RewriteRule . - [F,L]
  20. #RewriteRule .? - [F,L]
  21. #RewriteRule ^(.*)$ - [F,L]
  22. #RewriteRule .* - [R=503,L]
  23. RewriteRule . - [F,L]
  24.  
  25. SetEnvIfNoCase User-Agent ^$ unwanted_bot [NC,OR]
  26. SetEnvIfNoCase User-Agent "^Googlebot" unwanted_bot [NC,OR]
  27. SetEnvIfNoCase User-Agent "^robot" unwanted_bot [NC,OR]
  28. SetEnvIfNoCase User-Agent "^crawl" unwanted_bot [NC,OR]
  29. SetEnvIfNoCase User-Agent "^spider" unwanted_bot [NC,OR]
  30. SetEnvIfNoCase User-Agent "^Yahoo\ Slurp" unwanted_bot [NC,OR]
  31. SetEnvIfNoCase User-Agent "^discovery" unwanted_bot [NC,OR]
  32. SetEnvIfNoCase User-Agent "^MSNBot-Media" unwanted_bot [NC,OR]
  33. SetEnvIfNoCase User-Agent "^bot\*" unwanted_bot [NC,OR]
  34. SetEnvIfNoCase User-Agent "^\*bot" unwanted_bot [NC,OR]
  35. SetEnvIfNoCase User-Agent "^\*bot\*" unwanted_bot [NC]
  36.  
  37. BrowserMatchNoCase ^Googlebot unwanted_bot
  38. BrowserMatchNoCase ^robot unwanted_bot
  39. BrowserMatchNoCase ^crawl unwanted_bot
  40. BrowserMatchNoCase ^spider unwanted_bot
  41. BrowserMatchNoCase ^discovery unwanted_bot
  42. BrowserMatchNoCase ^MSNBot-Media unwanted_bot
  43. BrowserMatchNoCase ^bot\* unwanted_bot
  44. BrowserMatchNoCase ^\*bot unwanted_bot
  45. BrowserMatchNoCase ^\*bot\* unwanted_bot
  46.  
  47. Order Deny,Allow
  48. Deny from env=unwanted_bot
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!