API tools faq
paste
Guest User

robots.txt

a guest
Apr 13th, 2016
207
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 0.68 KB | None | 0 0
raw download clone embed print report
  1. User-agent: Teleport
  2. Disallow: /
  3.  
  4. User-agent: TeleportPro
  5. Disallow: /
  6.  
  7. User-agent: EmailCollector
  8. Disallow: /
  9.  
  10. User-agent: EmailSiphon
  11. Disallow: /
  12.  
  13. User-agent: WebBandit
  14. Disallow: /
  15.  
  16. User-agent: WebZIP
  17. Disallow: /
  18.  
  19. User-agent: WebReaper
  20. Disallow: /
  21.  
  22. User-agent: WebStripper
  23. Disallow: /
  24.  
  25. User-agent: Web Downloader
  26. Disallow: /
  27.  
  28. User-agent: WebCopier
  29. Disallow: /
  30.  
  31. User-agent: Offline Explorer Pro
  32. Disallow: /
  33.  
  34. User-agent: HTTrack Website Copier
  35. Disallow: /
  36.  
  37. User-agent: Offline Commander
  38. Disallow: /
  39.  
  40. User-agent: Leech
  41. Disallow: /
  42.  
  43. User-agent: WebSnake
  44. Disallow: /
  45.  
  46. User-agent: BlackWidow
  47. Disallow: /
  48.  
  49. User-agent: HTTP Weazel
  50. Disallow: /
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!