Advertisement
Guest User

rixstep robots.txt

a guest
Aug 7th, 2012
98
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 0.47 KB | None | 0 0
  1. Contents of: http://rixstep.com/robots.txt
  2. Date: 2012-08-07
  3.  
  4. User-agent: *
  5. Crawl-delay: 10
  6. Disallow: /7/
  7. User-agent: DirBuster-0.12
  8. Disallow: /
  9. User-agent: heritrix/1.8.0
  10. Disallow: /
  11. User-agent: ia_archiver
  12. Disallow: /
  13. User-agent: iGetter
  14. Disallow: /
  15. User-agent: larbin_2.6.3
  16. Disallow: /
  17. User-agent: MJ12bot
  18. Disallow: /
  19. User-agent: NASA Search
  20. Disallow: /
  21. User-agent: TurnitinBot
  22. Disallow: /
  23. User-agent: VoilaBot
  24. Disallow: /
  25. User-agent: yolinkBot
  26. Disallow: /
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement