Advertisement
Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- Contents of: http://rixstep.com/robots.txt
- Date: 2012-08-07
- User-agent: *
- Crawl-delay: 10
- Disallow: /7/
- User-agent: DirBuster-0.12
- Disallow: /
- User-agent: heritrix/1.8.0
- Disallow: /
- User-agent: ia_archiver
- Disallow: /
- User-agent: iGetter
- Disallow: /
- User-agent: larbin_2.6.3
- Disallow: /
- User-agent: MJ12bot
- Disallow: /
- User-agent: NASA Search
- Disallow: /
- User-agent: TurnitinBot
- Disallow: /
- User-agent: VoilaBot
- Disallow: /
- User-agent: yolinkBot
- Disallow: /
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement