API tools faq
paste
Guest User

Block Bad Robots - Robots.txt

a guest
Oct 20th, 2013
3,585
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
HTML 4.43 KB
raw download clone embed print report
  1. User-agent: BLEXBot
  2. User-agent: Wget/1.6
  3. User-agent: Wget/1.5.3
  4. User-agent: FairAd Client
  5. User-agent: URL_Spider_Pro
  6. User-agent: Mozilla
  7. User-agent: turingos
  8. User-agent: Python-urllib
  9. User-agent: Teleport
  10. User-agent: Stanford Comp Sci
  11. User-agent: grub-client
  12. User-agent: WebBandit/3.50
  13. User-agent: ia_archiver/1.6
  14. User-agent: ProPowerBot/2.14
  15. User-agent: naver
  16. User-agent: RepoMonkey
  17. User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 2000)
  18. User-agent: SpankBot
  19. User-agent: WebZip
  20. User-agent: moget/2.1
  21. User-agent: gigabot
  22. User-agent: Xenu's Link Sleuth 1.1c
  23. User-agent: spanner
  24. User-agent: NetMechanic
  25. User-agent: CheeseBot
  26. User-agent: Zeus 32297 Webster Pro V2.9 Win32
  27. User-agent: Jetbot
  28. User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 98)
  29. User-agent: sootle
  30. User-agent: WebCopier
  31. User-agent: LNSpiderguy
  32. User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95)
  33. User-agent: CopyRightCheck
  34. User-agent: mozilla
  35. User-agent: EmailWolf
  36. User-agent: Keyword Density/0.9
  37. User-agent: WebmasterWorld Extractor
  38. User-agent: CherryPickerSE/1.0
  39. User-agent: moget
  40. User-agent: es
  41. User-agent: Webster Pro
  42. User-agent: ExtractorPro
  43. User-agent: Microsoft URL Control - 5.01.4511
  44. User-agent: Openfind
  45. User-agent: exabot
  46. User-agent: Hatena Antenna
  47. User-agent: psbot
  48. User-agent: TheNomad
  49. User-agent: AhrefsBot
  50. User-agent: http://www.SearchEngineWorld.com bot
  51. User-agent: rogerbot
  52. User-agent: suzuran
  53. User-agent: WWW-Collector-E
  54. User-agent: EmailCollector
  55. User-agent: PerMan
  56. User-agent: WebAuto
  57. User-agent: GetRight/4.2
  58. User-agent: mozilla/4
  59. User-agent: VCI WebViewer VCI WebViewer Win32
  60. User-agent: Xenu's
  61. User-agent: ia_archiver
  62. User-agent: BackDoorBot/1.0
  63. User-agent: Jetbot/1.0
  64. User-agent: humanlinks
  65. User-agent: WebEnhancer
  66. User-agent: DittoSpyder
  67. User-agent: Harvest/1.5
  68. User-agent: TeleportPro
  69. User-agent: RMA
  70. User-agent: BecomeBot
  71. User-agent: mozilla/3
  72. User-agent: scooter
  73. User-agent: True_Robot
  74. User-agent: Aqua_Products
  75. User-agent: b2w/0.1
  76. User-agent: BotALot
  77. User-agent: WebmasterWorldForumBot
  78. User-agent: LinkScan/8.1a Unix
  79. User-agent: Enterprise_Search/1.0
  80. User-agent: Copernic
  81. User-agent: Flaming AttackBot
  82. User-agent: cosmos
  83. User-agent: NICErsPRO
  84. User-agent: Openbot
  85. User-agent: Microsoft URL Control
  86. User-agent: EmailSiphon
  87. User-agent: grub
  88. User-agent: Stanford CompClub
  89. User-agent: Teoma
  90. User-agent: Kenjin Spider
  91. User-agent: Foobot
  92. User-agent: True_Robot/1.0
  93. User-agent: LexiBot
  94. User-agent: Stanford
  95. User-agent: CherryPickerElite/1.0
  96. User-agent: MSIECrawler
  97. User-agent: Oracle Ultra Search
  98. User-agent: mozilla/5
  99. User-agent: LinkWalker
  100. User-agent: larbin
  101. User-agent: Zeus Link Scout
  102. User-agent: Stanford Spiderboys
  103. User-agent: Alexibot
  104. User-agent: Crescent
  105. User-agent: BlekkoBot
  106. User-agent: Mister PiX
  107. User-agent: Website Quester
  108. User-agent: searchpreview
  109. User-agent: http://www.WebmasterWorld.com bot
  110. User-agent: Wget
  111. User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
  112. User-agent: Web Image Collector
  113. User-agent: lwp-trivial/1.34
  114. User-agent: CherryPicker
  115. User-agent: QueryN Metasearch
  116. User-agent: Nutch
  117. User-agent: Mata Hari
  118. User-agent: dotbot
  119. User-agent: Bookmark search tool
  120. User-agent: The Intraformant
  121. User-agent: toCrawl/UrlDispatcher
  122. User-agent: Gaisbot
  123. User-agent: WebBandit
  124. User-agent: RepoMonkey Bait & Tackle/v1.01
  125. User-agent: MIIxpc/4.2
  126. User-agent: Bullseye/1.0
  127. User-agent: BuiltBotTough
  128. User-agent: Enterprise_Search
  129. User-agent: BlowFish/1.0
  130. User-agent: asterias
  131. User-agent: lwp-trivial
  132. User-agent: dumbot
  133. User-agent: Iron33/1.0.2
  134. User-agent: Zeus
  135. User-agent: JennyBot
  136. User-agent: VCI
  137. User-agent: URL Control
  138. User-agent: hloader
  139. User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 95)
  140. User-agent: EroCrawler
  141. User-agent: BunnySlippers
  142. User-agent: libWeb/clsHTTP
  143. User-agent: Radiation Retriever 1.1
  144. User-agent: Microsoft URL Control - 6.00.8169
  145. User-agent: Offline Explorer
  146. User-agent: MJ12bot
  147. User-agent: Stanford CompSciClub
  148. User-agent: LinkextractorPro
  149. User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows XP)
  150. User-agent: Szukacz/1.4
  151. User-agent: SiteSnagger
  152. User-agent: ProWebWalker
  153. User-agent: NetAnts
  154. User-agent: InfoNaviRobot
  155. User-agent: Telesoft
  156. User-agent: WebZip/4.0
  157. User-agent: WebSauger
  158. User-agent: MIIxpc
  159. User-agent: httplib
  160. User-agent: WebStripper
  161. User-agent: WebVac
  162. User-agent: URLy Warning
  163. User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows NT)
  164. User-agent: Openfind data gathere
  165. User-agent: looksmart
  166. Disallow: /
RAW Paste Data
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!