API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Feb 14th, 2015
86
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 4.15 KB | None | 0 0
raw download clone embed print report
  1. User-agent: asterias
  2. Disallow:/
  3. User-agent: BackDoorBot/1.0
  4. Disallow:/
  5. User-agent: Black Hole
  6. Disallow:/
  7. User-agent: BlowFish/1.0
  8. Disallow:/
  9. User-agent: BotALot
  10. Disallow:/
  11. User-agent: BuiltBotTough
  12. Disallow:/
  13. User-agent: Bullseye/1.0
  14. Disallow:/
  15. User-agent: BunnySlippers
  16. Disallow:/
  17. User-agent: Cegbfeieh
  18. Disallow:/
  19. User-agent: CheeseBot
  20. Disallow:/
  21. User-agent: CherryPicker
  22. Disallow:/
  23. User-agent: CherryPickerElite/1.0
  24. Disallow:/
  25. User-agent: CherryPickerSE/1.0
  26. Disallow:/
  27. User-agent: CopyRightCheck
  28. Disallow:/
  29. User-agent: cosmos
  30. Disallow:/
  31. User-agent: Crescent
  32. Disallow:/
  33. User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
  34. Disallow:/
  35. User-agent: DittoSpyder
  36. Disallow:/
  37. User-agent: EmailCollector
  38. Disallow:/
  39. User-agent: EmailSiphon
  40. Disallow:/
  41. User-agent: EmailWolf
  42. Disallow:/
  43. User-agent: EroCrawler
  44. Disallow:/
  45. User-agent: ExtractorPro
  46. Disallow:/
  47. User-agent: Foobot
  48. Disallow:/
  49. User-agent: Harvest/1.5
  50. Disallow:/
  51. User-agent: hloader
  52. Disallow:/
  53. User-agent: httplib
  54. Disallow:/
  55. User-agent: humanlinks
  56. Disallow:/
  57. User-agent: ia_archiver
  58. Disallow:/
  59. User-agent: InfoNaviRobot
  60. Disallow:/
  61. User-agent: JennyBot
  62. Disallow:/
  63. User-agent: Kenjin Spider
  64. Disallow:/
  65. User-agent: Keyword Density/0.9
  66. Disallow:/
  67. User-agent: LexiBot
  68. Disallow:/
  69. User-agent: libWeb/clsHTTP
  70. Disallow:/
  71. User-agent: LinkextractorPro
  72. Disallow:/
  73. User-agent: LinkScan/8.1a Unix
  74. Disallow:/
  75. User-agent: LinkWalker
  76. Disallow:/
  77. User-agent: LNSpiderguy
  78. Disallow:/
  79. User-agent: lwp-trivial
  80. Disallow:/
  81. User-agent: lwp-trivial/1.34
  82. Disallow:/
  83. User-agent: Mata Hari
  84. Disallow:/
  85. User-agent: Microsoft URL Control - 5.01.4511
  86. Disallow:/
  87. User-agent: Microsoft URL Control - 6.00.8169
  88. Disallow:/
  89. User-agent: MIIxpc
  90. Disallow:/
  91. User-agent: MIIxpc/4.2
  92. Disallow:/
  93. User-agent: Mister PiX
  94. Disallow:/
  95. User-agent: moget
  96. Disallow:/
  97. User-agent: moget/2.1
  98. Disallow:/
  99. User-agent: mozilla/4
  100. Disallow:/
  101. User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95)
  102. Disallow:/
  103. User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 95)
  104. Disallow:/
  105. User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 98)
  106. Disallow:/
  107. User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows NT)
  108. Disallow:/
  109. User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows XP)
  110. Disallow:/
  111. User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 2000)
  112. Disallow:/
  113. User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows ME)
  114. Disallow:/
  115. User-agent: mozilla/5
  116. Disallow:/
  117. User-agent: NetAnts
  118. Disallow:/
  119. User-agent: NICErsPRO
  120. Disallow:/
  121. User-agent: Offline Explorer
  122. Disallow:/
  123. User-agent: Openfind
  124. Disallow:/
  125. User-agent: Openfind data gathere
  126. Disallow:/
  127. User-agent: ProPowerBot/2.14
  128. Disallow:/
  129. User-agent: ProWebWalker
  130. Disallow:/
  131. User-agent: QueryN Metasearch
  132. Disallow:/
  133. User-agent: RepoMonkey
  134. Disallow:/
  135. User-agent: RepoMonkey Bait & Tackle/v1.01
  136. Disallow:/
  137. User-agent: RMA
  138. Disallow:/
  139. User-agent: SiteSnagger
  140. Disallow:/
  141. User-agent: SpankBot
  142. Disallow:/
  143. User-agent: spanner
  144. Disallow:/
  145. User-agent: suzuran
  146. Disallow:/
  147. User-agent: Szukacz/1.4
  148. Disallow:/
  149. User-agent: Teleport
  150. Disallow:/
  151. User-agent: TeleportPro
  152. Disallow:/
  153. User-agent: Telesoft
  154. Disallow:/
  155. User-agent: The Intraformant
  156. Disallow:/
  157. User-agent: TheNomad
  158. Disallow:/
  159. User-agent: TightTwatBot
  160. Disallow:/
  161. User-agent: Titan
  162. Disallow:/
  163. User-agent: toCrawl/UrlDispatcher
  164. Disallow:/
  165. User-agent: True_Robot
  166. Disallow:/
  167. User-agent: URLy Warning
  168. Disallow:/
  169. User-agent: VCI
  170. Disallow:/
  171. User-agent: Web Image Collector
  172. Disallow:/
  173. User-agent: WebAuto
  174. Disallow:/
  175. User-agent: WebBandit
  176. Disallow:/
  177. User-agent: WebCopier
  178. Disallow:/
  179. User-agent: WebEnhancer
  180. Disallow:/
  181. User-agent: WebmasterWorldForumBot
  182. Disallow:/
  183. User-agent: WebSauger
  184. Disallow:/
  185. User-agent: Website Quester
  186. Disallow:/
  187. User-agent: Webster Pro
  188. Disallow:/
  189. User-agent: WebStripper
  190. Disallow:/
  191. User-agent: WebZip/4.0
  192. Disallow:/
  193. User-agent: Wget
  194. Disallow:/
  195. User-agent: Wget/1.5.3
  196. Disallow:/
  197. User-agent: Wget/1.6
  198. Disallow:/
  199. User-agent: WWW-Collector-E
  200. Disallow:/
  201. User-agent: Xenu's
  202. Disallow:/
  203. User-agent: Xenu's Link Sleuth 1.1c
  204. Disallow:/
  205. User-agent: Zeus
  206. Disallow:/
  207. User-agent: Zeus 32297 Webster Pro V2.9 Win32
  208. Disallow:/
  209. User-agent: rogerbot/1.0
  210. Disallow:/
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!