FliegenToeter

Crawler in robots.txt aussperren

Jul 7th, 2014
509
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. User-agent: ia_archiver
  2. User-agent: OutfoxBot/0.5
  3. User-agent: complex_network_group
  4. User-agent: imparser09.yandex.ru
  5. User-agent: spider14.yandex.ru
  6. User-agent: yandex.ru
  7. User-agent: yandex
  8. User-agent: Alexibot
  9. User-agent: Baiduspider
  10. User-agent: Baiduspider-image
  11. User-agent: Baiduspider-video
  12. User-agent: Aqua_Products
  13. User-agent: BackDoorBot
  14. User-agent: BackDoorBot/1.0
  15. User-agent: BPImageWalker/2.0
  16. User-agent: Black.Hole
  17. User-agent: BlackWidow
  18. User-agent: BlowFish
  19. User-agent: BlowFish/1.0
  20. User-agent: Bookmark search tool
  21. User-agent: Bot mailto:craftbot@yahoo.com
  22. User-agent: BotALot
  23. User-agent: BotRightHere
  24. User-agent: BuiltBotTough
  25. User-agent: Bullseye
  26. User-agent: Bullseye/1.0
  27. User-agent: BunnySlippers
  28. User-agent: Cegbfeieh
  29. User-agent: CheeseBot
  30. User-agent: CherryPicker
  31. User-agent: CherryPickerElite/1.0
  32. User-agent: CherryPickerSE/1.0
  33. User-agent: ChinaClaw
  34. User-agent: Copernic
  35. User-agent: CopyRightCheck
  36. User-agent: Crescent
  37. User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
  38. User-agent: Custo
  39. User-agent: DISCo
  40. User-agent: DISCo Pump 3.0
  41. User-agent: DISCo Pump 3.2
  42. User-agent: DISCoFinder
  43. User-agent: DittoSpyder
  44. User-agent: Download Demon
  45. User-agent: Download Demon/3.2.0.8
  46. User-agent: Download Demon/3.5.0.11
  47. User-agent: EirGrabber
  48. User-agent: EmailCollector
  49. User-agent: EmailSiphon
  50. User-agent: EmailWolf
  51. User-agent: EroCrawler
  52. User-agent: Express WebPictures
  53. User-agent: Express WebPictures (www.express-soft.com)
  54. User-agent: ExtractorPro
  55. User-agent: EyeNetIE
  56. User-agent: FairAd Client
  57. User-agent: Flaming AttackBot
  58. User-agent: FlashGet
  59. User-agent: FlashGet WebWasher 3.2
  60. User-agent: Foobot
  61. User-agent: FrontPage
  62. User-agent: FrontPage [NC,OR]
  63. User-agent: Gaisbot
  64. User-agent: GetRight
  65. User-agent: GetRight/2.11
  66. User-agent: GetRight/3.1
  67. User-agent: GetRight/3.2
  68. User-agent: GetRight/3.3
  69. User-agent: GetRight/3.3.3
  70. User-agent: GetRight/3.3.4
  71. User-agent: GetRight/4.0.0
  72. User-agent: GetRight/4.1.0
  73. User-agent: GetRight/4.1.1
  74. User-agent: GetRight/4.1.2
  75. User-agent: GetRight/4.2
  76. User-agent: GetRight/4.2b (Portuguxeas)
  77. User-agent: GetRight/4.2c
  78. User-agent: GetRight/4.3
  79. User-agent: GetRight/4.5
  80. User-agent: GetRight/4.5a
  81. User-agent: GetRight/4.5b
  82. User-agent: GetRight/4.5b1
  83. User-agent: GetRight/4.5b2
  84. User-agent: GetRight/4.5b3
  85. User-agent: GetRight/4.5b6
  86. User-agent: GetRight/4.5b7
  87. User-agent: GetRight/4.5c
  88. User-agent: GetRight/4.5d
  89. User-agent: GetRight/4.5e
  90. User-agent: GetRight/5.0beta1
  91. User-agent: GetRight/5.0beta2
  92. User-agent: GetWeb!
  93. User-agent: Go!Zilla
  94. User-agent: Go!Zilla (www.gozilla.com)
  95. User-agent: Go!Zilla 3.3 (www.gozilla.com)
  96. User-agent: Go!Zilla 3.5 (www.gozilla.com)
  97. User-agent: Go-Ahead-Got-It
  98. User-agent: GrabNet
  99. User-agent: Grafula
  100. User-agent: HMView
  101. User-agent: HTTrack
  102. User-agent: HTTrack 3.0
  103. User-agent: HTTrack 3.0x
  104. User-agent: HTTrack [NC,OR]
  105. User-agent: Harvest
  106. User-agent: Harvest/1.5
  107. User-agent: Image Stripper
  108. User-agent: Image Sucker
  109. User-agent: Indy Library
  110. User-agent: Indy Library [NC,OR]
  111. User-agent: InfoNaviRobot
  112. User-agent: InterGET
  113. User-agent: Internet Ninja
  114. User-agent: Internet Ninja 4.0
  115. User-agent: Internet Ninja 5.0
  116. User-agent: Internet Ninja 6.0
  117. User-agent: Iron33/1.0.2
  118. User-agent: JOC Web Spider
  119. User-agent: JennyBot
  120. User-agent: JetCar
  121. User-agent: Kenjin Spider
  122. User-agent: Kenjin.Spider
  123. User-agent: Keyword Density/0.9
  124. User-agent: Keyword.Density
  125. User-agent: LNSpiderguy
  126. User-agent: LeechFTP
  127. User-agent: LexiBot
  128. User-agent: LinkScan/8.1a Unix
  129. User-agent: LinkScan/8.1a.Unix
  130. User-agent: LinkWalker
  131. User-agent: LinkextractorPro
  132. User-agent: MIDown tool
  133. User-agent: MIIxpc
  134. User-agent: MIIxpc/4.2
  135. User-agent: MSIECrawler
  136. User-agent: Mass Downloader
  137. User-agent: Mass Downloader/2.2
  138. User-agent: Mata Hari
  139. User-agent: Mata.Hari
  140. User-agent: Microsoft URL Control
  141. User-agent: Microsoft URL Control - 5.01.4511
  142. User-agent: Microsoft URL Control - 6.00.8169
  143. User-agent: Microsoft.URL
  144. User-agent: Mister PiX
  145. User-agent: Mister PiX version.dll
  146. User-agent: Mister Pix II 2.01
  147. User-agent: Mister Pix II 2.02a
  148. User-agent: Mister.PiX
  149. User-agent: NICErsPRO
  150. User-agent: NPBot
  151. User-agent: NPbot
  152. User-agent: Navroad
  153. User-agent: NearSite
  154. User-agent: Net Vampire
  155. User-agent: Net Vampire/3.0
  156. User-agent: NetAnts
  157. User-agent: NetAnts/1.10
  158. User-agent: NetAnts/1.23
  159. User-agent: NetAnts/1.24
  160. User-agent: NetAnts/1.25
  161. User-agent: NetMechanic
  162. User-agent: NetSpider
  163. User-agent: NetZIP
  164. User-agent: NetZip Downloader 1.0 Win32(Nov 12 1998)
  165. User-agent: NetZip-Downloader/1.0.62 (Win32; Dec 7 1998)
  166. User-agent: NetZippy+(http:/www.innerprise.net/usp-spider.asp)
  167. User-agent: Octopus
  168. User-agent: Offline Explorer
  169. User-agent: Offline Explorer/1.2
  170. User-agent: Offline Explorer/1.4
  171. User-agent: Offline Explorer/1.6
  172. User-agent: Offline Explorer/1.7
  173. User-agent: Offline Explorer/1.9
  174. User-agent: Offline Explorer/2.0
  175. User-agent: Offline Explorer/2.1
  176. User-agent: Offline Explorer/2.3
  177. User-agent: Offline Explorer/2.4
  178. User-agent: Offline Explorer/2.5
  179. User-agent: Offline Navigator
  180. User-agent: Offline.Explorer
  181. User-agent: Openbot
  182. User-agent: Openfind
  183. User-agent: Openfind data gatherer
  184. User-agent: Oracle Ultra Search
  185. User-agent: PageGrabber
  186. User-agent: Papa Foto
  187. User-agent: PerMan
  188. User-agent: ProPowerBot/2.14
  189. User-agent: ProWebWalker
  190. User-agent: Python-urllib
  191. User-agent: QueryN Metasearch
  192. User-agent: QueryN.Metasearch
  193. User-agent: RMA
  194. User-agent: Radiation Retriever 1.1
  195. User-agent: ReGet
  196. User-agent: RealDownload
  197. User-agent: RealDownload/4.0.0.40
  198. User-agent: RealDownload/4.0.0.41
  199. User-agent: RealDownload/4.0.0.42
  200. User-agent: RepoMonkey
  201. User-agent: RepoMonkey Bait & Tackle/v1.01
  202. User-agent: SiteSnagger
  203. User-agent: SlySearch
  204. User-agent: SmartDownload
  205. User-agent: SmartDownload/1.2.76 (Win32; Apr 1 1999)
  206. User-agent: SmartDownload/1.2.77 (Win32; Aug 17 1999)
  207. User-agent: SmartDownload/1.2.77 (Win32; Feb 1 2000)
  208. User-agent: SmartDownload/1.2.77 (Win32; Jun 19 2001)
  209. User-agent: SpankBot
  210. User-agent: Sqworm/2.9.85-BETA (beta_release; 20011115-775; i686-pc-linux
  211. User-agent: SuperBot
  212. User-agent: SuperBot/3.0 (Win32)
  213. User-agent: SuperBot/3.1 (Win32)
  214. User-agent: SuperHTTP
  215. User-agent: SuperHTTP/1.0
  216. User-agent: Surfbot
  217. User-agent: Szukacz/1.4
  218. User-agent: Teleport
  219. User-agent: Teleport Pro
  220. User-agent: Teleport Pro/1.29
  221. User-agent: Teleport Pro/1.29.1590
  222. User-agent: Teleport Pro/1.29.1634
  223. User-agent: Teleport Pro/1.29.1718
  224. User-agent: Teleport Pro/1.29.1820
  225. User-agent: Teleport Pro/1.29.1847
  226. User-agent: TeleportPro
  227. User-agent: Telesoft
  228. User-agent: The Intraformant
  229. User-agent: The.Intraformant
  230. User-agent: TheNomad
  231. User-agent: TightTwatBot
  232. User-agent: Titan
  233. User-agent: True_Robot
  234. User-agent: True_Robot/1.0
  235. User-agent: TurnitinBot
  236. User-agent: TurnitinBot/1.5
  237. User-agent: URL Control
  238. User-agent: URL_Spider_Pro
  239. User-agent: URLy Warning
  240. User-agent: URLy.Warning
  241. User-agent: VCI
  242. User-agent: VCI WebViewer VCI WebViewer Win32
  243. User-agent: VoidEYE
  244. User-agent: WWW-Collector-E
  245. User-agent: WWWOFFLE
  246. User-agent: Web Image Collector
  247. User-agent: Web Sucker
  248. User-agent: Web.Image.Collector
  249. User-agent: WebAuto
  250. User-agent: WebAuto/3.40 (Win98; I)
  251. User-agent: WebBandit
  252. User-agent: WebBandit/3.50
  253. User-agent: WebCapture 2.0
  254. User-agent: WebCopier
  255. User-agent: WebCopier v.2.2
  256. User-agent: WebCopier v2.5
  257. User-agent: WebCopier v2.6
  258. User-agent: WebCopier v2.7a
  259. User-agent: WebCopier v2.8
  260. User-agent: WebCopier v3.0
  261. User-agent: WebCopier v3.0.1
  262. User-agent: WebCopier v3.2
  263. User-agent: WebCopier v3.2a
  264. User-agent: WebEMailExtrac.*
  265. User-agent: WebEnhancer
  266. User-agent: WebFetch
  267. User-agent: WebGo IS
  268. User-agent: WebLeacher
  269. User-agent: WebReaper
  270. User-agent: WebReaper [info@webreaper.net]
  271. User-agent: WebReaper [webreaper@otway.com]
  272. User-agent: WebReaper v9.1 - www.otway.com/webreaper
  273. User-agent: WebReaper v9.7 - www.webreaper.net
  274. User-agent: WebReaper v9.8 - www.webreaper.net
  275. User-agent: WebReaper vWebReaper v7.3 - www,otway.com/webreaper
  276. User-agent: WebSauger
  277. User-agent: WebSauger 1.20b
  278. User-agent: WebSauger 1.20j
  279. User-agent: WebSauger 1.20k
  280. User-agent: WebStripper
  281. User-agent: WebStripper/2.03
  282. User-agent: WebStripper/2.10
  283. User-agent: WebStripper/2.12
  284. User-agent: WebStripper/2.13
  285. User-agent: WebStripper/2.15
  286. User-agent: WebStripper/2.16
  287. User-agent: WebStripper/2.19
  288. User-agent: WebWhacker
  289. User-agent: WebZIP
  290. User-agent: WebZIP/2.75 (http:/www.spidersoft.com)
  291. User-agent: WebZIP/3.65 (http:/www.spidersoft.com)
  292. User-agent: WebZIP/3.80 (http:/www.spidersoft.com)
  293. User-agent: WebZIP/4.0 (http:/www.spidersoft.com)
  294. User-agent: WebZIP/4.1 (http:/www.spidersoft.com)
  295. User-agent: WebZIP/4.21
  296. User-agent: WebZIP/4.21 (http:/www.spidersoft.com)
  297. User-agent: WebZIP/5.0
  298. User-agent: WebZIP/5.0 (http:/www.spidersoft.com)
  299. User-agent: WebZIP/5.0 PR1 (http:/www.spidersoft.com)
  300. User-agent: WebZip
  301. User-agent: WebZip/4.0
  302. User-agent: WebmasterWorldForumBot
  303. User-agent: Website Quester
  304. User-agent: Website Quester - www.asona.org
  305. User-agent: Website Quester - www.esalesbiz.com/extra/
  306. User-agent: Website eXtractor
  307. User-agent: Website eXtractor (http:/www.asona.org)
  308. User-agent: Website.Quester
  309. User-agent: Webster Pro
  310. User-agent: Webster.Pro
  311. User-agent: Wget
  312. User-agent: Wget/1.10.2
  313. User-agent: Wget/1.5.2
  314. User-agent: Wget/1.5.3
  315. User-agent: Wget/1.6
  316. User-agent: Wget/1.7
  317. User-agent: Wget/1.8
  318. User-agent: Wget/1.8.1
  319. User-agent: Wget/1.8.1+cvs
  320. User-agent: Wget/1.8.2
  321. User-agent: Wget/1.9-beta
  322. User-agent: Widow
  323. User-agent: Xaldon WebSpider
  324. User-agent: Xaldon WebSpider 2.5.b3
  325. User-agent: Xenu's
  326. User-agent: Xenu's Link Sleuth 1.1c
  327. User-agent: Zeus
  328. User-agent: Zeus 11389 Webster Pro V2.9 Win32
  329. User-agent: Zeus 11652 Webster Pro V2.9 Win32
  330. User-agent: Zeus 18018 Webster Pro V2.9 Win32
  331. User-agent: Zeus 26378 Webster Pro V2.9 Win32
  332. User-agent: Zeus 30747 Webster Pro V2.9 Win32
  333. User-agent: Zeus 32297 Webster Pro V2.9 Win32
  334. User-agent: Zeus 39206 Webster Pro V2.9 Win32
  335. User-agent: Zeus 41641 Webster Pro V2.9 Win32
  336. User-agent: Zeus 44238 Webster Pro V2.9 Win32
  337. User-agent: Zeus 51070 Webster Pro V2.9 Win32
  338. User-agent: Zeus 51674 Webster Pro V2.9 Win32
  339. User-agent: Zeus 51837 Webster Pro V2.9 Win32
  340. User-agent: Zeus 63567 Webster Pro V2.9 Win32
  341. User-agent: Zeus 6694 Webster Pro V2.9 Win32
  342. User-agent: Zeus 71129 Webster Pro V2.9 Win32
  343. User-agent: Zeus 82016 Webster Pro V2.9 Win32
  344. User-agent: Zeus 82900 Webster Pro V2.9 Win32
  345. User-agent: Zeus 84842 Webster Pro V2.9 Win32
  346. User-agent: Zeus 90872 Webster Pro V2.9 Win32
  347. User-agent: Zeus 94934 Webster Pro V2.9 Win32
  348. User-agent: Zeus 95245 Webster Pro V2.9 Win32
  349. User-agent: Zeus 95351 Webster Pro V2.9 Win32
  350. User-agent: Zeus 97371 Webster Pro V2.9 Win32
  351. User-agent: Zeus Link Scout
  352. User-agent: asterias
  353. User-agent: b2w/0.1
  354. User-agent: cosmos
  355. User-agent: eCatch
  356. User-agent: eCatch/3.0
  357. User-agent: hloader
  358. User-agent: httplib
  359. User-agent: humanlinks
  360. User-agent: larbin
  361. User-agent: larbin (samualt9@bigfoot.com)
  362. User-agent: larbin samualt9@bigfoot.com
  363. User-agent: larbin_2.6.2 (kabura@sushi.com)
  364. User-agent: larbin_2.6.2 (larbin2.6.2@unspecified.mail)
  365. User-agent: larbin_2.6.2 (listonATccDOTgatechDOTedu)
  366. User-agent: larbin_2.6.2 (vitalbox1@hotmail.com)
  367. User-agent: larbin_2.6.2 kabura@sushi.com
  368. User-agent: larbin_2.6.2 larbin2.6.2@unspecified.mail
  369. User-agent: larbin_2.6.2 larbin@correa.org
  370. User-agent: larbin_2.6.2 listonATccDOTgatechDOTedu
  371. User-agent: larbin_2.6.2 vitalbox1@hotmail.com
  372. User-agent: libWeb/clsHTTP
  373. User-agent: lwp-trivial
  374. User-agent: lwp-trivial/1.34
  375. User-agent: moget
  376. User-agent: moget/2.1
  377. User-agent: pavuk
  378. User-agent: pcBrowser
  379. User-agent: psbot
  380. User-agent: searchpreview
  381. User-agent: spanner
  382. User-agent: suzuran
  383. User-agent: tAkeOut
  384. User-agent: toCrawl/UrlDispatcher
  385. User-agent: turingos
  386. User-agent: webfetch/2.1.0
  387. User-agent: wget
  388. Disallow: /
RAW Paste Data