API tools faq
paste
Advertisement
Guest User

Untitled

a guest
May 20th, 2016
3,203
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 24.42 KB | None | 0 0
raw download clone embed print report
  1. Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/webcrawler.html) Gecko/2008032620
  2. ABACHOBot
  3. Accoona-AI-Agent/1.1.2 (aicrawler at accoonabot dot com)
  4. Accoona-AI-Agent/1.1.2
  5. Accoona-AI-Agent/1.1.1 (crawler at accoona dot com)
  6. Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0) AddSugarSpiderBot www.idealobserver.com
  7. Mozilla/5.0 (compatible; AnyApexBot/1.0; +http://www.anyapex.com/bot.html)
  8. Mozilla/4.0 (compatible; Arachmo)
  9. Mozilla/4.0 (compatible; B-l-i-t-z-B-O-T)
  10. Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
  11. Baiduspider+(+http://www.baidu.com/search/spider_jp.html)
  12. Baiduspider+(+http://www.baidu.com/search/spider.htm)
  13. BaiDuSpider
  14. Mozilla/5.0 (compatible; BecomeBot/3.0; MSIE 6.0 compatible; +http://www.become.com/site_owners.html)
  15. Mozilla/5.0 (compatible; BecomeBot/2.3; MSIE 6.0 compatible; +http://www.become.com/site_owners.html)
  16. Mozilla/5.0 (compatible; BeslistBot; nl; BeslistBot 1.0; http://www.beslist.nl/
  17. BillyBobBot/1.0 (+http://www.billybobbot.com/crawler/)
  18. Bimbot/1.0
  19. Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
  20. Mozilla/5.0 (compatible; bingbot/2.0 +http://www.bing.com/bingbot.htm)
  21. Mozilla/4.0 (compatible; BlitzBot)
  22. BlitzBOT@tricus.net (Mozilla compatible)
  23. BlitzBOT@tricus.com (Mozilla compatible)
  24. boitho.com-dc/0.85 ( http://www.boitho.com/dcbot.html )
  25. boitho.com-dc/0.83 ( http://www.boitho.com/dcbot.html )
  26. boitho.com-dc/0.82 ( http://www.boitho.com/dcbot.html )
  27. boitho.com-dc/0.81 ( http://www.boitho.com/dcbot.html )
  28. boitho.com-dc/0.79 ( http://www.boitho.com/dcbot.html )
  29. boitho.com-robot/1.1
  30. boitho.com-robot/1.0
  31. btbot/0.4 (+http://www.btbot.com/btbot.html)
  32. CatchBot/2.0; +http://www.catchbot.com
  33. CatchBot/1.0; +http://www.catchbot.com
  34. CatchBot/1.0; http://www.catchbot.com
  35. Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.2-Build-1)
  36. Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.2-Build-0)
  37. Mozilla/5.0 (compatible; Charlotte/1.1; http://www.searchme.com/support/)
  38. Mozilla/5.0 (compatible; Charlotte/1.0t; http://www.searchme.com/support/)
  39. Mozilla/5.0 (compatible; Charlotte/1.0b; http://www.searchme.com/support/)
  40. Mozilla/5.0 (compatible; Charlotte/1.0b; http://www.betaspider.com/)
  41. Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.11) Gecko/20080109 (Charlotte/0.9t; http://www.searchme.com/support/) (Charlotte/0.9t; http://www.searchme.com/support/)
  42. Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.11) Gecko/20080109 (Charlotte/0.9t; http://www.searchme.com/support/)
  43. Mozilla/5.0 (compatible; Charlotte/0.9t; http://www.searchme.com/support/)
  44. Mozilla/5.0 (compatible; Charlotte/0.9t; +http://www.searchme.com/support/)
  45. ConveraCrawler/0.9e (+http://ews.converasearch.com/crawl.htm)
  46. ConveraCrawler/0.9d (+http://www.authoritativeweb.com/crawl)
  47. ConveraCrawler/0.9d ( http://www.authoritativeweb.com/crawl)
  48. ConveraCrawler/0.9 (+http://www.authoritativeweb.com/crawl)
  49. cosmos/0.9_(robot@xyleme.com)
  50. Covario-IDS/1.0 (Covario; http://www.covario.com/ids; support at covario dot com)
  51. DataparkSearch/4.37-23012006 ( http://www.dataparksearch.org/)
  52. DataparkSearch/4.36 ( http://www.dataparksearch.org/)
  53. DataparkSearch/4.35-02122005 ( http://www.dataparksearch.org/)
  54. DataparkSearch/4.35 ( http://www.dataparksearch.org/)
  55. DiamondBot
  56. Mozilla/5.0 (compatible; discobot/1.0; +http://discoveryengine.com/discobot.html)
  57. Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, crawler@dotnetdotcom.org)
  58. DotBot/1.0.1 (http://www.dotnetdotcom.org/#info, crawler@dotnetdotcom.org)
  59. EmeraldShield.com WebBot (http://www.emeraldshield.com/webbot.aspx)
  60. envolk[ITS]spider/1.6 (+http://www.envolk.com/envolkspider.html)
  61. envolk[ITS]spider/1.6 ( http://www.envolk.com/envolkspider.html)
  62. EsperanzaBot(+http://www.esperanza.to/bot/)
  63. Exabot/2.0
  64. FAST Enterprise Crawler 6 used by Schibsted (webcrawl@schibstedsok.no)
  65. FAST Enterprise Crawler 6 / Scirus scirus-crawler@fast.no; http://www.scirus.com/srsapp/contactus/
  66. FAST Enteprise Crawler/6 (www dot fastsearch dot com)
  67. FAST-WebCrawler/3.8 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)
  68. FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
  69. FAST-WebCrawler/3.7 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)
  70. FAST-WebCrawler/3.6/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
  71. FAST-WebCrawler/3.6 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)
  72. FAST-WebCrawler/3.6
  73. FAST-WebCrawler/3.x Multimedia (mm dash crawler at fast dot no)
  74. FAST-WebCrawler/3.x Multimedia
  75. Mozilla/4.0 (compatible: FDSE robot)
  76. findlinks/2.0.1 (+http://wortschatz.uni-leipzig.de/findlinks/)
  77. findlinks/1.1.6-beta6 (+http://wortschatz.uni-leipzig.de/findlinks/)
  78. findlinks/1.1.6-beta4 (+http://wortschatz.uni-leipzig.de/findlinks/)
  79. findlinks/1.1.6-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/)
  80. findlinks/1.1.5-beta7 (+http://wortschatz.uni-leipzig.de/findlinks/)
  81. findlinks/1.1.4-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/)
  82. findlinks/1.1.3-beta9 (+http://wortschatz.uni-leipzig.de/findlinks/)
  83. findlinks/1.1.3-beta8 (+http://wortschatz.uni-leipzig.de/findlinks/)
  84. findlinks/1.1.3-beta6 (+http://wortschatz.uni-leipzig.de/findlinks/)
  85. findlinks/1.1.3-beta4 (+http://wortschatz.uni-leipzig.de/findlinks/)
  86. findlinks/1.1.3-beta2 (+http://wortschatz.uni-leipzig.de/findlinks/)
  87. findlinks/1.1.3-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/)
  88. findlinks/1.1.2-a5 (+http://wortschatz.uni-leipzig.de/findlinks/)
  89. findlinks/1.1.1-a5 (+http://wortschatz.uni-leipzig.de/findlinks/)
  90. findlinks/1.1.1-a1 (+http://wortschatz.uni-leipzig.de/findlinks/)
  91. findlinks/1.1.1 (+http://wortschatz.uni-leipzig.de/findlinks/)
  92. findlinks/1.1-a9 (+http://wortschatz.uni-leipzig.de/findlinks/)
  93. findlinks/1.1-a8 (+http://wortschatz.uni-leipzig.de/findlinks/)
  94. findlinks/1.1-a8 ( http://wortschatz.uni-leipzig.de/findlinks/)
  95. findlinks/1.1-a7 (+http://wortschatz.uni-leipzig.de/findlinks/)
  96. findlinks/1.1-a5 (+http://wortschatz.uni-leipzig.de/findlinks/)
  97. findlinks/1.1-a4 (+http://wortschatz.uni-leipzig.de/findlinks/)
  98. findlinks/1.1-a3 (+http://wortschatz.uni-leipzig.de/findlinks/)
  99. findlinks/1.1 (+http://wortschatz.uni-leipzig.de/findlinks/)
  100. findlinks/1.06 (+http://wortschatz.uni-leipzig.de/findlinks/)
  101. findlinks/1.0.9 (+http://wortschatz.uni-leipzig.de/findlinks/)
  102. findlinks/1.0.8 (+http://wortschatz.uni-leipzig.de/findlinks/)
  103. findlinks/1.0 (+http://wortschatz.uni-leipzig.de/findlinks/)
  104. Mozilla/4.0 compatible FurlBot/Furl Search 2.0 (FurlBot; http://www.furl.net; wn.furlbot@looksmart.net)
  105. FyberSpider (+http://www.fybersearch.com/fyberspider.php)
  106. FyberSpider
  107. g2Crawler nobody@airmail.net
  108. Gaisbot/3.0+(robot06@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php)
  109. Gaisbot/3.0+(robot05@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php)
  110. Gaisbot/3.0 (jerry_wu@openfind.com.tw; http://gais.cs.ccu.edu.tw/robot.php)
  111. GalaxyBot/1.0 (http://www.galaxy.com/galaxybot.html)
  112. genieBot (http://64.5.245.11/faq/faq.html)
  113. genieBot ((http://64.5.245.11/faq/faq.html))
  114. Gigabot/3.0 (http://www.gigablast.com/spider.html)
  115. Gigabot/2.0/gigablast.com/spider.html
  116. Gigabot/2.0 (http://www.gigablast.com/spider.html)
  117. Gigabot/2.0
  118. Gigabot/1.0
  119. Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1; .NET CLR 1.1.4322; Girafabot [girafa.com])
  120. Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Girafabot; girafabot at girafa dot com; http://www.girafa.com)
  121. Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; Girafabot; girafabot at girafa dot com; http://www.girafa.com)
  122. Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  123. Googlebot/2.1 (+http://www.googlebot.com/bot.html)
  124. Googlebot/2.1 (+http://www.google.com/bot.html)
  125. Googlebot-Image/1.0
  126. Mozilla/5.0 GurujiBot/1.0 (+http://www.guruji.com/en/WebmasterFAQ.html)
  127. Mozilla/5.0 GurujiBot/1.0 ( http://www.guruji.com/en/WebmasterFAQ.html)
  128. Mozilla/5.0 (compatible; GurujiBot/1.0; +http://www.guruji.com/en/WebmasterFAQ.html)
  129. GurujiBot/1.0 (+http://www.guruji.com/WebmasterFAQ.html)
  130. GurujiBot/1.0 (+http://www.guruji.com/en/WebmasterFAQ.html)
  131. HappyFunBot/1.1 ( http://www.happyfunsearch.com/bot.html)
  132. hl_ftien_spider_v1.1
  133. hl_ftien_spider
  134. holmes/3.9 (someurl.co.cc)
  135. holmes/3.12.4 (http://morfeo.centrum.cz/bot)
  136. holmes/3.12.3 (http://morfeo.centrum.cz/bot)
  137. holmes/3.12.2 (http://morfeo.centrum.cz/bot)
  138. holmes/3.12.1 (http://morfeo.centrum.cz/bot)
  139. htdig/3.1.6 (unconfigured@htdig.searchengine.maintainer)
  140. htdig/3.1.6 (mathieu.peltier@inrialpes.fr)
  141. htdig/3.1.5 (webmaster@online-medien.de)
  142. htdig/3.1.5 (root@localhost)
  143. htdig/3.1.5 (infosys@storm.rmi.org)
  144. htdig/3.1.5
  145. iaskspider/2.0(+http://iask.com/help/help_index.html)
  146. iaskspider
  147. ia_archiver/8.9 (Windows NT 3.1; en-US;)
  148. ia_archiver/8.9 (Windows 3.9; en-US;)
  149. ia_archiver/8.9 (Linux 1.0; en-US;)
  150. ia_archiver/8.8 (Windows XP 7.2; en-US;)
  151. ia_archiver/8.8 (Windows XP 3.0; en-US;)
  152. ia_archiver/8.2 (Windows 7.6; en-US;)
  153. ia_archiver/8.1 (Windows 2000 1.9; en-US;)
  154. ia_archiver/8.0 (Windows 2.4; en-US;)
  155. ia_archiver
  156. iCCrawler (http://www.iccenter.net/bot.htm)
  157. ichiro/4.0 (http://help.goo.ne.jp/door/crawler.html)
  158. ichiro/3.0 (http://help.goo.ne.jp/door/crawler.html)
  159. ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html)
  160. ichiro/2.0 (ichiro@nttr.co.jp)
  161. ichiro/2.0 (http://help.goo.ne.jp/door/crawler.html)
  162. igdeSpyder (compatible; igde.ru; +http://igde.ru/doc/tech.html)
  163. IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler/)
  164. IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler)
  165. IRLbot/2.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler)
  166. IRLbot/2.0 (+http://irl.cs.tamu.edu/crawler)
  167. IRLbot/2.0 ( http://irl.cs.tamu.edu/crawler)
  168. IssueCrawler
  169. Jaxified Bot (+http://www.jaxified.com/crawler/)
  170. Jyxobot/1
  171. Mozilla/5.0 (compatible; KoepaBot BETA; http://www.koepa.nl/bot.html)
  172. L.webis/0.87 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
  173. LapozzBot/1.4 (+http://robot.lapozz.com)
  174. Mozilla/5.0 larbin@unspecified.mail
  175. larbin_2.6.3 zumesun@hotmail.com
  176. larbin_2.6.3 tangyi858@163.com
  177. larbin_2.6.3 ltaa_web_crawler@groupes.epfl.ch
  178. larbin_2.6.3 larbin2.6.3@unspecified.mail
  179. larbin_2.6.3 gqnmgsp@ruc.edu.cn
  180. larbin_2.6.3 ghary@sohu.com
  181. larbin_2.6.3 capveg@cs.umd.edu
  182. larbin_2.6.3 (wgao@genieknows.com)
  183. larbin_2.6.3 (ltaa_web_crawler@groupes.epfl.ch)
  184. larbin_2.6.3 (larbin@behner.org)
  185. larbin_2.6.3 (larbin2.6.3@unspecified.mail)
  186. larbin_2.6.2 vitalbox1@hotmail.com
  187. larbin_2.6.2 pierre@micro-fun.ch
  188. larbin_2.6.2 listonATccDOTgatechDOTedu
  189. larbin_2.6.2 larbin@correa.org
  190. larbin_2.6.2 larbin2.6.2@unspecified.mail
  191. larbin_2.6.2 kalou@kalou.net
  192. larbin_2.6.2 dthunen@princeton.edu
  193. larbin_2.6.2 (vitalbox1@hotmail.com)
  194. larbin_2.6.2 (pierre@micro-fun.ch)
  195. larbin_2.6.2 (larbin@correa.org)
  196. larbin_2.6.2 (larbin2.6.2@unspecified.mail)
  197. larbin_2.6.1 larbin2.6.1@unspecified.mail
  198. larbin_2.5.0 (larbin2.5.0@unspecified.mail)
  199. larbin_xy250 larbin2.6.3@unspecified.mail
  200. larbin_test nobody@airmail.etn
  201. larbin_test (nobody@airmail.etn)
  202. LARBIN-EXPERIMENTAL efp@gmx.net
  203. ldspider (http://code.google.com/p/ldspider/wiki/Robots)
  204. LexxeBot/1.0 (lexxebot@lexxe.com)
  205. Linguee Bot (http://www.linguee.com/bot; bot@linguee.com)
  206. LinkWalker/2.0
  207. LinkWalker
  208. lmspider lmspider@scansoft.com
  209. lmspider (lmspider@scansoft.com)
  210. lwp-trivial/1.41
  211. lwp-trivial/1.38
  212. lwp-trivial/1.36
  213. lwp-trivial/1.35
  214. lwp-trivial/1.33
  215. http://www.mabontland.com
  216. magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)
  217. Mediapartners-Google/2.1
  218. Mozilla/5.0 (compatible; MJ12bot/v1.2.4; http://www.majestic12.co.uk/bot.php?+)
  219. Mozilla/5.0 (compatible; MJ12bot/v1.2.3; http://www.majestic12.co.uk/bot.php?+)
  220. MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+)
  221. MJ12bot/v1.0.7 (http://majestic12.co.uk/bot.php?+)
  222. MJ12bot/v1.0.6 (http://majestic12.co.uk/bot.php?+)
  223. MJ12bot/v1.0.5 (http://majestic12.co.uk/bot.php?+)
  224. Mnogosearch-3.1.21
  225. mogimogi/1.0
  226. Mozilla/5.0 (compatible; MojeekBot/2.0; http://www.mojeek.com/bot.html)
  227. MojeekBot/0.2 (archi; http://www.mojeek.com/bot.html)
  228. Moreoverbot/5.1 ( http://w.moreover.com; webmaster@moreover.com) Mozilla/5.0
  229. Moreoverbot/5.00 (+http://www.moreover.com; webmaster@moreover.com)
  230. Moreoverbot/5.00 (+http://www.moreover.com)
  231. Morning Paper 1.0 (robots.txt compliant!)
  232. msnbot/2.1
  233. msnbot/2.0b
  234. msnbot/1.1 (+http://search.msn.com/msnbot.htm)
  235. msnbot/1.1
  236. msnbot/1.0 (+http://search.msn.com/msnbot.htm)
  237. msnbot/0.9 (+http://search.msn.com/msnbot.htm)
  238. msnbot/0.11 ( http://search.msn.com/msnbot.htm)
  239. MSNBOT/0.1 (http://search.msn.com/msnbot.htm)
  240. MSRBOT (http://research.microsoft.com/research/sv/msrbot/)
  241. MSRBOT
  242. MVAClient
  243. Mozilla/5.0 (compatible; mxbot/1.0; +http://www.chainn.com/mxbot.html)
  244. Mozilla/5.0 (compatible; mxbot/1.0; http://www.chainn.com/mxbot.html)
  245. NetResearchServer/4.0(loopimprovements.com/robot.html)
  246. NetResearchServer/3.5(loopimprovements.com/robot.html)
  247. NetResearchServer/2.8(loopimprovements.com/robot.html)
  248. NetResearchServer/2.7(loopimprovements.com/robot.html)
  249. NetResearchServer/2.5(loopimprovements.com/robot.html)
  250. NetResearchServer(http://www.look.com)
  251. Mozilla/5.0 (compatible; NetSeer crawler/2.0; +http://www.netseer.com/crawler.html; crawler@netseer.com)
  252. NewsGator/2.5 (http://www.newsgator.com; Microsoft Windows NT 5.1.2600.0; .NET CLR 1.1.4322.2032)
  253. NewsGator/2.0 Bot (http://www.newsgator.com)
  254. NG-Search/0.9.8 (http://www.ng-search.com)
  255. NG-Search/0.86 (+http://www.ng-search.com)
  256. NG-Search/0.86 ( http://www.ng-search.com)
  257. nicebot
  258. noxtrumbot/1.0 (crawler@noxtrum.com)
  259. Nusearch Spider (www.nusearch.com)
  260. nuSearch Spider (compatible; MSIE 4.01; Windows NT)
  261. NutchCVS/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)
  262. NutchCVS/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)
  263. NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)
  264. NutchCVS/0.7.1 (Nutch running at UW; http://crawlers.cs.washington.edu/; sycrawl@cs.washington.edu)
  265. NutchCVS/0.7 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)
  266. NutchCVS/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)
  267. NutchCVS/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; jagdeepssandhu@hotmail.com)
  268. NutchCVS/0.05 (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)
  269. Nymesis/1.0 (http://nymesis.com)
  270. Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; obot)
  271. oegp v. 1.3.0
  272. omgilibot/0.4 +http://omgili.com
  273. omgilibot/0.3 +http://www.omgili.com/Crawler.html
  274. omgilibot/0.3 http://www.omgili.com/Crawler.html
  275. OmniExplorer_Bot/6.70 (+http://www.omni-explorer.com) WorldIndexer
  276. OmniExplorer_Bot/6.65a (+http://www.omni-explorer.com) WorldIndexer
  277. OmniExplorer_Bot/6.63b (+http://www.omni-explorer.com) WorldIndexer
  278. OmniExplorer_Bot/6.62 (+http://www.omni-explorer.com) WorldIndexer
  279. OmniExplorer_Bot/6.60 (+http://www.omni-explorer.com) WorldIndexer
  280. OmniExplorer_Bot/6.47 (+http://www.omni-explorer.com) WorldIndexer
  281. OmniExplorer_Bot/5.91c (+http://www.omni-explorer.com) WorldIndexer
  282. OmniExplorer_Bot/5.28 (+http://www.omni-explorer.com) WorldIndexer
  283. OmniExplorer_Bot/5.25 (+http://www.omni-explorer.com) WorldIndexer
  284. OmniExplorer_Bot/5.20 (+http://www.omni-explorer.com) WorldIndexer
  285. OmniExplorer_Bot/5.01 (+http://www.omni-explorer.com) WorldIndexer
  286. OmniExplorer_Bot/4.80 (+http://www.omni-explorer.com) WorldIndexer
  287. OmniExplorer_Bot/4.32 (+http://www.omni-explorer.com) WorldIndexer
  288. OOZBOT/0.20 ( -- ; http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
  289. OOZBOT/0.17 (--; http://www.setooz.com/oozbot.html; pvvpr at iiit dot ac dot in)
  290. Orbiter (+http://www.dailyorbit.com/bot.htm)
  291. PageBitesHyperBot/600 (http://www.pagebites.com/)
  292. Mozilla/5.0 (compatible; Peew/1.0; http://www.peew.de/crawler/)
  293. polybot 1.0 (http://cis.poly.edu/polybot/)
  294. Pompos/1.3 http://dir.com/pompos.html
  295. Pompos/1.2 http://pompos.iliad.fr
  296. Pompos/1.1 http://pompos.iliad.fr
  297. PostPost/1.0 (+http://postpo.st/crawlers)
  298. psbot/0.1 (+http://www.picsearch.com/bot.html)
  299. PycURL/7.23.1
  300. PycURL/7.19.7
  301. PycURL/7.19.5
  302. PycURL/7.19.3
  303. PycURL/7.19.0
  304. PycURL/7.18.2
  305. PycURL/7.18.0
  306. PycURL/7.16.4
  307. PycURL/7.15.5
  308. PycURL/7.13.2
  309. PycURL
  310. Qseero v1.0.0
  311. radian6_default_(www.radian6.com/crawler)
  312. RAMPyBot - www.giveRAMP.com/0.1 (RAMPyBot - www.giveRAMP.com; http://www.giveramp.com/bot.html; support@giveRAMP.com)
  313. RufusBot (Rufus Web Miner; http://64.124.122.252/feedback.html)
  314. SandCrawler - Compatibility Testing
  315. SBIder/0.8-dev (SBIder; http://www.sitesell.com/sbider.html; http://support.sitesell.com/contact-support.html)
  316. Mozilla/5.0 (compatible; ScoutJet; http://www.scoutjet.com/)
  317. Scrubby/2.2 (http://www.scrubtheweb.com/)
  318. Mozilla/5.0 (compatible; Scrubby/2.2; +http://www.scrubtheweb.com/)
  319. Mozilla/5.0 (compatible; Scrubby/2.2; http://www.scrubtheweb.com/)
  320. Scrubby/2.1 (http://www.scrubtheweb.com/)
  321. Mozilla/5.0 (compatible; Scrubby/2.1; +http://www.scrubtheweb.com/abs/meta-check.html)
  322. SearchSight/2.0 (http://SearchSight.com/)
  323. Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.2
  324. Seekbot/1.0 (http://www.seekbot.net/bot.html) HTTPFetcher/2.1
  325. Seekbot/1.0 (http://www.seekbot.net/bot.html) HTTPFetcher/0.3
  326. Seekbot/1.0 (http://www.seekbot.net/bot.html)
  327. semanticdiscovery/0.1
  328. Sensis Web Crawler (search_comments\at\sensis\dot\com\dot\au)
  329. Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) SEOChat::Bot v1.1
  330. SeznamBot/2.0 (+http://fulltext.seznam.cz/)
  331. SeznamBot/2.0 (+http://fulltext.sblog.cz/robot/)
  332. Shim-Crawler(Mozilla-compatible; http://www.logos.ic.i.u-tokyo.ac.jp/crawler/; crawl@logos.ic.i.u-tokyo.ac.jp)
  333. Shim-Crawler
  334. ShopWiki/1.0 ( +http://www.shopwiki.com/wiki/Help:Bot)
  335. Mozilla/4.0 (compatible: Shoula robot)
  336. silk/1.0 (+http://www.slider.com/silk.htm)/3.7
  337. Silk/1.0
  338. Mozilla/5.0 (compatible; SiteBot/0.1; +http://www.sitebot.org/robot/)
  339. Mozilla/5.0 (compatible; SiteBot/0.1; http://www.sitebot.org/robot/)
  340. Snappy/1.1 ( http://www.urltrends.com/ )
  341. sogou spider
  342. Sosospider+(+http://help.soso.com/webspider.htm)
  343. Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
  344. Mozilla/5.0 (compatible; Speedy Spider; http://www.entireweb.com/about/search_tech/speedy_spider/)
  345. Speedy Spider (Entireweb; Beta/1.3; http://www.entireweb.com/about/search_tech/speedyspider/)
  346. Speedy Spider (Entireweb; Beta/1.2; http://www.entireweb.com/about/search_tech/speedyspider/)
  347. Speedy Spider (Entireweb; Beta/1.1; http://www.entireweb.com/about/search_tech/speedyspider/)
  348. Speedy Spider (Entireweb; Beta/1.0; http://www.entireweb.com/about/search_tech/speedyspider/)
  349. Speedy Spider (Beta/1.0; www.entireweb.com)
  350. Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
  351. Speedy Spider (http://www.entireweb.com/about/search_tech/speedyspider/)
  352. Speedy Spider (http://www.entireweb.com)
  353. Sqworm/2.9.85-BETA (beta_release; 20011115-775; i686-pc-linux-gnu)
  354. StackRambler/2.0 (MSIE incompatible)
  355. StackRambler/2.0
  356. Mozilla/5.0 (compatible; suggybot v0.01a, http://blog.suggy.com/was-ist-suggy/suggy-webcrawler/)
  357. SurveyBot/2.3+(Whois+Source)
  358. SurveyBot/2.3 (Whois Source)
  359. SynooBot/0.7.1 (SynooBot; http://www.synoo.de/bot.html; webmaster@synoo.com)
  360. Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_crawling.html)
  361. Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)
  362. Mozilla/2.0 (compatible; Ask Jeeves/Teoma)
  363. TerrawizBot/1.0 (+http://www.terrawiz.com/bot.html)
  364. TheSuBot/0.2 (www.thesubot.de)
  365. TheSuBot/0.1 (www.thesubot.de)
  366. Thumbnail.CZ robot 1.1 (http://thumbnail.cz/why-no-robots-txt.html)
  367. TinEye/1.1 (http://tineye.com/crawler.html)
  368. TinEye
  369. truwoGPS/1.0 (GNU/Linux; U; i686; en-US; +http://www.lan4lano.net/browser.html )
  370. TurnitinBot/2.1 (http://www.turnitin.com/robot/crawlerinfo.html)
  371. TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html
  372. TurnitinBot/1.5 http://www.turnitin.com/robot/crawlerinfo.html
  373. TurnitinBot/1.5 (http://www.turnitin.com/robot/crawlerinfo.html)
  374. TurnitinBot/1.5 http://www.turnitin.com/robot/crawlerinfo.html
  375. TurnitinBot/1.5 (http://www.turnitin.com/robot/crawlerinfo.html)
  376. Mozilla/5.0 (compatible; TweetedTimes Bot/1.0; http://tweetedtimes.com)
  377. TwengaBot
  378. updated/0.1-beta (updated; http://www.updated.com; updated@updated.com)
  379. Mozilla/5.0 (compatible; Urlfilebot/2.2; +http://urlfile.com/bot.html)
  380. Mozilla/4.0 (compatible; Vagabondo/4.0Beta; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/; http://www.wise-guys.nl/)
  381. Mozilla/4.0 (compatible; Vagabondo/2.2; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)
  382. Mozilla/5.0 (compatible; Vagabondo/2.1; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)
  383. Mozilla/3.0 (Vagabondo/2.0 MT; webcrawler@NOSPAMexperimental.net; http://aanmelden.ilse.nl/?aanmeld_mode=webhints)
  384. Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot BETA 1.2 (http://www.voila.com/)
  385. Vortex/2.2 (+http://marty.anstey.ca/robots/vortex/)
  386. Vortex/2.2 ( http://marty.anstey.ca/robots/vortex/)
  387. VORTEX/1.2 ( http://marty.anstey.ca/robots/vortex/)
  388. voyager/2.0 (http://www.kosmix.com/crawler.html)
  389. voyager/1.0
  390. VYU2 (GNU; OpenRISC)
  391. webcollage/1.93
  392. webcollage/1.129
  393. webcollage/1.125
  394. webcollage/1.117
  395. webcollage/1.114
  396. Websquash.com (Add url robot)
  397. http://www.almaden.ibm.com/cs/crawler [wf84]
  398. WoFindeIch Robot 1.0(+http://www.search.wofindeich.com/robot.php)
  399. WoFindeIch Robot 1.0( http://www.search.wofindeich.com/robot.php)
  400. WomlpeFactory/0.1 (+http://www.Womple.com/bot.html)
  401. Xaldon_WebSpider/2.0.b1
  402. Xaldon_WebSpider/2.0.b1
  403. yacybot (x86 Windows XP 5.1; java 1.6.0_12; Europe/de) http://yacy.net/bot.html
  404. yacybot (x86 Windows XP 5.1; java 1.6.0_11; Europe/de) http://yacy.net/bot.html
  405. yacybot (x86 Windows XP 5.1; java 1.6.0; Europe/de) http://yacy.net/yacy/bot.html
  406. yacybot (x86 Windows 2000 5.0; java 1.6.0_16; Europe/de) http://yacy.net/bot.html
  407. yacybot (ppc Mac OS X 10.5.2; java 1.5.0_13; Europe/de) http://yacy.net/bot.html
  408. yacybot (ppc Mac OS X 10.4.10; java 1.5.0_07; Europe/de) http://yacy.net/bot.html
  409. yacybot (i386 Mac OS X 10.5.7; java 1.5.0_16; Europe/de) http://yacy.net/bot.html
  410. yacybot (i386 Linux 2.6.9-023stab046.2-smp; java 1.6.0_05; Europe/en) http://yacy.net/bot.html
  411. yacybot (i386 Linux 2.6.8-022stab070.5-enterprise; java 1.4.2-03; Europe/en) yacy.net
  412. yacybot (i386 Linux 2.6.31-16-generic; java 1.6.0_15; Europe/en) http://yacy.net/bot.html
  413. yacybot (i386 Linux 2.6.26-2-686; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
  414. yacybot (i386 Linux 2.6.24-28-generic; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
  415. yacybot (i386 Linux 2.6.24-24-generic; java 1.6.0_07; Europe/en) http://yacy.net/bot.html
  416. yacybot (i386 Linux 2.6.24-23-generic; java 1.6.0_16; Europe/en) http://yacy.net/bot.html
  417. yacybot (i386 Linux 2.6.23; java 1.6.0_17; Europe/en) http://yacy.net/bot.html
  418. yacybot (i386 Linux 2.6.23; java 1.6.0_04; Europe/en) http://yacy.net/bot.html
  419. yacybot (i386 Linux 2.6.22-14-generic; java 1.6.0_03; Europe/de) http://yacy.net/bot.html
  420. yacybot (amd64 Windows 7 6.1; java 1.6.0_17; Europe/de) http://yacy.net/bot.html
  421. yacybot (amd64 Linux 2.6.28-18-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
  422. yacybot (amd64 Linux 2.6.16-2-amd64-k8-smp; java 1.5.0_10; Europe/en) http://yacy.net/yacy/bot.html
  423. Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
  424. Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)
  425. YahooSeeker/1.2 (compatible; Mozilla 4.0; MSIE 5.5; yahooseeker at yahoo-inc dot com ; http://help.yahoo.com/help/us/shop/merchant/)
  426. YahooSeeker-Testing/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://search.yahoo.com/)
  427. Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
  428. Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)
  429. Yasaklibot/v1.2 (http://www.Yasakli.com/bot.php)
  430. Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)
  431. Yeti/1.0 (+http://help.naver.com/robots/)
  432. Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )
  433. yoogliFetchAgent/0.1
  434. Mozilla/5.0 (compatible; YoudaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; )
  435. Zao/0.1 (http://www.kototoi.org/zao/)
  436. Zao/0.1 (http://www.kototoi.org/zao/)
  437. Mozilla/4.0 (compatible; Zealbot 1.0)
  438. zspider/0.9-dev http://feedback.redkolibri.com/
  439. Mozilla/4.0 compatible ZyBorg/1.0 DLC (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)
  440. Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)
  441. Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker (wn.dlc@looksmart.net; http://www.WISEnutbot.com)
  442. Mozilla/4.0 compatible ZyBorg/1.0 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)
  443. Mozilla/4.0 compatible ZyBorg/1.0 (wn-16.zyborg@looksmart.net; http://www.WISEnutbot.com)
  444. Mozilla/4.0 compatible ZyBorg/1.0 (wn-14.zyborg@looksmart.net; http://www.WISEnutbot.com)
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!