API tools faq
paste
Guest User

Untitled

a guest
May 9th, 2016
192
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 45.33 KB | None | 0 0
raw download clone embed print report
  1. AdBot 1.0 (www.adnetik.com)
  2. AdMedia bot
  3. AdsBot-Google (+http://www.google.com/adsbot.html)
  4. AdsBot-Google (+http://www.google.com/adsbot.html)
  5. Affectv Robot v1.0/Nutch-1.6 (http://www.affectv.co.uk; austin at affectv dot co dot uk)
  6. Apple-iPhone4C1/1001.523 (compatible;acapbot/0.1)
  7. Apple-iPhone4C1/1001.523 (compatible;acapbot/0.1;treat like Googlebot)
  8. Apple-iPhone4C1/1001.523 (compatible;acapbot/0.1;treat like Googlebot)
  9. AppleNewsBot
  10. ArchiveTeam ArchiveBot/20150701.01 (wpull 1.2.1) and not Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWe
  11. ArchiveTeam ArchiveBot/20150715.01 (wpull 1.2.2) and not Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWe
  12. AugDiscovery-Bot/0.1
  13. BOT/0.1 (BOT for JCE)
  14. BOT/0.1 (BOT for JCE)
  15. BitActive.com Crawler (http://www.bitactive.com/bot.html)
  16. Bot
  17. Bot.AraTurka.com/0.0.1
  18. Bot.AraTurka.com/0.0.2
  19. BrandONbot (http://brandonmedia.net)
  20. BuckyOHare/1.3 (Googlebot/2.1; +https://hypefactors.com/webcrawler)
  21. BusinessBot: Nathan@lead-caddy.com
  22. Buzzbot/1.0 (Buzzbot; http://www.buzzstream.com; buzzbot@buzzstream.com)
  23. Buzzbot/1.0 (Buzzbot; http://www.buzzstream.com; buzzbot@buzzstream.com)
  24. Buzzbot/1.0 (Buzzbot; http://www.buzzstream.com; buzzbot@buzzstream.com)
  25. CCBot/2.0 (http://commoncrawl.org/faq/)
  26. CheburashkaSearchBot
  27. Clickagy Intelligence Bot v2
  28. Cliqzbot/0.1 (+http://cliqz.com/company/cliqzbot)
  29. Cliqzbot/0.1 (+http://cliqz.com/company/cliqzbot)
  30. CoinCornerBot/1.1 (+https://www.coincorner.com/BitcoinBot)
  31. CoinCornerBot/1.1 (+https://www.coincorner.com/BitcoinBot)
  32. Comodo-Webinspector-Crawler 2.2.0, http://www.comodorobot.com
  33. Comodo-Webinspector-Crawler 2.2.1, http://www.comodorobot.com
  34. ConBot
  35. ContextAd Bot 1.0
  36. CorenSearchBot/1.7 en libwww-perl/6.05
  37. DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
  38. DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
  39. Domain Re-Animator Bot (http://domainreanimator.com) - support@domainreanimator.com
  40. Domnutch-Bot/Nutch-1.0 (Domnutch; http://www.Nutch.de/)
  41. Elmer, the Thinglink ImageBot (http://www.thinglink.com/help/ThinglinkImageBot)
  42. Elmer, the Thinglink ImageBot (http://www.thinglink.com/help/ThinglinkImageBot)
  43. Experibot_v1
  44. Experibot_v1 (https://dl.dropboxusercontent.com/u/8024465/site/Info.html)
  45. Experibot_v1 [goo.gl/n6zrAf]
  46. Facebot/1.0
  47. Flamingo_SearchEngine (+http://www.flamingosearch.com/bot)
  48. Gigabot/1.0
  49. Gigabot/2.0
  50. Google Bot
  51. Googlebot
  52. Googlebot-Image/1.0
  53. Googlebot-News (2.3.4.1, ruby 1.9.3 (2014-05-14))
  54. Googlebot-Video/1.0
  55. Googlebot/1.2 (compatible)
  56. Googlebot/2.1 (+http://www.google.com/bot.html)
  57. Googlebot/2.1 (+http://www.google.com/bot.html)
  58. Googlebot/2.1 (+http://www.googlebot.com/bot.html)
  59. Googlebot/2.1 (+http://www.googlebot.com/bot.html)
  60. Googlebot/2.1 (+http://www.googlebot.com/bot.html)
  61. Googlebot/2.1; +http://www.google.com/bot.html)
  62. Googlebot/2.1; +http://www.google.com/bot.html)
  63. Googlebot/2.X (+http://www.googlebot.com/bot.html)
  64. Googlebot/2.X (+http://www.googlebot.com/bot.html)
  65. Googlebot/2.X (+http://www.googlebot.com/bot.html)
  66. GumGum Bot (http://gumgum.com; support@gumgum.com)
  67. GumGum-Bot/1.0 (http://gumgum.com; support@gumgum.com)
  68. IgnitionOneBot/Nutch-1.9 ( This is the IgnitionOne Company Bot for Web Crawling. IgnitionOne
  69. IgnitionOneBot/Nutch-1.9 ( This is the IgnitionOne Company Bot for Web Crawling. IgnitionOne
  70. InsightsCollector/0.1beta (http://www.conceptcreative.nl/bot.html)
  71. Kbot001
  72. Kemvibot/1.0 (http://kemvi.com)
  73. LexxeBot/1.0 (lexxebot@lexxe.com)
  74. LexxeBot/1.0 (lexxebot@lexxe.com)
  75. LinexBot/A-1.0 (www.linexsystems.com; support at linexsystems dot com)
  76. LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 +http://www.linkedin.com)
  77. LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/4.1 +http://www.linkedin.com)
  78. LinqiaMetadataDownloaderBot/1.0 (eng@linqia.com)
  79. LinqiaScrapeBot/1.0 (eng@linqia.com)
  80. LivelapBot/0.2 (http://site.livelap.com/crawler)
  81. LumpImageSearch/0.1 (+http://lump.co/about/bot)
  82. MOT-V3/ (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)
  83. MPDP-ALR-Search-Bot
  84. MSNBot/Nutch-1.5.1
  85. Mediatoolkitbot (complaints@mediatoolkit.com)
  86. Mediatoolkitbot (info@mediatoolkit.com)
  87. Mozilla (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)
  88. Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; Girafabot; girafabot at girafa dot com; http://www.gi
  89. Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; Girafabot; girafabot at girafa dot com; http://www.gi
  90. Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://www.changedetection.com/bot.html )
  91. Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; CrystalSemanticsBot http://www.crystalsemant
  92. Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0 ; Claritybot)
  93. Mozilla/5.0 (Linux; Android 5.1; CUBOT_NOTE_S Build/LMY47I) AppleWebKit/537.36 (KHTML, like Gecko) C
  94. Mozilla/5.0 (Linux; U; Android 4.2.2; en-us; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 (KHTML, like G
  95. Mozilla/5.0 (Linux; U; Android 4.2.2; en-us; CUBOT P9 Build/JDQ39) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30
  96. Mozilla/5.0 (Linux; U; Android 4.2.2; en-us; CUBOT X6 Build/JDQ39) AppleWebKit/534.30 (KHTML, like G
  97. Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.111 Safari/537.36 moatbot
  98. Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot)
  99. Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot)
  100. Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0
  101. Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0
  102. Mozilla/5.0 (Windows NT 5.1; U; Win64; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgr
  103. Mozilla/5.0 (Windows NT 5.1; U; Win64; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgr
  104. Mozilla/5.0 (Windows NT 6.1) (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
  105. Mozilla/5.0 (Windows NT 6.1) (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
  106. Mozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0) Gecko/20100101 Firefox/12.0 alsRobot2
  107. Mozilla/5.0 (Windows NT 6.1; Win64; x64) KomodiaBot/1.0
  108. Mozilla/5.0 (Windows NT 6.1; compatible; BDCbot/1.0; +http://ecommerce.bigdatacorp.com.br/faq.aspx)
  109. Mozilla/5.0 (Windows NT 6.2) Insitesbot/1.0
  110. Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv 11.0) like Gecko (compatible; Zombiebot/2.1; +http://ww
  111. Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv 11.0) like Gecko (compatible; Zombiebot/2.1; +http://www.zombiedomain.net/robot/)
  112. Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv 11.0) like Gecko (compatible; Zombiebot/2.1; +http://www.zombiedomain.net/robot/)
  113. Mozilla/5.0 (Windows NT 6.3;compatible; Leikibot/1.0; +http://www.leiki.com)
  114. Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ft
  115. Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ft
  116. Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; htt
  117. Mozilla/5.0 (X11; Linux x86_64 UXCrawlerBot http://zium.co)
  118. Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) TarantulaBot (Linux x86_64) W
  119. Mozilla/5.0 (X11; Linux x86_64; rv:14.0) Gecko/20100101 Firefox/14.0.1 moatbot
  120. Mozilla/5.0 (compatible; 200PleaseBot/1.0; +http://www.200please.com/bot)
  121. Mozilla/5.0 (compatible; 200PleaseBot/1.0; +http://www.200please.com/bot)
  122. Mozilla/5.0 (compatible; AMZNKAssocBot/4.0 +http://affiliate-program.amazon.com)
  123. Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)
  124. Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)
  125. Mozilla/5.0 (compatible; Applebot/0.3; +http://www.apple.com/go/applebot)
  126. Mozilla/5.0 (compatible; Applebot/0.3; +http://www.apple.com/go/applebot)
  127. Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)
  128. Mozilla/5.0 (compatible; BLEXBotTest/1.0; +http://webmeup.com/crawler.html)
  129. Mozilla/5.0 (compatible; BingLocalSearchBot/1.0)
  130. Mozilla/5.0 (compatible; Blekkobot; ScoutJet; +http://blekko.com/about/blekkobot)
  131. Mozilla/5.0 (compatible; Blekkobot; ScoutJet; +http://blekko.com/about/blekkobot)
  132. Mozilla/5.0 (compatible; CareerBot/1.1; +http://www.career-x.de/bot.html)
  133. Mozilla/5.0 (compatible; CareerBot/1.1; +http://www.career-x.de/bot.html)
  134. Mozilla/5.0 (compatible; Climatebot/1.0; +http://climate.k39.us/bot.html)
  135. Mozilla/5.0 (compatible; Climatebot/1.0; +http://climate.k39.us/bot.html)
  136. Mozilla/5.0 (compatible; Cliqzbot/0.1 +http://cliqz.com/company/cliqzbot)
  137. Mozilla/5.0 (compatible; Cliqzbot/0.1 +http://cliqz.com/company/cliqzbot)
  138. Mozilla/5.0 (compatible; Cliqzbot/1.0 +http://cliqz.com/company/cliqzbot)
  139. Mozilla/5.0 (compatible; Cliqzbot/1.0 +http://cliqz.com/company/cliqzbot)
  140. Mozilla/5.0 (compatible; CompSpyBot/1.0; +http://www.compspy.com/spider.html)
  141. Mozilla/5.0 (compatible; CrawlBot/1.0.0)
  142. Mozilla/5.0 (compatible; DeuSu/5.0.2; +https://deusu.de/robot.html)
  143. Mozilla/5.0 (compatible; Discordbot/1.0; +https://discordapp.com)
  144. Mozilla/5.0 (compatible; DomainSigmaCrawler/0.1; +http://domainsigma.com/robot)
  145. Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)
  146. Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)
  147. Mozilla/5.0 (compatible; DuckDuckGo-Favicons-Bot/1.0; +http://duckduckgo.com)
  148. Mozilla/5.0 (compatible; Edmunds SEO/DeepCrawl; http://deepcrawl.co.uk/bot.html) https://www.deepcra
  149. Mozilla/5.0 (compatible; EmailMarketingRobot/2.1; +http://www.email-marketing-robot.com/webmasters/)
  150. Mozilla/5.0 (compatible; EmailMarketingRobot/2.1; +http://www.email-marketing-robot.com/webmasters/)
  151. Mozilla/5.0 (compatible; EveryoneSocialBot/1.0; support@everyonesocial.com http://everyonesocial.com
  152. Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); +http://www.exabot.com/go/robot)
  153. Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); +http://www.exabot.com/go/robot)
  154. Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); +http://www.exabot.com/go/robot)
  155. Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)
  156. Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)
  157. Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)
  158. Mozilla/5.0 (compatible; ExchangleBot/3.0; +http://www.exchangle.com/add)
  159. Mozilla/5.0 (compatible; FatBot 2.0; http://www.thefind.com/crawler)
  160. Mozilla/5.0 (compatible; Feedspotbot/1.0; +http://www.feedspot.com/fs/bot)
  161. Mozilla/5.0 (compatible; Feedspotbot/1.0; +http://www.feedspot.com/fs/bot)
  162. Mozilla/5.0 (compatible; Findxbot/1.0; +http://www.findxbot.com)
  163. Mozilla/5.0 (compatible; Findxbot/1.0; +http://www.findxbot.com)
  164. Mozilla/5.0 (compatible; Gimme60bot/1.0; +http://gimme60.com)
  165. Mozilla/5.0 (compatible; Gimme60bot/1.0; +http://gimme60.com) Firefox/16.0
  166. Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +https://gimmeusa.com/crawler.html)
  167. Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +https://gimmeusa.com/pages/crawler)
  168. Mozilla/5.0 (compatible; Googlebot/2.1 +http://www.googlebot.com/bot.html)
  169. Mozilla/5.0 (compatible; Googlebot/2.1 +http://www.googlebot.com/bot.html)
  170. Mozilla/5.0 (compatible; Googlebot/2.1 +http://www.googlebot.com/bot.html)
  171. Mozilla/5.0 (compatible; Googlebot/2.1)
  172. Mozilla/5.0 (compatible; Googlebot/2.1; +http://import.io)
  173. Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html
  174. Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html
  175. Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  176. Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  177. Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)/1.8 (InfoSeek crawler; http
  178. Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)/1.8 (InfoSeek crawler; http
  179. Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)/1.8 (InfoSeek crawler; http://www.infoseek.com; crawler@infoseek.com)
  180. Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)/1.8 (InfoSeek crawler; http://www.infoseek.com; crawler@infoseek.com)
  181. Mozilla/5.0 (compatible; Googlebot/2.1; compatible; Googlebot/2.0; +http://www.google.com/bot.html)
  182. Mozilla/5.0 (compatible; Googlebot/2.1; compatible; Googlebot/2.0; +http://www.google.com/bot.html)
  183. Mozilla/5.0 (compatible; Googlebot/2.1; compatible; Googlebot/2.0; +http://www.google.com/bot.html)
  184. Mozilla/5.0 (compatible; Googlebot/2.1; http://deepcrawl.co.uk/bot.html)
  185. Mozilla/5.0 (compatible; Googlebot/2.1; http://deepcrawl.co.uk/bot.html)
  186. Mozilla/5.0 (compatible; Googlebot/2.1; https://www.deepcrawl.com/bot)
  187. Mozilla/5.0 (compatible; Googlebot/2.1; https://www.deepcrawl.com/bot)
  188. Mozilla/5.0 (compatible; Googlebot/2.1; startmebot/1.0; +https://start.me/bot)
  189. Mozilla/5.0 (compatible; Googlebot/2.1; startmebot/1.0; +https://start.me/bot)
  190. Mozilla/5.0 (compatible; Googlebot/2.1; startmebot/1.0; +https://start.me/bot)
  191. Mozilla/5.0 (compatible; Googlebot/2.1;)
  192. Mozilla/5.0 (compatible; Googlebot/2.1;+http://www.google.com/bot.html)
  193. Mozilla/5.0 (compatible; Googlebot/2.1;+http://www.google.com/bot.html)
  194. Mozilla/5.0 (compatible; GroupHigh/1.0; +http://www.grouphigh.com/bot.html)
  195. Mozilla/5.0 (compatible; HyperCrawl/0.2; +http://www.seograph.net/bot.html)
  196. Mozilla/5.0 (compatible; IstellaBot/1.18.81 +http://www.tiscali.it/)
  197. Mozilla/5.0 (compatible; IstellaBot/1.23.15 +http://www.tiscali.it/)
  198. Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails)
  199. Mozilla/5.0 (compatible; Kraken/0.1; http://linkfluence.net/; bot@linkfluence.net)
  200. Mozilla/5.0 (compatible; LinkisBot/1.0; bot@linkis.com)
  201. Mozilla/5.0 (compatible; LinkisBot/1.0; bot@linkis.com)
  202. Mozilla/5.0 (compatible; LinkpadBot/1.06; +http://www.linkpad.ru)
  203. Mozilla/5.0 (compatible; LinkpadBot/1.07; +http://www.linkpad.ru)
  204. Mozilla/5.0 (compatible; LinkpadBot/1.08; +http://www.linkpad.ru)
  205. Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
  206. Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
  207. Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots/2.0; +http://go.mail.ru/help/robots)
  208. Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots/2.0; +http://go.mail.ru/help/robots)
  209. Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots/2.0; +http://go.mail.ru/help/robots)
  210. Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots; +http://go.mail.ru/help/robots)
  211. Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots; +http://go.mail.ru/help/robots)
  212. Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots; +http://go.mail.ru/help/robots)
  213. Mozilla/5.0 (compatible; LoadTimeBot/0.9; +http://www.loadtime.net/bot.html)
  214. Mozilla/5.0 (compatible; LoadTimeBot/0.9; +http://www.loadtime.net/bot.html)
  215. Mozilla/5.0 (compatible; MJ12bot/v1.4.4; http://www.majestic12.co.uk/bot.php?+)
  216. Mozilla/5.0 (compatible; MJ12bot/v1.4.4; http://www.majestic12.co.uk/bot.php?+)
  217. Mozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+)
  218. Mozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+)
  219. Mozilla/5.0 (compatible; MixrankBot; crawler@mixrank.com)
  220. Mozilla/5.0 (compatible; Moatbot/2.2; +http://www.moat.com/pages/moatbot)
  221. Mozilla/5.0 (compatible; Moatbot/2.2; +http://www.moat.com/pages/moatbot)
  222. Mozilla/5.0 (compatible; MojeekBot/0.6; +https://www.mojeek.com/bot.html)
  223. Mozilla/5.0 (compatible; MojeekBot/0.6; +https://www.mojeek.com/bot.html)
  224. Mozilla/5.0 (compatible; MojeekBot/0.6; http://www.mojeek.com/bot.html)
  225. Mozilla/5.0 (compatible; MojeekBot/0.6; http://www.mojeek.com/bot.html)
  226. Mozilla/5.0 (compatible; Mojeekbot/0.6; +https://www.mojeek.com/bot.html)
  227. Mozilla/5.0 (compatible; Mojeekbot/0.6; +https://www.mojeek.com/bot.html)
  228. Mozilla/5.0 (compatible; OmniBot/1.0)
  229. Mozilla/5.0 (compatible; OpenHoseBot/2.1; +http://www.openhose.org/bot.html)
  230. Mozilla/5.0 (compatible; OpenHoseBot/2.1; +http://www.openhose.org/bot.html)
  231. Mozilla/5.0 (compatible; OptimizationCrawler/0.2; +http://www.domainoptima.com/robot)
  232. Mozilla/5.0 (compatible; OrangeBot/2.0; support.orangebot@orange.com)
  233. Mozilla/5.0 (compatible; OrangeBot/2.0; support.orangebot@orange.com)
  234. Mozilla/5.0 (compatible; OrangeBot/2.0; support.voilabot@orange.com)
  235. Mozilla/5.0 (compatible; OrangeBot/2.0; support.voilabot@orange.com)
  236. Mozilla/5.0 (compatible; PaperLiBot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)
  237. Mozilla/5.0 (compatible; Please Name Your robot; +http://173.13.143.74/bot.php)
  238. Mozilla/5.0 (compatible; Please Name Your robot; +http://173.13.143.74/bot.php)
  239. Mozilla/5.0 (compatible; Plukkie/1.5; http://www.botje.com/plukkie.htm)
  240. Mozilla/5.0 (compatible; Plukkie/1.6; http://www.botje.com/plukkie.htm)
  241. Mozilla/5.0 (compatible; PrivacyAwareBot/1.1; +http://www.privacyaware.org)
  242. Mozilla/5.0 (compatible; Qualidator.com Bot 0.9;)
  243. Mozilla/5.0 (compatible; RankActiveLinkBot; +https://rankactive.com/resources/rankactive-linkbot)
  244. Mozilla/5.0 (compatible; RankActiveLinkBot; +https://rankactive.com/resources/rankactive-linkbot)
  245. Mozilla/5.0 (compatible; SEOdiver/1.0; +http://www.seodiver.com/bot)
  246. Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)
  247. Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)
  248. Mozilla/5.0 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
  249. Mozilla/5.0 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
  250. Mozilla/5.0 (compatible; SearchmetricsBot; http://www.searchmetrics.com/en/searchmetrics-bot/)
  251. Mozilla/5.0 (compatible; SearchmetricsBot; http://www.searchmetrics.com/en/searchmetrics-bot/)
  252. Mozilla/5.0 (compatible; SemrushBot-SA/0.97; +http://www.semrush.com/bot.html)
  253. Mozilla/5.0 (compatible; SemrushBot-SA/0.97; +http://www.semrush.com/bot.html)
  254. Mozilla/5.0 (compatible; SemrushBot/0.97; +http://www.semrush.com/bot.html)
  255. Mozilla/5.0 (compatible; SemrushBot/0.97; +http://www.semrush.com/bot.html)
  256. Mozilla/5.0 (compatible; SemrushBot/0.97~bl; +http://www.semrush.com/bot.html)
  257. Mozilla/5.0 (compatible; SemrushBot/0.97~bl; +http://www.semrush.com/bot.html)
  258. Mozilla/5.0 (compatible; SemrushBot/0.98~bl; +http://www.semrush.com/bot.html)
  259. Mozilla/5.0 (compatible; SemrushBot/0.98~bl; +http://www.semrush.com/bot.html)
  260. Mozilla/5.0 (compatible; SemrushBot/0.99~bl; +http://www.semrush.com/bot.html)
  261. Mozilla/5.0 (compatible; SemrushBot/0.99~bl; +http://www.semrush.com/bot.html)
  262. Mozilla/5.0 (compatible; SemrushBot/1.1~bl; +http://www.semrush.com/bot.html)
  263. Mozilla/5.0 (compatible; SemrushBot/1.1~bl; +http://www.semrush.com/bot.html)
  264. Mozilla/5.0 (compatible; SemrushBot/1~bl; +http://www.semrush.com/bot.html)
  265. Mozilla/5.0 (compatible; SemrushBot/1~bl; +http://www.semrush.com/bot.html)
  266. Mozilla/5.0 (compatible; SeznamBot/3.2-test1; +http://fulltext.sblog.cz/)
  267. Mozilla/5.0 (compatible; SeznamBot/3.2-test2; +http://fulltext.sblog.cz/)
  268. Mozilla/5.0 (compatible; SeznamBot/3.2-test4; +http://fulltext.sblog.cz/)
  269. Mozilla/5.0 (compatible; SeznamBot/3.2; +http://fulltext.sblog.cz/)
  270. Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)
  271. Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)
  272. Mozilla/5.0 (compatible; SputnikBot/2.3; +http://corp.sputnik.ru/webmaster)
  273. Mozilla/5.0 (compatible; SputnikFaviconBot/1.2; +http://corp.sputnik.ru/webmaster)
  274. Mozilla/5.0 (compatible; SputnikImageBot/2.3; +http://corp.sputnik.ru/webmaster)
  275. Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html)
  276. Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html)
  277. Mozilla/5.0 (compatible; TWMBot/0.1; +http://thewebminer.com)
  278. Mozilla/5.0 (compatible; TWMBot/0.3; +http://thewebminer.com)
  279. Mozilla/5.0 (compatible; Taboolabot/3.7; +http://www.taboola.com)
  280. Mozilla/5.0 (compatible; Tapatalk CloudSearch Platform/1.0; http://www.tapatalk.com/bot.html)
  281. Mozilla/5.0 (compatible; TweetmemeBot/3.0; +http://tweetmeme.com/)
  282. Mozilla/5.0 (compatible; Twelvefoldbot/1.0 +http://www.twelvefold.com)
  283. Mozilla/5.0 (compatible; URLAppendBot/1.0; +http://www.profound.net/urlappendbot.html)
  284. Mozilla/5.0 (compatible; URLAppendBot/1.0; +http://www.profound.net/urlappendbot.html)
  285. Mozilla/5.0 (compatible; UnisterBot; crawler@unister.de)
  286. Mozilla/5.0 (compatible; Uptimebot/1.0; +http://www.uptime.com/uptimebot)
  287. Mozilla/5.0 (compatible; Uptimebot/1.0; +http://www.uptime.com/uptimebot)
  288. Mozilla/5.0 (compatible; Veoozbot/1.0; +http://www.veooz.com/veoozbot.html)
  289. Mozilla/5.0 (compatible; Veoozbot/1.0; +http://www.veooz.com/veoozbot.html)
  290. Mozilla/5.0 (compatible; WASALive-Bot ; http://blog.wasalive.com/wasalive-bots/)
  291. Mozilla/5.0 (compatible; WASALive-Bot ; http://blog.wasalive.com/wasalive-bots/)
  292. Mozilla/5.0 (compatible; WBSearchBot/1.1; +http://www.warebay.com/bot.html)
  293. Mozilla/5.0 (compatible; WBSearchBot/1.1; +http://www.warebay.com/bot.html)
  294. Mozilla/5.0 (compatible; WeViKaBot/1.0; +http://www.wevika.de/)
  295. Mozilla/5.0 (compatible; XoviBot/2.0; +http://www.xovibot.net/)
  296. Mozilla/5.0 (compatible; XoviBot/2.0; +http://www.xovibot.net/)
  297. Mozilla/5.0 (compatible; YandexAntivirus/2.0; +http://yandex.com/bots)
  298. Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
  299. Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
  300. Mozilla/5.0 (compatible; YandexDirect/3.0; +http://yandex.com/bots)
  301. Mozilla/5.0 (compatible; YandexFavicons/1.0; +http://yandex.com/bots)
  302. Mozilla/5.0 (compatible; YandexImageResizer/2.0; +http://yandex.com/bots)
  303. Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)
  304. Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots)
  305. Mozilla/5.0 (compatible; YesupBot/1.0; +http://www.yesup.net/bot.html)
  306. Mozilla/5.0 (compatible; YesupBot/1.0; +http://www.yesup.net/bot.html)
  307. Mozilla/5.0 (compatible; Yeti/1.1; +http://help.naver.com/robots/)
  308. Mozilla/5.0 (compatible; YioopBot; +http://173.13.143.74/bot.php)
  309. Mozilla/5.0 (compatible; YioopBot; +http://173.13.143.74/bot.php)
  310. Mozilla/5.0 (compatible; YoudaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; )
  311. Mozilla/5.0 (compatible; ZumBot/1.0; http://help.zum.com/inquiry)
  312. Mozilla/5.0 (compatible; archive.org_bot +http://archive.org/details/archive.org_bot)
  313. Mozilla/5.0 (compatible; archive.org_bot +http://archive.org/details/archive.org_bot)
  314. Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)
  315. Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)
  316. Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record; +http://archive.org/details/a
  317. Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)
  318. Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)
  319. Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
  320. Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
  321. Mozilla/5.0 (compatible; bnf.fr_bot; +http://www.bnf.fr/fr/outils/a.dl_web_capture_robot.html)
  322. Mozilla/5.0 (compatible; bnf.fr_bot; +http://www.bnf.fr/fr/outils/a.dl_web_capture_robot.html)
  323. Mozilla/5.0 (compatible; etoolsbot/1.0; +https://etools.io/bots/)
  324. Mozilla/5.0 (compatible; etoolsbot/1.0; +https://etools.io/bots/)
  325. Mozilla/5.0 (compatible; gimme60bot/1.0 ; +http://gimme60.com)
  326. Mozilla/5.0 (compatible; idmarch Automatic.beta/1.2; +http://www.idmarch.org/bot.html)
  327. Mozilla/5.0 (compatible; idmarch Automatic.beta/1.3; +http://www.idmarch.org/bot.html)
  328. Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/bots/)
  329. Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/bots/)
  330. Mozilla/5.0 (compatible; linkdexbot/2.2; +http://www.linkdex.com/bots/)
  331. Mozilla/5.0 (compatible; linkdexbot/2.2; +http://www.linkdex.com/bots/)
  332. Mozilla/5.0 (compatible; ltbot/0.3.6 +http://www.kdsl.tu-darmstadt.de/de/kdsl/research-program/crawl
  333. Mozilla/5.0 (compatible; mcsbot/2.0; +http://www.mediarithmics.com/)
  334. Mozilla/5.0 (compatible; meanpathbot/1.0; +http://www.meanpath.com/meanpathbot.html)
  335. Mozilla/5.0 (compatible; meanpathbot/1.0; +http://www.meanpath.com/meanpathbot.html)
  336. Mozilla/5.0 (compatible; memoryBot/1.20.311 +http://internetmemory.org/en/)
  337. Mozilla/5.0 (compatible; memoryBot/1.21.14 +http://mignify.com/bot.html)
  338. Mozilla/5.0 (compatible; memoryBot/1.21.14 +http://mignify.com/bot.html)
  339. Mozilla/5.0 (compatible; memoryBot/1.21.24 +http://internetmemory.org/en/)
  340. Mozilla/5.0 (compatible; memoryBot/1.21.46 +http://internetmemory.org/en/)
  341. Mozilla/5.0 (compatible; memorybot/1.21.14 +http://mignify.com/bot.html)
  342. Mozilla/5.0 (compatible; memorybot/1.21.14 +http://mignify.com/bot.html)
  343. Mozilla/5.0 (compatible; musobot/1.0; info@muso.com; +http://www.muso.com)
  344. Mozilla/5.0 (compatible; oBot/2.3.1; +http://filterdb.iss.net/crawler/)
  345. Mozilla/5.0 (compatible; oBot/2.3.1; http://filterdb.iss.net/crawler/)
  346. Mozilla/5.0 (compatible; parsijoo-bot; +http://www.parsijoo.ir/; ehsanmousa@parsijoo.ir)
  347. Mozilla/5.0 (compatible; pmoz.info ODP link checker; +http://pmoz.info/doc/botinfo.htm)
  348. Mozilla/5.0 (compatible; quakbot/432.12 +http://iamreallynoone.me)
  349. Mozilla/5.0 (compatible; quakbot/432.12 +http://iamreallynoone.me/educational/scientific/crawl)
  350. Mozilla/5.0 (compatible; redditbot/1.0; +http://www.reddit.com/feedback)
  351. Mozilla/5.0 (compatible; seplinkbot/1.0 )
  352. Mozilla/5.0 (compatible; smrjbot/0.0.20)
  353. Mozilla/5.0 (compatible; spbot/4.1.0; +http://OpenLinkProfiler.org/bot )
  354. Mozilla/5.0 (compatible; spbot/4.1.0; +http://OpenLinkProfiler.org/bot )
  355. Mozilla/5.0 (compatible; spbot/4.2.0; +http://OpenLinkProfiler.org/bot )
  356. Mozilla/5.0 (compatible; spbot/4.2.0; +http://OpenLinkProfiler.org/bot )
  357. Mozilla/5.0 (compatible; spbot/4.3.0; +http://OpenLinkProfiler.org/bot )
  358. Mozilla/5.0 (compatible; spbot/4.3.0; +http://OpenLinkProfiler.org/bot )
  359. Mozilla/5.0 (compatible; spbot/4.4.0; +http://OpenLinkProfiler.org/bot )
  360. Mozilla/5.0 (compatible; spbot/4.4.0; +http://OpenLinkProfiler.org/bot )
  361. Mozilla/5.0 (compatible; spbot/4.4.2; +http://OpenLinkProfiler.org/bot )
  362. Mozilla/5.0 (compatible; spbot/4.4.2; +http://OpenLinkProfiler.org/bot )
  363. Mozilla/5.0 (compatible; spbot/5.0.1; +http://OpenLinkProfiler.org/bot )
  364. Mozilla/5.0 (compatible; spbot/5.0.1; +http://OpenLinkProfiler.org/bot )
  365. Mozilla/5.0 (compatible; spbot/5.0; +http://OpenLinkProfiler.org/bot )
  366. Mozilla/5.0 (compatible; spbot/5.0; +http://OpenLinkProfiler.org/bot )
  367. Mozilla/5.0 (compatible; special_archiver/3.1.1 +http://www.archive.org/details/archive.org_bot)
  368. Mozilla/5.0 (compatible; sukibot_heritrix/3.1.1 +http://suki.ling.helsinki.fi/eng/webmasters.html)
  369. Mozilla/5.0 (compatible; uMBot-FC/1.0; mailto: crawling@ubermetrics-technologies.com)
  370. Mozilla/5.0 (compatible; uMBot-LN/1.0; mailto: crawling@ubermetrics-technologies.com)
  371. Mozilla/5.0 (compatible; yoozBot-2.2; http://yooz.ir; info@yooz.ir)
  372. Mozilla/5.0 (compatible; zitebot support [at] zite [dot] com +http://zite.com)
  373. Mozilla/5.0 (compatible;acapbot/0.1)
  374. Mozilla/5.0 (compatible;acapbot/0.1.;treat like Googlebot)
  375. Mozilla/5.0 (compatible;acapbot/0.1.;treat like Googlebot)
  376. Mozilla/5.0 (compatible;acapbot/0.1;treat like Googlebot)
  377. Mozilla/5.0 (compatible;acapbot/0.1;treat like Googlebot)
  378. Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
  379. Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
  380. Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  381. Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  382. Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  383. Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  384. Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview Analytics) Chrome/27.0.1453 Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  385. Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview Analytics) Chrome/27.0.1453 Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  386. Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com
  387. Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36
  388. Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36
  389. Mozilla/5.0 CommonCrawler Node DQ42QAR43VA2BC4KN7EBC63YS5HJEKKKYFAGWB7QCGRXLI5NWYLAMM3EFNB2XC2.OFBOT
  390. Mozilla/5.0 CommonCrawler Node DU57F65256BFBY3T77KNFXMBOTUD5AOGUQZSCFXO26P3S63WJ72YLWX3CIAQPHO.IX3YM
  391. Mozilla/5.0 CommonCrawler Node HZIGADHAW6QJCJAXTX7MSCXEY5IPGMT4SHUEK57TADOBP5WWPEXBFIG2NWBOT6F.7OQLF
  392. Mozilla/5.0 CommonCrawler Node IRIGGBOTAYCC6D75KXTF7SRLKGDSNECLF7PQTX3NRYGWYF5WMA3ALMSMDVDLZGT.C4YVJ
  393. Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)
  394. Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)
  395. Mozilla/5.0+(compatible;+Coallbot/2.1;++http://www.coall.com/bot.html)
  396. Mozilla/5.0+(compatible;+Coallbot/2.1;++http://www.coall.com/bot.html)
  397. Mozilla/5.0+(compatible;+Robot/1.2)
  398. Mozilla/5.0+(compatible;+searchbot/2.1;++search-engine-crawler)
  399. Mozilla/5.1 (compatible; CBot/1.0; +http://downloaflare.biz)
  400. Mozilla/5.1 (compatible; CBot/1.0; +http://overx50.com)
  401. Mozilla/5.1 (compatible; DBot/7.0; +http://getdownload.ws)
  402. Mozilla/5.1 (compatible; DBot/7.0; +http://inload.in)
  403. Mozilla/xintellibot
  404. NerdyBot
  405. NetzCheckBot/1.0 (security seal - page; http://chihuahua-people.com.netzcheck.com; bot@netzcheck.com
  406. NetzCheckBot/1.0 (security seal - page; http://chihuahua-people.com.netzcheck.com; bot@netzcheck.com
  407. NextGenSearchBot 1 (for information visit http://www.zoominfo.com/About/misc/NextGenSearchBot.aspx)
  408. NextGenSearchBot 1 (for information visit http://www.zoominfo.com/About/misc/NextGenSearchBot.aspx)
  409. Not a Java Bot
  410. Open Web Analytics Bot 1.5.4
  411. Open Web Analytics Bot 1.5.5
  412. PagesInventory (robot +http://www.pagesinventory.com)
  413. Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)
  414. PostDesk-ReadBot/0.1 +http://postdesk.com
  415. PostReleaseBOT (http://www.postrelease.com)
  416. QuerySeekerSpider ( http://queryseeker.com/bot.html )
  417. RSSingBot (http://www.rssing.com)
  418. RivalSeek.com-Bot
  419. SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
  420. SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
  421. SMTBot (similartech.com/smtbot)
  422. SMTBot (similartech.com/smtbot)
  423. SafeDNS search bot/Nutch-1.9 (https://www.safedns.com/searchbot; support [at] safedns [dot] com)
  424. SafeDNS search bot/Nutch-1.9 (https://www.safedns.com/searchbot; support [at] safedns [dot] com)
  425. SafeDNSBot (https://www.safedns.com/searchbot)
  426. SafeDNSBot (https://www.safedns.com/searchbot)
  427. ScooperBot www.customscoop.com
  428. ScreenerBot Crawler Beta 2.0 (+http://www.ScreenerBot.com)
  429. ScreenerBot Crawler Beta 2.0 (+http://www.ScreenerBot.com)
  430. SemrushBot
  431. SentiBot www.sentibot.eu (compatible with Googlebot)
  432. SentiBot www.sentibot.eu (compatible with Googlebot)
  433. SentiBot www.sentibot.eu (compatible with Googlebot)
  434. Seoterritory.com bot
  435. SeznamBot/3.0 (+http://fulltext.sblog.cz/)
  436. ShowyouBot (http://showyou.com/crawler)
  437. Slack-ImgProxy 1.128 (+https://api.slack.com/robots)
  438. Slack-ImgProxy 1.133 (+https://api.slack.com/robots)
  439. Slackbot 1.0 (+https://api.slack.com/robots)
  440. Slackbot 1.0 (+https://api.slack.com/robots)
  441. Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)
  442. Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)
  443. SpiderBot/1.0
  444. Spiderbot/Nutch-1.7
  445. Superfeedr bot/2.0 http://superfeedr.com - Checking HTML reverse link.
  446. TelegramBot (like TwitterBot)
  447. TelegramBot (like TwitterBot)
  448. TinEye-bot/0.02 (see http://www.tineye.com/crawler.html)
  449. TinEye-bot/0.51 (see http://www.tineye.com/crawler.html)
  450. TjoosBot/1.0 (http://www.tjoos.com/Page-Content/34/TjoosBot)
  451. TjoosBot/1.0 (http://www.tjoos.com/Page-Content/34/TjoosBot)
  452. Treato-Bot/1.0 (TREATO LTD, www.treato.com, mailto: botadmin@treato.com)
  453. Treato-Bot/1.0 (TREATO LTD, www.treato.com, mailto: botadmin@treato.com)
  454. TurnitinBot (https://turnitin.com/robot/crawlerinfo.html)
  455. TurnitinBot (https://turnitin.com/robot/crawlerinfo.html)
  456. TurnitinBot/3.0 (http://www.turnitin.com/robot/crawlerinfo.html)
  457. TurnitinBot/3.0 (http://www.turnitin.com/robot/crawlerinfo.html)
  458. Twitterbot/1.0
  459. UXCrawlerBot
  460. VeBot (+http://www.veinteractive.com/vebot)
  461. VeBot (+http://www.veinteractive.com/vebot)
  462. VegeBot (we follow your robots.txt settings before crawling, you can slow down the bot by change the
  463. VegeBot (we follow your robots.txt settings before crawling, you can slow down the bot by change the
  464. VegeBot (we follow your robots.txt settings before crawling, you can slow down the bot by change the
  465. WeSEE:Ads/PageBot (http://www.wesee.com/bot/)
  466. WeSEE:Ads/PageBot (http://www.wesee.com/bot/)
  467. WeSEE:Ads/PictureBot (http://www.wesee.com/bot/)
  468. WeSEE:Ads/PictureBot (http://www.wesee.com/bot/)
  469. WeSEE:Search/0.1 (Alpha, http://www.wesee.com/bot/)
  470. WeSEE_Bot:we_help_monitize_your_site (http://www.wesee.com/bot/)
  471. WeSEE_Bot:we_help_monitize_your_site (http://www.wesee.com/bot/)
  472. Who.is Bot
  473. WordChampBot
  474. WorldBrewBot/2.1 (+http://www.marketbrew.com/)
  475. Wotbox/2.01 (+http://www.wotbox.com/bot/)
  476. XoviBot/2.0
  477. Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)
  478. Yeti/1.1 (Naver Corp.; http://help.naver.com/robots/)
  479. ZumBot/1.0 (ZUM Search; http://help.zum.com/inquiry)
  480. [('User-agent', 'Mozilla/5.0 (compatible; ZumBot/1.0; http://help.zum.com/inquiry)')]
  481. abot v1.0 http://code.google.com/p/abot
  482. abot v1.0 http://code.google.com/p/abot
  483. acapbot
  484. adbeat_bot
  485. betaBot
  486. bhc.collectionBot
  487. bhcBot
  488. bitlybot
  489. bitlybot/3.0 (+http://bit.ly/)
  490. calculon spider/0.1.0 mailto:abuse@yieldbot.com
  491. datagnionbot (+http://www.datagnion.com/bot.html)
  492. datagnionbot (+http://www.datagnion.com/bot.html)
  493. focusbot/1.1
  494. hivaBot/hivaBot-1.0 (Iranian Search Engine bot; http://gorgor.ir; info@gorgor.ir)
  495. hivaBot/hivaBot-1.0 (Iranian Search Engine bot; http://gorgor.ir; info@gorgor.ir)
  496. hivaBot/hivaBot-1.0 (Iranian Search Engine bot; http://gorgor.ir; info@gorgor.ir)
  497. istellabot-nutch/Nutch-1.10
  498. istellabot/Nutch-1.10
  499. istellabot/Nutch-1.11
  500. istellabot/t.1
  501. jaribot
  502. linguatools-bot/Nutch-1.6 (searching for translated pages; http://www.linguatools.de/linguatoolsbot.
  503. linguatools-bot/Nutch-1.6 (searching for translated pages; http://www.linguatools.de/linguatoolsbot.
  504. linkapediabot (+http://www.linkapedia.com)
  505. mediawords bot (http://cyber.law.harvard.edu)
  506. mfibot/1.1 (http://www.mfisoft.ru/analyst/; <admin@mfisoft.ru>; en-RU)
  507. mozilla/4.0 (compatible; msie 8.0; windows nt 5.1; trident/4.0) mj12bot
  508. mozilla/4.0 (compatible; msie 8.0; windows nt 5.1; trident/4.0) nextgensearchbot
  509. mozilla/4.0 (compatible; msie 8.0; windows nt 5.1; trident/4.0) psbot
  510. mozilla/4.0 (compatible; msie 8.0; windows nt 5.1; trident/4.0) zealbot
  511. mozilla/5.0 (compatible; msie 9.0; windows nt 6.0; trident/5.0) abot
  512. mozilla/5.0 (compatible; msie 9.0; windows nt 6.0; trident/5.0) linkbot
  513. mozilla/5.0 (compatible; msie 9.0; windows nt 6.0; trident/5.0) webbot
  514. mozilla/5.0 (compatible; msie 9.0; windows nt 6.0; wow64; trident/5.0) NaverBot
  515. mozilla/5.0 (compatible; msie 9.0; windows nt 6.0; wow64; trident/5.0) linkbot
  516. mozilla/5.0 (compatible; msie 9.0; windows nt 6.0; wow64; trident/5.0) terrawizbot
  517. mozilla/5.0 (compatible; msie 9.0; windows nt 6.0; wow64; trident/5.0) webbot
  518. mozilla/5.0 (compatible; msie 9.0; windows nt 6.1; trident/5.0) nbot
  519. mozilla/5.0 (compatible; msie 9.0; windows nt 6.1; trident/5.0) npbot
  520. mozilla/5.0 (compatible; msie 9.0; windows nt 6.1; win64; x64; trident/5.0) NaverBot
  521. mozilla/5.0 (compatible; msie 9.0; windows nt 6.1; win64; x64; trident/5.0) nextgensearchbot
  522. mozilla/5.0 (compatible; msie 9.0; windows nt 6.1; win64; x64; trident/5.0) psbot
  523. mozilla/5.0 (compatible; msie 9.0; windows nt 6.1; wow64; trident/5.0) abot
  524. mozilla/5.0 (compatible; msie 9.0; windows nt 6.1; wow64; trident/5.0) mj12bot
  525. mozilla/5.0 (compatible; msie 9.0; windows nt 6.1; wow64; trident/5.0) nbot
  526. mozilla/5.0 (compatible; msie 9.0; windows nt 6.1; wow64; trident/5.0) nextgensearchbot
  527. mozilla/5.0 (compatible; msie 9.0; windows nt 6.1; wow64; trident/5.0) terrawizbot
  528. mozilla/5.0 (compatible; msie 9.0; windows nt 6.1; wow64; trident/5.0) webbot
  529. mozilla/5.0 (compatible; msie 9.0; windows nt 6.1; wow64; trident/5.0; funwebproducts) linkbot
  530. mozilla/5.0 (compatible; msie 9.0; windows nt 6.1; wow64; trident/5.0; funwebproducts) mj12bot
  531. mozilla/5.0 (compatible; msie 9.0; windows nt 6.1; wow64; trident/5.0; funwebproducts) npbot
  532. mozilla/5.0 (compatible; msie 9.0; windows nt 6.1; wow64; trident/5.0; funwebproducts) terrawizbot
  533. mozilla/5.0 (compatible; msie 9.0; windows nt 6.1; wow64; trident/5.0; np06) abot
  534. mozilla/5.0 (compatible; msie 9.0; windows nt 6.1; wow64; trident/5.0; np06) nbot
  535. mozilla/5.0 (compatible; msie 9.0; windows nt 6.1; wow64; trident/5.0; np06) terrawizbot
  536. mozilla/5.0 (compatible; msie 9.0; windows nt 6.1; wow64; trident/5.0; ytb730) NaverBot
  537. mozilla/5.0 (compatible; msie 9.0; windows nt 6.1; wow64; trident/5.0; ytb730) abot
  538. mozilla/5.0 (compatible; msie 9.0; windows nt 6.1; wow64; trident/5.0; ytb730) mj12bot
  539. mozilla/5.0 (compatible; msie 9.0; windows nt 6.1; wow64; trident/5.0; ytb730) psbot
  540. mozilla/5.0 (compatible; msie 9.0; windows nt 6.1; wow64; trident/5.0; ytb730) webbot
  541. mozilla/5.0 (macintosh; intel mac os x 10.6; rv:12.0) gecko/20100101 firefox/12.0 linkbot
  542. mozilla/5.0 (macintosh; intel mac os x 10.6; rv:12.0) gecko/20100101 firefox/12.0 nbot
  543. mozilla/5.0 (macintosh; intel mac os x 10.6; rv:12.0) gecko/20100101 firefox/12.0 nextgensearchbot
  544. mozilla/5.0 (macintosh; intel mac os x 10.6; rv:12.0) gecko/20100101 firefox/12.0 webbot
  545. mozilla/5.0 (macintosh; intel mac os x 10.7; rv:12.0) gecko/20100101 firefox/12.0 linkbot
  546. mozilla/5.0 (macintosh; intel mac os x 10.7; rv:12.0) gecko/20100101 firefox/12.0 mj12bot
  547. mozilla/5.0 (macintosh; intel mac os x 10.7; rv:12.0) gecko/20100101 firefox/12.0 nextgensearchbot
  548. mozilla/5.0 (macintosh; intel mac os x 10.7; rv:12.0) gecko/20100101 firefox/12.0 terrawizbot
  549. mozilla/5.0 (windows nt 5.1; rv:11.0) gecko/20100101 firefox/11.0 nbot
  550. mozilla/5.0 (windows nt 5.1; rv:11.0) gecko/20100101 firefox/11.0 nextgensearchbot
  551. mozilla/5.0 (windows nt 5.1; rv:11.0) gecko/20100101 firefox/11.0 psbot
  552. mozilla/5.0 (windows nt 5.1; rv:11.0) gecko/20100101 firefox/11.0 webbot
  553. mozilla/5.0 (windows nt 5.1; rv:11.0) gecko/20100101 firefox/11.0 zealbot
  554. mozilla/5.0 (windows nt 5.1; rv:12.0) gecko/20100101 firefox/12.0 abot
  555. mozilla/5.0 (windows nt 5.1; rv:12.0) gecko/20100101 firefox/12.0 linkbot
  556. mozilla/5.0 (windows nt 5.1; rv:12.0) gecko/20100101 firefox/12.0 npbot
  557. mozilla/5.0 (windows nt 5.1; rv:12.0) gecko/20100101 firefox/12.0 terrawizbot
  558. mozilla/5.0 (windows nt 6.0; rv:12.0) gecko/20100101 firefox/12.0 NaverBot
  559. mozilla/5.0 (windows nt 6.0; rv:12.0) gecko/20100101 firefox/12.0 nbot
  560. mozilla/5.0 (windows nt 6.0; rv:12.0) gecko/20100101 firefox/12.0 nextgensearchbot
  561. mozilla/5.0 (windows nt 6.0; rv:12.0) gecko/20100101 firefox/12.0 npbot
  562. mozilla/5.0 (windows nt 6.0; rv:12.0) gecko/20100101 firefox/12.0 psbot
  563. mozilla/5.0 (windows nt 6.0; wow64; rv:12.0) gecko/20100101 firefox/12.0 abot
  564. mozilla/5.0 (windows nt 6.0; wow64; rv:12.0) gecko/20100101 firefox/12.0 linkbot
  565. mozilla/5.0 (windows nt 6.0; wow64; rv:12.0) gecko/20100101 firefox/12.0 mj12bot
  566. mozilla/5.0 (windows nt 6.0; wow64; rv:12.0) gecko/20100101 firefox/12.0 npbot
  567. mozilla/5.0 (windows nt 6.0; wow64; rv:12.0) gecko/20100101 firefox/12.0 psbot
  568. mozilla/5.0 (windows nt 6.1; rv:11.0) gecko/20100101 firefox/11.0 NaverBot
  569. mozilla/5.0 (windows nt 6.1; rv:11.0) gecko/20100101 firefox/11.0 linkbot
  570. mozilla/5.0 (windows nt 6.1; rv:11.0) gecko/20100101 firefox/11.0 nbot
  571. mozilla/5.0 (windows nt 6.1; rv:11.0) gecko/20100101 firefox/11.0 nextgensearchbot
  572. mozilla/5.0 (windows nt 6.1; rv:11.0) gecko/20100101 firefox/11.0 terrawizbot
  573. mozilla/5.0 (windows nt 6.1; rv:12.0) gecko/20100101 firefox/12.0 mj12bot
  574. mozilla/5.0 (windows nt 6.1; rv:12.0) gecko/20100101 firefox/12.0 nbot
  575. mozilla/5.0 (windows nt 6.1; rv:12.0) gecko/20100101 firefox/12.0 nextgensearchbot
  576. mozilla/5.0 (windows nt 6.1; rv:12.0) gecko/20100101 firefox/12.0 psbot
  577. mozilla/5.0 (windows nt 6.1; rv:12.0) gecko/20100101 firefox/12.0 webbot
  578. mozilla/5.0 (windows nt 6.1; wow64; rv:10.0.2) gecko/20100101 firefox/10.0.2 NaverBot
  579. mozilla/5.0 (windows nt 6.1; wow64; rv:10.0.2) gecko/20100101 firefox/10.0.2 abot
  580. mozilla/5.0 (windows nt 6.1; wow64; rv:10.0.2) gecko/20100101 firefox/10.0.2 nbot
  581. mozilla/5.0 (windows nt 6.1; wow64; rv:10.0.2) gecko/20100101 firefox/10.0.2 nextgensearchbot
  582. mozilla/5.0 (windows nt 6.1; wow64; rv:10.0.2) gecko/20100101 firefox/10.0.2 zealbot
  583. mozilla/5.0 (windows nt 6.1; wow64; rv:11.0) gecko/20100101 firefox/11.0 abot
  584. mozilla/5.0 (windows nt 6.1; wow64; rv:11.0) gecko/20100101 firefox/11.0 nextgensearchbot
  585. mozilla/5.0 (windows nt 6.1; wow64; rv:11.0) gecko/20100101 firefox/11.0 psbot
  586. mozilla/5.0 (windows nt 6.1; wow64; rv:11.0) gecko/20100101 firefox/11.0 terrawizbot
  587. mozilla/5.0 (windows nt 6.1; wow64; rv:11.0) gecko/20100101 firefox/11.0 webbot
  588. mozilla/5.0 (windows nt 6.1; wow64; rv:12.0) gecko/20100101 firefox/12.0 nextgensearchbot
  589. mozilla/5.0 (windows nt 6.1; wow64; rv:12.0) gecko/20100101 firefox/12.0 terrawizbot
  590. mozilla/5.0 (windows nt 6.1; wow64; rv:12.0) gecko/20100101 firefox/12.0 zealbot
  591. mozilla/5.0 (windows nt 6.1; wow64; rv:6.0) gecko/20100101 firefox/6.0 linkbot
  592. mozilla/5.0 (windows nt 6.1; wow64; rv:6.0) gecko/20100101 firefox/6.0 mj12bot
  593. mozilla/5.0 (windows nt 6.1; wow64; rv:6.0) gecko/20100101 firefox/6.0 nbot
  594. mozilla/5.0 (windows nt 6.1; wow64; rv:6.0) gecko/20100101 firefox/6.0 nextgensearchbot
  595. mozilla/5.0 (windows nt 6.1; wow64; rv:6.0) gecko/20100101 firefox/6.0 zealbot
  596. mozilla/5.0 (windows nt 6.1; wow64; rv:9.0.1) gecko/20100101 firefox/9.0.1 abot
  597. mozilla/5.0 (windows nt 6.1; wow64; rv:9.0.1) gecko/20100101 firefox/9.0.1 mj12bot
  598. mozilla/5.0 (windows nt 6.1; wow64; rv:9.0.1) gecko/20100101 firefox/9.0.1 npbot
  599. mozilla/5.0 (windows nt 6.1; wow64; rv:9.0.1) gecko/20100101 firefox/9.0.1 psbot
  600. mozilla/5.0 (windows nt 6.1; wow64; rv:9.0.1) gecko/20100101 firefox/9.0.1 webbot
  601. msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)
  602. msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)
  603. msnbot-media/2.0b (+http://search.msn.com/msnbot.htm)
  604. msnbot-media/2.0b (+http://search.msn.com/msnbot.htm)
  605. msnbot/2.0b (+http://search.msn.com/msnbot.htm)
  606. msnbot/2.0b (+http://search.msn.com/msnbot.htm)
  607. niki-bot
  608. omgilibot/0.4 +http://omgili.com
  609. page-crawler bot 1.0 - http://varocarbas.com/page-crawler_bot.php
  610. page-crawler bot 1.0 - http://varocarbas.com/page-crawler_bot.php
  611. psbot-image (+http://www.picsearch.com/bot.html)
  612. psbot-image (+http://www.picsearch.com/bot.html)
  613. psbot-page (+http://www.picsearch.com/bot.html)
  614. psbot-page (+http://www.picsearch.com/bot.html)
  615. psbot/0.1 (+http://www.picsearch.com/bot.html)
  616. psbot/0.1 (+http://www.picsearch.com/bot.html)
  617. robots
  618. rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+shiny@moz.com)
  619. rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+shiny@moz.com)
  620. rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+shiny@moz.com)
  621. rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-wherecat@moz.com)
  622. rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-wherecat@moz.com)
  623. rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-wherecat@moz.com)
  624. rogerbot/1.0 (http://www.moz.com/dp/rogerbot, rogerbot-crawler@moz.com)
  625. rogerbot/1.0 (http://www.moz.com/dp/rogerbot, rogerbot-crawler@moz.com)
  626. rogerbot/1.0 (http://www.moz.com/dp/rogerbot, rogerbot-crawler@moz.com)
  627. semanticbot
  628. ssearch_bot (sSearch Crawler; http://www.semantissimo.de)
  629. studylib download bot, see http://studylib.docsio.net/public/bot for additional info
  630. studylib download bot, see http://studylib.docsio.net/public/bot for additional info
  631. studylib download bot, see http://studylib.net/public/bot for additional info
  632. studylib download bot, see http://studylib.net/public/bot for additional info
  633. supremesearch.net Spider-Bot/4.0
  634. tbot-nutch/Nutch-1.10
  635. tigerbot
  636. uipbot/1.0 (uipbot@semasio.net)
  637. uipbot/1.0 (uipbot@semasio.net)
  638. wonderbot/JS 1.0
  639. woobot/1.1
  640. woobot/2.0
  641. yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
  642. yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
  643. yacybot (/global; amd64 Linux 4.3.0-1-amd64; java 1.7.0_91; Europe/fi) http://yacy.net/bot.html
  644. yacybot (/global; amd64 Linux 4.3.0-1-amd64; java 1.7.0_91; Europe/fi) http://yacy.net/bot.html
  645. yacybot (/global; amd64 Linux 4.4.0-1-amd64; java 1.7.0_91; America/en) http://yacy.net/bot.html
  646. yacybot (/global; amd64 Linux 4.4.0-1-amd64; java 1.7.0_91; America/en) http://yacy.net/bot.html
  647. yacybot (/global; amd64 Linux 4.4.6-gentoo; java 1.8.0_91; Europe/de) http://yacy.net/bot.html
  648. yacybot (/global; amd64 Linux 4.4.6-gentoo; java 1.8.0_91; Europe/de) http://yacy.net/bot.html
  649. yacybot (/global; amd64 Linux 4.5.0ct; java 1.7.0_95; Europe/de) http://yacy.net/bot.html
  650. yacybot (/global; amd64 Linux 4.5.0ct; java 1.7.0_95; Europe/de) http://yacy.net/bot.html
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!