API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Sep 21st, 2019
379
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 29.31 KB | None | 0 0
raw download clone embed print report
  1. bot
  2. @empty
  3. googlebot
  4. google
  5. google
  6. Google
  7. bot
  8. Bot
  9. ads
  10. Ads
  11. googlebot
  12. GoogleBot
  13. Mediapartners
  14. AdsBot
  15. adwords
  16. Adwords
  17. News
  18. news
  19. video
  20. Video
  21. image
  22. Image
  23. Mediapartners-Google
  24. AdsBot-Google-Mobile
  25. APIs-Google
  26. AdsBot-Google
  27. Googlebot-Image
  28. Googlebot-News
  29. Googlebot-Video
  30. AdsBot-Google-Mobile-Apps
  31. startmebot
  32. StartMeBot
  33. Page
  34. page
  35. Speed
  36. speed
  37. insights
  38. Insights
  39. preview
  40. Preview
  41. analytics
  42. Analytics
  43. acapbot
  44. import
  45. Import
  46. Crawl
  47. crawl
  48. crawler
  49. Crawler
  50. OnPage
  51. onpage
  52. OnPageBot
  53. onpagebot
  54. deepcrawl
  55. DeepCrawl
  56. Bing
  57. bing
  58. yahoo
  59. Yahoo
  60. bingbot
  61. BingBot
  62. yahoobot
  63. YahooBot
  64. webhistory
  65. WebHistoryBot
  66. webhistorybot
  67. bot.html
  68. http://www.google.com/bot.html
  69. 66.249.64.13 - crawl-66-249-64-13.googlebot.com
  70. 66.249.64.13
  71. crawl-66-249-64-13.googlebot.com
  72. googlebot.com
  73. google.com
  74. https://whatis.contentkingapp.com
  75. contentkingapp
  76. ContentKingApp
  77. whatis
  78. WhatIs
  79. facebook
  80. Facebook
  81. facebook.com
  82. Googlebot/2.1
  83. Googlebot/3.0
  84. Googlebot/3
  85. Googlebot/2
  86. Googlebot/wannabe
  87. wannabe
  88. WannaBe
  89. https://xfrocks.com/resources/30
  90. crawl-66-249-66-1.googlebot.com
  91. crawl-66-249-66-1
  92. GoogleBot2/1
  93. Googlebot/20.1
  94. Googlebot/4.0
  95. Gridspider/12.6
  96. Gridspider
  97. https://deepcrawl.com/bot
  98. Googlebot/2.X
  99. googlebot 2,12
  100. BotThief
  101. googlebot.com
  102. Google's spider
  103. infoseek
  104. InfoSeek
  105. bottheif
  106. 66.249.64.14 - crawl-66-249-64-14.googlebot.com
  107. 66.249.64.14
  108. crawl-66-249-64-14.googlebot.com
  109. Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  110. Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  111. Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  112. Mozilla/5.0 (compatible; Googlebot/2.1; startmebot/1.0; +https://start.me/bot)
  113. Googlebot/2.1 (+http://www.googlebot.com/bot.html)
  114. Googlebot/2.1 (+http://www.google.com/bot.html)
  115. Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Chrome/27.0.1453 Safari/537.36 GoogleBot/2.1
  116. Mozilla/5.0 (iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Version/6.0 Mobile/10A525 Safari/8536.25 GoogleBot/2.1
  117. Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)
  118. Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview Analytics) Chrome/27.0.1453 Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  119. Mozilla/5.0 (compatible;acapbot/0.1;treat like Googlebot)
  120. Mozilla/5.0 (compatible; Googlebot/2.1; +http://import.io)
  121. Googlebot-Image/1.0
  122. Googlebot
  123. Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.4 (KHTML, like Gecko; Google Page Speed Insights) Chrome/22.0.1229 Safari/537.4 GoogleBot/2.1
  124. Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview Analytics) Chrome/41.0.2272.118 Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  125. Google Crawler: Googlebot/2.1 (+http://www.google.com/bot.html)
  126. OnPageBot (compatible; Googlebot 2.1; +https://bot.onpage.org/)
  127. Mozilla/5.0 (compatible; Googlebot/2.1 +http://www.googlebot.com/bot.html)
  128. Googlebot/2.1; +http://www.google.com/bot.html)
  129. Mozilla/5.0 (compatible;acapbot/0.1.;treat like Googlebot)
  130. Mozilla/5.0 (iPhone; U; CPU iPhone 0S 3.0 like Mac 0S X; en-us; compatible; Googlebot/2.1; http://www.google.com/bot.html; AppleWebKit/528.18(KHTML,like Gecko) Version/4.0 Mobile/7A341 Safari/528.16 UNTRUSTED/1.0
  131. Apple-iPhone4C1/1001.523 (compatible;acapbot/0.1;treat like Googlebot)
  132. Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)/Nutch-2.3.1
  133. Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html
  134. Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36
  135. Googlebot-News
  136. Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  137. GoogleBot/3.0
  138. Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  139. Mozilla/5.0 (compatible; Googlebot/2.1; https://deepcrawl.com/bot)
  140. Mozilla/5.0 (compatible; Googlebot/2.1; https://www.deepcrawl.com/bot)
  141. Googlebot-Video/1.0
  142. Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) UNTRUSTED/1.0
  143. Mozilla/5.0 (compatible; WebHistoryBot/1.2.1 IS NOT SE bot like Googlebot/2.1; +http://www.google.com/bot.html,Yahoo! Slurp or Bingbot)
  144. Googlebot/2.1 (+http://www.googlebot.com/bot.html) <iframe src="whatismyip.com"></iframe>
  145. Googlebot/2.1
  146. Googlebot 2.1
  147. Googlebot/2.1; http://www.google.com/bot.html
  148. Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  149. Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) AlexaToolbar/alxf-2.17
  150. Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) FirePHP/0.7.4
  151. NokiaC1-01/2.0 (06.15) Profile/MIDP-2.1 Configuration/CLDC-1.1 Mozilla/5.0 (iPhone; U; CPU iPhone 0S 3.0 like Mac 0S X; en-us; compatible; Googlebot/2.1; http://www.google.com/bot.html; AppleWebKit/528.18(KHTML,like Gecko) Version/4.0 Mobile/7A341 Safari/528.16 UNTRUSTED/1.0
  152. Mozilla/5.0 (iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/537.4 (KHTML, like Gecko; Google Page Speed Insights) Version/6.0 Mobile/10A525 Safari/8536.25 GoogleBot/2.1
  153. Googlebot (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  154. Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  155. Googlebot/2.1 (+http://www.googlebot.com/bot.html) (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko
  156. BotThief (compatible; Googlebot/2.1; Location RU)
  157. Mozilla/5.0+(compatible;+googlebot/2.1;++http://www.google.com/bot.html)
  158. Mozilla/5.0 (compatible; Googlebot/2.1; actually this is http://ismyblogworking.com/)
  159. Mozilla/5.0 (+http://www.googlebot.com/bot.html; Windows NT 6.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2486.0 Safari/537.36 Edge/13.9200
  160. http://www.googlebot.com/bot.html
  161. * Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  162. Mozilla/5.0 (compatible; Googlebot/2.1; http://deepcrawl.co.uk/bot.html)
  163. Googlebot/2.1+(+http://www.googlebot.com/bot.html)
  164. Googlebot/2.1 (+http://www.googlebot.com/bot.html
  165. Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  166. Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html )
  167. Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +https://www.deepcrawl.com/bot)
  168. Mozilla/5.0 (iPhone; U; CPU iPhone 0S 6.0 like Mac 0S X; en-us; compatible; Googlebot/2.1; http://www.google.com/bot.html; AppleWebKit/528.18(KHTML,like Gecko) Version/4.0 Mobile/7A341 Safari/528.16 UNTRUSTED/1.0
  169. Aensland Mobile Browser /5.0 (iPhone; CPU iPhone OS 7_1 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  170. 66.249.64.13 - crawl-66-249-64-13.googlebot.com
  171. Mozilla/5.0 (compatible; Googlebot/2.1; +https://whatis.contentkingapp.com/)
  172. Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)
  173. Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36.
  174. Mozilla/5.0 (compatible; Googlebot/3.2; +http://www.facebook.com)
  175. Googlebot/2.1 Compatible: http://www.googlebot.com/bot.html
  176. Googlebot/2.1 ( http://www.googlebot.com/bot.html)
  177. Googlebot/2.1" "Compatible"="+http://www.googlebot.com/bot.html
  178. Mozilla/5.0 (compatible; Googlebot/2.1
  179. Googlebot/2.1 (http://www.googlebot.com/bot.html)
  180. Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  181. Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  182. Googlebot/2.1; + http://www.google.com/bot.html
  183. Mozilla/5.0 (compatible; Googlebot/wannabe; +https://xfrocks.com/resources/30/)
  184. Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  185. (compatible; Googlebot/2.1; http://www.google.com/bot.html)
  186. Mozilla/5.0 (compatible; Googlebot/2.1; +www.google.com/bot.html)
  187. Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +https://deepcrawl.com/bot)
  188. Mozilla/5.0 (compatible; Macintosh; Intel Mac OS X) AppleWebKit/534.46.0 (KHTML, like Gecko) Googlebot/2.1 Safari/7534.34
  189. Googlebot-News (2.3.4.1, ruby 1.9.3 (2014-05-14))
  190. Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  191. Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  192. Googlebot/2.1 (+http://www.google.com/bot.html
  193. Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  194. crawl-66-249-66-1.googlebot.com
  195. Googlebot-Image/1.0 (+http://www.googlebot.com/bot.html)
  196. GoogleBot2/1
  197. Googlebot/Nutch-1.7
  198. Mozilla/5.0 (compatible; Googlebot/20.1; +http://www.google.com/bot.html)
  199. Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.78 Safari/537.36 Googlebot-News
  200. Googlebot/4.0 (Blackberry; ExtremeOS 10_3_1 PPC; rv:12.6) Crocodile/20040327 Gridspider/12.6
  201. Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  202. Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) https://deepcrawl.com/bot
  203. Nokia2700c-2/2.0 (09.98) Profile/MIDP-2.1 Configuration/CLDC-1.1 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)/UCWEB8.2.0.116/69/352 UNTRUSTED/1.0
  204. Mozilla/5.0 (compatible; Googlebot/2.1;)
  205. Mozilla/4.0 (Googlebot/2.1 (+http://www.googlebot.com/bot.html); MSIE 6.0; )
  206. { "ua": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36", "browser": { "name": "Safari" }, "engine": { "version": "537.36", "name": "WebKit" }, "os": {}, "device": {}, "cpu": {} }
  207. Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  208. Mozilla/5.0 (Linux; Android 2.3.2; Nexus One; U; id-en) Build/FRF9 applewebkit/523.18 presto/2.2.1.40 Version/4.0 mobile safari/523.18 Googlebot/20.1.254.89 + http://www.google.com/bot.html UCBrowser8.4.0.159/69/352 UNTRUSTED/1.0
  209. Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; http://www.google.com/bot.html)
  210. Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.2.2.0 Safari/537.31 Googlebot/2.1 (+http://www.googlebot.com/bot.html)
  211. Mozilla/11.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  212. Mozilla/5.0 (compatible; Googlebot/6.1; +http://www.google.com/bot.html)
  213. Mozilla/5.0 (Android; U; Android; en-id; compatible; Googlebot/10.0) AppleWebKit/534.1+ (KHTML, Like Gecko) Version/6.0.0.141 Mobile Safari/534.1+ UNTRUSTED/1.0
  214. Googlebot/2.X (+http://www.googlebot.com/bot.html)
  215. Nokia6303iclassic/5.0 (10.80) Profile/MIDP-2.1 Configuration/CLDC-1.1 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) UNTRUSTED/1.0
  216. Googlebot/2.1 (+http://googlebot.com/bot.html)
  217. Mozilla/5.0 (Windows NT 6.2; WOW64; rv:32.0; compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  218. Googlebot/2.1 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET4.0C; .NET4.0E)
  219. Mozilla/5.0 (compatible; Googlebot/2.1; https://www.deepcrawl.com/bot) https://deepcrawl.com/bot
  220. googlebot 2,12
  221. Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36 Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  222. Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.5 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  223. Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.71 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  224. Googlebot/2.1+(+http://www.googlebot.com/bot.html)"
  225. NokiaX2-00/5.0 (04.25) Profile/MIDP-2.1 Configuration/CLDC-1.1 UCWEB/2.0 (Java; U; MIDP-2.0; id; Googlebot) U2/1.0.0 UCBrowser/9.4.1.377 U2/1.0.0 Mobile UNTRUSTED/1.0
  226. Mozilla/5.0 (compatible; Googlebot/1.0; +http://www.google.com/bot.html)
  227. Mozilla/5.0 (compatible; Googlebot/2.1;+http://www.google.com/bot.html)
  228. Mozilla/5.0 (compatible? Googlebot/2.1? + Http://www.google.com/bot.html)
  229. Mozilla/5.0 AppleWebKit/27.0.1453 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/27.0.1453
  230. Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.95 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) or (rarely used): Googlebot/2.1 (+http://www.google.com/bot.html)
  231. Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)aaa
  232. Mozilla/4.0 (compatible; MSIE 6.0; Googlebot/2.1; +http://www.google.com/bot.html)
  233. Googlebot/2.1; +
  234. Googlebot/2.1(+http://www.googlebot.com/bot.html)
  235. GoogleBot/2.9 (Fedora 3.4; en-US;)
  236. GoogleBot/6.6 (OS 10.5 7.0; en-US;)
  237. Mozilla/5.0 (compatiable Googlebot/2.1 +http//www.google.com/bot hrml)
  238. GoogleBot/5.5 (Windows 95 5.1; en-US;)
  239. Mozilla/5.0 (iPhone;<br /> U; CPU iPhone 0S 3.0<br /> like Mac 0S X; en-us;<br /> compatible;<br /> Googlebot/2.1;<br /> http://<br /> www.google.com/<br /> bot.html;<br /> AppleWebKit/528.18(KHTML,like<br /> Gecko) Version/4.0<br /> Mobile/7A341<br /> Safari/528.16
  240. Googlebot/2.1 (+http://www.googlebot.com/bot.html).
  241. Mozilla/5.0 (X11; U; Linux armv7l; en-GB; rv:1.9.2.3pre) Gecko/20100723 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  242. Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1667.0 Safari/537.36
  243. Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  244. Mozilla/23.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  245. Mozilla/5.0 (compatible;+Googlebot/2.1;+http://www.google.com/bot.html)
  246. Googlebot/2.1 (+http://www.googlebot.com/bot.html) <iframe></iframe>
  247. Googlebot/2.1(Windows 3.1)
  248. "Googlebot/2.1",gzip(gfe) (via translate.google.com)
  249. googlebot2.1
  250. +http://www.googlebot.com/bot.html
  251. Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)/Nutch-1.14-SNAPSHOT
  252. Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) https://deepcrawl.com/bot
  253. Mozilla/5.0 (compatible; Googlebot/2.5; +http://www.google.com/bot.html)
  254. Googlebot/2.1 (+http://www.googlebot.com/bot.html) <iframe src="http://www.whatismyip.com/"></iframe>
  255. Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  256. Googlebot/2.1 (+http:\\www.googlebot.com/bot.html)
  257. Mozilla/5.0 (Macintosh; Intel Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Safari/8536.25 Googlebot/2.1 scarlet/1.0 (development)
  258. Mozilla/5.0 (iPad; CPU OS 9_2_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Mobile/13D11 Googlebot/2.1 scarlet/1.0 (development)
  259. Googlebot/2.1 (+https://www.googlebot.com/bot.html)
  260. Mozilla/6.0 (iPhone; CPU iPhone OS 8_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/8.0 Mobile/10A5376e Safari/8536.25 Googlebot/2.1 scarlet/1.0 (development)
  261. Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.1.0.0 Safari/537.36 Googlebot/2.1 (+http://www.google.com/bot.html)
  262. Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.1.0.0 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  263. BotThief (compatible; Googlebot/2.1; Location US)
  264. Mozilla/5.0 (X11; U; Linux armv7l; en-GB; rv:25.0; Presto/2.12.388) Gecko/20111216 Firefox/27.0 Googlebot/2.1 Maemo Browser 1.7.4.8 RX-51 N900
  265. Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html; Evolute5.9; JK5)
  266. Mozilla/5.0 (X11; U; Linux armv7l; en-US; rv:1.9.2.3pre; Presto/2.12.388; Macintosh; Intel Mac OS X 10.6; rv:25.0) Gecko/20111216 Firefox/13.0.1 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Maemo Browser 1.7.4.8 RX-51 N900
  267. Mozilla/7.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  268. Mozilla/9.0 (compatible; Googlebot/3.7; +http://www.google.com/bot.html)
  269. Googlebot/2.1 ([www.googlebot.com])
  270. Googlebot (iPhone; X11; Linux x86_64; compatible; WOW64; Gecko/20130331; MSIE 10.0; Android 2.2; Windows NT 10.0) Firefox/49.0
  271. Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html; Trident/7.0; rv:11.0) like Gecko
  272. Googlebot (Google's spider)
  273. Googlebot/2.1+
  274. googlebot/2.1 (+http //www.google.com/bot.html)
  275. Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible ; Googlebot/2.1 ; +http://www.google.com/bot.html)
  276. Hidden/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  277. Mozilla/6.0 (compatible; Googlebot/3.9; +http://www.google.com/bot.html)
  278. Mozella/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  279. NokiaC1-01/2.0 (04.40) Profile/MIDP-2.1 Configuration/CLDC-1.1 Mozilla/5.0 (iPhone; U; CPU iPhone 0S 3.0 like Mac 0S X; en-us; compatible; Googlebot/2.1; http://www.tapanwap.com/index.html; AppleWebKit/528.18(KHTML,like Gecko) Version/4.0 Mobile/7A341 Safari/528.16 UNTRUSTED/1.0
  280. Mozilla/5.0 (Windows; U; Windows NT 9.5; en-us; compatible; Googlebot/2.1; http://www.google.com/bot.html; AppleWebKit/534.19(KHTML,like Gecko) Chrome/11.0.661.0 Safari/534.19 UNTRUSTED/1.0
  281. Googlebot/2.2 (+http://www.googlebot.com/bot.html)
  282. Googlebot-Image/1.0 (Prism; XKeyscore)
  283. (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  284. Googlebot/2.1 (+http://www.myteltech.blogspot.in/bot.html)
  285. Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)/1.8 (InfoSeek crawler; http://www.infoseek.com; crawler@infoseek.com)
  286. Googlebot/2.1 (+http://www.google.com/bot.html; For abuse: kilobyte2007@gmail.com)
  287. Mozilla/5.0 (compatible; Googlebot/
  288. Mozilla/5.0 (compatible; Googlebot/7.0
  289. Hacker Anonnymous (+http://www.googlebot.com/bot.html
  290. =Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  291. Mozilla/5.0 (compatible; Googlebot/2.1;
  292. GoogleBot/15.18 (Ubuntu 5.3; ar_JO;)
  293. Mozilla/5.0 (compatible; U; Googlebot/9.8.4.62; + http://www.google.com/bot; en) AppleWebKit/534.1+ (KHTML, Like Gecko) Version/6.0.0.141 Mobile Safari/534.1+ UNTRUSTED/1.0
  294. Mozilla/5.0 (compatible; Googlebot/20.0.1105.2; + http://www.google.com/bot.html) UNTRUSTED/1.0
  295. Mozilla/5.0 (Linux; Android 2.3.2; Nexus One; Googlebot/20.0.1105.2; + http://www.google.com/bot.html) UNTRUSTED/1.0
  296. Googlebot/4.0 (MSIE 4.01; ExtremeOS 10_3_1 PPC; rv:12.6) Crocodile/20040327 Gridspider/12.6
  297. Google Chrome/5.0 (Linux; Android 8.0.0 Pixel XL Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  298. Google Chrome/61 (Linux; Android 8.0.0 Pixel XL Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  299. Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) support@google.com
  300. Mozilla/5.0 (iPhone; U; CPU iPhone 0S 3.0 like Mac 0S X; en-us; compatible; Googlebot/2.1; http://www.google.com/bot.html; AppleWebKit/528.18(KHTML,like Gecko) Version/4.0 Mobile/7A341 Safari/528.16 UNTRUSTED/1.0 3gpp-gba
  301. Mozilla/5.0 (iPad; U; ipad/12; en-us; compatible; Googlebot/2.1; http://www.google.com/bot.html; AppleWebKit/528.18(KHTML,like Gecko) Version/4.0 Mobile/7A341 Safari/528.16 UNTRUSTED/1.0
  302. Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  303. 66.249.64.14 - crawl-66-249-64-14.googlebot.com
  304. Mozilla/5.0 (Windows; U; Windows NT 10.0; en-US) AppleWebKit/604.1.38 (KHTML, like Gecko) QtWebEngine/5.10.1 googlebot/604.1.38
  305. Nokia2700c-2/2.0 (09.98) Profile/MIDP-2.1 Configuration/CLDC-1.1 Mozilla/5.0 (iPhone; U; CPU iPhone 0S 3.0 like Mac 0S X; en-us; compatible; Googlebot/2.1; http://www.google.com/bot.html; AppleWebKit/528.18(KHTML,like Gecko) Version/4.0 Mobile/7A341 Safari/528.16/UCWEB8.2.0.116/69/352 UNTRUSTED/1.0
  306. Mozilla/4.0 (Googlebot/2.1; MSIE 6.0; http://www.googlebot.com/bot.html)
  307. Mozilla/4.0 (http://www.googlebot.com/bot.html; Googlebot/2.1; )
  308. Googlebot/2.1 (Windoze; N; Windows 95; ZUSE3; rv1.3.10) Trident/20100101 Icefox/11.0b4
  309. Googlebot-Image 1.0
  310. NokiaC1-01/2.0 (06.15) Profile/MIDP-2.1 Configuration/CLDC-1.1 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) UNTRUSTED/1.0
  311. Googlebot-Image 2.1
  312. Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) UNTRUSTED/1.0 3gpp-gba
  313. Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  314. Nokia7210Supernova/2.0 (07.23) Profile/MIDP-2.1 Configuration/CLDC-1.1 Googlebot/2.1#(+http://www.google.com/ bot.html)/UCWEB7.5.0.66/69/352 UNTRUSTED/1.0
  315. Googlebot/2.0
  316. Mozilla/5.0 (compatiable Googlebot/2.1 + http//www.Google.com/not html)
  317. Googlebot/2.X (http://www.googlebot.com/bot.html)
  318. ​Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  319. Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.15 Safari/537.36, : Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  320. Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) AlexaToolbar/alxf-2.18
  321. Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.104 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  322. Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) ASProxy/5.5.0
  323. Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.1 Safari/537.36 Googlebot/2.1
  324. Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36 Googlebot/2.1 (+http://www.googlebot.com/bot.html)
  325. Googlebot/2.1 (+http://www.googlebot.com/bot.html (Windows NT 6.3; Win64; x64; Trident/7.0; rv:11.0) like Gecko
  326. Mozilla/5.0 (compatible; Googlebot[b]new[/b]/2.1; +http://www.google.com/bot.html)
  327. mozilla/5.0 (compatible Googlebot/2.1 +http,//www.google.com/bot html)
  328. Googlebot 2.1(+http://www.googlebot.com/bot.html)
  329. Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html; Trident/5.0)
  330. Mozilla/5.0 (compatible; Googlebot/3333; +http://www.google.com/bot.html)
  331. Mozilla/5.0 (NokiaC5-00; compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  332. Googlebot/2.1 (+http://www.googlebot.com/bot.html) AlexaToolbar/alxf-2.19
  333. Mozilla/5.0 (Linux; U; Android) Googlebot/2.1
  334. Mozilla/5.0 (compatible; Googlebot<b>New</b>/2.1; +http://www.google.com/bot.html)
  335. Mozilla/5.0 (Android; Android; en-us; compatible; Googlebot/2.1; http://www.google.com/bot.html; AppleWebKit/528.18(KHTML,like Gecko) Version/4.0 Mobile/7A341 Safari/528.16 UNTRUSTED/1.0
  336. Mozilla/5.0 (Android; U; Android; en-id; compatible; Googlebot 10.0) AppleWebKit/534.1+ (KHTML, Like Gecko) Version/6.0.0.141 Mobile Safari/534.1+ UNTRUSTED/1.0
  337. Googlebot/Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:28.0) Gecko/20100101 Firefox/28.0
  338. Googlebot (Windows NT 6.1; Win64; x64)
  339. Mozilla/5.0 (Android; U; Googlebot/2.1; en-id) AppleWebKit/534.1+ (KHTML, Like Gecko) Version/6.0.0.141 Mobile Safari/534.1+ UNTRUSTED/1.0
  340. [Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) or (rarely used): Googlebot/2.1 (+http://www.google.com/bot.html)]
  341. Mozilla/5.0 (compatible; <B>Googlebot</B>/2.1; +http://www.google.com/bot.html)
  342. Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.9oo91e.qjz9zk/bot.html)
  343. Mozilla/5.0 (compatible; Googlebot/2.1; +http://google.com/)
  344. Mozilla/5.0 (compatible; Googlebot/2.0; +http://www.google.com/bot.html)
  345. Mozilla/5.0 (compatible; Googlebot/2.2; +http://www.google.com/bot.html)
  346. Mozilla/5.0 (compatible; Googlebot/2.1; ̣+http://www.google.com/bot.html)
  347. Mozilla/5.0 (compatible; Googlebot/2.1; +http//www.google.com/bot.html) (compatible; MSIE 6.0; Windows CE; IEMobile 7.6)
  348. Googlebot/2.10
  349. Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) or
  350. Googlebot/2.109
  351. Googlebot/2.109aer
  352. Googlebot-News (2.3.3, ruby 1.9.3 (2013-11-22))
  353. Googlebot/a
  354. Mozilla/5.0 (compatible; Googlebot/3.1; +http://www.google.com/bot.html)
  355. Mozilla/35.0.1 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  356. Mozilla/5.0 (compatible; Googlebot/0.1; +http://www.google.com/bot.html)
  357. Mozilla/5.0 GoogleBot/3.0
  358. Mozilla/5.0 (Linux; U; CPU Android 3.0 like Mac 0S X; en-us; compatible; Googlebot/2.1; http://www.google.com/bot.html; AppleWebKit/528.18(KHTML,like Gecko) Version/4.0 Mobile/7A341 Safari/528.16 UNTRUSTED/1.0
  359. Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  360. Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36 Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  361. Mozilla/5.0 (Linux; U; Android 2.2; en-us; Nexus One; googlebot/2.1; +http://www.google.com/bot; AppleWebKit/528.18(KHTML,like Gecko) Version/4.0 Mobile/7A341 Safari/528.16 presto/2.2.40 build/tipartimur UNTRUSTED/1.0
  362. Googlebot/2.1 (+hi; Android; Mobile;)
  363. HitLeap Viewer=HitLeap Viewer 2.2 FireFox 6.0=Mozilla/5.0 (Windows NT 5.1; rv:6.0.2) Gecko/20100101 Firefox/6.0.2IE8(Windows XP)=Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; CIBA; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)IE6=Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)Mac Safari 4=Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_8; en-us) AppleWebKit/531.21.8 (KHTML, like Gecko) Version/4.0.4 Safari/5Google Bot=Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)iPhone 3.0=Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Version/4.0 Mobile/7A341 Safari/528.16iPhone 4S=Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_0 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8A293 Safari/6531.22.7Nokia C5-00(S60v3)=Mozilla/5.0 (SymbianOS/9.3; Series60/3.2 NokiaC5-00/032.010; Profile/MIDP-2.1 Configuration/CLDC-1.1 ) AppleWebKit/525 (KHTML, like Gecko ) Version/3.0 BrowserNG/7.2.6.2 3gpp-gbaHTC Legend(Android)=Mozilla/5.0 (Linux; U; Android 2.2; zh-cn; HTC Legend Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1iPad=Mozilla/5.0 (iPad; U; CPU OS 3_2 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Version/4.0.4 Mobile/7B334b Safari/531.21.10
  364. GoogleBot 1.0
  365. Mozilla/5.0 (compatible; Googlebot/2.4; +http://www.google.com/bot.html)
  366. Rizostronza/1.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  367. Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)
  368. GoogleBot/1.0
  369. Googlebot/2.1 (+http://www.google.com/bot.html) Mozilla/5.0 (compatible); Googlebot/2.1; (+http://www.google.com/bot.html) Googlebot-Image/1.0
  370. Opera/18.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  371. Googlebot/2.1 (+https://www.google.com/bot.html)
  372. Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.60 Safari/537.36 Googlebot/2.1 (+http://www.googlebot.com/bot.html)
  373. Mozilla/5.0 (X11; CrOS x86_64 12239.92.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.136 Safari/537.36
  374. CrOS
  375. X11
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!