API tools faq
paste
archosaur2

www.google.com/robots.txt

archosaur2
Jan 28th, 2013
230
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Robots 7.31 KB | None | 0 0
raw download clone embed print report
  1. User-agent: *
  2. Disallow: /search
  3. Disallow: /sdch
  4. Disallow: /groups
  5. Disallow: /images
  6. Disallow: /catalogs
  7. Allow: /catalogs/about
  8. Allow: /catalogs/p?
  9. Disallow: /catalogues
  10. Disallow: /news
  11. Allow: /news/directory
  12. Disallow: /nwshp
  13. Disallow: /setnewsprefs?
  14. Disallow: /index.html?
  15. Disallow: /?
  16. Allow: /?hl=
  17. Disallow: /?hl=*&
  18. Disallow: /addurl/image?
  19. Disallow: /pagead/
  20. Disallow: /relpage/
  21. Disallow: /relcontent
  22. Disallow: /imgres
  23. Disallow: /imglanding
  24. Disallow: /sbd
  25. Disallow: /keyword/
  26. Disallow: /u/
  27. Disallow: /univ/
  28. Disallow: /cobrand
  29. Disallow: /custom
  30. Disallow: /advanced_group_search
  31. Disallow: /googlesite
  32. Disallow: /preferences
  33. Disallow: /setprefs
  34. Disallow: /swr
  35. Disallow: /url
  36. Disallow: /default
  37. Disallow: /m?
  38. Disallow: /m/?
  39. Disallow: /m/blogs?
  40. Disallow: /m/directions?
  41. Disallow: /m/ig
  42. Disallow: /m/images?
  43. Disallow: /m/imgres?
  44. Disallow: /m/local?
  45. Disallow: /m/movies?
  46. Disallow: /m/news?
  47. Disallow: /m/news/i?
  48. Disallow: /m/place?
  49. Disallow: /m/products?
  50. Disallow: /m/products/
  51. Disallow: /m/setnewsprefs?
  52. Disallow: /m/search?
  53. Disallow: /m/swmloptin?
  54. Disallow: /m/trends
  55. Disallow: /m/video?
  56. Disallow: /wml?
  57. Disallow: /wml/?
  58. Disallow: /wml/search?
  59. Disallow: /xhtml?
  60. Disallow: /xhtml/?
  61. Disallow: /xhtml/search?
  62. Disallow: /xml?
  63. Disallow: /imode?
  64. Disallow: /imode/?
  65. Disallow: /imode/search?
  66. Disallow: /jsky?
  67. Disallow: /jsky/?
  68. Disallow: /jsky/search?
  69. Disallow: /pda?
  70. Disallow: /pda/?
  71. Disallow: /pda/search?
  72. Disallow: /sprint_xhtml
  73. Disallow: /sprint_wml
  74. Disallow: /pqa
  75. Disallow: /palm
  76. Disallow: /gwt/
  77. Disallow: /purchases
  78. Disallow: /hws
  79. Disallow: /bsd?
  80. Disallow: /linux?
  81. Disallow: /mac?
  82. Disallow: /microsoft?
  83. Disallow: /unclesam?
  84. Disallow: /answers/search?q=
  85. Disallow: /local?
  86. Disallow: /local_url
  87. Disallow: /shihui?
  88. Disallow: /shihui/
  89. Disallow: /froogle?
  90. Disallow: /products?
  91. Disallow: /products/
  92. Disallow: /froogle_
  93. Disallow: /product_
  94. Disallow: /products_
  95. Disallow: /products;
  96. Disallow: /print
  97. Disallow: /books/
  98. Disallow: /bkshp?*q=*
  99. Disallow: /books?*q=*
  100. Disallow: /books?*output=*
  101. Disallow: /books?*pg=*
  102. Disallow: /books?*jtp=*
  103. Disallow: /books?*jscmd=*
  104. Disallow: /books?*buy=*
  105. Disallow: /books?*zoom=*
  106. Allow: /books?*q=related:*
  107. Allow: /books?*q=editions:*
  108. Allow: /books?*q=subject:*
  109. Allow: /books/about
  110. Allow: /booksrightsholders
  111. Allow: /books?*zoom=1*
  112. Allow: /books?*zoom=5*
  113. Disallow: /ebooks/
  114. Disallow: /ebooks?*q=*
  115. Disallow: /ebooks?*output=*
  116. Disallow: /ebooks?*pg=*
  117. Disallow: /ebooks?*jscmd=*
  118. Disallow: /ebooks?*buy=*
  119. Disallow: /ebooks?*zoom=*
  120. Allow: /ebooks?*q=related:*
  121. Allow: /ebooks?*q=editions:*
  122. Allow: /ebooks?*q=subject:*
  123. Allow: /ebooks?*zoom=1*
  124. Allow: /ebooks?*zoom=5*
  125. Disallow: /patents?
  126. Disallow: /patents/related/
  127. Allow: /patents?id=
  128. Allow: /patents?vid=
  129. Disallow: /scholar
  130. Disallow: /citations?
  131. Allow: /citations?user=
  132. Allow: /citations?view_op=new_profile
  133. Allow: /citations?view_op=top_venues
  134. Disallow: /complete
  135. Disallow: /s?
  136. Disallow: /sponsoredlinks
  137. Disallow: /videosearch?
  138. Disallow: /videopreview?
  139. Disallow: /videoprograminfo?
  140. Disallow: /maps?
  141. Disallow: /mapstt?
  142. Disallow: /mapslt?
  143. Disallow: /maps/stk/
  144. Disallow: /maps/br?
  145. Disallow: /mapabcpoi?
  146. Disallow: /maphp?
  147. Disallow: /mapprint?
  148. Disallow: /maps/api/js/StaticMapService.GetMapImage?
  149. Disallow: /maps/api/staticmap?
  150. Disallow: /mld?
  151. Disallow: /staticmap?
  152. Disallow: /places/
  153. Allow: /places/$
  154. Disallow: /maps/place
  155. Disallow: /help/maps/streetview/partners/welcome/
  156. Disallow: /help/maps/indoormaps/partners/
  157. Disallow: /lochp?
  158. Disallow: /center
  159. Disallow: /ie?
  160. Disallow: /sms/demo?
  161. Disallow: /katrina?
  162. Disallow: /blogsearch?
  163. Disallow: /blogsearch/
  164. Disallow: /blogsearch_feeds
  165. Disallow: /advanced_blog_search
  166. Disallow: /reader/
  167. Allow: /reader/play
  168. Disallow: /uds/
  169. Disallow: /chart?
  170. Disallow: /transit?
  171. Disallow: /mbd?
  172. Disallow: /extern_js/
  173. Disallow: /calendar/feeds/
  174. Disallow: /calendar/ical/
  175. Disallow: /cl2/feeds/
  176. Disallow: /cl2/ical/
  177. Disallow: /coop/directory
  178. Disallow: /coop/manage
  179. Disallow: /trends?
  180. Disallow: /trends/music?
  181. Disallow: /trends/hottrends?
  182. Disallow: /trends/viz?
  183. Disallow: /notebook/search?
  184. Disallow: /musica
  185. Disallow: /musicad
  186. Disallow: /musicas
  187. Disallow: /musicl
  188. Disallow: /musics
  189. Disallow: /musicsearch
  190. Disallow: /musicsp
  191. Disallow: /musiclp
  192. Disallow: /browsersync
  193. Disallow: /call
  194. Disallow: /archivesearch?
  195. Disallow: /archivesearch/url
  196. Disallow: /archivesearch/advanced_search
  197. Disallow: /base/reportbadoffer
  198. Disallow: /urchin_test/
  199. Disallow: /movies?
  200. Disallow: /codesearch?
  201. Disallow: /codesearch/feeds/search?
  202. Disallow: /wapsearch?
  203. Disallow: /safebrowsing
  204. Allow: /safebrowsing/diagnostic
  205. Allow: /safebrowsing/report_badware/
  206. Allow: /safebrowsing/report_error/
  207. Allow: /safebrowsing/report_phish/
  208. Disallow: /reviews/search?
  209. Disallow: /orkut/albums
  210. Allow: /jsapi
  211. Disallow: /views?
  212. Disallow: /c/
  213. Disallow: /cbk
  214. Allow: /cbk?output=tile&cb_client=maps_sv
  215. Disallow: /recharge/dashboard/car
  216. Disallow: /recharge/dashboard/static/
  217. Disallow: /translate_a/
  218. Disallow: /translate_c
  219. Disallow: /translate_f
  220. Disallow: /translate_static/
  221. Disallow: /translate_suggestion
  222. Disallow: /profiles/me
  223. Allow: /profiles
  224. Disallow: /s2/profiles/me
  225. Allow: /s2/profiles
  226. Allow: /s2/photos
  227. Allow: /s2/static
  228. Disallow: /s2
  229. Allow: /s2/search/social
  230. Disallow: /transconsole/portal/
  231. Disallow: /gcc/
  232. Disallow: /aclk
  233. Disallow: /cse?
  234. Disallow: /cse/home
  235. Disallow: /cse/panel
  236. Disallow: /cse/manage
  237. Disallow: /tbproxy/
  238. Disallow: /imesync/
  239. Disallow: /shenghuo/search?
  240. Disallow: /support/forum/search?
  241. Disallow: /reviews/polls/
  242. Disallow: /hosted/images/
  243. Disallow: /ppob/?
  244. Disallow: /ppob?
  245. Disallow: /ig/add?
  246. Disallow: /adwordsresellers
  247. Disallow: /accounts/o8
  248. Allow: /accounts/o8/id
  249. Disallow: /topicsearch?q=
  250. Disallow: /xfx7/
  251. Disallow: /squared/api
  252. Disallow: /squared/search
  253. Disallow: /squared/table
  254. Disallow: /toolkit/
  255. Allow: /toolkit/*.html
  256. Disallow: /globalmarketfinder/
  257. Allow: /globalmarketfinder/*.html
  258. Disallow: /qnasearch?
  259. Disallow: /app/updates
  260. Disallow: /sidewiki/entry/
  261. Disallow: /quality_form?
  262. Disallow: /labs/popgadget/search
  263. Disallow: /buzz/post
  264. Disallow: /compressiontest/
  265. Disallow: /analytics/reporting/
  266. Disallow: /analytics/admin/
  267. Disallow: /analytics/web/
  268. Disallow: /analytics/feeds/
  269. Disallow: /analytics/settings/
  270. Disallow: /alerts/
  271. Disallow: /ads/preferences/
  272. Allow: /ads/preferences/html/
  273. Allow: /ads/preferences/plugin
  274. Disallow: /ads/search
  275. Disallow: /settings/ads/onweb/
  276. Disallow: /phone/compare/?
  277. Allow: /alerts/manage
  278. Allow: /alerts/remove
  279. Disallow: /travel/clk
  280. Disallow: /hotelfinder/rpc
  281. Disallow: /flights/rpc
  282. Disallow: /commercesearch/services/
  283. Disallow: /evaluation/
  284. Disallow: /chrome/browser/mobile/tour
  285. Disallow: /plus/history/tasks/
  286. Disallow: /compare/*/apply*
  287. Disallow: /forms/perks/
  288. Disallow: /baraza/*/search
  289. Disallow: /baraza/*/report
  290. Disallow: /now/topics/
  291. Disallow: /shopping/suppliers/search
  292. Sitemap: http://www.google.com/culturalinstitute/sitemap.xml
  293. Sitemap: http://www.google.com/hostednews/sitemap_index.xml
  294. Sitemap: http://www.google.com/sitemaps_webmasters.xml
  295. Sitemap: http://www.google.com/ventures/sitemap_ventures.xml
  296. Sitemap: http://www.gstatic.com/dictionary/static/sitemaps/sitemap_index.xml
  297. Sitemap: http://www.gstatic.com/earth/gallery/sitemaps/sitemap.xml
  298. Sitemap: http://www.gstatic.com/s2/sitemaps/profiles-sitemap.xml
  299. Sitemap: http://www.gstatic.com/trends/websites/sitemaps/sitemapindex.xml
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!