API tools faq
paste
Advertisement
FR0ST_LXIX

robots

FR0ST_LXIX
Jun 16th, 2016
99
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 3.94 KB | None | 0 0
raw download clone embed print report
  1. User-agent: Zeus
  2. Disallow: /
  3. User-agent: BredoLab
  4. Disallow: /
  5. User-agent: Mariposa
  6. Disallow: /
  7. User-agent: Marina Botnet
  8. Disallow: /
  9. User-agent: Mega-D
  10. Disallow: /
  11. User-agent: Festi
  12. Disallow: /
  13. User-agent: Gumblar
  14. Disallow: /
  15. User-agent: Akbot
  16. Disallow: /
  17. User-agent:google*.com
  18. Disallow:/
  19. User-agent:*google.com
  20. Disallow:/
  21. User-agent: Kelihos
  22. Disallow: /
  23. User-agent: Boatnet
  24. Disallow: /
  25. User-agent: Sally
  26. Disallow: /
  27. User-agent: BobBot
  28. Disallow: /
  29. User-agent: AhrefsBot
  30. Disallow: /
  31. User-agent: dotbot
  32. Crawl-delay: 120
  33. Disallow: /
  34. User-agent: DotBot
  35. Crawl-delay: 120
  36. Disallow: /
  37. User-agent: MJ12bot
  38. Disallow: /
  39. User-agent: Yandex
  40. Crawl-delay: 120
  41. Disallow: /
  42. User-agent: https://fbstatic-a.akamaihd.net/
  43. Disallow: /
  44. User-agent: facebook.*
  45. Disallow: /
  46. User-agent: *facebook.*
  47. Disallow: /
  48. User-agent: facebook*.*
  49. Disallow: /
  50. User-agent: akamaihd
  51. Disallow: /
  52. User-agent: *bot
  53. Disallow: /
  54. User-agent: bot*
  55. Disallow: /
  56. User-agnet: *.gov
  57. Disallow: /
  58. User-agent: Police*.*
  59. Disallow: /
  60. User-agent: https://akamaihd.net
  61. Disallow: /
  62. User-agent: Googlebot
  63. Disallow: /lst/ng/
  64. User-agent: Yandex/1.01.001
  65. Crawl-delay: 120
  66. Disallow: /
  67. User-agent: Baiduspider
  68. Disallow: /
  69. User-agent: Baiduspider-video
  70. Disallow: /
  71. User-agent: Baiduspider-image
  72. Disallow: /
  73. User-agent: Exabot
  74. Disallow: /
  75. User-agent: Offline Explorer
  76. Disallow: /
  77. User-agent: VoilaBot
  78. Disallow: /
  79. User-agent: sogou spider
  80. Crawl-delay: 120
  81. Disallow: /
  82. User-agent: Sogou web spider
  83. Crawl-delay: 120
  84. Disallow: /
  85. User-agent: SputnikBot
  86. Disallow: /
  87. User-agent: FatBot
  88. Disallow: /
  89. User-agent: FatBot 2.0
  90. Disallow: /
  91. User-agent: Fetch
  92. Disallow: /
  93. User-agent: XoviBot
  94. Disallow: /
  95. User-agent: Zao
  96. Disallow: /
  97. User-agent: sistrix
  98. Disallow: /
  99. User-agent: 007ac9
  100. Disallow: /
  101. User-agent: BacklinkCrawler
  102. Disallow: /
  103. User-agent: SEOkicks-Robot
  104. Disallow: /
  105. User-agent: spbot
  106. Disallow: /
  107. User-agent: SiteSnagger
  108. Disallow: /
  109. User-agent: magpie-crawler
  110. Disallow: /
  111. User-agent: MSIECrawler
  112. Disallow: /
  113. User-agent: Teleport
  114. Disallow: /
  115. User-agent: UbiCrawler
  116. Disallow: /
  117. User-agent: Zealbot
  118. Disallow: /
  119. User-agent: w3af
  120. Disallow: /
  121. User-agent: WebStripper
  122. Disallow: /
  123. User-agent: WebCopier
  124. Disallow: /
  125. User-agent: lwp
  126. Disallow: /
  127. User-agent: *
  128. Disallow: /cgi-bin/
  129. # rest of agents
  130. User-agent: *
  131. Disallow:/cgi-bin*
  132. Disallow: /adminpanel.asp
  133. Disallow: /administratorlogin.asp
  134. Disallow: /adminLogin/
  135. Disallow: /admin-login.asp
  136. Disallow: /admin/account.asp
  137. Disallow: /cgi-bin
  138. Disallow: /wp/wp-admin/
  139. Disallow: /wp-admin
  140. Disallow: /wp-login/
  141. Disallow: /wp-includes
  142. Disallow: /wp-content/
  143. Disallow: /*/trackback/
  144. Disallow: /*/feed/
  145. Disallow: /*page
  146. Disallow: /admi
  147. Disallow: /de/DE/Widget/
  148. Disallow: /fr/FR/Widget/
  149. Disallow: /en/GB/Widget/
  150. Disallow: /nl/BE/Widget/
  151. Disallow: /fr/BE/Widget/
  152. Disallow: /dk/DK/Widget/
  153. Disallow: /es/ES/Widget/
  154. Disallow: /en/IE/Widget/
  155. Disallow: /it/IT/Widget/
  156. Disallow: /nl/NL/Widget/
  157. Disallow: /no/NO/Widget/
  158. Disallow: /pl/PL/Widget/
  159. Disallow: /fi/FI/Widget/
  160. Disallow: /se/SE/Widget/
  161. Disallow: /de/AT/Widget/
  162. Disallow: /en/EU/Widget/
  163. Disallow: /us/US/Widget/
  164. Disallow: /us/CA/Widget/
  165. Disallow: /fr/CA/Widget/
  166. Disallow: /fr/US/Widget/
  167. Disallow: /de/DE/Ajax/
  168. Disallow: /fr/FR/Ajax/
  169. Disallow: /en/GB/Ajax/
  170. Disallow: /nl/BE/Ajax/
  171. Disallow: /fr/BE/Ajax/
  172. Disallow: /dk/DK/Ajax/
  173. Disallow: /es/ES/Ajax/
  174. Disallow: /en/IE/Ajax/
  175. Disallow: /it/IT/Ajax/
  176. Disallow: /nl/NL/Ajax/
  177. Disallow: /no/NO/Ajax/
  178. Disallow: /pl/PL/Ajax/
  179. Disallow: /fi/FI/Ajax/
  180. Disallow: /se/SE/Ajax/
  181. Disallow: /de/AT/Ajax/
  182. Disallow: /en/EU/Ajax/
  183. Disallow: /us/US/Ajax/
  184. Disallow: /us/CA/Ajax/
  185. Disallow: /fr/CA/Ajax/
  186. Disallow: /fr/US/Ajax/
  187. Disallow: /us-ca/
  188. Disallow: /fr-ca/
  189. Disallow: /api/
  190. Disallow: /log/
  191. Disallow: /start-C3639
  192. Disallow: /start-C6220
  193. Disallow: /*-C6220*
  194. Disallow: /*-C3639*
  195. Disallow: /*-C4407*
  196. Disallow: /*-C3375*
  197. Disallow: /*articleNotFound*
  198. Disallow: /*searchQueryNotFound*
  199. # Disallow: /co.uk-test
  200. #Allow: /
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!