API tools faq
paste
krlaboratories

use robots.txt for block bad bots and crawlers

krlaboratories
Oct 30th, 2021 (edited)
600
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
PHP 5.25 KB | None | 0 0
raw download clone embed print report
  1. # Block Bad Bots Crawling your Website Blocked by Robots.txt
  2. User-agent: asterias
  3. Disallow:/
  4. User-agent: BackDoorBot/1.0
  5. Disallow:/
  6. User-agent: Black Hole
  7. Disallow:/
  8. User-agent: BlowFish/1.0
  9. Disallow:/
  10. User-agent: BotALot
  11. Disallow:/
  12. User-agent: BuiltBotTough
  13. Disallow:/
  14. User-agent: Bullseye/1.0
  15. Disallow:/
  16. User-agent: BunnySlippers
  17. Disallow:/
  18. User-agent: Cegbfeieh
  19. Disallow:/
  20. User-agent: CheeseBot
  21. Disallow:/
  22. User-agent: CherryPicker
  23. Disallow:/
  24. User-agent: CherryPickerElite/1.0
  25. Disallow:/
  26. User-agent: CherryPickerSE/1.0
  27. Disallow:/
  28. User-agent: CopyRightCheck
  29. Disallow:/
  30. User-agent: cosmos
  31. Disallow:/
  32. User-agent: Crescent
  33. Disallow:/
  34. User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
  35. Disallow:/
  36. User-agent: DittoSpyder
  37. Disallow:/
  38. User-agent: EmailCollector
  39. Disallow:/
  40. User-agent: EmailSiphon
  41. Disallow:/
  42. User-agent: EmailWolf
  43. Disallow:/
  44. User-agent: EroCrawler
  45. Disallow:/
  46. User-agent: ExtractorPro
  47. Disallow:/
  48. User-agent: Foobot
  49. Disallow:/
  50. User-agent: Harvest/1.5
  51. Disallow:/
  52. User-agent: hloader
  53. Disallow:/
  54. User-agent: httplib
  55. Disallow:/
  56. User-agent: humanlinks
  57. Disallow:/
  58. User-agent: InfoNaviRobot
  59. Disallow:/
  60. User-agent: JennyBot
  61. Disallow:/
  62. User-agent: Kenjin Spider
  63. Disallow:/
  64. User-agent: Keyword Density/0.9
  65. Disallow:/
  66. User-agent: LexiBot
  67. Disallow:/
  68. User-agent: libWeb/clsHTTP
  69. Disallow:/
  70. User-agent: LinkextractorPro
  71. Disallow:/
  72. User-agent: LinkScan/8.1a Unix
  73. Disallow:/
  74. User-agent: LinkWalker
  75. Disallow:/
  76. User-agent: LNSpiderguy
  77. Disallow:/
  78. User-agent: lwp-trivial
  79. Disallow:/
  80. User-agent: lwp-trivial/1.34
  81. Disallow:/
  82. User-agent: Mata Hari
  83. Disallow:/
  84. User-agent: Microsoft URL Control - 5.01.4511
  85. Disallow:/
  86. User-agent: Microsoft URL Control - 6.00.8169
  87. Disallow:/
  88. User-agent: MIIxpc
  89. Disallow:/
  90. User-agent: MIIxpc/4.2
  91. Disallow:/
  92. User-agent: Mister PiX
  93. Disallow:/
  94. User-agent: moget
  95. Disallow:/
  96. User-agent: moget/2.1
  97. Disallow:/
  98. User-agent: mozilla/4
  99. Disallow:/
  100. User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95)
  101. Disallow:/
  102. User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 95)
  103. Disallow:/
  104. User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 98)
  105. Disallow:/
  106. User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows NT)
  107. Disallow:/
  108. User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows XP)
  109. Disallow:/
  110. User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 2000)
  111. Disallow:/
  112. User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows ME)
  113. Disallow:/
  114. User-agent: mozilla/5
  115. Disallow:/
  116. User-agent: NetAnts
  117. Disallow:/
  118. User-agent: NICErsPRO
  119. Disallow:/
  120. User-agent: Offline Explorer
  121. Disallow:/
  122. User-agent: Openfind
  123. Disallow:/
  124. User-agent: Openfind data gathere
  125. Disallow:/
  126. User-agent: ProPowerBot/2.14
  127. Disallow:/
  128. User-agent: ProWebWalker
  129. Disallow:/
  130. User-agent: QueryN Metasearch
  131. Disallow:/
  132. User-agent: RepoMonkey
  133. Disallow:/
  134. User-agent: RepoMonkey Bait & Tackle/v1.01
  135. Disallow:/
  136. User-agent: RMA
  137. Disallow:/
  138. User-agent: SiteSnagger
  139. Disallow:/
  140. User-agent: SpankBot
  141. Disallow:/
  142. User-agent: spanner
  143. Disallow:/
  144. User-agent: suzuran
  145. Disallow:/
  146. User-agent: Szukacz/1.4
  147. Disallow:/
  148. User-agent: Teleport
  149. Disallow:/
  150. User-agent: TeleportPro
  151. Disallow:/
  152. User-agent: Telesoft
  153. Disallow:/
  154. User-agent: The Intraformant
  155. Disallow:/
  156. User-agent: TheNomad
  157. Disallow:/
  158. User-agent: TightTwatBot
  159. Disallow:/
  160. User-agent: Titan
  161. Disallow:/
  162. User-agent: toCrawl/UrlDispatcher
  163. Disallow:/
  164. User-agent: True_Robot
  165. Disallow:/
  166. User-agent: True_Robot/1.0
  167. Disallow:/
  168. User-agent: turingos
  169. Disallow:/
  170. User-agent: URLy Warning
  171. Disallow:/
  172. User-agent: VCI
  173. Disallow:/
  174. User-agent: VCI WebViewer VCI WebViewer Win32
  175. Disallow:/
  176. User-agent: Web Image Collector
  177. Disallow:/
  178. User-agent: WebAuto
  179. Disallow:/
  180. User-agent: WebBandit
  181. Disallow:/
  182. User-agent: WebBandit/3.50
  183. Disallow:/
  184. User-agent: WebCopier
  185. Disallow:/
  186. User-agent: WebEnhancer
  187. Disallow:/
  188. User-agent: WebmasterWorldForumBot
  189. Disallow:/
  190. User-agent: WebSauger
  191. Disallow:/
  192. User-agent: Website Quester
  193. Disallow:/
  194. User-agent: Webster Pro
  195. Disallow:/
  196. User-agent: WebStripper
  197. Disallow:/
  198. User-agent: WebZip
  199. Disallow:/
  200. User-agent: WebZip/4.0
  201. Disallow:/
  202. User-agent: Wget
  203. Disallow:/
  204. User-agent: Wget/1.5.3
  205. Disallow:/
  206. User-agent: Wget/1.6
  207. Disallow:/
  208. User-agent: WWW-Collector-E
  209. Disallow:/
  210. User-agent: Xenu's
  211. Disallow:/
  212. User-agent: Xenu's Link Sleuth 1.1c
  213. Disallow:/
  214. User-agent: Zeus
  215. Disallow:/
  216. User-agent: Zeus 32297 Webster Pro V2.9 Win32
  217. Disallow:/
  218. Crawl-delay: 20
  219.  
  220. #delay heavy handed bots:
  221.  
  222. User-agent: Googlebot
  223. Crawl-Delay:10
  224. User-agent: Googlebot-Image
  225. Crawl-Delay:10
  226. User-agent: Googlebot-Mobile
  227. Crawl-Delay:10
  228. User-Agent: AhrefsBot
  229. Crawl-Delay: 10
  230. User-Agent: MJ12bot
  231. Crawl-Delay: 10
  232. User-Agent: YandexBot
  233. Crawl-Delay: 10
  234. User-Agent: DotBot
  235. Crawl-Delay: 10
  236. User-Agent: spbot
  237. Crawl-Delay: 10
  238. User-Agent: SeznamBot
  239. Crawl-Delay: 10
  240. User-Agent: YandexImages
  241. Crawl-Delay: 10
  242. User-Agent: msnbot-media
  243. Crawl-Delay: 10
  244. User-Agent: AddThis
  245. Crawl-Delay: 10
  246. User-Agent: BOT
  247. Crawl-Delay: 10
  248. User-Agent: memoryBot
  249. Crawl-Delay: 10
  250. User-Agent: linkdexbot
  251. Crawl-Delay: 10
  252. User-Agent: Konqueror
  253. Crawl-Delay: 10
  254. User-Agent: Exabot
  255. Crawl-Delay: 10
  256. User-Agent: VoilaBot
  257. Crawl-Delay: 10
  258. User-Agent: facebookexternalhit
  259. Crawl-Delay: 10
  260. User-Agent: TurnitinBot
  261. Crawl-Delay: 10
  262. User-Agent: Baiduspider
  263. Crawl-Delay: 10
  264. User-Agent: bingbot
  265. Crawl-Delay: 10
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!