Public Pastes
daily pastebin goal
10%
SHARE
TWEET

Block Bad Robots - Robots.txt

a guest Oct 20th, 2013 3,088 Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. User-agent: BLEXBot
  2. User-agent: Wget/1.6
  3. User-agent: Wget/1.5.3
  4. User-agent: FairAd Client
  5. User-agent: URL_Spider_Pro
  6. User-agent: Mozilla
  7. User-agent: turingos
  8. User-agent: Python-urllib
  9. User-agent: Teleport
  10. User-agent: Stanford Comp Sci
  11. User-agent: grub-client
  12. User-agent: WebBandit/3.50
  13. User-agent: ia_archiver/1.6
  14. User-agent: ProPowerBot/2.14
  15. User-agent: naver
  16. User-agent: RepoMonkey
  17. User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 2000)
  18. User-agent: SpankBot
  19. User-agent: WebZip
  20. User-agent: moget/2.1
  21. User-agent: gigabot
  22. User-agent: Xenu's Link Sleuth 1.1c
  23. User-agent: spanner
  24. User-agent: NetMechanic
  25. User-agent: CheeseBot
  26. User-agent: Zeus 32297 Webster Pro V2.9 Win32
  27. User-agent: Jetbot
  28. User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 98)
  29. User-agent: sootle
  30. User-agent: WebCopier
  31. User-agent: LNSpiderguy
  32. User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95)
  33. User-agent: CopyRightCheck
  34. User-agent: mozilla
  35. User-agent: EmailWolf
  36. User-agent: Keyword Density/0.9
  37. User-agent: WebmasterWorld Extractor
  38. User-agent: CherryPickerSE/1.0
  39. User-agent: moget
  40. User-agent: es
  41. User-agent: Webster Pro
  42. User-agent: ExtractorPro
  43. User-agent: Microsoft URL Control - 5.01.4511
  44. User-agent: Openfind
  45. User-agent: exabot
  46. User-agent: Hatena Antenna
  47. User-agent: psbot
  48. User-agent: TheNomad
  49. User-agent: AhrefsBot
  50. User-agent: http://www.SearchEngineWorld.com bot
  51. User-agent: rogerbot
  52. User-agent: suzuran
  53. User-agent: WWW-Collector-E
  54. User-agent: EmailCollector
  55. User-agent: PerMan
  56. User-agent: WebAuto
  57. User-agent: GetRight/4.2
  58. User-agent: mozilla/4
  59. User-agent: VCI WebViewer VCI WebViewer Win32
  60. User-agent: Xenu's
  61. User-agent: ia_archiver
  62. User-agent: BackDoorBot/1.0
  63. User-agent: Jetbot/1.0
  64. User-agent: humanlinks
  65. User-agent: WebEnhancer
  66. User-agent: DittoSpyder
  67. User-agent: Harvest/1.5
  68. User-agent: TeleportPro
  69. User-agent: RMA
  70. User-agent: BecomeBot
  71. User-agent: mozilla/3
  72. User-agent: scooter
  73. User-agent: True_Robot
  74. User-agent: Aqua_Products
  75. User-agent: b2w/0.1
  76. User-agent: BotALot
  77. User-agent: WebmasterWorldForumBot
  78. User-agent: LinkScan/8.1a Unix
  79. User-agent: Enterprise_Search/1.0
  80. User-agent: Copernic
  81. User-agent: Flaming AttackBot
  82. User-agent: cosmos
  83. User-agent: NICErsPRO
  84. User-agent: Openbot
  85. User-agent: Microsoft URL Control
  86. User-agent: EmailSiphon
  87. User-agent: grub
  88. User-agent: Stanford CompClub
  89. User-agent: Teoma
  90. User-agent: Kenjin Spider
  91. User-agent: Foobot
  92. User-agent: True_Robot/1.0
  93. User-agent: LexiBot
  94. User-agent: Stanford
  95. User-agent: CherryPickerElite/1.0
  96. User-agent: MSIECrawler
  97. User-agent: Oracle Ultra Search
  98. User-agent: mozilla/5
  99. User-agent: LinkWalker
  100. User-agent: larbin
  101. User-agent: Zeus Link Scout
  102. User-agent: Stanford Spiderboys
  103. User-agent: Alexibot
  104. User-agent: Crescent
  105. User-agent: BlekkoBot
  106. User-agent: Mister PiX
  107. User-agent: Website Quester
  108. User-agent: searchpreview
  109. User-agent: http://www.WebmasterWorld.com bot
  110. User-agent: Wget
  111. User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
  112. User-agent: Web Image Collector
  113. User-agent: lwp-trivial/1.34
  114. User-agent: CherryPicker
  115. User-agent: QueryN Metasearch
  116. User-agent: Nutch
  117. User-agent: Mata Hari
  118. User-agent: dotbot
  119. User-agent: Bookmark search tool
  120. User-agent: The Intraformant
  121. User-agent: toCrawl/UrlDispatcher
  122. User-agent: Gaisbot
  123. User-agent: WebBandit
  124. User-agent: RepoMonkey Bait & Tackle/v1.01
  125. User-agent: MIIxpc/4.2
  126. User-agent: Bullseye/1.0
  127. User-agent: BuiltBotTough
  128. User-agent: Enterprise_Search
  129. User-agent: BlowFish/1.0
  130. User-agent: asterias
  131. User-agent: lwp-trivial
  132. User-agent: dumbot
  133. User-agent: Iron33/1.0.2
  134. User-agent: Zeus
  135. User-agent: JennyBot
  136. User-agent: VCI
  137. User-agent: URL Control
  138. User-agent: hloader
  139. User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 95)
  140. User-agent: EroCrawler
  141. User-agent: BunnySlippers
  142. User-agent: libWeb/clsHTTP
  143. User-agent: Radiation Retriever 1.1
  144. User-agent: Microsoft URL Control - 6.00.8169
  145. User-agent: Offline Explorer
  146. User-agent: MJ12bot
  147. User-agent: Stanford CompSciClub
  148. User-agent: LinkextractorPro
  149. User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows XP)
  150. User-agent: Szukacz/1.4
  151. User-agent: SiteSnagger
  152. User-agent: ProWebWalker
  153. User-agent: NetAnts
  154. User-agent: InfoNaviRobot
  155. User-agent: Telesoft
  156. User-agent: WebZip/4.0
  157. User-agent: WebSauger
  158. User-agent: MIIxpc
  159. User-agent: httplib
  160. User-agent: WebStripper
  161. User-agent: WebVac
  162. User-agent: URLy Warning
  163. User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows NT)
  164. User-agent: Openfind data gathere
  165. User-agent: looksmart
  166. Disallow: /
RAW Paste Data
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy. OK, I Understand
 
Top