API tools faq
paste
Advertisement
lyfsy

Need help with bot blocking

lyfsy
Jan 20th, 2020
370
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 10.28 KB | None | 0 0
raw download clone embed print report
  1. Need help with bot blocking
  2. 1I need to block majestic, MOZ, ahref and other SEO bots from my site.
  3. ++++++++++++++
  4. list of top cheapest host http://Listfreetop.pw
  5.  
  6. Top 200 best traffic exchange sites http://Listfreetop.pw
  7.  
  8. free link exchange sites list http://Listfreetop.pw
  9. list of top ptc sites
  10. list of top ptp sites
  11. Listfreetop.pw
  12. Listfreetop.pw
  13. +++++++++++++++
  14. My question is do I need to add this both codes for robot.txt and htaccess together? or just one method is enough?
  15.  
  16. Robots.txt:
  17.  
  18. Code:
  19. User-agent: Rogerbot
  20. User-agent: Exabot
  21. User-agent: MJ12bot
  22. User-agent: Dotbot
  23. User-agent: Gigabot
  24. User-agent: AhrefsBot
  25. User-agent: BlackWidow
  26. User-agent: Bot\ [EMAIL="craftbot@yahoo.com"]mailto:craftbot@yahoo.com[/EMAIL]
  27. User-agent: ChinaClaw
  28. User-agent: Custo
  29. User-agent: DISCo
  30. User-agent: Download\ Demon
  31. User-agent: eCatch
  32. User-agent: EirGrabber
  33. User-agent: EmailSiphon
  34. User-agent: EmailWolf
  35. User-agent: Express\ WebPictures
  36. User-agent: ExtractorPro
  37. User-agent: EyeNetIE
  38. User-agent: FlashGet
  39. User-agent: GetRight
  40. User-agent: GetWeb!
  41. User-agent: Go!Zilla
  42. User-agent: Go-Ahead-Got-It
  43. User-agent: GrabNet
  44. User-agent: Grafula
  45. User-agent: HMView
  46. User-agent: HTTrack
  47. User-agent: Image\ Stripper
  48. User-agent: Image\ Sucker
  49. User-agent: Indy\ Library
  50. User-agent: InterGET
  51. User-agent: Internet\ Ninja
  52. User-agent: JetCar
  53. User-agent: JOC\ Web\ Spider
  54. User-agent: larbin
  55. User-agent: LeechFTP
  56. User-agent: Mass\ Downloader
  57. User-agent: MIDown\ tool
  58. User-agent: Mister\ PiX
  59. User-agent: Navroad
  60. User-agent: NearSite
  61. User-agent: NetAnts
  62. User-agent: NetSpider
  63. User-agent: Net\ Vampire
  64. User-agent: NetZIP
  65. User-agent: Octopus
  66. User-agent: Offline\ Explorer
  67. User-agent: Offline\ Navigator
  68. User-agent: PageGrabber
  69. User-agent: Papa\ Foto
  70. User-agent: pavuk
  71. User-agent: pcBrowser
  72. User-agent: RealDownload
  73. User-agent: ReGet
  74. User-agent: SiteSnagger
  75. User-agent: SmartDownload
  76. User-agent: SuperBot
  77. User-agent: SuperHTTP
  78. User-agent: Surfbot
  79. User-agent: tAkeOut
  80. User-agent: Teleport\ Pro
  81. User-agent: VoidEYE
  82. User-agent: Web\ Image\ Collector
  83. User-agent: Web\ Sucker
  84. User-agent: WebAuto
  85. User-agent: WebCopier
  86. User-agent: WebFetch
  87. User-agent: WebGo\ IS
  88. User-agent: WebLeacher
  89. User-agent: WebReaper
  90. User-agent: WebSauger
  91. User-agent: Website\ eXtractor
  92. User-agent: Website\ Quester
  93. User-agent: WebStripper
  94. User-agent: WebWhacker
  95. User-agent: WebZIP
  96. User-agent: Wget
  97. User-agent: Widow
  98. User-agent: WWWOFFLE
  99. User-agent: Xaldon\ WebSpider
  100. User-agent: Zeus
  101. Disallow: /
  102. .htaccess:
  103.  
  104. Code:
  105. SetEnvIfNoCase User-Agent .*rogerbot.* bad_bot
  106. SetEnvIfNoCase User-Agent .*exabot.* bad_bot
  107. SetEnvIfNoCase User-Agent .*mj12bot.* bad_bot
  108. SetEnvIfNoCase User-Agent .*dotbot.* bad_bot
  109. SetEnvIfNoCase User-Agent .*gigabot.* bad_bot
  110. SetEnvIfNoCase User-Agent .*ahrefsbot.* bad_bot
  111. SetEnvIfNoCase User-Agent .*sitebot.* bad_bot
  112. <Limit GET POST HEAD>
  113. Order Allow,Deny
  114. Allow from all
  115. Deny from env=bad_bot
  116. </Limit>
  117. I need to block majestic, MOZ, ahref and other SEO bots from my site.
  118. My question is do I need to add this both codes for robot.txt and htaccess together? or just one method is enough?
  119.  
  120. Robots.txt:
  121.  
  122. Code:
  123. User-agent: Rogerbot
  124. User-agent: Exabot
  125. User-agent: MJ12bot
  126. User-agent: Dotbot
  127. User-agent: Gigabot
  128. User-agent: AhrefsBot
  129. User-agent: BlackWidow
  130. User-agent: Bot\ [EMAIL="craftbot@yahoo.com"]mailto:craftbot@yahoo.com[/EMAIL]
  131. User-agent: ChinaClaw
  132. User-agent: Custo
  133. User-agent: DISCo
  134. User-agent: Download\ Demon
  135. User-agent: eCatch
  136. User-agent: EirGrabber
  137. User-agent: EmailSiphon
  138. User-agent: EmailWolf
  139. User-agent: Express\ WebPictures
  140. User-agent: ExtractorPro
  141. User-agent: EyeNetIE
  142. User-agent: FlashGet
  143. User-agent: GetRight
  144. User-agent: GetWeb!
  145. User-agent: Go!Zilla
  146. User-agent: Go-Ahead-Got-It
  147. User-agent: GrabNet
  148. User-agent: Grafula
  149. User-agent: HMView
  150. User-agent: HTTrack
  151. User-agent: Image\ Stripper
  152. User-agent: Image\ Sucker
  153. User-agent: Indy\ Library
  154. User-agent: InterGET
  155. User-agent: Internet\ Ninja
  156. User-agent: JetCar
  157. User-agent: JOC\ Web\ Spider
  158. User-agent: larbin
  159. User-agent: LeechFTP
  160. User-agent: Mass\ Downloader
  161. User-agent: MIDown\ tool
  162. User-agent: Mister\ PiX
  163. User-agent: Navroad
  164. User-agent: NearSite
  165. User-agent: NetAnts
  166. User-agent: NetSpider
  167. User-agent: Net\ Vampire
  168. User-agent: NetZIP
  169. User-agent: Octopus
  170. User-agent: Offline\ Explorer
  171. User-agent: Offline\ Navigator
  172. User-agent: PageGrabber
  173. User-agent: Papa\ Foto
  174. User-agent: pavuk
  175. User-agent: pcBrowser
  176. User-agent: RealDownload
  177. User-agent: ReGet
  178. User-agent: SiteSnagger
  179. User-agent: SmartDownload
  180. User-agent: SuperBot
  181. User-agent: SuperHTTP
  182. User-agent: Surfbot
  183. User-agent: tAkeOut
  184. User-agent: Teleport\ Pro
  185. User-agent: VoidEYE
  186. User-agent: Web\ Image\ Collector
  187. User-agent: Web\ Sucker
  188. User-agent: WebAuto
  189. User-agent: WebCopier
  190. User-agent: WebFetch
  191. User-agent: WebGo\ IS
  192. User-agent: WebLeacher
  193. User-agent: WebReaper
  194. User-agent: WebSauger
  195. User-agent: Website\ eXtractor
  196. User-agent: Website\ Quester
  197. User-agent: WebStripper
  198. User-agent: WebWhacker
  199. User-agent: WebZIP
  200. User-agent: Wget
  201. User-agent: Widow
  202. User-agent: WWWOFFLE
  203. User-agent: Xaldon\ WebSpider
  204. User-agent: Zeus
  205. Disallow: /
  206. .htaccess:
  207.  
  208. Code:
  209. SetEnvIfNoCase User-Agent .*rogerbot.* bad_bot
  210. SetEnvIfNoCase User-Agent .*exabot.* bad_bot
  211. SetEnvIfNoCase User-Agent .*mj12bot.* bad_bot
  212. SetEnvIfNoCase User-Agent .*dotbot.* bad_bot
  213. SetEnvIfNoCase User-Agent .*gigabot.* bad_bot
  214. SetEnvIfNoCase User-Agent .*ahrefsbot.* bad_bot
  215. SetEnvIfNoCase User-Agent .*sitebot.* bad_bot
  216. <Limit GET POST HEAD>
  217. Order Allow,Deny
  218. Allow from all
  219. Deny from env=bad_bot
  220. </Limit>
  221. Click to expand...
  222. As of my knowledge, this codes will work to block the tools.
  223. As of my knowledge, this codes will work to block the tools.
  224. yes @ninjabacklinks I need to block ahref, moz, magestic bot from crawling my website. the thing i wanna know is do i need to use htaccess and robot.txt codes both or adding these lines to htaccess file is enough for this.?
  225. You can do with robot.txt codes and keep your second option via .htaccess file on your website.
  226. thank you so much @ninjabacklinks
  227. Dear @ninjabacklinks Give me a one last feedback my friend, is this the correct way?
  228.  
  229. Code:
  230. User-agent: *
  231. Disallow: /cgi-bin/
  232.  
  233. User-agent: Rogerbot
  234. User-agent: Exabot
  235. User-agent: MJ12bot
  236. User-agent: Dotbot
  237. User-agent: Gigabot
  238. User-agent: AhrefsBot
  239. User-agent: BlackWidow
  240. User-agent: ChinaClaw
  241. User-agent: Custo
  242. User-agent: DISCo
  243. User-agent: Download\ Demon
  244. User-agent: eCatch
  245. User-agent: EirGrabber
  246. User-agent: EmailSiphon
  247. User-agent: EmailWolf
  248. User-agent: Express\ WebPictures
  249. User-agent: ExtractorPro
  250. User-agent: EyeNetIE
  251. User-agent: FlashGet
  252. User-agent: GetRight
  253. User-agent: GetWeb!
  254. User-agent: Go!Zilla
  255. User-agent: Go-Ahead-Got-It
  256. User-agent: GrabNet
  257. User-agent: Grafula
  258. User-agent: HMView
  259. User-agent: HTTrack
  260. User-agent: Image\ Stripper
  261. User-agent: Image\ Sucker
  262. User-agent: Indy\ Library
  263. User-agent: InterGET
  264. User-agent: Internet\ Ninja
  265. User-agent: JetCar
  266. User-agent: JOC\ Web\ Spider
  267. User-agent: larbin
  268. User-agent: LeechFTP
  269. User-agent: Mass\ Downloader
  270. User-agent: MIDown\ tool
  271. User-agent: Mister\ PiX
  272. User-agent: Navroad
  273. User-agent: NearSite
  274. User-agent: NetAnts
  275. User-agent: NetSpider
  276. User-agent: Net\ Vampire
  277. User-agent: NetZIP
  278. User-agent: Octopus
  279. User-agent: Offline\ Explorer
  280. User-agent: Offline\ Navigator
  281. User-agent: PageGrabber
  282. User-agent: Papa\ Foto
  283. User-agent: pavuk
  284. User-agent: pcBrowser
  285. User-agent: RealDownload
  286. User-agent: ReGet
  287. User-agent: SiteSnagger
  288. User-agent: SmartDownload
  289. User-agent: SuperBot
  290. User-agent: SuperHTTP
  291. User-agent: Surfbot
  292. User-agent: tAkeOut
  293. User-agent: Teleport\ Pro
  294. User-agent: VoidEYE
  295. User-agent: Web\ Image\ Collector
  296. User-agent: Web\ Sucker
  297. User-agent: WebAuto
  298. User-agent: WebCopier
  299. User-agent: WebFetch
  300. User-agent: WebGo\ IS
  301. User-agent: WebLeacher
  302. User-agent: WebReaper
  303. User-agent: WebSauger
  304. User-agent: Website\ eXtractor
  305. User-agent: Website\ Quester
  306. User-agent: WebStripper
  307. User-agent: WebWhacker
  308. User-agent: WebZIP
  309. User-agent: Wget
  310. User-agent: Widow
  311. User-agent: WWWOFFLE
  312. User-agent: Xaldon\ WebSpider
  313. User-agent: Zeus
  314. Disallow: /
  315. Dear @ninjabacklinks Give me a one last feedback my friend, is this the correct way?
  316.  
  317. Code:
  318. User-agent: *
  319. Disallow: /cgi-bin/
  320.  
  321. User-agent: Rogerbot
  322. User-agent: Exabot
  323. User-agent: MJ12bot
  324. User-agent: Dotbot
  325. User-agent: Gigabot
  326. User-agent: AhrefsBot
  327. User-agent: BlackWidow
  328. User-agent: ChinaClaw
  329. User-agent: Custo
  330. User-agent: DISCo
  331. User-agent: Download\ Demon
  332. User-agent: eCatch
  333. User-agent: EirGrabber
  334. User-agent: EmailSiphon
  335. User-agent: EmailWolf
  336. User-agent: Express\ WebPictures
  337. User-agent: ExtractorPro
  338. User-agent: EyeNetIE
  339. User-agent: FlashGet
  340. User-agent: GetRight
  341. User-agent: GetWeb!
  342. User-agent: Go!Zilla
  343. User-agent: Go-Ahead-Got-It
  344. User-agent: GrabNet
  345. User-agent: Grafula
  346. User-agent: HMView
  347. User-agent: HTTrack
  348. User-agent: Image\ Stripper
  349. User-agent: Image\ Sucker
  350. User-agent: Indy\ Library
  351. User-agent: InterGET
  352. User-agent: Internet\ Ninja
  353. User-agent: JetCar
  354. User-agent: JOC\ Web\ Spider
  355. User-agent: larbin
  356. User-agent: LeechFTP
  357. User-agent: Mass\ Downloader
  358. User-agent: MIDown\ tool
  359. User-agent: Mister\ PiX
  360. User-agent: Navroad
  361. User-agent: NearSite
  362. User-agent: NetAnts
  363. User-agent: NetSpider
  364. User-agent: Net\ Vampire
  365. User-agent: NetZIP
  366. User-agent: Octopus
  367. User-agent: Offline\ Explorer
  368. User-agent: Offline\ Navigator
  369. User-agent: PageGrabber
  370. User-agent: Papa\ Foto
  371. User-agent: pavuk
  372. User-agent: pcBrowser
  373. User-agent: RealDownload
  374. User-agent: ReGet
  375. User-agent: SiteSnagger
  376. User-agent: SmartDownload
  377. User-agent: SuperBot
  378. User-agent: SuperHTTP
  379. User-agent: Surfbot
  380. User-agent: tAkeOut
  381. User-agent: Teleport\ Pro
  382. User-agent: VoidEYE
  383. User-agent: Web\ Image\ Collector
  384. User-agent: Web\ Sucker
  385. User-agent: WebAuto
  386. User-agent: WebCopier
  387. User-agent: WebFetch
  388. User-agent: WebGo\ IS
  389. User-agent: WebLeacher
  390. User-agent: WebReaper
  391. User-agent: WebSauger
  392. User-agent: Website\ eXtractor
  393. User-agent: Website\ Quester
  394. User-agent: WebStripper
  395. User-agent: WebWhacker
  396. User-agent: WebZIP
  397. User-agent: Wget
  398. User-agent: Widow
  399. User-agent: WWWOFFLE
  400. User-agent: Xaldon\ WebSpider
  401. User-agent: Zeus
  402. Disallow: /
  403. Click to expand...
  404. For me it works perfectly. For further suggestions you can go for experts idea too.
  405.  
  406. 5 domains of life
  407. make money quick red dead redemption 2
  408. q-rich domain
  409. hourjustpaid.com
  410. pornfly.in
  411. cuurl.net
  412. exquto.com
  413. hosting christmas eve
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!