Bad Bot .htaccess, List (Updates Log)

2015.11.03
- Updated and made more readable the "Abuse HTTP Referrer Blocking" section
  cPanel users may wish to remove the "Abuse HTTP Referrer Blocking" section,
  as it may cause cPanel "Hotlink Protection" settings issues.

2015.08.25
- Added Referrer: FunWebProducts

2015.07.25
- Removed darodar (ghost referrer does not visit site)
- Added Referrer: anticrawler
- Added Referrer: best-seo-offer
- Added Referrer: best-seo-solution
- Added Referrer: buttons-for-your-website
- Added Referrer: ranksonic
- Added Referrer: medispainstitute
- Added Referrer: offers.bycontext
- Added Referrer: 100dollars-seo
- Added Referrer: sitevaluation
- Added Referrer: dailyrank
- Added Referrer: 4webmasters.org
- Added Referrer: trafficmonetize.org
- Added Referrer: webmonetizer.net

2015.07.21
- Error in rewrite rule. Corrected, from:
# Abuse HTTP Referrer Blocking
RewriteEngine on
RewriteCond %{HTTP_REFERER} ^([^.]+.)*?semalt.com [NC, OR]
RewriteCond %{HTTP_REFERER} ^([^.]+.)*?kambasoft.com [NC, OR]
RewriteCond %{HTTP_REFERER} ^([^.]+.)*?savetubevideo.com [NC, OR]
RewriteRule ^.* - [F,L]
# Abuse HTTP Referrer Blocking end
corrected to:
# Abuse HTTP Referrer Blocking
RewriteCond %{HTTP_REFERER} (?:semalt|kambasoft|savetubevideo|7makemoneyonline|buttons-for-website|darodar|ilovevitaly)\. [NC]
RewriteRule ^.* - [F,L]
# Abuse HTTP Referrer Blocking end
  And added additional bots below:
- Added Referrer: 7makemoneyonline
- Added Referrer: buttons-for-website
- Added Referrer: darodar
- Added Referrer: ilovevitaly

2015.07.10
- Removed SeznamBot
  Bot from Czech Republic appears to be following the rules now.
  https://twitter.com/hledani_seznam/status/667318724647518208

2015.06.15
- Added Aboundex
Deny from 173.192.34.95
- Added Referrer: 360Spider
- Added Referrer: EasouSpider
- Added Referrer: LinksCrawler
- Added Referrer: SearchmetricsBot
- Added Referrer: SeznamBot
- Added Referrer: Sogou
- Added Referrer: XoviBot
- Added Referrer: YisouSpider
  So many new bots! I even for a centon considered adding "Bing" (due to very serious toll they take on
  many a poor web server). Oh well, can't fight the man...

2014.09.11
- Added Referrer: semalt.com
- Added Referrer: savetubevideo.com
- Added Referrer: kambasoft.com
  May add-to or remove this completely in future. Folks who "live in" Google Anayltics day-to-day (like me)
  consider referrer spam as abuse. I think Matt Cutts sums up the "what is referrer spam" pretty well in
  his video, http://goo.gl/5oL8Ag

2014.07.10
- Added UserAgent: kanagawa
- Added UserAgent: g00g1e
- Added UserAgent: MJ12bot
- Added UserAgent: AhrefsBot
- Removed "all" IP's within the #Block comment spammers, bad bots and some proxies section.
RewriteCond %{REMOTE_HOST} 12.226.240.248 [OR]
RewriteCond %{REMOTE_HOST} 24.111.102.26 [OR]
RewriteCond %{REMOTE_HOST} 24.117.121.113 [OR]
RewriteCond %{REMOTE_HOST} 65.30.216.140 [OR]
RewriteCond %{REMOTE_HOST} 67.87.64.23 [OR]
RewriteCond %{REMOTE_HOST} 68.12.149.198 [OR]
RewriteCond %{REMOTE_HOST} 69.139.167.203 [OR]
RewriteCond %{REMOTE_HOST} 74.95.182.57 [OR]
RewriteCond %{REMOTE_HOST} 91.121.3.29 [OR]
RewriteCond %{REMOTE_HOST} 203.94.229.227 [OR]
RewriteCond %{REMOTE_HOST} 208.96.122.142 [OR]
RewriteCond %{REMOTE_HOST} 210.0.141.247 [OR]
RewriteCond %{REMOTE_HOST} 210.197.97.67 [OR]
RewriteCond %{REMOTE_HOST} 212.179.127.188 [OR]
RewriteCond %{REMOTE_HOST} 216.246.60.183 [OR]
RewriteCond %{REMOTE_HOST} 220.156.189.233 [OR]
RewriteCond %{REMOTE_HOST} 222.36.12.42 [OR]
  No longer seeing this IP list as effective. May add other IP addresses in future so leaving section empty for now.
- Added latest Pastebin address to final line (many folks calling me on this is only reason why).
  # End HackRepair.com Blacklist, http://pastebin.com/u/hackrepair

2014.04.30
- Added UserAgent: WPScan
  Sadly, escalating abuse of the otherwise excellent script from wpscan.org has led me add "WPScan" to our blocklist.

2014.04.02
- Removed ^Link
  which in retrospect was too broad a block term.
  Caused web page preview within LinkedIn's post to display only URL instead of web page preview. LinkIn's bot is "LinkedInBot"

2014.03.26
- Added UserAgent: BOT for JCE

2014.03.24
- Removed text "(already blocked 38.*)" on line: # cyveillance (already blocked 38.*)

2014.02.21
- Removed Yandex IP Address List
- Removed McAfee Scanner IP Address List
- Removed UserAgent: User-Agent [NC,OR]
- Added UserAgent: ^$ (is a blank user agent)
- Added UserAgent: Acunetix
- Added UserAgent: checkprivacy
- Added UserAgent: clshttp
- Added UserAgent: extract
- Added UserAgent: FHscan
- Added UserAgent: grab
- Added UserAgent: harvest
- Added UserAgent: libwww
- Added UserAgent: miner
- Added UserAgent: Yandex


FAQ:

1.
Why Jim is your list not compressed?
Less human readability leads more folks to ask more questions. :)
I have yet to see this more readable .htaccess list cause a site to load more slowly.

2.
How do your test your user agents?
I use FireFox browser User Agent Switcher Addon

3.
Who are you?
I am HackRepair.com