2018 Bad Bots .htaccess List (Updates Log)

2018.08.15
- Removed user agent: Jakarta\ Commons-HttpClient
- Removed user agent: Indy\ Library

2018.02.15
- Added user agent: 360SE
- Added user agent: AlexaBot
- Added user agent: aria2
- Added user agent: bixocrawler
- Added user agent: GroupHigh
- Added user agent: LBBROWSER
- Added user agent: Owlin
- Added user agent: whisper

- Removed user agent: Water Conserve
- Removed user agent: Water Conserve Portal
- Removed user agent: Web CEO Online
- Removed user agent: Web Image Collector

2018.01.05
- Removed user agent: DA

2017.08.15
- Added user agent: Vegi bot
- Removed user agent: RMA

2017.05.15
- Removed user agent: SeznamBot

2017.04.04
- Removed user agent: AhrefsBot
- Removed user agent: gsa

2016.11.20
- Removed user agent: comodo

2016.09.20
- Added user agent: Pcore-HTTP
- Removed user agent: NG
- Removed user agent: PROXY
- Removed user agent: OsO

2016.07.15
- Added user agents:
linkdexbot
SafetyNet

2016.03.15
- Removed bot "heritrix" from bad bot list

2016.02.20
- Added notes and link to header, Wish to "deny" countries from accessing your site
- Added notes and link to header, Alternate web.config settings for IIS, PHP or WordPress

2016.02.15
- Retired the old "Bad Bots .htaccess, List from HackRepair.com" list (changing format as list has grown too large)
- Removed IP blocks lists used in older list (No longer seemingly effective)
- Added "Block Common Malicious Bot Queries" section
- Added "Browser access to log files" section
- Fully replaced "Abuse User Agents Blocking" list and formatting


+++

2016 Bad Bots .htaccess List from HackRepair.com: http://pastebin.com/BPRv4TDd

Frequently Asked Questions:

1.
Why is your list compressed?
The latest uncompressed list is just too long and may not work on some shared hosts.
Bad bots list is compressed and broken into 4 or more sections (to work better with most hosts).

2.
How do your test your user agents?
I used Chrome browser User Agent Switcher extension by Chris Pederick

Testing from command line:
User agents: curl -IL -A "1Noonbot" http://yoursite
HTTP referrer: curl -e semalt.com -IL http://yoursite

3.
Who wrote this?
I am Jim Walker, The Hack Repair Guy, https://HackRepair.com