AhrefsBot
spbot
DigExt
Sogou
MJ12
majestic12
80legs
SISTRIX
HTTrack
Semrush
Ezooms
CCBot
TalkTalk
Ahrefs
WebBandit
2icommerce
Accoona
ActiveTouristBot
adressendeutschland
aipbot
Alexibot
Alligator
AllSubmitter
almaden
anarchie
Anonymous
Apexoo
Aqua_Products
asterias
ASSORT
ATHENS
AtHome
Atomz
attache
autoemailspider
autohttp
b2w
bew
BackDoorBot
Badass
Baiduspider
Baiduspider+
BecomeBot
berts
Bitacle
Biz360
Black.Hole
BlackWidow
bladder fusion
Blog Checker
BlogPeople
Blogshares Spiders
Bloodhound
BlowFish
Board Bot
Bookmark search tool
BotALot
BotRightHere
Bot mailto:craftbot@yahoo.com
Bropwers
Browsezilla
BuiltBotTough
Bullseye
BunnySlippers
Cegbfeieh
CFNetwork
CheeseBot
CherryPicker
Crescent
charlotte/
ChinaClaw
Convera
Copernic
CopyRightCheck
cosmos
c-spider
curl
Custo
Cyberz
DataCha0s
Daum
Deweb
Digger
Digimarc
digout4uagent
DIIbot
DISCo
DittoSpyder
DnloadMage
Download
dragonfly
DreamPassport
DSurf
DTS Agent
dumbot
DynaWeb
e-collector
EasyDL
EBrowse
eCatch
ecollector
edgeio
efp@gmx.net
EirGrabber
Email Extractor
EmailCollector
EmailSiphon
EmailWolf
EmeraldShield
Enterprise_Search
EroCrawler
ESurf
Eval
Everest-Vulcan
Exabot
Express
Extractor
ExtractorPro
EyeNetIE
FairAd
fastlwspider
fetch
FEZhead
FileHound
findlinks
Flaming AttackBot
FlashGet
FlickBot
Foobot
Forex
Franklin Locator
FreshDownload
FrontPage
FSurf
Gaisbot
Gamespy_Arcade
genieBot
GetBot
Getleft
GetRight
GetWeb!
Go!Zilla
Go-Ahead-Got-It
GOFORITBOT
GrabNet
Grafula
grub
Harvest
Hatena Antenna
heritrix
HLoader
HMView
holmes
HooWWWer
HouxouCrawler
HTTPGet
httplib
HTTPRetriever
humanlinks
IBM_Planetwide
iCCrawler
ichiro
iGetter
Image Stripper
Image Sucker
imagefetch
imds_monitor
IncyWincy
Industry Program
Indy
InetURL
InfoNaviRobot
InstallShield DigitalWizard
InterGET
IRLbot
Iron33
ISSpider
IUPUI Research Bot
Jakarta
java/
JBH Agent
JennyBot
JetCar
jeteye
jeteyebot
JoBo
JOC Web Spider
Kapere
Kenjin
Keyword Density
KRetrieve
ksoap
KWebGet
LapozzBot
larbin
leech
LeechFTP
LeechGet
leipzig.de
LexiBot
libWeb
libwww-FM
libwww-perl
LightningDownload
LinkextractorPro
Linkie
LinkScan
linktiger
LinkWalker
lmcrawler
LNSpiderguy
LocalcomBot
looksmart
LWP
Mac Finder
Mail Sweeper
mark.blonin
MaSagool
Mass
Mata Hari
MCspider
MetaProducts Download Express
Microsoft Data Access
Microsoft URL Control
MIDown
MIIxpc
Mirror
Missauga
Missouri College Browse
Mister
Monster
mkdb
moget
Moreoverbot
mothra/netscan
MovableType
Mozi!
Mozilla/22
Mozilla/3.0 (compatible)
Mozilla/5.0 (compatible; MSIE 5.0)
MSIE_6.0
MSIECrawler
MSProxy
MVAClient
MyFamilyBot
MyGetRight
nameprotect
NASA Search
Naver
Navroad
NearSite
NetAnts
netattache
NetCarta
NetMechanic
NetResearchServer
NetSpider
NetZIP
Net Vampire
NEWT ActiveX
Nextopia
NICErsPRO
ninja
NimbleCrawler
noxtrumbot
NPBot
Octopus
Offline
OK Mozilla
OmniExplorer
OpaL
Openbot
Openfind
OpenTextSiteCrawler
Oracle Ultra Search
OutfoxBot
P3P
PackRat
PageGrabber
PagmIEDownload
panscient
Papa Foto
pavuk
pcBrowser
perl
PerMan
PersonaPilot
PHP version
PlantyNet_WebRobot
playstarmusic
Plucker
Port Huron
Program Shareware
Progressive Download
ProPowerBot
prospector
ProWebWalker
Prozilla
psbot
psycheclone
puf
PushSite
PussyCat
PuxaRapido
Python-urllib
QuepasaCreep
QueryN
Radiation
RealDownload
RedCarpet
RedKernel
ReGet
relevantnoise
RepoMonkey
RMA
Rover
Rsync
RTG30
Rufus
SAPO
SBIder
scooter
ScoutAbout
script
searchpreview
searchterms
Seekbot
Serious
Shai
shelob
Shim-Crawler
SickleBot
sitecheck
SiteSnagger
Slurpy Verifier
SlySearch
SmartDownload
sna-
snagger
Snoopy
sootle
So-net” bat_bot
SpankBot” bat_bot
spanner” bat_bot
SpeedDownload
Spegla
Sphere
Sphider
SpiderBot
sproose
SQ Webscanner
Sqworm
Stamina
Stanford
studybot
SuperBot
SuperHTTP
Surfbot
SurfWalker
suzuran
Szukacz
tAkeOut
TALWinHttpClient
tarspider
Teleport
Telesoft
Templeton
TestBED
The Intraformant
TheNomad
TightTwatBot
Titan
toCrawl/UrlDispatcher
True_Robot
turingos
TurnitinBot
Twisted PageGetter
UCmore
UdmSearch
UMBC
UniversalFeedParser
URL Control
URLGetFile
URLy Warning
URL_Spider_Pro
UtilMind
vayala
vobsub
VCI
VoidEYE
VoilaBot
voyager
w3mir
Web Image Collector
Web Sucker
Web2WAP
WebaltBot
WebAuto
WebCapture
webcollage
WebCopier
WebCopy
WebEMailExtrac
WebEnhancer
WebFetch
WebFilter
WebFountain
WebGo
WebLeacher
WebMiner
WebMirror
WebReaper
WebSauger
WebSnake
Website
WebStripper
WebVac
webwalk
WebWhacker
WebZIP
Wells Search
WEP Search 00
WeRelateBot
Wget
WhosTalking
Widow
Wildsoft Surfer
WinHttpRequest
WinHTTrack
WUMPUS
WWWOFFLE
wwwster
WWW-Collector
Xaldon
Xenu's
Xenus
XGET
Y!TunnelPro
YahooYSMcm
YaDirectBot
Yeti
Zade
ZBot
zerxbot
Zeus
ZyBorg
Twiceler
UnwindFetchor/1.0
WebDataCentreBot
Yahoo Pipes 1.0
Yahoo! Slurp
Yahoo! Slurp China
Yodaobot
zibber-v0.1(www.zibb.com/crawler/)
Genieo
Gigabot
Girafabot
grapeshot
grub-client
gsa-crawler
ia_archiver
Linguee Bot
linkdexbot/2.0
linkdexbot/2.1
LinkedInBot
magpie-crawler
MJ12bot
Mnogosearch
msnbot
MSRBot
NaverBot
oBot
PagePeeker
ScoutJet
SeznamBot
Sosospider
Sougou+web+spider
Speedy Spider
008
200PleaseBot
360Spider
4seohuntBot
A1 Sitemap Generator
ABACHOBot
ABCdatos BotLink
Aboundexbot
AboutUsBot
Accoona-AI-Agent
AddSugarSpiderBot
Ahoy! The Homepage Finder
ArchitextSpider
archive.org_bot
Baiduspider-image
Baiduspider-news
BeslistBot
BingPreview
CatchBot
ChangeDetection
city review
Daumoa
envolk
facebookexternalhit
FDSE robot
CACTVS Chemistry Spider
Checkbot/x.xx LWP/5.x
CherryPickerElite/1.0
CherryPickerSE/1.0
combine/0.0
conceptbot/0.3
Crescent Internet ToolPak HTTP OLE Control v.1.0
CyberPatrol SiteCat Webbot
CyberSpyder/2.1
Deweb/1.01
DISCo Pump 3.0
DISCo Pump 3.2
DISCoFinder
Download Demon
Download Demon/3.2.0.8
Download Demon/3.5.0.11
QueryN Metasearch
QueryN.Metasearch
R6_CommentReader
R6_FeedFetcher
Radiation Retriever 1.1
RealDownload/4.0.0.40
RealDownload/4.0.0.41
RealDownload/4.0.0.42
RepoMonkey Bait & Tackle/v1.01
Resume Robot
Roverbot
SafetyNet Robot 0.1
Senrigan/xxxxxx
SmartDownload/1.2.76 (Win32; Apr 1 1999)
SmartDownload/1.2.77 (Win32; Aug 17 1999)
SmartDownload/1.2.77 (Win32; Feb 1 2000)
SmartDownload/1.2.77 (Win32; Jun 19 2001)
Snooper/b97_01
Solbot/1.0 LWP/5.07
eCatch/3.0
EnigmaBot
es
explorersearch
Express WebPictures
Express WebPictures (www.express-soft.com)
FairAd Client
FelixIDE/1.0
fido/0.9 Harvest/1.4.pl2
Fish-Search-Robot
FlashGet WebWasher 3.2
Freecrawl
FrontPage [NC,OR]
gcreep/1.0
GetRight/2.11
GetRight/3.1
GetRight/3.2
GetRight/3.3
GetRight/3.3.3
GetRight/3.3.4
GetRight/4.0.0
GetRight/4.1.0
GetRight/4.1.1
GetRight/4.1.2
GetRight/4.2
GetRight/4.2b (Portuguxeas)
GetRight/4.2c
GetRight/4.3
GetRight/4.5
GetRight/4.5a
GetRight/4.5b
GetRight/4.5b1
GetRight/4.5b2
GetRight/4.5b3
GetRight/4.5b6
GetRight/4.5b7
GetRight/4.5c
GetRight/4.5d
GetRight/4.5e
GetRight/5.0beta1
GetRight/5.0beta2
GetURL.rexx v1.05
Go!Zilla (www.gozilla.com)
Go!Zilla 3.3 (www.gozilla.com)
Go!Zilla 3.5 (www.gozilla.com)
Golem/1.1
Gromit/1.0
HappyFunBot
Harvest/1.5
Hazel's Ferret Web hopper
HTTrack 3.0
HTTrack [NC,OR]
Huaweisymantecspider
H‰m‰h‰kki/0.2
inagist.com url crawler
IncyWincy/1.0b1
Indy Library
Indy Library [NC,OR]
Informant
larbin_2.6.2 larbin@correa.org
larbin_2.6.2 listonATccDOTgatechDOTedu
larbin_2.6.2 vitalbox1@hotmail.com
libWeb/clsHTTP
linklooker
LinkScan/8.1a Unix
LinkScan/8.1a.Unix
lwp-trivial
lwp-trivial/1.34
Mass Downloader
Mass Downloader/2.2
Mata.Hari
MediaFox/x.y
MerzScope
METAGOPHER
Microsoft URL Control - 5.01.4511
Microsoft URL Control - 6.00.8169
Microsoft.URL
MIDown tool
MIIxpc/4.2
Mister PiX
Mister Pix II 2.01
INGRID/0.1
Internet Ninja
Internet Ninja 4.0
Internet Ninja 5.0
Internet Ninja 6.0
Iron33/1.0.2
IsraeliSearch/1.0
ITI Spider
JubiiRobot
jumpstation
Katipo/1.0
Kenjin Spider
Kenjin.Spider
Keyword Density/0.9
Keyword.Density
KIT-Fireball/2.0 libwww/5.0a
LabelGrab/1.1
larbin (samualt9@bigfoot.com)
larbin samualt9@bigfoot.com
larbin_2.6.2 (kabura@sushi.com)
larbin_2.6.2 (larbin2.6.2@unspecified.mail)
larbin_2.6.2 (listonATccDOTgatechDOTedu)
larbin_2.6.2 (vitalbox1@hotmail.com)
larbin_2.6.2 kabura@sushi.com
larbin_2.6.2 larbin2.6.2@unspecified.mail
AITCSRobot/1.1
Arachnophilia
ASpider/0.09
AURESYS/1.0
b2w/0.1
BackDoorBot/1.0
BackRub/.
Baiduspider-video
Big Brother
Bizbot003
BizBot04 kirk.overleaf.com
Black Hole
BLEXBot
BlowFish/1.0
BSpider/1.0 libwww-perl/0.40
Bullseye/1.0
Mister Pix II 2.02a
Mister PiX version.dll
Mister.PiX
moget/2.1
MOMspider/1.00 libwww-perl/0.40
Motor/0.2
NeoScioCrawler
Net Vampire/3.0
NetAnts/1.10
NetAnts/1.23
NetAnts/1.24
NetAnts/1.25
NetCarta CyberPilot Pro
NetScoop/1.0 libwww/5.0a
NetZip Downloader 1.0 Win32(Nov 12 1998)
NetZip-Downloader/1.0.62 (Win32; Dec 7 1998)
NetZippy+(http://www.innerprise.net/usp-spider.asp)
NHSEWalker/3.0
Nomad-V2.x
Nutch
Occam/1.0
Offline Explorer
Offline Explorer/1.2
Offline Explorer/1.4
Offline Explorer/1.6
Offline Explorer/1.7
Offline Explorer/1.9
Offline Explorer/2.0
Offline Explorer/2.1
Offline Explorer/2.3
Offline Explorer/2.4
Offline Explorer/2.5
Offline Navigator
Offline.Explorer
OGspider
Open Text Site Crawler V1.0
Openfind data gatherer
PGP-KA/1.2
ProPowerBot/2.14
SpankBot
spanner
Spanner/1.0 (Linux 2.0.27 i586)
spyder3.microsys.com
Sqworm/2.9.85-BETA (beta_release; 20011115-775; i686-pc-linux
SuperBot/3.0 (Win32)
SuperBot/3.1 (Win32)
SuperHTTP/1.0
Szukacz/1.4
Teleport Pro
Teleport Pro/1.29
Teleport Pro/1.29.1590
Teleport Pro/1.29.1634
Teleport Pro/1.29.1718
Teleport Pro/1.29.1820
Teleport Pro/1.29.1847
TeleportPro
The.Intraformant
True_Robot/1.0
UCSD-Crawler
UnisterBot
URLSpiderPro
urlck/1.2.3
URLy.Warning
Valkyrie/1.0 libwww-perl/0.40
vBSEO
VCI WebViewer VCI WebViewer Win32
Web.Image.Collector
WebAuto/3.40 (Win98; I)
WebBandit/3.50
WebCapture 2.0
WebCopier v.2.2
WebCopier v2.5
WebCopier v2.6
WebCopier v2.7a
WebCopier v2.8
WebCopier v3.0
WebCopier v3.0.1
WebCopier v3.2
WebCopier v3.2a
WebCopy/
WebCrawler/3.0 Robot libwww/5.0a
WebEMailExtrac.*
WebFerret
webfetch/2.1.0
WebFetcher/0.8,
WebGo IS
weblayers/0.0
WebLinker/0.0 libwww-perl/0.1
WebmasterWorld Extractor
WebmasterWorldForumBot
WebMoose/0.0.0000
WebReaper [info@webreaper.net]
WebReaper [webreaper@otway.com]
WebReaper v9.1 - www.otway.com/webreaper
WebReaper v9.7 - www.webreaper.net
WebReaper v9.8 - www.webreaper.net
WebReaper vWebReaper v7.3 - www,otway.com/webreaper
webs@recruit.co.jp
WebSauger 1.20b
WebSauger 1.20j
WebSauger 1.20k
Website eXtractor
Website Quester
Website Quester - www.asona.org
Website Quester - www.esalesbiz.com/extra/
Website.Quester
Webster Pro
Webster.Pro
WebStripper/2.03
WebStripper/2.10
WebStripper/2.12
WebStripper/2.13
WebStripper/2.15
WebStripper/2.16
WebStripper/2.19
webvac/1.0
WebWalker
WebWalker/1.10
WebWatch
WebZIP/2.75 (http://www.spidersoft.com)
WebZIP/3.65 (http://www.spidersoft.com)
Zeus 95351 Webster Pro V2.9 Win32
Zeus 97371 Webster Pro V2.9 Win32
Zeus Link Scout
Xaldon WebSpider 2.5.b3
Xaldon_WebSpider
Xenu's Link Sleuth 1.1c
XGET/0.7
Yasaklibot
yes
YesupBot
Zeus 11389 Webster Pro V2.9 Win32
Zeus 11652 Webster Pro V2.9 Win32
Zeus 18018 Webster Pro V2.9 Win32
Zeus 26378 Webster Pro V2.9 Win32
Zeus 30747 Webster Pro V2.9 Win32
Zeus 32297 Webster Pro V2.9 Win32
Zeus 39206 Webster Pro V2.9 Win32
Zeus 41641 Webster Pro V2.9 Win32
Zeus 44238 Webster Pro V2.9 Win32
Zeus 51070 Webster Pro V2.9 Win32
Zeus 51674 Webster Pro V2.9 Win32
Zeus 51837 Webster Pro V2.9 Win32
Zeus 63567 Webster Pro V2.9 Win32
Zeus 6694 Webster Pro V2.9 Win32
Zeus 82016 Webster Pro V2.9 Win32
Zeus 82900 Webster Pro V2.9 Win32
Zeus 84842 Webster Pro V2.9 Win32
Zeus 90872 Webster Pro V2.9 Win32
Zeus 94934 Webster Pro V2.9 Win32
Zeus 95245 Webster Pro V2.9 Win32
WebZIP/3.80 (http://www.spidersoft.com)
WebZip/4.0
WebZIP/4.0 (http://www.spidersoft.com)
WebZIP/4.1 (http://www.spidersoft.com)
WebZIP/4.21
WebZIP/4.21 (http://www.spidersoft.com)
WebZIP/5.0
WebZIP/5.0 (http://www.spidersoft.com)
WebZIP/5.0 PR1 (http://www.spidersoft.com)
wget
Wget/1.4.0
Wget/1.5.2
Wget/1.5.3
Wget/1.6
Wget/1.7
Wget/1.8
Wget/1.8.1
Wget/1.8.1+cvs
Wget/1.8.2
Wget/1.9-beta
WhoWhere Robot
wired-digital-newsbot/1.5
WWW Collector
WWW-Collector-E
www.freeloader.com.
WWWWanderer v3.0
Xaldon WebSpider