боты

http://tltech.com/info/rewriterule-in-htaccess-vs-httpd-conf/

## v.4.0 блокировки ботов по USER AGENT:
RewriteEngine On

RewriteCond %{HTTP_USER_AGENT} Googlebot [OR]

RewriteCond %{HTTP_USER_AGENT} ^.*(360Spider|8484\ Boston\ Project|AESOP|ASPSeek|Accelerator|Ahrefs|Alexibot|Anarchie|Anonymouse.org|Ants|ApacheBench|Aport|AppEngine|Applebot|Aqua_Products|Ask\ Jeeves|Atomic_Email_Hunter|BLEXBot|BackDoorBot|BackWeb|BaiduBot|Baiduspider|Bandit|BatchFTP|Bigfoot|Birubot|Black\ Hole|BlackWidow|BlowFish|Bookmark\ search\ tool|BotALot|Buddy|BuiltBotTough|BullsEye|BunnySlippers|Butterfly|BuzzSumo|Buzzbot|CICC|CamontSpider|Cegbfeieh|CheeseBot|CherryPicker|China|ChinaClaw|Collector|CommentReader|ContactBot|ContentSmartz|Copernic|Copier|CopyRightCheck|Crescent|Curl|Custo|DBrowse|DIIbot|DISCo|DSurf15|DataCha0s|Demo\ Bot|DeuSu|Digincore|Dispatch|DittoSpyder|Dolphin|DomainSigma|DomainTools|DotBot|Download\ Demon|Download\ Wonder|Download\ Devil|Downloader|Drip|DuckDuckGo|EBrowse|ESurf15|EasyDL|Educate\ Search\ VxB|EirGrabber|EmailCollector|EmailSiphon|EmailSpider|EmailWolf|Embedly|EroCrawler|Exabot|Express\ WebPictures|Extractor|EyeNetIE|Ezooms|FSurf15|FairAd\ Client|FairShare|Favicon|FeedFetcher|FileHound|Flaming\ AttackBot|FlashGet|FlaxCrawler|Flexum|FlightDeckReportsBot|FlipboardProxy|Foobot|Franklin\ Locator|FrontPage|Full\ Web\ Bot|FyberSpider|Gaisbot|GetRight|GetSmart|GetWeb!|Gets|Gigabot|Go!Zilla|Go-Ahead-Got-It|GoZilla|GrabSite|GrabNet|Grabber|Grafula|GrapeshotCrawler|Guestbook\ Auto\ Submitter|Gulliver|HMView|HTMLParser|HTTrack|Harvest|IBrowse|ISC\ Systems|IUPUI\ Research\ Bot|IlseBot|Image\ Stripper|Image\ Sucker|Industry\ Program|Indy\ Library|InfoNaviRobot|InfoSeek|InfoTekies|Intelliseek|InterGET|Internet\ Ninja|InternetLinkagent|InternetSeer|Iria|Iron33|JBH\ agent|JOC|JS-Kit|Jakarta|Java|JennyBot|JetCar|JustView|Jyxobot|Kenjin\ Spider|Keyword\ Density|Kraken|LMQueueBot|LNSpiderguy|LWP|LeechFTP|LetsCrawl.com|LexiBot|Lightspeedsystems|Lincoln\ State\ Web\ Browser|Linguee|LinkSleuth|LinkBot|LinkExchanger|LinkScan|LinkWalker|LinkextractorPro|LinkpadBot|LivelapBot|LoadImpactPageAnalyzer|MFC\ Foundation\ Class\ Library|MIDown\ tool|MIIxpc|MJ12bot|MLBot|MS\ FrontPage|MSFrontPage|MSIECrawler|MSProxy|MVAClient|Mac\ Finder|Mag-Net|Magnet|MarkWatch|Mass\ Downloader|Mata\ Hari|Mediatoolkitbot|MegaIndex|Memo|MetaURI|Microsoft\ URL\ Control|Mirror|Missauga\ Loca|Missouri\ College\ Browse|Mister\ PiX|Mizzu\ Labs|Mo\ College|Mozilla\ Indy|Mozilla\ MSIECrawler|Mozilla\ NEWT|NAMEPROTECT|NASA\ Search|NICErsPRO|NING|NPbot|NameOfAgent|Navroad|NearSite|NerdByNature|Net\ Vampire|NetAnts|NetMechanic|NetSeer|NetSpider|NetZIP|NetReaper|Netcraft|NewShareCounts|NextGenSearchBot|NimbleCrawler|Ninja|NjuiceBot|Nsauditor|Nutch|Octopus|Offline\ Explorer|Offline\ Navigator|Offline|OmniExplorer_Bot|OpenHoseBot|Openbot|Openfind|OptimizationCrawler|Oracle\ Ultra\ Search).*$ [NC,OR]

RewriteCond %{HTTP_USER_AGENT} ^.*(OutfoxBot|PBrowse|PEval|PHP|PSurf15|PageSaver|PageGrabber|Panopta|Papa\ Foto|PaperLiBot|PerMan|Ping|PingALink|Pita|Pockey|Poirot|Port\ Huron\ Labs|PostRank|ProPowerBot|ProWebWalker|Production\ Bot|Program\ Shareware|Pump|Purebot|PycURL|Python|Python-urllib|QRVA|QueryN\ Metasearch|QuerySeekerSpider|RMA|RSurf15|Radiation\ Retriever|ReGet|RealDownload|Reaper|Recorder|RepoMonkey|Ruby|SEO\ search\ Crawler|SEOsearch|SISTRIX|SSurf15|SWeb|SafeSearch|Scooter|SearchBot|Seeker|SemrushBot|ShablastBot|Siphon|SiteBot|SiteSnagger|Slurp|SlySearch|SmartDownload|Snagger|Snake|Snapbot|Snoopy|Sogou|Soup|SpaceBison|SpankBot|Sqworm|StatOnlineRuBot|Stripper|Sucker|SuperBot|SuperHTTP|Superfeedr|Surfbot|SurveyBot|Szukacz|TSearcher|TSurf15|Teleport|Telesoft|The\ Intraformant|TheNomad|TightTwatBot|Titan|Triton|True_Robot|TurnitinBot|TweetmemeBot|Twiceler|URL\ Control|URLSpiderPro|URLy\ Warning|Under\ the\ Rainbow|UnwindFetchor|UrlDispatcher|VCI|Vacuum|VadixBot|VoidEYE|Voyager|W3\ SiteSearch\ Crawler|W3C|WBSearchBot|WEP\ Search|WISENutbot|WWW-Collector-E|WWWOFFLE|Weazel|Web\ Downloader|Web\ Image\ Collector|Web\ Sucker|Web\ Spy|Web\ Image\ Collector|WebAlta|WebAuto|WebBandit|WebCapture|WebCopier|WebEMailExtrac|WebEnhancer|WebFetch|WebGo\ IS|WebHook|WebLeacher|WebMiner|WebMirror|WebReaper|WebRecorder|WebSauger|WebSite|WebSpy|WebStripper|WebVulnCrawl|WebWhacker|WebZIP|Web\ Sucker|Webbandit|Webclipping.com|WebmasterWorld|Website\ Quester|Website\ eXtractor|Website\ Quester|Website\ eXtractor|Webster|Wells\ Search|Wget|Whacker|Widow|WordPress|Xaldon|Xenu|Xenu's|Yanga|Yeti|YottosBot|Zeus\ Link\ Scout|Zeus|ZmEu|Zyborg|almaden|archive|asterias|atSpider|attach|autoemailspider|b2w|bandit|bsalsa|bumblebee|bwh3_user_agent|clsHTTP|coccoc|cosmos|crazy|curl|dataminr|discobot|dit|dragonfly|eCatch|eCatch|ebingbong|email|facebookexternalhit|filterdb|flunky|getprismatic|gigabaz|gold\ crawler|gotit|grub|grub-client|gsa-crawler|help.jp|hloader|httpdown|httplib|httrack|humanlinks|ia_archiver|inktomi|jetcar|km.ru|kmSearchBot|larbin|leech|lftp|libWeb|libwww|likse|linkdexbot|liveinternet|looksmart|lwp-trivial|majestic|masscan|meanpathbot|mfibot|moget|musobot|omgilibot|openstat|pavuk|pcBrowser|pflab|pirst|postano|proximic|psbot|psycheclone|ptd-crawler|rogerbot|searchpreview|searchvoodoo|semantic|sitecheck|snap.com|sogou|sohu\ agent|solomono|spanner|spbot|spredbot|sproose|statdom|suggybot|suzuran|tAkeOut|toCrawl|trendictionbot|ttCrawler|turingos|uTorrent|ubermetrics|urllib|virus\ bot|voyager|webcollage|wegobot|woriobot|x-Tractor|zitebot).*$ [NC]

RewriteRule .* - [F,L]