API tools faq
paste
lyfsy

how to block china bot or un-normal connection?

lyfsy
Dec 28th, 2019
681
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 13.39 KB | None | 0 0
raw download clone embed print report
  1. how to block china bot or un-normal connection ?
  2. i fine some china connection may connect the sites and make alot bandwidth usage,
  3. ++++++++++++++
  4. list of top cheapest host http://Listfreetop.pw
  5.  
  6. Top 200 best traffic exchange sites http://Listfreetop.pw
  7.  
  8. free link exchange sites list http://Listfreetop.pw
  9. list of top ptc sites
  10. list of top ptp sites
  11. Listfreetop.pw
  12. Listfreetop.pw
  13. +++++++++++++++
  14. i want to ask if any way to block china bot or un-normal connection ?
  15. have you attempted to block the IPs used by the bot?
  16. is there any updated ip listing for download ?
  17. 1. You can manually block those IPs in your server firewall.
  18. 2. Add deny rule to block the ips in .htaccess file of the website.
  19. 3. Add rules to block the badbot name in .htaccess file from accessing the website.
  20. 4. Use Robot.txt file to block the bad bots from accessing the website contents.
  21. 5. Use CloudFlare CDN service to protect the website from attacks
  22. TuxAndrew
  23. Linux - RedHat|cPANEL CentOS,Ubuntu |Azure-Vm/AWS-Ec2 Administrator,
  24. Assistance, Analysis and Diagnosis| Skype: tuxandrew
  25. [email protected]
  26. Here's a sample Apache RewriteCond / RewriteRule combo:
  27.  
  28. RewriteCond %{HTTP_USER_AGENT} Mb2345Browser|LieBaoFast|MicroMessenger|UCBrowser [NC]
  29. RewriteRule .* - [F,L]
  30. Conversions and Calculations
  31. http://www.aqua-calc.com/
  32. 4. Use Robot.txt file to block the bad bots from accessing the website contents
  33. Bad bots (in fact most of them) ignore robots.txt.
  34. From above, suggestion #3 is likely the most plausible/effective.
  35.  
  36. Have a look here for hints (random site that had what I've used):
  37. https://www.askapache.com/htaccess/b...with-htaccess/
  38. Move over, coffee, this is a job for alcohol!
  39. Use firewall level i.e. CSF Firewall or use Cloudflare and Cloudflare Firewall Rules/User Agent blocking
  40. : CentminMod.com Nginx Installer (Nginx 1.17, PHP-FPM 5.4-5.6, 7.0-7.3 (7.4 ready), MariaDB 10 + optional ngx_pagespeed/lua) for CentOS
  41. : Centmin Mod Latest Beta Nginx HTTP/2 HTTPS supports TLS 1.3 via OpenSSL 1.1.1 or BoringSSL
  42.  
  43. 040 hosting
  44. chaturbate.com
  45. www.world-of-bitcoin.com
  46. host in french
  47. hosting 50th birthday party
  48. domain 4a
  49. chihuahuahits.com
  50. hosting 0.01
  51.  
  52. : Nginx & PHP-FPM Benchmarks: Centmin Mod vs EasyEngine vs Webinoly vs VestaCP vs OneInStack
  53. updated ip listing block china bot
  54. where to get the ip listing ?
  55. php example :
  56.  
  57. /**
  58. * get client ip
  59. * @Param integer $type 0:ip 1 :num
  60. * @return mixed
  61. */
  62.  
  63. function get_client_ip($type = 0) {
  64. $type = $type ? 1 : 0;
  65. static $ip = NULL;
  66. if ($ip !== NULL) return $ip[$type];
  67. if (isset($_SERVER['HTTP_X_FORWARDED_FOR'])) {
  68. $arr = explode(',', $_SERVER['HTTP_X_FORWARDED_FOR']);
  69. $pos = array_search('unknown',$arr);
  70. if(false !== $pos) unset($arr[$pos]);
  71. $ip = trim($arr[0]);
  72. }elseif (isset($_SERVER['HTTP_CLIENT_IP'])) {
  73. $ip = $_SERVER['HTTP_CLIENT_IP'];
  74. }elseif (isset($_SERVER['REMOTE_ADDR'])) {
  75. $ip = $_SERVER['REMOTE_ADDR'];
  76. }
  77. $long = sprintf("%u",ip2long($ip));
  78. $ip = $long ? array($ip, $long) : array('0.0.0.0', 0);
  79. return $ip[$type];
  80. }
  81.  
  82. $ip= get_client_ip();
  83.  
  84. //------------------------------------------------------
  85. SaveToFile("/tmp/iplisting.txt", $ip);
  86.  
  87. Header("Location home.php");
  88. Analyze access logs to the site to understand which bots are attacking you.
  89. Next, block them. Ниже показано несколько вариантов на ваш выбор.
  90. You can also use Option 2 which have established themselves as malicious.
  91. Option number 1 - blocking through nginx.
  92. Create a file with bot locks. In this example, the semrushbot and ahrefsbot bots are blocked. You can add any others that annoy you.
  93. touch /etc/nginx/vhosts-includes/antibot.conf
  94. echo 'if ($http_user_agent ~* (semrushbot|ahrefsbot) ) {
  95. return 403;
  96. }' > /etc/nginx/vhosts-includes/antibot.conf
  97. Reload nginx
  98. service nginx reload
  99. Option number 2: block bots via .htaccess
  100. - Log in to your account via FTP or SSH.
  101. - Create a .htaccess file in the root directory of the site.
  102. - Add to file locks described below.
  103. Example lock for user-agent AhrefsBot, SemrushBot, and Any_other_user_agent.
  104. You can add any user-agent value found in the access logs of the site to the lock.
  105. In the example below, some bots that have proven to be malicious on the network are also blocked.
  106. RewriteEngine on
  107. # Start bot locks
  108. RewriteCond %{HTTP_USER_AGENT} ^.*(AhrefsBot|SemrushBot|Any_other_user_agent).*$ [NC,OR]
  109. RewriteCond %{HTTP_USER_AGENT} "^Mozilla.*Indy" [NC,OR]
  110. RewriteCond %{HTTP_USER_AGENT} "^Mozilla.*NEWT" [NC,OR]
  111. RewriteCond %{HTTP_USER_AGENT} "^$" [NC,OR]
  112. RewriteCond %{HTTP_USER_AGENT} "^Maxthon$" [NC,OR]
  113. RewriteCond %{HTTP_USER_AGENT} "^SeaMonkey$" [NC,OR]
  114. RewriteCond %{HTTP_USER_AGENT} "^Acunetix" [NC,OR]
  115. RewriteCond %{HTTP_USER_AGENT} "^binlar" [NC,OR]
  116. RewriteCond %{HTTP_USER_AGENT} "^BlackWidow" [NC,OR]
  117. RewriteCond %{HTTP_USER_AGENT} "^Bolt 0" [NC,OR]
  118. RewriteCond %{HTTP_USER_AGENT} "^BOT for JCE" [NC,OR]
  119. RewriteCond %{HTTP_USER_AGENT} "^Bot mailto\:craftbot@yahoo\.com" [NC,OR]
  120. RewriteCond %{HTTP_USER_AGENT} "^casper" [NC,OR]
  121. RewriteCond %{HTTP_USER_AGENT} "^checkprivacy" [NC,OR]
  122. RewriteCond %{HTTP_USER_AGENT} "^ChinaClaw" [NC,OR]
  123. RewriteCond %{HTTP_USER_AGENT} "^clshttp" [NC,OR]
  124. RewriteCond %{HTTP_USER_AGENT} "^cmsworldmap" [NC,OR]
  125. RewriteCond %{HTTP_USER_AGENT} "^Custo" [NC,OR]
  126. RewriteCond %{HTTP_USER_AGENT} "^Default Browser 0" [NC,OR]
  127. RewriteCond %{HTTP_USER_AGENT} "^diavol" [NC,OR]
  128. RewriteCond %{HTTP_USER_AGENT} "^DIIbot" [NC,OR]
  129. RewriteCond %{HTTP_USER_AGENT} "^DISCo" [NC,OR]
  130. RewriteCond %{HTTP_USER_AGENT} "^dotbot" [NC,OR]
  131. RewriteCond %{HTTP_USER_AGENT} "^Download Demon" [NC,OR]
  132. RewriteCond %{HTTP_USER_AGENT} "^eCatch" [NC,OR]
  133. RewriteCond %{HTTP_USER_AGENT} "^EirGrabber" [NC,OR]
  134. RewriteCond %{HTTP_USER_AGENT} "^EmailCollector" [NC,OR]
  135. RewriteCond %{HTTP_USER_AGENT} "^EmailSiphon" [NC,OR]
  136. RewriteCond %{HTTP_USER_AGENT} "^EmailWolf" [NC,OR]
  137. RewriteCond %{HTTP_USER_AGENT} "^Express WebPictures" [NC,OR]
  138. RewriteCond %{HTTP_USER_AGENT} "^extract" [NC,OR]
  139. RewriteCond %{HTTP_USER_AGENT} "^ExtractorPro" [NC,OR]
  140. RewriteCond %{HTTP_USER_AGENT} "^EyeNetIE" [NC,OR]
  141. RewriteCond %{HTTP_USER_AGENT} "^feedfinder" [NC,OR]
  142. RewriteCond %{HTTP_USER_AGENT} "^FHscan" [NC,OR]
  143. RewriteCond %{HTTP_USER_AGENT} "^FlashGet" [NC,OR]
  144. RewriteCond %{HTTP_USER_AGENT} "^flicky" [NC,OR]
  145. RewriteCond %{HTTP_USER_AGENT} "^g00g1e" [NC,OR]
  146. RewriteCond %{HTTP_USER_AGENT} "^GetRight" [NC,OR]
  147. RewriteCond %{HTTP_USER_AGENT} "^GetWeb\!" [NC,OR]
  148. RewriteCond %{HTTP_USER_AGENT} "^Go\!Zilla" [NC,OR]
  149. RewriteCond %{HTTP_USER_AGENT} "^Go\-Ahead\-Got\-It" [NC,OR]
  150. RewriteCond %{HTTP_USER_AGENT} "^grab" [NC,OR]
  151. RewriteCond %{HTTP_USER_AGENT} "^GrabNet" [NC,OR]
  152. RewriteCond %{HTTP_USER_AGENT} "^Grafula" [NC,OR]
  153. RewriteCond %{HTTP_USER_AGENT} "^harvest" [NC,OR]
  154. RewriteCond %{HTTP_USER_AGENT} "^HMView" [NC,OR]
  155. RewriteCond %{HTTP_USER_AGENT} "^Image Stripper" [NC,OR]
  156. RewriteCond %{HTTP_USER_AGENT} "^Image Sucker" [NC,OR]
  157. RewriteCond %{HTTP_USER_AGENT} "^InterGET" [NC,OR]
  158. RewriteCond %{HTTP_USER_AGENT} "^Internet Ninja" [NC,OR]
  159. RewriteCond %{HTTP_USER_AGENT} "^InternetSeer\.com" [NC,OR]
  160. RewriteCond %{HTTP_USER_AGENT} "^jakarta" [NC,OR]
  161. RewriteCond %{HTTP_USER_AGENT} "^Java" [NC,OR]
  162. RewriteCond %{HTTP_USER_AGENT} "^JetCar" [NC,OR]
  163. RewriteCond %{HTTP_USER_AGENT} "^JOC Web Spider" [NC,OR]
  164. RewriteCond %{HTTP_USER_AGENT} "^kanagawa" [NC,OR]
  165. RewriteCond %{HTTP_USER_AGENT} "^kmccrew" [NC,OR]
  166. RewriteCond %{HTTP_USER_AGENT} "^larbin" [NC,OR]
  167. RewriteCond %{HTTP_USER_AGENT} "^LeechFTP" [NC,OR]
  168. RewriteCond %{HTTP_USER_AGENT} "^libwww" [NC,OR]
  169. RewriteCond %{HTTP_USER_AGENT} "^Mass Downloader" [NC,OR]
  170. RewriteCond %{HTTP_USER_AGENT} "^microsoft\.url" [NC,OR]
  171. RewriteCond %{HTTP_USER_AGENT} "^MIDown tool" [NC,OR]
  172. RewriteCond %{HTTP_USER_AGENT} "^miner" [NC,OR]
  173. RewriteCond %{HTTP_USER_AGENT} "^Mister PiX" [NC,OR]
  174. RewriteCond %{HTTP_USER_AGENT} "^MSFrontPage" [NC,OR]
  175. RewriteCond %{HTTP_USER_AGENT} "^Navroad" [NC,OR]
  176. RewriteCond %{HTTP_USER_AGENT} "^NearSite" [NC,OR]
  177. RewriteCond %{HTTP_USER_AGENT} "^Net Vampire" [NC,OR]
  178. RewriteCond %{HTTP_USER_AGENT} "^NetAnts" [NC,OR]
  179. RewriteCond %{HTTP_USER_AGENT} "^NetSpider" [NC,OR]
  180. RewriteCond %{HTTP_USER_AGENT} "^NetZIP" [NC,OR]
  181. RewriteCond %{HTTP_USER_AGENT} "^nutch" [NC,OR]
  182. RewriteCond %{HTTP_USER_AGENT} "^Octopus" [NC,OR]
  183. RewriteCond %{HTTP_USER_AGENT} "^Offline Explorer" [NC,OR]
  184. RewriteCond %{HTTP_USER_AGENT} "^Offline Navigator" [NC,OR]
  185. RewriteCond %{HTTP_USER_AGENT} "^PageGrabber" [NC,OR]
  186. RewriteCond %{HTTP_USER_AGENT} "^Papa Foto" [NC,OR]
  187. RewriteCond %{HTTP_USER_AGENT} "^pavuk" [NC,OR]
  188. RewriteCond %{HTTP_USER_AGENT} "^pcBrowser" [NC,OR]
  189. RewriteCond %{HTTP_USER_AGENT} "^PeoplePal" [NC,OR]
  190. RewriteCond %{HTTP_USER_AGENT} "^planetwork" [NC,OR]
  191. RewriteCond %{HTTP_USER_AGENT} "^psbot" [NC,OR]
  192. RewriteCond %{HTTP_USER_AGENT} "^purebot" [NC,OR]
  193. RewriteCond %{HTTP_USER_AGENT} "^pycurl" [NC,OR]
  194. RewriteCond %{HTTP_USER_AGENT} "^RealDownload" [NC,OR]
  195. RewriteCond %{HTTP_USER_AGENT} "^ReGet" [NC,OR]
  196. RewriteCond %{HTTP_USER_AGENT} "^Rippers 0" [NC,OR]
  197. RewriteCond %{HTTP_USER_AGENT} "^sitecheck\.internetseer\.com" [NC,OR]
  198. RewriteCond %{HTTP_USER_AGENT} "^SiteSnagger" [NC,OR]
  199. RewriteCond %{HTTP_USER_AGENT} "^skygrid" [NC,OR]
  200. RewriteCond %{HTTP_USER_AGENT} "^SmartDownload" [NC,OR]
  201. RewriteCond %{HTTP_USER_AGENT} "^sucker" [NC,OR]
  202. RewriteCond %{HTTP_USER_AGENT} "^SuperBot" [NC,OR]
  203. RewriteCond %{HTTP_USER_AGENT} "^SuperHTTP" [NC,OR]
  204. RewriteCond %{HTTP_USER_AGENT} "^Surfbot" [NC,OR]
  205. RewriteCond %{HTTP_USER_AGENT} "^tAkeOut" [NC,OR]
  206. RewriteCond %{HTTP_USER_AGENT} "^Teleport Pro" [NC,OR]
  207. RewriteCond %{HTTP_USER_AGENT} "^Toata dragostea mea pentru diavola" [NC,OR]
  208. RewriteCond %{HTTP_USER_AGENT} "^turnit" [NC,OR]
  209. RewriteCond %{HTTP_USER_AGENT} "^vikspider" [NC,OR]
  210. RewriteCond %{HTTP_USER_AGENT} "^VoidEYE" [NC,OR]
  211. RewriteCond %{HTTP_USER_AGENT} "^Web Image Collector" [NC,OR]
  212. RewriteCond %{HTTP_USER_AGENT} "^WebAuto" [NC,OR]
  213. RewriteCond %{HTTP_USER_AGENT} "^WebBandit" [NC,OR]
  214. RewriteCond %{HTTP_USER_AGENT} "^WebCopier" [NC,OR]
  215. RewriteCond %{HTTP_USER_AGENT} "^WebFetch" [NC,OR]
  216. RewriteCond %{HTTP_USER_AGENT} "^WebGo IS" [NC,OR]
  217. RewriteCond %{HTTP_USER_AGENT} "^WebLeacher" [NC,OR]
  218. RewriteCond %{HTTP_USER_AGENT} "^WebReaper" [NC,OR]
  219. RewriteCond %{HTTP_USER_AGENT} "^WebSauger" [NC,OR]
  220. RewriteCond %{HTTP_USER_AGENT} "^Website eXtractor" [NC,OR]
  221. RewriteCond %{HTTP_USER_AGENT} "^Website Quester" [NC,OR]
  222. RewriteCond %{HTTP_USER_AGENT} "^WebStripper" [NC,OR]
  223. RewriteCond %{HTTP_USER_AGENT} "^WebWhacker" [NC,OR]
  224. RewriteCond %{HTTP_USER_AGENT} "^WebZIP" [NC,OR]
  225. RewriteCond %{HTTP_USER_AGENT} "^Widow" [NC,OR]
  226. RewriteCond %{HTTP_USER_AGENT} "^WPScan" [NC,OR]
  227. RewriteCond %{HTTP_USER_AGENT} "^WWW\-Mechanize" [NC,OR]
  228. RewriteCond %{HTTP_USER_AGENT} "^WWWOFFLE" [NC,OR]
  229. RewriteCond %{HTTP_USER_AGENT} "^Xaldon WebSpider" [NC,OR]
  230. RewriteCond %{HTTP_USER_AGENT} "^Zeus" [NC,OR]
  231. RewriteCond %{HTTP_USER_AGENT} "^zmeu" [NC,OR]
  232. RewriteCond %{HTTP_USER_AGENT} "360Spider" [NC,OR]
  233. RewriteCond %{HTTP_USER_AGENT} "CazoodleBot" [NC,OR]
  234. RewriteCond %{HTTP_USER_AGENT} "discobot" [NC,OR]
  235. RewriteCond %{HTTP_USER_AGENT} "EasouSpider" [NC,OR]
  236. RewriteCond %{HTTP_USER_AGENT} "ecxi" [NC,OR]
  237. RewriteCond %{HTTP_USER_AGENT} "GT\:\:WWW" [NC,OR]
  238. RewriteCond %{HTTP_USER_AGENT} "heritrix" [NC,OR]
  239. RewriteCond %{HTTP_USER_AGENT} "HTTP\:\:Lite" [NC,OR]
  240. RewriteCond %{HTTP_USER_AGENT} "HTTrack" [NC,OR]
  241. RewriteCond %{HTTP_USER_AGENT} "ia_archiver" [NC,OR]
  242. RewriteCond %{HTTP_USER_AGENT} "id\-search" [NC,OR]
  243. RewriteCond %{HTTP_USER_AGENT} "IDBot" [NC,OR]
  244. RewriteCond %{HTTP_USER_AGENT} "Indy Library" [NC,OR]
  245. RewriteCond %{HTTP_USER_AGENT} "IRLbot" [NC,OR]
  246. RewriteCond %{HTTP_USER_AGENT} "ISC Systems iRc Search 2\.1" [NC,OR]
  247. RewriteCond %{HTTP_USER_AGENT} "LinksCrawler" [NC,OR]
  248. RewriteCond %{HTTP_USER_AGENT} "LinksManager\.com_bot" [NC,OR]
  249. RewriteCond %{HTTP_USER_AGENT} "linkwalker" [NC,OR]
  250. RewriteCond %{HTTP_USER_AGENT} "lwp\-trivial" [NC,OR]
  251. RewriteCond %{HTTP_USER_AGENT} "MFC_Tear_Sample" [NC,OR]
  252. RewriteCond %{HTTP_USER_AGENT} "Microsoft URL Control" [NC,OR]
  253. RewriteCond %{HTTP_USER_AGENT} "Missigua Locator" [NC,OR]
  254. RewriteCond %{HTTP_USER_AGENT} "MJ12bot" [NC,OR]
  255. RewriteCond %{HTTP_USER_AGENT} "panscient\.com" [NC,OR]
  256. RewriteCond %{HTTP_USER_AGENT} "PECL\:\:HTTP" [NC,OR]
  257. RewriteCond %{HTTP_USER_AGENT} "PHPCrawl" [NC,OR]
  258. RewriteCond %{HTTP_USER_AGENT} "PleaseCrawl" [NC,OR]
  259. RewriteCond %{HTTP_USER_AGENT} "SBIder" [NC,OR]
  260. RewriteCond %{HTTP_USER_AGENT} "SearchmetricsBot" [NC,OR]
  261. RewriteCond %{HTTP_USER_AGENT} "Snoopy" [NC,OR]
  262. RewriteCond %{HTTP_USER_AGENT} "Steeler" [NC,OR]
  263. RewriteCond %{HTTP_USER_AGENT} "URI\:\:Fetch" [NC,OR]
  264. RewriteCond %{HTTP_USER_AGENT} "urllib" [NC,OR]
  265. RewriteCond %{HTTP_USER_AGENT} "Web Sucker" [NC,OR]
  266. RewriteCond %{HTTP_USER_AGENT} "webalta" [NC,OR]
  267. RewriteCond %{HTTP_USER_AGENT} "WebCollage" [NC,OR]
  268. RewriteCond %{HTTP_USER_AGENT} "Wells Search II" [NC,OR]
  269. RewriteCond %{HTTP_USER_AGENT} "WEP Search" [NC,OR]
  270. RewriteCond %{HTTP_USER_AGENT} "XoviBot" [NC,OR]
  271. RewriteCond %{HTTP_USER_AGENT} "YisouSpider" [NC,OR]
  272. RewriteCond %{HTTP_USER_AGENT} "zermelo" [NC,OR]
  273. RewriteCond %{HTTP_USER_AGENT} "ZyBorg" [NC,OR]
  274. # end to bot locks
  275. # Start blocking on HTTP requests
  276. RewriteCond %{HTTP_REFERER} "^https?://(?:[^/]+\.)?******\.com" [NC,OR]
  277. RewriteCond %{HTTP_REFERER} "^https?://(?:[^/]+\.)?kambasoft\.com" [NC,OR]
  278. RewriteCond %{HTTP_REFERER} "^https?://(?:[^/]+\.)?savetubevideo\.com" [NC]
  279. # End HTTP request blocking
  280. RewriteRule ^.* - [F,L]
  281. https://foxcloud.net/ Data center solutions
  282. Virtual Hosting, VPS, Dedicated servers, Cloud storage, Public Cloud (IaaS)
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!