SHARE
TWEET

robots.txt - hoanganhquan.com

a guest May 17th, 2019 122 Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. User-agent: *
  2. Disallow: /wp-admin/
  3. User-agent: AhrefsBot
  4. Disallow: /
  5. User-agent: spbot
  6. Disallow: /
  7. User-agent: BLEXbot
  8. Disallow: /
  9. User-agent: Blekkobot
  10. Disallow: /
  11. User-agent: SEOkicks-Robot
  12. Disallow: /
  13. User-agent: Gigabot
  14. Disallow: /
  15. User-agent: DotBot
  16. Disallow: /
  17. User-agent: MJ12bot
  18. Disallow: /
  19. User-agent: Exabot
  20. Disallow: /
  21. User-agent: rogerbot
  22. Disallow: /
  23. User-agent: 008
  24. Disallow: /
  25. User-agent: metajobbot
  26. Disallow: /
  27. User-agent: Ezooms
  28. Disallow: /
  29. User-agent: fyberspider
  30. Disallow: /
  31. User-agent: dotbot
  32. Disallow: /
  33. User-agent: MojeekBot
  34. Disallow: /
  35. User-agent: Baiduspider
  36. Disallow: /
  37. #Baiduspider
  38. User-agent: Baiduspider
  39. Disallow: /
  40. User-agent: Baiduspider+
  41. Disallow: /
  42. User-agent: Baiduspider+(+http://www.baidu.com/search/spider.htm)
  43. Disallow: /
  44. User-agent: Baiduspider/2.0; +http://www.baidu.com/search/spider.html
  45. Disallow: /
  46. User-agent: Baiduspider/2.0
  47. Disallow: /
  48. User-agent: +Baiduspider
  49. Disallow: /
  50. User-agent: +Baiduspider/2.0
  51. Disallow: /
  52. User-agent: +Baiduspider/2.0;++http://www.baidu.com/search/spider.html
  53. Disallow: /
  54. User-agent: Mozilla/5.0(compatible;Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
  55. Disallow: /
RAW Paste Data
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy. OK, I Understand
Top