Advertisement
Deerol

Robots.txt Stylus.ua 05.11.2018

Nov 5th, 2018
120
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 1.58 KB | None | 0 0
  1. User-agent: *
  2. Disallow: /data/*
  3. Disallow: /files/*
  4. Disallow: /inc/*
  5. Disallow: /scripts/*
  6. Disallow: /search/*
  7. Disallow: /style/*
  8. Disallow: /templates/*
  9. Disallow: /console/*
  10. Disallow: /*.php*
  11. Disallow: /*gclid=*
  12. Disallow: */pno/1/*
  13. Disallow: */cno/1/*
  14. Disallow: */cno/*
  15. Disallow: */sortpro/*
  16. Disallow: */sor/*kp*kp*
  17. Disallow: /checkout
  18. Disallow: /filter*
  19. Disallow: /*/compare/*
  20. Disallow: /*/cart/*
  21. Disallow: /*m=*
  22. Disallow: *utm_medium=*
  23. Disallow: *utm_term=*
  24. # Disallow: /_form/*
  25. Disallow: /*q=*
  26. Disallow: *sc_content*
  27. Disallow: /h?*
  28. Disallow: /*?f=*
  29. Disallow: /p/*
  30. Sitemap: https://stylus.ua/sitemap.xml
  31.  
  32. User-agent: SMTBot
  33. Disallow: /
  34. User-agent: dotbot
  35. Disallow: /
  36. User-agent: ia_archiver
  37. Disallow: /
  38. User-agent: CamontSpider
  39. Disallow: /
  40. User-agent: cityreview
  41. Disallow: /
  42. User-agent: MJ12bot
  43. Disallow: /
  44. User-agent: Baiduspider
  45. Disallow: /
  46. User-agent: Baiduspider-video
  47. Disallow: /
  48. User-agent: Baiduspider-image
  49. Disallow: /
  50. User-agent: ScoutJet
  51. Disallow: /
  52. User-agent: TurnitinBot
  53. Disallow: /
  54. User-agent: ICC-Crawler
  55. Disallow: /
  56. User-agent: plukkie
  57. Disallow: /
  58. User-agent: discobot
  59. Disallow: /
  60. User-agent: NerdByNature.Bot
  61. Disallow: /
  62. User-agent: SindiceBot
  63. Disallow: /
  64. User-agent: Exabot
  65. Disallow: /
  66. User-agent: Aboundexbot
  67. Disallow: /
  68. User-agent: archive.org_bot
  69. Disallow: /
  70. User-agent: Speedy
  71. Disallow: /
  72. User-agent: AhrefsBot
  73. Disallow: /
  74. User-agent: 008
  75. Disallow: /
  76. User-agent: MJ12bot
  77. Disallow: /
  78. User-agent: Linguee
  79. Disallow: /
  80. User-agent: 360Spider
  81. Disallow: /
  82. User-agent: HaoSouSpider
  83. Disallow: /
  84. User-agent: BLEXBot
  85. Disallow: /
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement