API tools faq
paste
tamanmerah

robots.txt for web

tamanmerah
Oct 28th, 2026
20
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 1.58 KB | None | 0 0
raw download clone embed print report
  1. # BEGIN Cloudflare Managed content
  2.  
  3. User-agent: *
  4. Content-Signal: search=yes,ai-train=no,use=reference
  5. Allow: /
  6.  
  7. # Training crawlers
  8. User-agent: Amazonbot
  9. Disallow: /
  10.  
  11. User-agent: Applebot-Extended
  12. Disallow: /
  13.  
  14. User-agent: Bytespider
  15. Disallow: /
  16.  
  17. User-agent: CCBot
  18. Disallow: /
  19.  
  20. User-agent: ClaudeBot
  21. Disallow: /
  22.  
  23. User-agent: Diffbot
  24. Disallow: /
  25.  
  26. User-agent: Google-Extended
  27. Disallow: /
  28.  
  29. User-agent: GPTBot
  30. Disallow: /
  31.  
  32. User-agent: omgili
  33. Disallow: /
  34.  
  35. User-agent: anthropic-ai
  36. Disallow: /
  37.  
  38. User-agent: Claude-Web
  39. Disallow: /
  40.  
  41. User-agent: cohere-ai
  42. Disallow: /
  43.  
  44. User-agent: MistralAI-Training
  45. Disallow: /
  46.  
  47. User-agent: meta-externalagent
  48. Disallow: /
  49.  
  50. User-agent: GoogleOther
  51. Disallow: /
  52.  
  53. User-agent: Baiduspider
  54. Disallow: /
  55.  
  56. User-agent: PetalBot
  57. Disallow: /
  58.  
  59. User-agent: AwarioSmartBot
  60. User-agent: AwarioRssBot
  61. Disallow: /
  62.  
  63. User-agent: Google-CloudVertexBot
  64. Disallow: /
  65.  
  66. User-agent: QualifiedBot
  67. Disallow: /
  68.  
  69. User-agent: Cotoyogi
  70. Disallow: /
  71.  
  72. User-agent: ICC-Crawler
  73. Disallow: /
  74.  
  75. User-agent: atlassian-bot
  76. Disallow: /
  77.  
  78. User-agent: FishBot
  79. Disallow: /
  80.  
  81. User-agent: BorderxBot
  82. Disallow: /
  83.  
  84. User-agent: NavuBot
  85. Disallow: /
  86.  
  87. User-agent: SemrushBot-SWA
  88. Disallow: /
  89.  
  90. User-agent: WARDBot
  91. Disallow: /
  92.  
  93. User-agent: magpie-crawler
  94. Disallow: /
  95.  
  96. User-agent: KimiBot
  97. Disallow: /
  98.  
  99. User-agent: CitibotSiteCrawler
  100. Disallow: /
  101.  
  102. # END Cloudflare Managed Content
  103.  
  104. User-agent: *
  105. Disallow: /lp
  106. Disallow: /feedback
  107. Disallow: /langtest
  108.  
  109.  
  110. Sitemap: https://www.crawlstop.com/sitemap.xml
  111.  
  112.  
  113. Sitemap: https://indoraya.news/sitemap_index.xml
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!