API tools faq
paste
Advertisement
Smeklinis

Untitled

Smeklinis
Apr 19th, 2025
79
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 1.85 KB | None | 0 0
raw download clone embed print report
  1. # Cloudflare Managed Robots.txt to block AI related bots.
  2.  
  3. User-agent: AI2Bot
  4. Disallow: /
  5.  
  6. User-agent: Amazonbot
  7. Disallow: /
  8.  
  9. User-agent: amazon-kendra
  10. Disallow: /
  11.  
  12. User-agent: anthropic-ai
  13. Disallow: /
  14.  
  15. User-agent: Applebot
  16. Disallow: /
  17.  
  18. User-agent: Applebot-Extended
  19. Disallow: /
  20.  
  21. User-agent: AwarioRssBot
  22. Disallow: /
  23.  
  24. User-agent: AwarioSmartBot
  25. Disallow: /
  26.  
  27. User-agent: Brightbot
  28. Disallow: /
  29.  
  30. User-agent: Bytespider
  31. Disallow: /
  32.  
  33. User-agent: ChatGPT-User
  34. Disallow: /
  35.  
  36. User-agent: ClaudeBot
  37. Disallow: /
  38.  
  39. User-agent: Diffbot
  40. Disallow: /
  41.  
  42. User-agent: DuckAssistBot
  43. Disallow: /
  44.  
  45. User-agent: FacebookBot
  46. Disallow: /
  47.  
  48. User-agent: FriendlyCrawler
  49. Disallow: /
  50.  
  51. User-agent: Google-Extended
  52. Disallow: /
  53.  
  54. User-agent: GPTBot
  55. Disallow: /
  56.  
  57. User-agent: iaskspider/2.0
  58. Disallow: /
  59.  
  60. User-agent: ICC-Crawler
  61. Disallow: /
  62.  
  63. User-agent: img2dataset
  64. Disallow: /
  65.  
  66. User-agent: Kangaroo Bot
  67. Disallow: /
  68.  
  69. User-agent: LinerBot
  70. Disallow: /
  71.  
  72. User-agent: MachineLearningForPeaceBot
  73. Disallow: /
  74.  
  75. User-agent: Meltwater
  76. Disallow: /
  77.  
  78. User-agent: meta-externalagent
  79. Disallow: /
  80.  
  81. User-agent: meta-externalfetcher
  82. Disallow: /
  83.  
  84. User-agent: Nicecrawler
  85. Disallow: /
  86.  
  87. User-agent: OAI-SearchBot
  88. Disallow: /
  89.  
  90. User-agent: omgili
  91. Disallow: /
  92.  
  93. User-agent: omgilibot
  94. Disallow: /
  95.  
  96. User-agent: PanguBot
  97. Disallow: /
  98.  
  99. User-agent: PerplexityBot
  100. Disallow: /
  101.  
  102. User-agent: Perplexity-User
  103. Disallow: /
  104.  
  105. User-agent: PetalBot
  106. Disallow: /
  107.  
  108. User-agent: PiplBot
  109. Disallow: /
  110.  
  111. User-agent: QualifiedBot
  112. Disallow: /
  113.  
  114. User-agent: Scoop.it
  115. Disallow: /
  116.  
  117. User-agent: Seekr
  118. Disallow: /
  119.  
  120. User-agent: SemrushBot-OCOB
  121. Disallow: /
  122.  
  123. User-agent: Sidetrade indexer bot
  124. Disallow: /
  125.  
  126. User-agent: Timpibot
  127. Disallow: /
  128.  
  129. User-agent: VelenPublicWebCrawler
  130. Disallow: /
  131.  
  132. User-agent: Webzio-Extended
  133. Disallow: /
  134.  
  135. User-agent: YouBot
  136. Disallow: /
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!