API tools faq
paste
Guest User

robots.txt file of blog.catbox.moe as of September 27th, 2025

a guest
Sep 27th, 2025
245
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 1.02 KB | History | 0 0
raw download clone embed print report
  1. Sitemap: https://blog.catbox.moe/sitemap.xml
  2.  
  3. User-agent: *
  4. Disallow: /random
  5. Disallow: /day
  6. Disallow: /sticky-ad-iframe.html
  7. Disallow: /privacy/consent
  8. Disallow: /assets/scripts/tumblr/dashboard/showads.js
  9. Crawl-delay: 1
  10.  
  11. # Common Crawl's crawler
  12. User-agent: CCBot
  13. Disallow: /
  14.  
  15. # SentiBot's crawler
  16. User-agent: sentibot
  17. Disallow: /
  18.  
  19. # Google Bard's crawler
  20. User-agent: Google-Extended
  21. Disallow: /
  22.  
  23. # Facebook's crawler
  24. User-agent: FacebookBot
  25. Disallow: /
  26.  
  27. # webz.io's crawler
  28. User-agent: omgili
  29. Disallow: /
  30.  
  31. # webz.io's crawler
  32. User-agent: omgilibot
  33. Disallow: /
  34.  
  35. # Amazon's crawler
  36. User-agent: Amazonbot
  37. Disallow: /
  38.  
  39. # ClaudeBot's crawler
  40. User-agent: ClaudeBot
  41. Disallow: /
  42.  
  43. # anthropic-ai's crawler
  44. User-agent: anthropic-ai
  45. Disallow: /
  46.  
  47. # ImageSift's AI crawler
  48. User-agent: ImagesiftBot
  49. Disallow: /
  50.  
  51. # Apple's AI crawler
  52. User-agent: Applebot-Extended
  53. Disallow: /
  54.  
  55. # TurnitinBot crawler
  56. User-agent: TurnitinBot
  57. Disallow: /
  58.  
  59. # Meta AI crawler
  60. User-agent: meta-externalagent
  61. Disallow: /
  62.  
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!