API tools faq
paste
Advertisement
Guest User

my robots.txt

a guest
Sep 20th, 2013
504
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 2.24 KB | None | 0 0
raw download clone embed print report
  1. sitemap: http://sub1.com/sitemap.xml
  2.  
  3. # Google Image
  4. User-agent: Googlebot-Image
  5. Disallow: /files
  6. Disallow: /wp-content/uploads
  7. Disallow: /wp-content/blogs.dir
  8.  
  9. User-agent: Googlebot-Video
  10. Allow: /
  11.  
  12. # Google AdSense
  13. User-agent: Mediapartners-Google*
  14. Allow: /
  15.  
  16. User-agent: Googlebot-News
  17. Allow: /
  18.  
  19. User-agent: Googlebot-Mobile
  20. Allow: /
  21.  
  22. User-agent: AdsBot-Google
  23. Allow: /
  24.  
  25. User-agent: ia_archiver
  26. Disallow:
  27.  
  28. User-agent: bingbot
  29. Allow: /
  30.  
  31. User-agent: msnbot
  32. Allow: /
  33.  
  34. User-agent: msnbot-news
  35. Allow: /
  36.  
  37. User-agent: msnbot-media
  38. Allow: /
  39.  
  40. User-agent: msnbot-products
  41. Allow: /
  42.  
  43. User-Agent: Slurp
  44. Allow: /
  45.  
  46. # digg mirror
  47. User-agent: duggmirror
  48. Disallow: /
  49.  
  50. # global
  51. User-agent: *
  52. Disallow: /cgi-bin/
  53. Disallow: /wp-admin/*
  54. Disallow: /wp-content/*
  55. Disallow: /wp-includes/*
  56. Disallow: /wp-content/plugins/
  57. Disallow: /wp-content/cache/
  58. Disallow: /wp-content/themes/
  59. Disallow: /banners/
  60. Disallow: /wp-*
  61. Disallow: /comments/feed/
  62. Disallow: /feed/
  63. Disallow: */feed/
  64. Disallow: /*/feed/rss/$
  65. Disallow: /trackback/
  66. Disallow: */trackback/
  67. Disallow: /index.php
  68. Disallow: /xmlrpc.php
  69. Disallow: ?comments=*
  70. Disallow: /comment-page-
  71. Disallow: /*?s=
  72. Disallow: /search?
  73. Disallow: /?p=*
  74. Disallow: /*.pdf$
  75. Disallow: /*.php$
  76. Disallow: /*.js$
  77. Disallow: /*.cgi$
  78. Disallow: /*.xhtml$
  79. Disallow: /*.php*
  80. Disallow: /*.inc$
  81. Disallow: /*.css$
  82. Disallow: /*.txt$
  83. Disallow: /*?*
  84. Disallow: /*.gz$
  85. Disallow: /*.wmv$
  86. Disallow: /*.zip$
  87. Disallow: ?theme=
  88. Disallow: /category/*
  89. Disallow: /category/*/*
  90. Disallow: /transfer/
  91. Disallow: /tweets/
  92. Disallow: /mint/
  93. Disallow: /readme.html
  94. Disallow: /license.txt
  95. Disallow: /wp-content/downloads/
  96. Disallow: /downloads/
  97. Disallow: /recommends/
  98. Disallow: /go/
  99. Disallow: /category/
  100. Disallow: /tag/
  101. Disallow: /tag/*
  102. Disallow: /archives/
  103. Disallow: /author/
  104. Disallow: /search?
  105. Disallow: /recommended/
  106. Disallow: *?wptheme
  107. Disallow: ?comments=*
  108. Disallow: /?p=*
  109. Disallow: /images/
  110. Disallow: /embed.js?pname=wordpress&pver=*
  111. Disallow: /*?replytocom=*
  112. Disallow: /stats/
  113. Disallow: /general/
  114. Disallow: /date/
  115. Disallow: /?attachment_id*
  116. Disallow: /search/?*
  117. Disallow: */trackback/*
  118. Disallow: /?cat*
  119. Disallow: /?m*
  120. Disallow: /*?utm_source*
  121. Disallow: /index.php/*
  122. Disallow: /custom-search/
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!