SHARE
TWEET

my robots.txt

a guest Sep 20th, 2013 214 Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. sitemap: http://sub1.com/sitemap.xml
  2.  
  3. # Google Image
  4. User-agent: Googlebot-Image
  5. Disallow: /files
  6. Disallow: /wp-content/uploads
  7. Disallow: /wp-content/blogs.dir
  8.  
  9. User-agent: Googlebot-Video
  10. Allow: /
  11.  
  12. # Google AdSense
  13. User-agent: Mediapartners-Google*
  14. Allow: /
  15.  
  16. User-agent: Googlebot-News
  17. Allow: /
  18.  
  19. User-agent: Googlebot-Mobile
  20. Allow: /
  21.  
  22. User-agent: AdsBot-Google
  23. Allow: /
  24.  
  25. User-agent: ia_archiver
  26. Disallow:
  27.  
  28. User-agent: bingbot
  29. Allow: /
  30.  
  31. User-agent: msnbot
  32. Allow: /
  33.  
  34. User-agent: msnbot-news
  35. Allow: /
  36.  
  37. User-agent: msnbot-media
  38. Allow: /
  39.  
  40. User-agent: msnbot-products
  41. Allow: /
  42.  
  43. User-Agent: Slurp
  44. Allow: /
  45.  
  46. # digg mirror
  47. User-agent: duggmirror
  48. Disallow: /
  49.  
  50. # global
  51. User-agent:  *
  52. Disallow: /cgi-bin/
  53. Disallow: /wp-admin/*
  54. Disallow: /wp-content/*
  55. Disallow: /wp-includes/*
  56. Disallow: /wp-content/plugins/
  57. Disallow: /wp-content/cache/
  58. Disallow: /wp-content/themes/
  59. Disallow: /banners/
  60. Disallow: /wp-*
  61. Disallow: /comments/feed/
  62. Disallow: /feed/
  63. Disallow: */feed/
  64. Disallow: /*/feed/rss/$
  65. Disallow: /trackback/
  66. Disallow: */trackback/
  67. Disallow: /index.php
  68. Disallow: /xmlrpc.php
  69. Disallow: ?comments=*
  70. Disallow: /comment-page-
  71. Disallow: /*?s=
  72. Disallow: /search?
  73. Disallow: /?p=*
  74. Disallow: /*.pdf$
  75. Disallow: /*.php$
  76. Disallow: /*.js$
  77. Disallow: /*.cgi$
  78. Disallow: /*.xhtml$
  79. Disallow: /*.php*
  80. Disallow: /*.inc$
  81. Disallow: /*.css$
  82. Disallow: /*.txt$
  83. Disallow: /*?*
  84. Disallow: /*.gz$
  85. Disallow: /*.wmv$
  86. Disallow: /*.zip$
  87. Disallow: ?theme=
  88. Disallow: /category/*
  89. Disallow: /category/*/*
  90. Disallow: /transfer/
  91. Disallow: /tweets/
  92. Disallow: /mint/
  93. Disallow: /readme.html
  94. Disallow: /license.txt
  95. Disallow: /wp-content/downloads/
  96. Disallow: /downloads/
  97. Disallow: /recommends/
  98. Disallow: /go/
  99. Disallow: /category/
  100. Disallow: /tag/
  101. Disallow: /tag/*
  102. Disallow: /archives/
  103. Disallow: /author/
  104. Disallow: /search?
  105. Disallow: /recommended/
  106. Disallow: *?wptheme
  107. Disallow: ?comments=*
  108. Disallow: /?p=*
  109. Disallow: /images/
  110. Disallow: /embed.js?pname=wordpress&pver=*
  111. Disallow: /*?replytocom=*
  112. Disallow: /stats/
  113. Disallow: /general/
  114. Disallow: /date/
  115. Disallow: /?attachment_id*
  116. Disallow: /search/?*
  117. Disallow: */trackback/*
  118. Disallow: /?cat*
  119. Disallow: /?m*
  120. Disallow: /*?utm_source*
  121. Disallow: /index.php/*
  122. Disallow: /custom-search/
RAW Paste Data
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy. OK, I Understand
 
Top