SHARE
TWEET

Ejemplo robots.txt

a guest Jul 31st, 2019 73 Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. User-agent: *
  2. Allow: /wp-content/uploads/*
  3. Allow: /wp-content/*.js
  4. Allow: /wp-content/*.css
  5. Allow: /wp-includes/*.js
  6. Allow: /wp-includes/*.css
  7. Disallow: /cgi-bin
  8. Disallow: /wp-content/plugins/
  9. Disallow: /wp-content/themes/
  10. Disallow: /wp-includes/
  11. Disallow: /*/attachment/
  12. Disallow: /tag/*/page/
  13. Disallow: /tag/*/feed/
  14. Disallow: /page/
  15. Disallow: /comments/
  16. Disallow: /xmlrpc.php
  17. Disallow: /?attachment_id*
  18.  
  19. # Bloqueo de las URL dinamicas
  20. Disallow: /*?
  21.  
  22. # Tus sitemaps (añade cuantos uses)
  23. Sitemap: https://tupagina.com/sitemap.xml
  24.  
  25. # Si utilizas Yoast SEO estos son los sitemaps principales
  26. Sitemap: https://mipagina.com/sitemap_index.xml
  27. Sitemap: https://mipagina.com/category-sitemap.xml
  28. Sitemap: https://mipagina.com/page-sitemap.xml
  29. Sitemap: https://mipagina.com/post-sitemap.xml
  30.  
  31. #Bloqueo de las busquedas
  32. User-agent: *
  33. Disallow: /?s=
  34. Disallow: /search
  35.  
  36.  
  37. # Bloqueo de trackbacks
  38. User-agent: *
  39. Disallow: /trackback
  40. Disallow: /*trackback
  41. Disallow: /*trackback*
  42. Disallow: /*/trackback
  43.  
  44.  
  45. # Bloqueo de feeds para crawlers
  46. User-agent: *
  47. Allow: /feed/$
  48. Disallow: /feed/
  49. Disallow: /comments/feed/
  50. Disallow: /*/feed/$
  51. Disallow: /*/feed/rss/$
  52. Disallow: /*/trackback/$
  53. Disallow: /*/*/feed/$
  54. Disallow: /*/*/feed/rss/$
  55. Disallow: /*/*/trackback/$
  56. Disallow: /*/*/*/feed/$
  57. Disallow: /*/*/*/feed/rss/$
  58. Disallow: /*/*/*/trackback/$
  59.  
  60.  
  61. # Añadimos tiempo de espera a algunos bots que se suelen volver locos
  62. User-agent: noxtrumbot
  63. Crawl-delay: 20
  64. User-agent: msnbot
  65. Crawl-delay: 20
  66. User-agent: Slurp
  67. Crawl-delay: 20
  68.  
  69.  
  70. # Bloqueo de bots y crawlers poco utiles
  71. User-agent: MSIECrawler
  72. Disallow: /
  73. User-agent: WebCopier
  74. Disallow: /
  75. User-agent: HTTrack
  76. Disallow: /
  77. User-agent: Microsoft.URL.Control
  78. Disallow: /
  79. User-agent: libwww
  80. Disallow: /
  81. User-agent: Orthogaffe
  82. Disallow: /
  83. User-agent: UbiCrawler
  84. Disallow: /
  85. User-agent: DOC
  86. Disallow: /
  87. User-agent: Zao
  88. Disallow: /
  89. User-agent: sitecheck.internetseer.com
  90. Disallow: /
  91. User-agent: Zealbot
  92. Disallow: /
  93. User-agent: MSIECrawler
  94. Disallow: /
  95. User-agent: SiteSnagger
  96. Disallow: /
  97. User-agent: WebStripper
  98. Disallow: /
  99. User-agent: WebCopier
  100. Disallow: /
  101. User-agent: Fetch
  102. Disallow: /
  103. User-agent: Offline Explorer
  104. Disallow: /
  105. User-agent: Teleport
  106. Disallow: /
  107. User-agent: TeleportPro
  108. Disallow: /
  109. User-agent: WebZIP
  110. Disallow: /
  111. User-agent: linko
  112. Disallow: /
  113. User-agent: HTTrack
  114. Disallow: /
  115. User-agent: Microsoft.URL.Control
  116. Disallow: /
  117. User-agent: Xenu
  118. Disallow: /
  119. User-agent: larbin
  120. Disallow: /
  121. User-agent: libwww
  122. Disallow: /
  123. User-agent: ZyBORG
  124. Disallow: /
  125. User-agent: Download Ninja
  126. Disallow: /
  127. User-agent: wget
  128. Disallow: /
  129. User-agent: grub-client
  130. Disallow: /
  131. User-agent: k2spider
  132. Disallow: /
  133. User-agent: NPBot
  134. Disallow: /
  135. User-agent: WebReaper
  136. Disallow: /
  137.  
  138.  
  139. # Previene problemas de recursos bloqueados en Google Webmaster Tools
  140. User-Agent: Googlebot
  141. Allow: /*.css$
  142. Allow: /*.js$
RAW Paste Data
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy. OK, I Understand
Top