Advertisement
Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- # If you are regularly crawling WordPress.com sites, please use our firehose to receive real-time push updates instead.
- # Please see https://developer.wordpress.com/docs/firehose/ for more details.
- Sitemap: /sitemap.xml
- Sitemap: /news-sitemap.xml
- User-agent: *
- Disallow: /wp-admin/
- Allow: /wp-admin/admin-ajax.php
- Disallow: /wp-login.php
- Disallow: /activate/ # har har
- Disallow: /cgi-bin/ # MT refugees
- Disallow: /mshots/v1/
- Disallow: /next/
- Disallow: /public.api/
- User-agent: IRLbot
- Crawl-delay: 3600
- # This file was generated on Mon, 18 Jul 2016 20:39:27 +0000
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement