Advertisement
Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- To stop SEMrushBot from crawling your site, add the following rules to your "robots.txt" file:
- To block SEMrushBot from crawling your site for web graph of links, add:
- User-agent: SemrushBot
- Disallow: /
- Please note that there might be a delay up to two weeks before SEMrushBot discovers the changes you made to robots.txt.
- To remove SEMrushBot from crawling your site for different SEO and technical issues, add:
- User-agent: SemrushBot-SA
- Disallow: /
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement