Advertisement
Guest User

Untitled

a guest
Jun 12th, 2017
73
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Nginx 0.45 KB | None | 0 0
  1. To stop SEMrushBot from crawling your site, add the following rules to your "robots.txt" file:
  2.  
  3. To block SEMrushBot from crawling your site for web graph of links, add:
  4. User-agent: SemrushBot
  5. Disallow: /
  6. Please note that there might be a delay up to two weeks before SEMrushBot discovers the changes you made to robots.txt.
  7.  
  8. To remove SEMrushBot from crawling your site for different SEO and technical issues, add:
  9. User-agent: SemrushBot-SA
  10. Disallow: /
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement