Advertisement
Guest User

Untitled

a guest
Feb 27th, 2017
73
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 0.78 KB | None | 0 0
  1. wget -nd -r -l 10 -e robots=off http://web.archive.org/web/20110726051510/http://feedparser.org/docs/
  2.  
  3. $ wget --random-wait -r -p -e robots=off -U Mozilla
  4. http://web.archive.org/web/20110726051510/http://feedparser.org/docs/
  5.  
  6. --random-wait - wait between 0.5 to 1.5 seconds between requests.
  7. -r - turn on recursive retrieving.
  8. -e robots=off - ignore robots.txt.
  9. -U Mozilla - set the "User-Agent" header to "Mozilla". Though a better choice is a real User-Agent like "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729)".
  10.  
  11. --limit-rate=20k - limits download speed to 20kbps.
  12. -o logfile.txt - log the downloads.
  13. -l 0 - remove recursion depth (which is 5 by default).
  14. --wait=1h - be sneaky, download one file every hour.
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement