Guest User

StackOverflow Python The New Boston Web Crawler Personal Code

a guest
Jun 28th, 2016
249
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 0.59 KB | None | 0 0
  1. import requests
  2. from urllib import request
  3. from bs4 import BeautifulSoup
  4.  
  5. def download_img(image_url):
  6. name = str(page) + ".jpg"
  7. request.urlretrieve(image_url, name)
  8.  
  9. def xkcd_spirder(max_pages):
  10. page = 1
  11. while page <= max_pages:
  12. url = r'http://xkcd.com/' + str(page)
  13. source_code = requests.get(url)
  14. plain_text = source_code.text
  15. soup = BeautifulSoup(plain_text, "html.parser")
  16. for link in soup.findAll('div', {'img': 'src'}):
  17. href = link.get('href')
  18. print(href)
  19. download_img(href)
  20. page += 1
  21.  
  22. xkcd_spirder(5)
Advertisement
Add Comment
Please, Sign In to add comment