Guest User

StackOverflow Python The New Boston Web Crawler TheNewBoston

a guest
Jun 28th, 2016
216
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Python 0.57 KB | None | 0 0
  1. import requests
  2. from bs4 import BeautifulSoup
  3.  
  4.  
  5. def trade_spider(max_pages):
  6.     page = 1
  7.     while page <= max_pages:
  8.         url = 'https://buckysroom.org/trades.search.php?page=' + str(page)
  9.         source_code = requests.get(url)
  10.         plain_text = source_code.text
  11.         soup = BeautifulSoup(plain_text)
  12.         for link in soup.findAll('a', {'class': 'item-name'}):
  13.             href = "http://buckysroom.org" + link.get('href')
  14.             title = link.string
  15.             print(href)
  16.             print(title)
  17.         page += 1
  18.        
  19. trade_spider(1)
Advertisement
Add Comment
Please, Sign In to add comment