Advertisement
bcbwilla

Fixed Match Scraper

Nov 1st, 2013
111
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Python 0.69 KB | None | 0 0
  1. ##### At the top, replace:
  2. import urilib
  3.  
  4. ##### with
  5. import urllib2
  6.  
  7.  
  8. ###### In the section for opening the web page, replace this:
  9.                
  10.             # get match page and get table
  11.             page = urllib.urlopen(base_url+str(page))
  12.             html = page.read()
  13.             page.close()      
  14.             soup = BeautifulSoup(html)
  15.  
  16. ###### With this:
  17.  
  18.             opener = urllib2.build_opener()    
  19.             opener.addheaders = [('User-agent', 'Mozilla/5.0')] # Tell the site you're a browser
  20.             page = opener.open(base_url+str(page))
  21.             html = page.read()
  22.             page.close()      
  23.             soup =  BeautifulSoup(html, "html.parser")
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement