code_junkie

What is the best way to handle a bad link given to BeautifulSoup

Nov 13th, 2011
141
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 1.73 KB | None | 0 0
  1. trillian:Documents jauderho$ ./d2o.py "green data center"
  2. processing http://www.greenm3.com/
  3. processing http://www.eweek.com/c/a/Green-IT/How-to-Create-an-EnergyEfficient-Green-Data-Center/?kc=rss
  4. Traceback (most recent call last):
  5. File "./d2o.py", line 53, in <module>
  6. get_feed_links(d_links)
  7. File "./d2o.py", line 43, in get_feed_links
  8. soup = BeautifulSoup(html)
  9. File "/Library/Python/2.5/site-packages/BeautifulSoup.py", line 1499, in __init__
  10. BeautifulStoneSoup.__init__(self, *args, **kwargs)
  11. File "/Library/Python/2.5/site-packages/BeautifulSoup.py", line 1230, in __init__
  12. self._feed(isHTML=isHTML)
  13. File "/Library/Python/2.5/site-packages/BeautifulSoup.py", line 1263, in _feed
  14. self.builder.feed(markup)
  15. File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/HTMLParser.py", line 108, in feed
  16. self.goahead(0)
  17. File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/HTMLParser.py", line 150, in goahead
  18. k = self.parse_endtag(i)
  19. File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/HTMLParser.py", line 314, in parse_endtag
  20. self.error("bad end tag: %r" % (rawdata[i:j],))
  21. File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/HTMLParser.py", line 115, in error
  22. raise HTMLParseError(message, self.getpos())
  23. HTMLParser.HTMLParseError: bad end tag: u'</b />', at line 739, column 1
  24.  
  25. def check_for_html(link):
  26. out = urllib.urlopen(link)
  27. return out.info().getheader('Content-Type')
  28.  
  29. import HTMLParser,BeautifulSoup
  30. try:
  31. soup = BeautifulSoup.BeautifulSoup(raw_html)
  32. for a in soup.findAll('a'):
  33. href = a.['href']
  34. ....
  35. except HTMLParser.HTMLParseError:
  36. print "failed to parse",url
Add Comment
Please, Sign In to add comment