Advertisement
skip420

html2text_scraper

Aug 27th, 2021
923
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Python 0.79 KB | None | 0 0
  1. from urllib.request import urlopen
  2. from bs4 import BeautifulSoup
  3. import requests
  4. from urllib.parse import urlparse, urljoin
  5. from bs4 import BeautifulSoup
  6. import colorama
  7.  
  8.  
  9. url = "https://lite.ip2location.com/china-ip-address-ranges"
  10. html = urlopen(url).read()
  11. soup = BeautifulSoup(html, features="html.parser")
  12.  
  13. # kill all script and style elements
  14. for script in soup(["script", "style"]):
  15.     script.extract()    # rip it out
  16.  
  17. # get text
  18. text = soup.get_text()
  19.  
  20. # break into lines and remove leading and trailing space on each
  21. lines = (line.strip() for line in text.splitlines())
  22. # break multi-headlines into a line each
  23. chunks = (phrase.strip() for line in lines for phrase in line.split("  "))
  24. # drop blank lines
  25. text = '\n'.join(chunk for chunk in chunks if chunk)
  26.  
  27.    
  28. print(text)
  29.  
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement