Advertisement
anjanesh

Using selenium and BeautifulSoup

Jun 28th, 2023
778
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Python 0.93 KB | Source Code | 0 0
  1. from selenium import webdriver
  2. from selenium.webdriver.chrome.options import Options
  3. from bs4 import BeautifulSoup
  4. import time
  5.  
  6. # Configure Selenium to run in headless mode
  7. chrome_options = Options()
  8. chrome_options.add_argument("--headless")
  9.  
  10. # Set up Selenium WebDriver with the configured options
  11. driver = webdriver.Chrome(options=chrome_options)
  12.  
  13. url = "https://adviserinfo.sec.gov/individual/summary/1973578"
  14.  
  15. # Open the URL using Selenium
  16. driver.get(url)
  17.  
  18. time.sleep(5)
  19.  
  20. # Get the page source after dynamic content has loaded
  21. page_source = driver.page_source
  22.  
  23. # Create a BeautifulSoup object to parse the HTML content
  24. soup = BeautifulSoup(page_source, "html.parser")
  25.  
  26. # Find the span element containing the data
  27. span_element = soup.find("span", class_="text-lg sm:text-sm font-semibold")
  28.  
  29. # Extract the data
  30. data = span_element.text.strip() if span_element else None
  31.  
  32. print(data)
  33.  
  34. # Close the browser
  35. driver.quit()
Tags: parse
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement