Guest User

Untitled

a guest
Jan 17th, 2018
89
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 1.38 KB | None | 0 0
  1. URLs = loadit()
  2. start = time.time()
  3. with ThreadPoolExecutor(max_workers=10) as executor:
  4. # start the load operations and mark each future with its URL
  5. future_to_url = {executor.submit(load_url, url): url for url in URLs}
  6. for future in concurrent.futures.as_completed(future_to_url):
  7. url = future_to_url[future]
  8. try:
  9. data = future.result()
  10. print(data.values())
  11. # scraped_data = restaurant_parse(link)
  12. # time.sleep(random.randrange(3, 5))
  13. writeit(outName, data.values())
  14. except Exception as exc:
  15. print('%r generated an exception: %s' % (url, exc))
  16. end = time.time()
  17. print(end - start)
  18.  
  19. ubuntu-dev@ubuntu:~$ lscpu
  20. Architecture: x86_64
  21. CPU op-mode(s): 32-bit, 64-bit
  22. Byte Order: Little Endian
  23. CPU(s): 3
  24. On-line CPU(s) list: 0-2
  25. Thread(s) per core: 1
  26. Core(s) per socket: 3
  27. Socket(s): 1
  28. NUMA node(s): 1
  29. Vendor ID: GenuineIntel
  30. CPU family: 6
  31. Model: 61
  32. Model name: Intel(R) Core(TM) i7-5557U CPU @ 3.10GHz
  33. Stepping: 4
  34. CPU MHz: 3100.000
  35. BogoMIPS: 6200.00
  36. Hypervisor vendor: KVM
  37. Virtualization type: full
  38. L1d cache: 32K
  39. L1i cache: 32K
  40. L2 cache: 256K
  41. L3 cache: 4096K
  42. NUMA node0 CPU(s): 0-2
Add Comment
Please, Sign In to add comment