Advertisement
Guest User

Untitled

a guest
May 2nd, 2016
77
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 3.44 KB | None | 0 0
  1. from urllib import request
  2.  
  3. urls = (
  4. 'http://math.stackexchange.com/election/5?tab=primary',
  5. 'http://serverfault.com/election/5?tab=primary',
  6. 'http://stackoverflow.com/election/7?tab=primary',
  7. )
  8.  
  9. for url in urls:
  10. print('fetching {} ...'.format(url))
  11. request.urlopen(url).read()
  12.  
  13. fetching http://math.stackexchange.com/election/5?tab=primary ...
  14. fetching http://serverfault.com/election/5?tab=primary ...
  15. fetching http://stackoverflow.com/election/7?tab=primary ...
  16. Traceback (most recent call last):
  17. File "examples/t.py", line 11, in <module>
  18. request.urlopen(url).read()
  19. File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 161, in urlopen
  20. return opener.open(url, data, timeout)
  21. File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 469, in open
  22. response = meth(req, response)
  23. File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 579, in http_response
  24. 'http', request, response, code, msg, hdrs)
  25. File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 507, in error
  26. return self._call_chain(*args)
  27. File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 441, in _call_chain
  28. result = func(*args)
  29. File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 587, in http_error_default
  30. raise HTTPError(req.full_url, code, msg, hdrs, fp)
  31. urllib.error.HTTPError: HTTP Error 403: Forbidden
  32.  
  33. from urllib import request
  34.  
  35. urls = (
  36. 'http://math.stackexchange.com/election/5?tab=primary',
  37. 'http://serverfault.com/election/5?tab=primary',
  38. 'http://stackoverflow.com/election/7?tab=primary',
  39. )
  40.  
  41. for url in urls:
  42. print('fetching {} ...'.format(url))
  43. try:
  44. request.urlopen(url).read()
  45. except:
  46. print('got an exception, changing user-agent to urllib3 default')
  47. req = request.Request(url)
  48. req.add_header('User-Agent', 'Python-urllib/3.4')
  49. try:
  50. request.urlopen(req)
  51. except:
  52. print('got another exception, changing user-agent to something else')
  53. req.add_header('User-Agent', 'not-Python-urllib/3.4')
  54. request.urlopen(req)
  55.  
  56. fetching http://math.stackexchange.com/election/5?tab=primary ...
  57. success with url: http://math.stackexchange.com/election/5?tab=primary
  58.  
  59. fetching http://serverfault.com/election/5?tab=primary ...
  60. success with url: http://serverfault.com/election/5?tab=primary
  61.  
  62. fetching http://stackoverflow.com/election/7?tab=primary ...
  63. got an exception, changing user-agent to urllib default
  64. got another exception, changing user-agent to something else
  65. success with url: http://stackoverflow.com/election/7?tab=primary
  66.  
  67. import requests
  68.  
  69. urls = (
  70. 'http://math.stackexchange.com/election/5?tab=primary',
  71. 'http://serverfault.com/election/5?tab=primary',
  72. 'http://stackoverflow.com/election/7?tab=primary',
  73. )
  74.  
  75. for url in urls:
  76. print('fetching {} ...'.format(url))
  77. data = requests.get(url)
  78.  
  79. import requests
  80.  
  81. urls = (
  82. 'http://math.stackexchange.com/election/5?tab=primary',
  83. 'http://serverfault.com/election/5?tab=primary',
  84. 'http://stackoverflow.com/election/7?tab=primary',
  85. )
  86. with requests.Session() as session:
  87. for url in urls:
  88. print('fetching {} ...'.format(url))
  89. data = session.get(url)
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement