Advertisement
Guest User

asd

a guest
Jul 7th, 2011
99
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Python 1.21 KB | None | 0 0
  1. Here is the pooling:
  2.  
  3. for host in ri.get_next_most_crowded_host():
  4.     results = pool.apply_async(self.__get_host_urls, [ri])
  5.  
  6. And the exception:
  7.  
  8. Exception in thread Thread-2:
  9. Traceback (most recent call last):
  10.   File "/usr/lib/python3.2/threading.py", line 736, in _bootstrap_inner
  11.     self.run()
  12.   File "/usr/lib/python3.2/threading.py", line 689, in run
  13.     self._target(*self._args, **self._kwargs)
  14.   File "/usr/lib/python3.2/multiprocessing/pool.py", line 338, in _handle_tasks
  15.     put(task)
  16. _pickle.PicklingError: Can't pickle <class 'method'>: attribute lookup builtins.method failed
  17.  
  18. Here is the functions trying to be called:
  19. ...
  20. def __get_host_urls(self, ri):
  21.    pool = Comm.ThreadPool(self.poolsize)
  22.    sp = Comm.SocketPool(size=self.poolsize)
  23.    results = []
  24.    for lpi in ri.links[host]:
  25.        pool.add_task(self.__get_url_by_sp, host, self.count, sp, lpi, results)
  26.        self.count += 1
  27.    pool.wait_completion()
  28.    return results
  29.  
  30. def __get_url_by_sp(self, host, nr, sp, lpi, resultlist):
  31.    try:
  32.        resp = sp.urlopen(nr, host, lpi.url)
  33.        lpi.set_info('html', resp)
  34.        resultlist.append(lpi)
  35.    except Exception as e:
  36.        self.logger.exception(e)
  37. ...
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement