Pastebin launched a little side project called VERYVIRAL.com, check it out ;-) Want more features on Pastebin? Sign Up, it's FREE!
Guest

asd

By: a guest on Jul 7th, 2011  |  syntax: Python  |  size: 1.21 KB  |  views: 26  |  expires: Never
download  |  raw  |  embed  |  report abuse  |  print
Text below is selected. Please press Ctrl+C to copy to your clipboard. (⌘+C on Mac)
  1. Here is the pooling:
  2.  
  3. for host in ri.get_next_most_crowded_host():
  4.     results = pool.apply_async(self.__get_host_urls, [ri])
  5.  
  6. And the exception:
  7.  
  8. Exception in thread Thread-2:
  9. Traceback (most recent call last):
  10.   File "/usr/lib/python3.2/threading.py", line 736, in _bootstrap_inner
  11.     self.run()
  12.   File "/usr/lib/python3.2/threading.py", line 689, in run
  13.     self._target(*self._args, **self._kwargs)
  14.   File "/usr/lib/python3.2/multiprocessing/pool.py", line 338, in _handle_tasks
  15.     put(task)
  16. _pickle.PicklingError: Can't pickle <class 'method'>: attribute lookup builtins.method failed
  17.  
  18. Here is the functions trying to be called:
  19. ...
  20. def __get_host_urls(self, ri):
  21.    pool = Comm.ThreadPool(self.poolsize)
  22.    sp = Comm.SocketPool(size=self.poolsize)
  23.    results = []
  24.    for lpi in ri.links[host]:
  25.        pool.add_task(self.__get_url_by_sp, host, self.count, sp, lpi, results)
  26.        self.count += 1
  27.    pool.wait_completion()
  28.    return results
  29.  
  30. def __get_url_by_sp(self, host, nr, sp, lpi, resultlist):
  31.    try:
  32.        resp = sp.urlopen(nr, host, lpi.url)
  33.        lpi.set_info('html', resp)
  34.        resultlist.append(lpi)
  35.    except Exception as e:
  36.        self.logger.exception(e)
  37. ...