daily pastebin goal
31%
SHARE
TWEET

Untitled

a guest Aug 19th, 2018 52 Never
Upgrade to PRO!
ENDING IN00days00hours00mins00secs
  1. Read txt file with multi-threaded in python
  2. import os.path
  3. from multiprocessing import Pool
  4. import sys
  5. import time
  6.  
  7. def process_file(name):
  8.     ''' Process one file: count number of lines and words '''
  9.     linecount=0
  10.     wordcount=0
  11.     with open(name, 'r') as inp:
  12.         for line in inp:
  13.             linecount+=1
  14.             wordcount+=len(line.split(' '))
  15.  
  16.     return name, linecount, wordcount
  17.  
  18. def process_files_parallel(arg, dirname, names):
  19.     ''' Process each file in parallel via Poll.map() '''
  20.     pool=Pool()
  21.     results=pool.map(process_file, [os.path.join(dirname, name) for name in names])
  22.  
  23. def process_files(arg, dirname, names):
  24.     ''' Process each file in via map() '''
  25.     results=map(process_file, [os.path.join(dirname, name) for name in names])
  26.  
  27. if __name__ == '__main__':
  28.     start=time.time()
  29.     os.path.walk('input/', process_files, None)
  30.     print "process_files()", time.time()-start
  31.  
  32.     start=time.time()
  33.     os.path.walk('input/', process_files_parallel, None)
  34.     print "process_files_parallel()", time.time()-start
  35.    
  36. $ python process_files.py
  37. process_files() 1.71218085289
  38. process_files_parallel() 1.28905105591
RAW Paste Data
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy. OK, I Understand
 
Top