daily pastebin goal
59%
SHARE
TWEET

Untitled

a guest Aug 19th, 2018 52 Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. Read txt file with multi-threaded in python
  2. import os.path
  3. from multiprocessing import Pool
  4. import sys
  5. import time
  6.  
  7. def process_file(name):
  8.     ''' Process one file: count number of lines and words '''
  9.     linecount=0
  10.     wordcount=0
  11.     with open(name, 'r') as inp:
  12.         for line in inp:
  13.             linecount+=1
  14.             wordcount+=len(line.split(' '))
  15.  
  16.     return name, linecount, wordcount
  17.  
  18. def process_files_parallel(arg, dirname, names):
  19.     ''' Process each file in parallel via Poll.map() '''
  20.     pool=Pool()
  21.     results=pool.map(process_file, [os.path.join(dirname, name) for name in names])
  22.  
  23. def process_files(arg, dirname, names):
  24.     ''' Process each file in via map() '''
  25.     results=map(process_file, [os.path.join(dirname, name) for name in names])
  26.  
  27. if __name__ == '__main__':
  28.     start=time.time()
  29.     os.path.walk('input/', process_files, None)
  30.     print "process_files()", time.time()-start
  31.  
  32.     start=time.time()
  33.     os.path.walk('input/', process_files_parallel, None)
  34.     print "process_files_parallel()", time.time()-start
  35.    
  36. $ python process_files.py
  37. process_files() 1.71218085289
  38. process_files_parallel() 1.28905105591
RAW Paste Data
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy. OK, I Understand
 
Top