Advertisement
Guest User

ipython read files in cluster

a guest
Jul 24th, 2016
58
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 0.87 KB | None | 0 0
  1. fileName = '/mnt/eftdata2/pos_ptlf_201309%(i)02d.txt'
  2. names = [fileName % {'i':i} for i in range(5,12)]
  3. strDate = '2013-09-%(j)02d'
  4. dates = [strDate %{'j':j} for j in range(5,12)]
  5.  
  6.  
  7. # Connect to the IPython cluster
  8. # set 7 cluster engines, 1 cluster preprocess 1 daily data (total 7 daily dataset per week)
  9. c = Client(profile='titaClusters')
  10. c[:].run('chunk.py')
  11. c[:].block = True
  12. numC = len(c)
  13.  
  14. dat01 = c[0].apply_sync(extract_data,names[0])
  15. clear_clients(c[0])
  16. dat02 = c[1].apply_sync(extract_data,names[1])
  17. clear_clients(c[1])
  18. dat03 = c[2].apply_sync(extract_data,names[2])
  19. clear_clients(c[2])
  20. dat04 = c[3].apply_sync(extract_data,names[3])
  21. clear_clients(c[3])
  22. dat05 = c[4].apply_sync(extract_data,names[4])
  23. clear_clients(c[4])
  24. dat06 = c[5].apply_sync(extract_data,names[5])
  25. clear_clients(c[5])
  26. dat07 = c[6].apply_sync(extract_data,names[6])
  27. clear_clients(c[6])
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement