API tools faq
paste
CAROJASQ

3 predictors GPU log

CAROJASQ
Oct 23rd, 2018
56
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 14.56 KB | None | 0 0
raw download clone embed print report
  1. Running calculations on GPU
  2. Doing regressions for 1 predictors (54) regressions
  3. Number of possible combinations are 54, batch size is 2447000
  4. Generating 2447000 combs for this batch
  5. Processing from 0 to 54 regressions in this batch
  6. For this batch 0 models are invalid
  7. Doing regressions for 2 predictors (1431) regressions
  8. Number of possible combinations are 1431, batch size is 1819000
  9. Generating 1819000 combs for this batch
  10. Processing from 0 to 1431 regressions in this batch
  11. For this batch 0 models are invalid
  12. Doing regressions for 3 predictors (24804) regressions
  13. Number of possible combinations are 24804, batch size is 1441000
  14. Generating 1441000 combs for this batch
  15. Processing from 0 to 24804 regressions in this batch
  16. For this batch 0 models are invalid
  17. 26289 Regressions has been done, tt 1.91709589958, te: 0.336488008499
  18. Using GPU to do regressions took 2.74310398102
  19. Timer unit: 1e-06 s
  20.  
  21. Total time: 2.6961 s
  22. File: /home/nvera/Cris/HMMMR/src/batched_regression.py
  23. Function: find_best_models_gpu at line 182
  24.  
  25. Line # Hits Time Per Hit % Time Line Contents
  26. ==============================================================
  27. 182 def find_best_models_gpu(file_name='../TestData/Y=2X1+3X2+4X3+5_with_shitty.csv', min_predictors=1, max_predictors=4, metric=None, window=None, handle=None, max_batch_size=None, **kwargs):
  28. 183 """
  29. 184
  30. 185 :param file_name: File name containing data, the format is the following
  31. 186 Columns: Contains data for N-2 predictors, 1 column full of 1s and 1 column with outcome data
  32. 187 Columns 1 to N-2 contains predictors data
  33. 188 The N-1 column is always full of 1s (due the constant on the model)
  34. 189 The N column contains Y data
  35. 190 Rows: The first row contains the name of the predictor
  36. 191 The next rows contains the observations (They need to be real values, no empty/nans are allowed
  37. 192 :param max_predictors: Max numbers of predictors to test in the regression. Should b N-2 at max
  38. 193 :return: Ordered array (by RMSE) of tuples containing (predictors_combination, RMSE)
  39. 194 """
  40. 195 1 2.0 2.0 0.0 tt = te = 0 # total time
  41. 196 1 145957.0 145957.0 5.4 handle = handle if handle else cublas.cublasCreate()
  42. 197 1 12844.0 12844.0 0.5 XY = np.loadtxt(open(file_name, "rb"), delimiter=",", skiprows=1, dtype=np.float32)
  43. 198 1 120.0 120.0 0.0 X = np.delete(XY, XY.shape[1] - 1, 1)
  44. 199 1 3.0 3.0 0.0 Y = XY[:, -1]
  45. 200 1 1.0 1.0 0.0 combs_rmse = None
  46. 201 1 1.0 1.0 0.0 done_regressions = 0
  47. 202 1 15.0 15.0 0.0 with open(file_name, 'rb') as f:
  48. 203 1 53.0 53.0 0.0 col_names = np.array(f.readline().strip().split(','))
  49. 204 4 11.0 2.8 0.0 for n_predictors in range(min_predictors, max_predictors+1):
  50. 205 3 1020.0 340.0 0.0 _print_memory_usage("Initial State: ")
  51. 206 3 945.0 315.0 0.0 max_batch_size = _get_max_batch_size(n_predictors+1, Y.size)
  52. 207 3 65.0 21.7 0.0 iterator = get_combinatorial_iterator(X, n_predictors)
  53. 208 3 11.0 3.7 0.0 index_combinations = get_column_index_combinations(iterator, X, max_batch_size=max_batch_size) # n predictors - 1 constant
  54. 209 3 75.0 25.0 0.0 s_i = ncr(X.shape[1]-1, n_predictors) # Number of possible combinations
  55. 210 3 107.0 35.7 0.0 print "Doing regressions for {} predictors ({}) regressions".format(n_predictors, s_i)
  56. 211 3 38.0 12.7 0.0 print "Number of possible combinations are {}, batch size is {}".format(s_i, max_batch_size)
  57. 212 3 5.0 1.7 0.0 i = 0
  58. 213 6 55196.0 9199.3 2.0 for current_combinations in index_combinations:
  59. 214 3 67.0 22.3 0.0 print "Processing from {} to {} regressions in this batch".format(i, i + len(current_combinations))
  60. 215 3 8.0 2.7 0.0 ss = time()
  61. 216 3 181394.0 60464.7 6.7 Xs = get_X_matrices_from_combinations(X, current_combinations)
  62. 217 3 146398.0 48799.3 5.4 XTs = get_Xt_matrices_from_combinations(X.T, current_combinations)
  63. 218 3 8659.0 2886.3 0.3 YsObs = get_Ys_matrices(Y, len(current_combinations))
  64. 219 3 13.0 4.3 0.0 te += time() - ss
  65. 220 3 4.0 1.3 0.0 ss = time()
  66. 221 3 1917051.0 639017.0 71.1 regression_results = massive_multilineal_regresion(Xs, XTs, YsObs, handle=handle)
  67. 222 3 28.0 9.3 0.0 tt += time() - ss
  68. 223 3 18535.0 6178.3 0.7 regression_results['predictors_combinations'] = np.array(current_combinations, dtype=np.int32)
  69. 224 # If the matrix had not inverse then the model is invalid
  70. 225 3 828.0 276.0 0.0 invalid_models = np.where(regression_results['inv_results'].get() != 0)[0]
  71. 226 3 156.0 52.0 0.0 print "For this batch {} models are invalid".format(len(invalid_models))
  72. 227 # Cleaning invalid model results
  73. 228 3 1372.0 457.3 0.1 regression_results['predictors_combinations'] = np.delete(regression_results['predictors_combinations'], invalid_models, 0)
  74. 229 3 1152.0 384.0 0.0 regression_results['beta_coefficients'] = np.delete(regression_results['beta_coefficients'], invalid_models, 0)
  75. 230 3 326.0 108.7 0.0 regression_results['rmse'] = np.delete(regression_results['rmse'], invalid_models, 0)
  76. 231 3 12836.0 4278.7 0.5 regression_results['ys_sim'] = np.delete(regression_results['ys_sim'], invalid_models, 0)
  77. 232 3 8523.0 2841.0 0.3 regression_results['ys_obs'] = np.delete(regression_results['ys_obs'], invalid_models, 0)
  78. 233 26292 86269.0 3.3 3.2 combinations_cols_names = np.array([col_names[x] for x in regression_results['predictors_combinations']])
  79. 234 3 7.0 2.3 0.0 if combs_rmse is None:
  80. 235 1 154.0 154.0 0.0 combs_rmse = np.array(list(zip(combinations_cols_names, regression_results['rmse'])))
  81. 236 else:
  82. 237 2 73196.0 36598.0 2.7 combs_rmse = np.vstack((combs_rmse, np.array(list(zip(combinations_cols_names, regression_results['rmse'])))))
  83. 238 3 10.0 3.3 0.0 i += len(current_combinations)
  84. 239 3 6.0 2.0 0.0 done_regressions += len(current_combinations)
  85. 240 1 44.0 44.0 0.0 print "{} Regressions has been done, tt {}, te: {}".format(done_regressions, tt, te)
  86. 241 1 22590.0 22590.0 0.8 ordered_combs = combs_rmse[combs_rmse[:, 1].argsort()]
  87. 242 1 1.0 1.0 0.0 return ordered_combs
  88.  
  89. Total time: 0 s
  90. File: /home/nvera/Cris/HMMMR/src/numpy_multiple_regression.py
  91. Function: find_best_models_cpu at line 78
  92.  
  93. Line # Hits Time Per Hit % Time Line Contents
  94. ==============================================================
  95. 78 def find_best_models_cpu(file_name='../TestData/Y=2X1+3X2+4X3+5_with_shitty.csv', min_predictors=1, max_predictors=4, handle=None, **kwargs):
  96. 79 """
  97. 80
  98. 81 :param file_name: File name containing data, the format is the following
  99. 82 Columns: Contains data for N-2 predictors, 1 column full of 1s and 1 column with outcome data
  100. 83 Columns 1 to N-2 contains predictors data
  101. 84 The N-1 column is always full of 1s (due the constant on the model)
  102. 85 The N column contains Y data
  103. 86 Rows: The first row contains the name of the predictor
  104. 87 The next rows contains the observations (They need to be real values, no empty/nans are allowed
  105. 88 :param max_predictors: Max numbers of predictors to test in the regression. Should b N-2 at max
  106. 89 :return: Ordered array (by RMSE) of tuples containing (predictors_combination, RMSE)
  107. 90 """
  108. 91 XY = np.loadtxt(open(file_name, "rb"), delimiter=",", skiprows=1, dtype=np.float32)
  109. 92 X = np.delete(XY, XY.shape[1] - 1, 1)
  110. 93 Y = XY[:, -1]
  111. 94 combs_rmse = None
  112. 95 done_regressions = 0
  113. 96 invalid_regressions = 0
  114. 97 with open(file_name, 'rb') as f:
  115. 98 col_names = np.array(f.readline().strip().split(','))
  116. 99 for n_predictors in range(min_predictors, max_predictors+1):
  117. 100 index_combinations = get_column_index_combinations(X, n_predictors) # n predictors - 1 constant
  118. 101 s_i = ncr(X.shape[1]-1, n_predictors) # Number of possible combinations
  119. 102 print "Doing regressions for {} predictors ({}) regressions".format(n_predictors, s_i)
  120. 103 for comb in index_combinations:
  121. 104 try:
  122. 105 X1, X1t = get_X_Xt_matrix(X, comb)
  123. 106 regression = numpy_regression(X1, X1t, Y)
  124. 107 combinations_cols_names = np.array([col_names[x] for x in comb])
  125. 108 result = np.array([[combinations_cols_names, regression['metric']]])
  126. 109
  127. 110 if combs_rmse is None:
  128. 111 combs_rmse = result
  129. 112 else:
  130. 113 combs_rmse = np.vstack([combs_rmse, result])
  131. 114 except:
  132. 115 invalid_regressions += 1
  133. 116 done_regressions += s_i
  134. 117 print "{} Regressions has been done, {} invalid".format(done_regressions, invalid_regressions)
  135. 118 ordered_combs = combs_rmse[combs_rmse[:, 1].argsort()]
  136. 119 return ordered_combs
  137.  
  138. Total time: 5.09919 s
  139. File: massive_multilinear_regresions.py
  140. Function: perform_regressions at line 53
  141.  
  142. Line # Hits Time Per Hit % Time Line Contents
  143. ==============================================================
  144. 53 @do_profile(follow=[find_best_models_gpu, find_best_models_cpu])
  145. 54 def perform_regressions():
  146. 55 1 3.0 3.0 0.0 start_time = time()
  147. 56 1 3262.0 3262.0 0.1 input_file, window, max_predictors, min_predictors, metric, output_file, device, max_batch_size = parse_arguments()
  148. 57 1 1.0 1.0 0.0 if device == "gpu":
  149. 58 1 39.0 39.0 0.0 print "Running calculations on GPU"
  150. 59 1 2739787.0 2739787.0 53.7 ordered_combs = find_best_models_gpu(file_name=input_file, min_predictors=min_predictors, max_predictors=max_predictors, metric=metric, window=window, max_batch_size=max_batch_size)
  151. 60 1 32.0 32.0 0.0 print "Using GPU to do regressions took {}".format(time() - start_time)
  152. 61 elif device == "cpu":
  153. 62 ordered_combs = find_best_models_cpu(file_name=input_file, min_predictors=min_predictors, max_predictors=max_predictors, metric=metric, window=window, max_batch_size=max_batch_size)
  154. 63 1 1063.0 1063.0 0.0 df = pd.DataFrame(ordered_combs)
  155. 64 1 2354999.0 2354999.0 46.2 df.to_csv("/tmp/{}".format(output_file))
  156.  
  157. python massive_multilinear_regresions.py -i -mp 3 -np 1 -d gpu 6,38s user 7,71s system 84% cpu 16,650 total
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!