API tools faq
paste
Advertisement
mike19951208

Untitled

mike19951208
Aug 18th, 2019
392
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 249.17 KB | None | 0 0
raw download clone embed print report
  1. WARNING: Logging before InitGoogleLogging() is written to STDERR
  2. I0819 14:26:04.340821 12127 parallel.cpp:47] P2PManager::Init @ ntu329-ESC8000-G3-Series-Invalid-entry-length-16-Fixed-up-to-11
  3. I0819 14:26:04.808403 12127 caffe.cpp:709] This is NVCaffe 0.17.2 started at Mon Aug 19 14:26:04 2019
  4. I0819 14:26:04.808634 12127 caffe.cpp:711] CuDNN version: 7103
  5. I0819 14:26:04.808640 12127 caffe.cpp:712] CuBLAS version: 9000
  6. I0819 14:26:04.808645 12127 caffe.cpp:713] CUDA version: 9000
  7. I0819 14:26:04.808648 12127 caffe.cpp:714] CUDA driver version: 9000
  8. I0819 14:26:04.808655 12127 caffe.cpp:715] Arguments:
  9. [0]: ./build/tools/caffe
  10. [1]: train
  11. [2]: --solver=examples/mnist/lenet_solver_fp16.prototxt
  12. [3]: -gpu
  13. [4]: all
  14. I0819 14:26:04.811754 12127 caffe.cpp:220] Using GPUs 0, 1, 2, 3, 4, 5, 6, 7
  15. I0819 14:26:04.852030 12127 gpu_memory.cpp:105] GPUMemory::Manager initialized
  16. I0819 14:26:04.853935 12127 gpu_memory.cpp:107] Total memory: 11714691072, Free: 11253448704, dev_info[0]: total=11714691072 free=11253448704
  17. I0819 14:26:04.855993 12127 gpu_memory.cpp:107] Total memory: 11715084288, Free: 11253448704, dev_info[1]: total=11714691072 free=11253448704
  18. I0819 14:26:04.858072 12127 gpu_memory.cpp:107] Total memory: 11715084288, Free: 11253448704, dev_info[2]: total=11714691072 free=11253448704
  19. I0819 14:26:04.860294 12127 gpu_memory.cpp:107] Total memory: 11715084288, Free: 11253448704, dev_info[3]: total=11714691072 free=11253448704
  20. I0819 14:26:04.862391 12127 gpu_memory.cpp:107] Total memory: 11715084288, Free: 11253448704, dev_info[4]: total=11714691072 free=11253448704
  21. I0819 14:26:04.864476 12127 gpu_memory.cpp:107] Total memory: 11715084288, Free: 11253448704, dev_info[5]: total=11714691072 free=11253448704
  22. I0819 14:26:04.866564 12127 gpu_memory.cpp:107] Total memory: 11715084288, Free: 11253448704, dev_info[6]: total=11714691072 free=11253448704
  23. I0819 14:26:04.868671 12127 gpu_memory.cpp:107] Total memory: 11715084288, Free: 11253448704, dev_info[7]: total=11714691072 free=11253448704
  24. I0819 14:26:04.870052 12127 caffe.cpp:227] GPU 0: GeForce GTX 1080 Ti
  25. I0819 14:26:04.871790 12127 caffe.cpp:227] GPU 1: GeForce GTX 1080 Ti
  26. I0819 14:26:04.873723 12127 caffe.cpp:227] GPU 2: GeForce GTX 1080 Ti
  27. I0819 14:26:04.875409 12127 caffe.cpp:227] GPU 3: GeForce GTX 1080 Ti
  28. I0819 14:26:04.877002 12127 caffe.cpp:227] GPU 4: GeForce GTX 1080 Ti
  29. I0819 14:26:04.878690 12127 caffe.cpp:227] GPU 5: GeForce GTX 1080 Ti
  30. I0819 14:26:04.880345 12127 caffe.cpp:227] GPU 6: GeForce GTX 1080 Ti
  31. I0819 14:26:04.881867 12127 caffe.cpp:227] GPU 7: GeForce GTX 1080 Ti
  32. I0819 14:26:04.881983 12127 solver.cpp:40] Solver data type: FLOAT16
  33. I0819 14:26:04.911844 12127 solver.cpp:43] Initializing solver from parameters:
  34. test_iter: 100
  35. test_interval: 500
  36. base_lr: 0.006
  37. display: 100
  38. max_iter: 20000
  39. lr_policy: "inv"
  40. gamma: 0.0001
  41. power: 0.75
  42. momentum: 0.9
  43. weight_decay: 0.0005
  44. solver_mode: GPU
  45. device_id: 0
  46. net: "examples/mnist/lenet_train_test_fp16.prototxt"
  47. train_state {
  48. level: 0
  49. stage: ""
  50. }
  51. solver_data_type: FLOAT16
  52. I0819 14:26:04.912662 12127 solver.cpp:85] Creating training net from net file: examples/mnist/lenet_train_test_fp16.prototxt
  53. I0819 14:26:04.913134 12127 net.cpp:459] The NetState phase (0) differed from the phase (1) specified by a rule in layer mnist
  54. I0819 14:26:04.913177 12127 net.cpp:459] The NetState phase (0) differed from the phase (1) specified by a rule in layer accuracy
  55. I0819 14:26:04.913367 12127 net.cpp:83] Initializing net from parameters:
  56. name: "LeNet-fp16"
  57. state {
  58. phase: TRAIN
  59. level: 0
  60. stage: ""
  61. }
  62. default_forward_type: FLOAT16
  63. default_backward_type: FLOAT16
  64. default_forward_math: FLOAT16
  65. default_backward_math: FLOAT16
  66. global_grad_scale: 1
  67. layer {
  68. name: "mnist"
  69. type: "Data"
  70. top: "data"
  71. top: "label"
  72. include {
  73. phase: TRAIN
  74. }
  75. transform_param {
  76. scale: 0.00390625
  77. }
  78. data_param {
  79. source: "examples/mnist/mnist_train_lmdb"
  80. batch_size: 8
  81. backend: LMDB
  82. }
  83. }
  84. layer {
  85. name: "conv1"
  86. type: "Convolution"
  87. bottom: "data"
  88. top: "conv1"
  89. param {
  90. lr_mult: 1
  91. }
  92. param {
  93. lr_mult: 2
  94. }
  95. convolution_param {
  96. num_output: 20
  97. kernel_size: 5
  98. stride: 1
  99. weight_filler {
  100. type: "gspt_xavier"
  101. }
  102. bias_filler {
  103. type: "gspt_constant"
  104. }
  105. }
  106. }
  107. layer {
  108. name: "pool1"
  109. type: "Pooling"
  110. bottom: "conv1"
  111. top: "pool1"
  112. pooling_param {
  113. pool: MAX
  114. kernel_size: 2
  115. stride: 2
  116. }
  117. }
  118. layer {
  119. name: "conv2"
  120. type: "Convolution"
  121. bottom: "pool1"
  122. top: "conv2"
  123. param {
  124. lr_mult: 1
  125. }
  126. param {
  127. lr_mult: 2
  128. }
  129. convolution_param {
  130. num_output: 50
  131. kernel_size: 5
  132. stride: 1
  133. weight_filler {
  134. type: "gspt_xavier"
  135. }
  136. bias_filler {
  137. type: "gspt_constant"
  138. }
  139. }
  140. }
  141. layer {
  142. name: "pool2"
  143. type: "Pooling"
  144. bottom: "conv2"
  145. top: "pool2"
  146. pooling_param {
  147. pool: MAX
  148. kernel_size: 2
  149. stride: 2
  150. }
  151. }
  152. layer {
  153. name: "ip1"
  154. type: "InnerProduct"
  155. bottom: "pool2"
  156. top: "ip1"
  157. param {
  158. lr_mult: 1
  159. }
  160. param {
  161. lr_mult: 2
  162. }
  163. inner_product_param {
  164. num_output: 500
  165. weight_filler {
  166. type: "gspt_xavier"
  167. }
  168. bias_filler {
  169. type: "gspt_constant"
  170. }
  171. }
  172. }
  173. layer {
  174. name: "relu1"
  175. type: "ReLU"
  176. bottom: "ip1"
  177. top: "ip1"
  178. }
  179. layer {
  180. name: "ip2"
  181. type: "InnerProduct"
  182. bottom: "ip1"
  183. top: "ip2"
  184. param {
  185. lr_mult: 1
  186. }
  187. param {
  188. lr_mult: 2
  189. }
  190. inner_product_param {
  191. num_output: 10
  192. weight_filler {
  193. type: "gspt_xavier"
  194. }
  195. bias_filler {
  196. type: "gspt_constant"
  197. }
  198. }
  199. }
  200. layer {
  201. name: "loss"
  202. type: "SoftmaxWithLoss"
  203. bottom: "ip2"
  204. bottom: "label"
  205. top: "loss"
  206. }
  207. I0819 14:26:04.913514 12127 layer_factory.hpp:172] Creating layer 'mnist' of type 'Data'
  208. I0819 14:26:04.913532 12127 layer_factory.hpp:184] Layer's types are Ftype:FLOAT16 Btype:FLOAT Fmath:FLOAT16 Bmath:FLOAT16
  209. I0819 14:26:04.913703 12127 internal_thread.cpp:18] Starting 1 internal thread(s) on device 0
  210. I0819 14:26:04.914325 12127 net.cpp:202] Created Layer mnist (0)
  211. I0819 14:26:04.914341 12127 net.cpp:544] mnist -> data
  212. I0819 14:26:04.914397 12127 net.cpp:544] mnist -> label
  213. I0819 14:26:04.914441 12127 data_reader.cpp:59] Sample Data Reader threads: 1, out queues: 1, depth: 8
  214. I0819 14:26:04.916122 12127 internal_thread.cpp:18] Starting 1 internal thread(s) on device 0
  215. I0819 14:26:04.916152 12173 blocking_queue.cpp:40] Data layer prefetch queue empty
  216. I0819 14:26:04.917896 12174 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_train_lmdb
  217. I0819 14:26:04.919813 12127 data_layer.cpp:200] [n0.d0.r0] Output data size: 8, 1, 28, 28
  218. I0819 14:26:04.919858 12127 internal_thread.cpp:18] Starting 1 internal thread(s) on device 0
  219. I0819 14:26:04.919941 12127 net.cpp:262] Setting up mnist
  220. I0819 14:26:04.919961 12127 net.cpp:269] TRAIN Top shape for layer 0 'mnist' 8 1 28 28 (6272)
  221. I0819 14:26:04.919975 12127 net.cpp:269] TRAIN Top shape for layer 0 'mnist' 8 (8)
  222. I0819 14:26:04.919986 12127 layer_factory.hpp:172] Creating layer 'conv1' of type 'Convolution'
  223. I0819 14:26:04.919994 12127 layer_factory.hpp:184] Layer's types are Ftype:FLOAT16 Btype:FLOAT16 Fmath:FLOAT16 Bmath:FLOAT16
  224. I0819 14:26:04.921756 12127 net.cpp:202] Created Layer conv1 (1)
  225. I0819 14:26:04.921785 12127 net.cpp:574] conv1 <- data
  226. I0819 14:26:04.921828 12127 net.cpp:544] conv1 -> conv1
  227. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  228. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  229. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  230. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  231. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  232. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  233. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  234. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  235. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  236. in math_function.cpp, line 730, tmp_value = 1048576
  237. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  238. definition of type is half.
  239. gspt r[i] = 0.0625 (Q_frac=24
  240. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  241. In blob.cpp, blob.Q_frac= 24
  242. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  243. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  244. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  245. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  246. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  247. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  248. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  249. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  250. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  251. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  252. in math_function.cpp, line 1227, tmp_value = 0
  253. definition of type is half [FILL ZERO].
  254. gspt r[i] = 0 (Q_frac=22
  255. In blob.cpp, blob.Q_frac= 22
  256. I0819 14:26:05.810344 12127 net.cpp:262] Setting up conv1
  257. I0819 14:26:05.810415 12127 net.cpp:269] TRAIN Top shape for layer 1 'conv1' 8 20 24 24 (92160)
  258. I0819 14:26:05.810488 12127 layer_factory.hpp:172] Creating layer 'pool1' of type 'Pooling'
  259. I0819 14:26:05.810503 12127 layer_factory.hpp:184] Layer's types are Ftype:FLOAT16 Btype:FLOAT16 Fmath:FLOAT16 Bmath:FLOAT16
  260. I0819 14:26:05.810540 12127 net.cpp:202] Created Layer pool1 (2)
  261. I0819 14:26:05.810556 12127 net.cpp:574] pool1 <- conv1
  262. I0819 14:26:05.810569 12127 net.cpp:544] pool1 -> pool1
  263. I0819 14:26:05.810711 12127 net.cpp:262] Setting up pool1
  264. I0819 14:26:05.810725 12127 net.cpp:269] TRAIN Top shape for layer 2 'pool1' 8 20 12 12 (23040)
  265. I0819 14:26:05.810734 12127 layer_factory.hpp:172] Creating layer 'conv2' of type 'Convolution'
  266. I0819 14:26:05.810751 12127 layer_factory.hpp:184] Layer's types are Ftype:FLOAT16 Btype:FLOAT16 Fmath:FLOAT16 Bmath:FLOAT16
  267. I0819 14:26:05.810786 12127 net.cpp:202] Created Layer conv2 (3)
  268. I0819 14:26:05.810794 12127 net.cpp:574] conv2 <- pool1
  269. I0819 14:26:05.810801 12127 net.cpp:544] conv2 -> conv2
  270. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  271. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  272. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  273. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  274. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  275. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  276. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  277. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  278. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  279. in math_function.cpp, line 730, tmp_value = 1048576
  280. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  281. definition of type is half.
  282. gspt r[i] = 0.015625 (Q_frac=26
  283. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  284. In blob.cpp, blob.Q_frac= 26
  285. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  286. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  287. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  288. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  289. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  290. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  291. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  292. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  293. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  294. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  295. in math_function.cpp, line 1227, tmp_value = 0
  296. definition of type is half [FILL ZERO].
  297. gspt r[i] = 0 (Q_frac=22
  298. In blob.cpp, blob.Q_frac= 22
  299. I0819 14:26:05.822500 12127 net.cpp:262] Setting up conv2
  300. I0819 14:26:05.822523 12127 net.cpp:269] TRAIN Top shape for layer 3 'conv2' 8 50 8 8 (25600)
  301. I0819 14:26:05.822582 12127 layer_factory.hpp:172] Creating layer 'pool2' of type 'Pooling'
  302. I0819 14:26:05.822590 12127 layer_factory.hpp:184] Layer's types are Ftype:FLOAT16 Btype:FLOAT16 Fmath:FLOAT16 Bmath:FLOAT16
  303. I0819 14:26:05.822615 12127 net.cpp:202] Created Layer pool2 (4)
  304. I0819 14:26:05.822623 12127 net.cpp:574] pool2 <- conv2
  305. I0819 14:26:05.822631 12127 net.cpp:544] pool2 -> pool2
  306. I0819 14:26:05.822722 12127 net.cpp:262] Setting up pool2
  307. I0819 14:26:05.822737 12127 net.cpp:269] TRAIN Top shape for layer 4 'pool2' 8 50 4 4 (6400)
  308. I0819 14:26:05.822743 12127 layer_factory.hpp:172] Creating layer 'ip1' of type 'InnerProduct'
  309. I0819 14:26:05.822751 12127 layer_factory.hpp:184] Layer's types are Ftype:FLOAT16 Btype:FLOAT16 Fmath:FLOAT16 Bmath:FLOAT16
  310. I0819 14:26:05.822767 12127 net.cpp:202] Created Layer ip1 (5)
  311. I0819 14:26:05.822774 12127 net.cpp:574] ip1 <- pool2
  312. I0819 14:26:05.822782 12127 net.cpp:544] ip1 -> ip1
  313. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  314. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  315. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  316. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  317. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  318. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  319. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  320. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  321. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  322. in math_function.cpp, line 730, tmp_value = 1048576
  323. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  324. definition of type is half.
  325. gspt r[i] = 0.0078125 (Q_frac=27
  326. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  327. In blob.cpp, blob.Q_frac= 27
  328. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  329. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  330. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  331. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  332. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  333. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  334. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  335. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  336. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  337. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  338. in math_function.cpp, line 1227, tmp_value = 0
  339. definition of type is half [FILL ZERO].
  340. gspt r[i] = 0 (Q_frac=22
  341. In blob.cpp, blob.Q_frac= 22
  342. I0819 14:26:05.976035 12127 net.cpp:262] Setting up ip1
  343. I0819 14:26:05.976068 12127 net.cpp:269] TRAIN Top shape for layer 5 'ip1' 8 500 (4000)
  344. I0819 14:26:05.976094 12127 layer_factory.hpp:172] Creating layer 'relu1' of type 'ReLU'
  345. I0819 14:26:05.976104 12127 layer_factory.hpp:184] Layer's types are Ftype:FLOAT16 Btype:FLOAT16 Fmath:FLOAT16 Bmath:FLOAT16
  346. I0819 14:26:05.976135 12127 net.cpp:202] Created Layer relu1 (6)
  347. I0819 14:26:05.976143 12127 net.cpp:574] relu1 <- ip1
  348. I0819 14:26:05.976152 12127 net.cpp:529] relu1 -> ip1 (in-place)
  349. I0819 14:26:05.976186 12127 net.cpp:262] Setting up relu1
  350. I0819 14:26:05.976192 12127 net.cpp:269] TRAIN Top shape for layer 6 'relu1' 8 500 (4000)
  351. I0819 14:26:05.976198 12127 layer_factory.hpp:172] Creating layer 'ip2' of type 'InnerProduct'
  352. I0819 14:26:05.976204 12127 layer_factory.hpp:184] Layer's types are Ftype:FLOAT16 Btype:FLOAT16 Fmath:FLOAT16 Bmath:FLOAT16
  353. I0819 14:26:05.976214 12127 net.cpp:202] Created Layer ip2 (7)
  354. I0819 14:26:05.976220 12127 net.cpp:574] ip2 <- ip1
  355. I0819 14:26:05.976228 12127 net.cpp:544] ip2 -> ip2
  356. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  357. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  358. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  359. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  360. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  361. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  362. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  363. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  364. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  365. in math_function.cpp, line 730, tmp_value = 1048576
  366. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  367. definition of type is half.
  368. gspt r[i] = 0.015625 (Q_frac=26
  369. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  370. In blob.cpp, blob.Q_frac= 26
  371. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  372. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  373. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  374. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  375. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  376. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  377. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  378. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  379. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  380. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  381. in math_function.cpp, line 1227, tmp_value = 0
  382. definition of type is half [FILL ZERO].
  383. gspt r[i] = 0 (Q_frac=22
  384. In blob.cpp, blob.Q_frac= 22
  385. I0819 14:26:05.979912 12127 net.cpp:262] Setting up ip2
  386. I0819 14:26:05.979929 12127 net.cpp:269] TRAIN Top shape for layer 7 'ip2' 8 10 (80)
  387. I0819 14:26:05.979940 12127 layer_factory.hpp:172] Creating layer 'loss' of type 'SoftmaxWithLoss'
  388. I0819 14:26:05.979948 12127 layer_factory.hpp:184] Layer's types are Ftype:FLOAT16 Btype:FLOAT16 Fmath:FLOAT16 Bmath:FLOAT16
  389. I0819 14:26:05.979971 12127 net.cpp:202] Created Layer loss (8)
  390. I0819 14:26:05.979979 12127 net.cpp:574] loss <- ip2
  391. I0819 14:26:05.979984 12127 net.cpp:574] loss <- label
  392. I0819 14:26:05.979992 12127 net.cpp:544] loss -> loss
  393. I0819 14:26:05.980211 12127 net.cpp:262] Setting up loss
  394. I0819 14:26:05.980224 12127 net.cpp:269] TRAIN Top shape for layer 8 'loss' (1)
  395. I0819 14:26:05.980228 12127 net.cpp:273] with loss weight 1
  396. I0819 14:26:05.980237 12127 net.cpp:338] loss needs backward computation.
  397. I0819 14:26:05.980242 12127 net.cpp:338] ip2 needs backward computation.
  398. I0819 14:26:05.980247 12127 net.cpp:338] relu1 needs backward computation.
  399. I0819 14:26:05.980250 12127 net.cpp:338] ip1 needs backward computation.
  400. I0819 14:26:05.980255 12127 net.cpp:338] pool2 needs backward computation.
  401. I0819 14:26:05.980260 12127 net.cpp:338] conv2 needs backward computation.
  402. I0819 14:26:05.980264 12127 net.cpp:338] pool1 needs backward computation.
  403. I0819 14:26:05.980268 12127 net.cpp:338] conv1 needs backward computation.
  404. I0819 14:26:05.980273 12127 net.cpp:340] mnist does not need backward computation.
  405. I0819 14:26:05.980278 12127 net.cpp:382] This network produces output loss
  406. I0819 14:26:05.980299 12127 net.cpp:405] Top memory (TRAIN) required for data: 323124 diff: 335684
  407. I0819 14:26:05.980305 12127 net.cpp:408] Bottom memory (TRAIN) required for data: 323120 diff: 335680
  408. I0819 14:26:05.980309 12127 net.cpp:411] Shared (in-place) memory (TRAIN) by data: 8000 diff: 8000
  409. I0819 14:26:05.980314 12127 net.cpp:414] Parameters memory (TRAIN) required for data: 862160 diff: 862160
  410. I0819 14:26:05.980319 12127 net.cpp:417] Parameters shared memory (TRAIN) by data: 0 diff: 0
  411. I0819 14:26:05.980342 12127 net.cpp:423] Network initialization done.
  412. I0819 14:26:05.980953 12127 solver.cpp:174] Creating test net (#0) specified by net file: examples/mnist/lenet_train_test_fp16.prototxt
  413. I0819 14:26:05.980998 12127 net.cpp:459] The NetState phase (1) differed from the phase (0) specified by a rule in layer mnist
  414. W0819 14:26:05.981026 12127 parallel.cpp:319] Batch size must be divisible by the number of solvers (GPUs): it's been adjusted from 100 to 104
  415. I0819 14:26:05.981329 12127 net.cpp:83] Initializing net from parameters:
  416. name: "LeNet-fp16"
  417. state {
  418. phase: TEST
  419. }
  420. default_forward_type: FLOAT16
  421. default_backward_type: FLOAT16
  422. default_forward_math: FLOAT16
  423. default_backward_math: FLOAT16
  424. global_grad_scale: 1
  425. layer {
  426. name: "mnist"
  427. type: "Data"
  428. top: "data"
  429. top: "label"
  430. include {
  431. phase: TEST
  432. }
  433. transform_param {
  434. scale: 0.00390625
  435. }
  436. data_param {
  437. source: "examples/mnist/mnist_test_lmdb"
  438. batch_size: 13
  439. backend: LMDB
  440. }
  441. }
  442. layer {
  443. name: "conv1"
  444. type: "Convolution"
  445. bottom: "data"
  446. top: "conv1"
  447. param {
  448. lr_mult: 1
  449. }
  450. param {
  451. lr_mult: 2
  452. }
  453. convolution_param {
  454. num_output: 20
  455. kernel_size: 5
  456. stride: 1
  457. weight_filler {
  458. type: "gspt_xavier"
  459. }
  460. bias_filler {
  461. type: "gspt_constant"
  462. }
  463. }
  464. }
  465. layer {
  466. name: "pool1"
  467. type: "Pooling"
  468. bottom: "conv1"
  469. top: "pool1"
  470. pooling_param {
  471. pool: MAX
  472. kernel_size: 2
  473. stride: 2
  474. }
  475. }
  476. layer {
  477. name: "conv2"
  478. type: "Convolution"
  479. bottom: "pool1"
  480. top: "conv2"
  481. param {
  482. lr_mult: 1
  483. }
  484. param {
  485. lr_mult: 2
  486. }
  487. convolution_param {
  488. num_output: 50
  489. kernel_size: 5
  490. stride: 1
  491. weight_filler {
  492. type: "gspt_xavier"
  493. }
  494. bias_filler {
  495. type: "gspt_constant"
  496. }
  497. }
  498. }
  499. layer {
  500. name: "pool2"
  501. type: "Pooling"
  502. bottom: "conv2"
  503. top: "pool2"
  504. pooling_param {
  505. pool: MAX
  506. kernel_size: 2
  507. stride: 2
  508. }
  509. }
  510. layer {
  511. name: "ip1"
  512. type: "InnerProduct"
  513. bottom: "pool2"
  514. top: "ip1"
  515. param {
  516. lr_mult: 1
  517. }
  518. param {
  519. lr_mult: 2
  520. }
  521. inner_product_param {
  522. num_output: 500
  523. weight_filler {
  524. type: "gspt_xavier"
  525. }
  526. bias_filler {
  527. type: "gspt_constant"
  528. }
  529. }
  530. }
  531. layer {
  532. name: "relu1"
  533. type: "ReLU"
  534. bottom: "ip1"
  535. top: "ip1"
  536. }
  537. layer {
  538. name: "ip2"
  539. type: "InnerProduct"
  540. bottom: "ip1"
  541. top: "ip2"
  542. param {
  543. lr_mult: 1
  544. }
  545. param {
  546. lr_mult: 2
  547. }
  548. inner_product_param {
  549. num_output: 10
  550. weight_filler {
  551. type: "gspt_xavier"
  552. }
  553. bias_filler {
  554. type: "gspt_constant"
  555. }
  556. }
  557. }
  558. layer {
  559. name: "accuracy"
  560. type: "Accuracy"
  561. bottom: "ip2"
  562. bottom: "label"
  563. top: "accuracy"
  564. include {
  565. phase: TEST
  566. }
  567. }
  568. layer {
  569. name: "loss"
  570. type: "SoftmaxWithLoss"
  571. bottom: "ip2"
  572. bottom: "label"
  573. top: "loss"
  574. }
  575. I0819 14:26:05.981458 12127 layer_factory.hpp:172] Creating layer 'mnist' of type 'Data'
  576. I0819 14:26:05.981469 12127 layer_factory.hpp:184] Layer's types are Ftype:FLOAT16 Btype:FLOAT Fmath:FLOAT16 Bmath:FLOAT16
  577. I0819 14:26:05.981510 12127 internal_thread.cpp:18] Starting 1 internal thread(s) on device 0
  578. I0819 14:26:05.981616 12127 net.cpp:202] Created Layer mnist (0)
  579. I0819 14:26:05.981624 12127 net.cpp:544] mnist -> data
  580. I0819 14:26:05.981634 12127 net.cpp:544] mnist -> label
  581. I0819 14:26:05.981652 12127 data_reader.cpp:59] Data Reader threads: 1, out queues: 1, depth: 13
  582. I0819 14:26:05.981688 12127 internal_thread.cpp:18] Starting 1 internal thread(s) on device 0
  583. I0819 14:26:05.985260 12177 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_test_lmdb
  584. I0819 14:26:05.985519 12127 data_layer.cpp:200] (n0.d0.r0) Output data size: 13, 1, 28, 28
  585. I0819 14:26:05.985556 12127 internal_thread.cpp:18] Starting 1 internal thread(s) on device 0
  586. I0819 14:26:05.986106 12127 net.cpp:262] Setting up mnist
  587. I0819 14:26:05.986117 12127 net.cpp:269] TEST Top shape for layer 0 'mnist' 13 1 28 28 (10192)
  588. I0819 14:26:05.986125 12127 net.cpp:269] TEST Top shape for layer 0 'mnist' 13 (13)
  589. I0819 14:26:05.986131 12127 layer_factory.hpp:172] Creating layer 'label_mnist_1_split' of type 'Split'
  590. I0819 14:26:05.986137 12127 layer_factory.hpp:184] Layer's types are Ftype:FLOAT16 Btype:FLOAT16 Fmath:FLOAT16 Bmath:FLOAT16
  591. I0819 14:26:05.986166 12127 net.cpp:202] Created Layer label_mnist_1_split (1)
  592. I0819 14:26:05.986171 12127 net.cpp:574] label_mnist_1_split <- label
  593. I0819 14:26:05.986181 12127 net.cpp:544] label_mnist_1_split -> label_mnist_1_split_0
  594. I0819 14:26:05.986191 12127 net.cpp:544] label_mnist_1_split -> label_mnist_1_split_1
  595. I0819 14:26:05.987752 12178 data_layer.cpp:105] (n0.d0.r0) Parser threads: 1
  596. I0819 14:26:05.987764 12127 net.cpp:262] Setting up label_mnist_1_split
  597. I0819 14:26:05.987776 12178 data_layer.cpp:107] (n0.d0.r0) Transformer threads: 1
  598. I0819 14:26:05.987787 12127 net.cpp:269] TEST Top shape for layer 1 'label_mnist_1_split' 13 (13)
  599. I0819 14:26:05.987794 12127 net.cpp:269] TEST Top shape for layer 1 'label_mnist_1_split' 13 (13)
  600. I0819 14:26:05.987802 12127 layer_factory.hpp:172] Creating layer 'conv1' of type 'Convolution'
  601. I0819 14:26:05.987808 12127 layer_factory.hpp:184] Layer's types are Ftype:FLOAT16 Btype:FLOAT16 Fmath:FLOAT16 Bmath:FLOAT16
  602. I0819 14:26:05.987841 12127 net.cpp:202] Created Layer conv1 (2)
  603. I0819 14:26:05.987848 12127 net.cpp:574] conv1 <- data
  604. I0819 14:26:05.987854 12127 net.cpp:544] conv1 -> conv1
  605. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  606. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  607. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  608. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  609. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  610. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  611. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  612. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  613. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  614. in math_function.cpp, line 730, tmp_value = 1048576
  615. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  616. definition of type is half.
  617. gspt r[i] = 0.0625 (Q_frac=24
  618. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  619. In blob.cpp, blob.Q_frac= 24
  620. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  621. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  622. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  623. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  624. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  625. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  626. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  627. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  628. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  629. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  630. in math_function.cpp, line 1227, tmp_value = 0
  631. definition of type is half [FILL ZERO].
  632. gspt r[i] = 0 (Q_frac=22
  633. In blob.cpp, blob.Q_frac= 22
  634. I0819 14:26:05.990432 12127 net.cpp:262] Setting up conv1
  635. I0819 14:26:05.990456 12127 net.cpp:269] TEST Top shape for layer 2 'conv1' 13 20 24 24 (149760)
  636. I0819 14:26:05.990481 12127 layer_factory.hpp:172] Creating layer 'pool1' of type 'Pooling'
  637. I0819 14:26:05.990489 12127 layer_factory.hpp:184] Layer's types are Ftype:FLOAT16 Btype:FLOAT16 Fmath:FLOAT16 Bmath:FLOAT16
  638. I0819 14:26:05.990506 12127 net.cpp:202] Created Layer pool1 (3)
  639. I0819 14:26:05.990512 12127 net.cpp:574] pool1 <- conv1
  640. I0819 14:26:05.990520 12127 net.cpp:544] pool1 -> pool1
  641. I0819 14:26:05.990665 12127 net.cpp:262] Setting up pool1
  642. I0819 14:26:05.990681 12127 net.cpp:269] TEST Top shape for layer 3 'pool1' 13 20 12 12 (37440)
  643. I0819 14:26:05.990689 12127 layer_factory.hpp:172] Creating layer 'conv2' of type 'Convolution'
  644. I0819 14:26:05.990694 12127 layer_factory.hpp:184] Layer's types are Ftype:FLOAT16 Btype:FLOAT16 Fmath:FLOAT16 Bmath:FLOAT16
  645. I0819 14:26:05.990754 12127 net.cpp:202] Created Layer conv2 (4)
  646. I0819 14:26:05.990761 12127 net.cpp:574] conv2 <- pool1
  647. I0819 14:26:05.990767 12127 net.cpp:544] conv2 -> conv2
  648. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  649. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  650. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  651. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  652. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  653. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  654. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  655. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  656. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  657. in math_function.cpp, line 730, tmp_value = 1048576
  658. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  659. definition of type is half.
  660. gspt r[i] = 0.015625 (Q_frac=26
  661. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  662. In blob.cpp, blob.Q_frac= 26
  663. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  664. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  665. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  666. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  667. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  668. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  669. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  670. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  671. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  672. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  673. in math_function.cpp, line 1227, tmp_value = 0
  674. definition of type is half [FILL ZERO].
  675. gspt r[i] = 0 (Q_frac=22
  676. In blob.cpp, blob.Q_frac= 22
  677. I0819 14:26:05.999617 12127 net.cpp:262] Setting up conv2
  678. I0819 14:26:05.999635 12127 net.cpp:269] TEST Top shape for layer 4 'conv2' 13 50 8 8 (41600)
  679. I0819 14:26:05.999650 12127 layer_factory.hpp:172] Creating layer 'pool2' of type 'Pooling'
  680. I0819 14:26:05.999656 12127 layer_factory.hpp:184] Layer's types are Ftype:FLOAT16 Btype:FLOAT16 Fmath:FLOAT16 Bmath:FLOAT16
  681. I0819 14:26:05.999672 12127 net.cpp:202] Created Layer pool2 (5)
  682. I0819 14:26:05.999680 12127 net.cpp:574] pool2 <- conv2
  683. I0819 14:26:05.999686 12127 net.cpp:544] pool2 -> pool2
  684. I0819 14:26:05.999768 12127 net.cpp:262] Setting up pool2
  685. I0819 14:26:05.999778 12127 net.cpp:269] TEST Top shape for layer 5 'pool2' 13 50 4 4 (10400)
  686. I0819 14:26:05.999783 12127 layer_factory.hpp:172] Creating layer 'ip1' of type 'InnerProduct'
  687. I0819 14:26:05.999788 12127 layer_factory.hpp:184] Layer's types are Ftype:FLOAT16 Btype:FLOAT16 Fmath:FLOAT16 Bmath:FLOAT16
  688. I0819 14:26:05.999801 12127 net.cpp:202] Created Layer ip1 (6)
  689. I0819 14:26:05.999806 12127 net.cpp:574] ip1 <- pool2
  690. I0819 14:26:05.999812 12127 net.cpp:544] ip1 -> ip1
  691. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  692. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  693. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  694. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  695. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  696. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  697. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  698. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  699. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  700. in math_function.cpp, line 730, tmp_value = 1048576
  701. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  702. definition of type is half.
  703. gspt r[i] = 0.0078125 (Q_frac=27
  704. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  705. In blob.cpp, blob.Q_frac= 27
  706. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  707. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  708. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  709. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  710. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  711. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  712. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  713. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  714. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  715. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  716. in math_function.cpp, line 1227, tmp_value = 0
  717. definition of type is half [FILL ZERO].
  718. gspt r[i] = 0 (Q_frac=22
  719. In blob.cpp, blob.Q_frac= 22
  720. I0819 14:26:06.132076 12127 net.cpp:262] Setting up ip1
  721. I0819 14:26:06.132113 12127 net.cpp:269] TEST Top shape for layer 6 'ip1' 13 500 (6500)
  722. I0819 14:26:06.132139 12127 layer_factory.hpp:172] Creating layer 'relu1' of type 'ReLU'
  723. I0819 14:26:06.132150 12127 layer_factory.hpp:184] Layer's types are Ftype:FLOAT16 Btype:FLOAT16 Fmath:FLOAT16 Bmath:FLOAT16
  724. I0819 14:26:06.132164 12127 net.cpp:202] Created Layer relu1 (7)
  725. I0819 14:26:06.132171 12127 net.cpp:574] relu1 <- ip1
  726. I0819 14:26:06.132180 12127 net.cpp:529] relu1 -> ip1 (in-place)
  727. I0819 14:26:06.132201 12127 net.cpp:262] Setting up relu1
  728. I0819 14:26:06.132208 12127 net.cpp:269] TEST Top shape for layer 7 'relu1' 13 500 (6500)
  729. I0819 14:26:06.132233 12127 layer_factory.hpp:172] Creating layer 'ip2' of type 'InnerProduct'
  730. I0819 14:26:06.132238 12127 layer_factory.hpp:184] Layer's types are Ftype:FLOAT16 Btype:FLOAT16 Fmath:FLOAT16 Bmath:FLOAT16
  731. I0819 14:26:06.132254 12127 net.cpp:202] Created Layer ip2 (8)
  732. I0819 14:26:06.132261 12127 net.cpp:574] ip2 <- ip1
  733. I0819 14:26:06.132266 12127 net.cpp:544] ip2 -> ip2
  734. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  735. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  736. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  737. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  738. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  739. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  740. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  741. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  742. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  743. in math_function.cpp, line 730, tmp_value = 1048576
  744. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  745. definition of type is half.
  746. gspt r[i] = 0.015625 (Q_frac=26
  747. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  748. In blob.cpp, blob.Q_frac= 26
  749. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  750. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  751. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  752. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  753. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  754. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  755. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  756. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  757. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  758. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  759. in math_function.cpp, line 1227, tmp_value = 0
  760. definition of type is half [FILL ZERO].
  761. gspt r[i] = 0 (Q_frac=22
  762. In blob.cpp, blob.Q_frac= 22
  763. I0819 14:26:06.134950 12127 net.cpp:262] Setting up ip2
  764. I0819 14:26:06.134966 12127 net.cpp:269] TEST Top shape for layer 8 'ip2' 13 10 (130)
  765. I0819 14:26:06.134982 12127 layer_factory.hpp:172] Creating layer 'ip2_ip2_0_split' of type 'Split'
  766. I0819 14:26:06.134989 12127 layer_factory.hpp:184] Layer's types are Ftype:FLOAT16 Btype:FLOAT16 Fmath:FLOAT16 Bmath:FLOAT16
  767. I0819 14:26:06.134999 12127 net.cpp:202] Created Layer ip2_ip2_0_split (9)
  768. I0819 14:26:06.135005 12127 net.cpp:574] ip2_ip2_0_split <- ip2
  769. I0819 14:26:06.135011 12127 net.cpp:544] ip2_ip2_0_split -> ip2_ip2_0_split_0
  770. I0819 14:26:06.135020 12127 net.cpp:544] ip2_ip2_0_split -> ip2_ip2_0_split_1
  771. I0819 14:26:06.135076 12127 net.cpp:262] Setting up ip2_ip2_0_split
  772. I0819 14:26:06.135087 12127 net.cpp:269] TEST Top shape for layer 9 'ip2_ip2_0_split' 13 10 (130)
  773. I0819 14:26:06.135092 12127 net.cpp:269] TEST Top shape for layer 9 'ip2_ip2_0_split' 13 10 (130)
  774. I0819 14:26:06.135097 12127 layer_factory.hpp:172] Creating layer 'accuracy' of type 'Accuracy'
  775. I0819 14:26:06.135102 12127 layer_factory.hpp:184] Layer's types are Ftype:FLOAT16 Btype:FLOAT16 Fmath:FLOAT16 Bmath:FLOAT16
  776. I0819 14:26:06.135118 12127 net.cpp:202] Created Layer accuracy (10)
  777. I0819 14:26:06.135124 12127 net.cpp:574] accuracy <- ip2_ip2_0_split_0
  778. I0819 14:26:06.135130 12127 net.cpp:574] accuracy <- label_mnist_1_split_0
  779. I0819 14:26:06.135136 12127 net.cpp:544] accuracy -> accuracy
  780. I0819 14:26:06.135172 12127 net.cpp:262] Setting up accuracy
  781. I0819 14:26:06.135183 12127 net.cpp:269] TEST Top shape for layer 10 'accuracy' (1)
  782. I0819 14:26:06.135188 12127 layer_factory.hpp:172] Creating layer 'loss' of type 'SoftmaxWithLoss'
  783. I0819 14:26:06.135193 12127 layer_factory.hpp:184] Layer's types are Ftype:FLOAT16 Btype:FLOAT16 Fmath:FLOAT16 Bmath:FLOAT16
  784. I0819 14:26:06.135210 12127 net.cpp:202] Created Layer loss (11)
  785. I0819 14:26:06.135215 12127 net.cpp:574] loss <- ip2_ip2_0_split_1
  786. I0819 14:26:06.135221 12127 net.cpp:574] loss <- label_mnist_1_split_1
  787. I0819 14:26:06.135226 12127 net.cpp:544] loss -> loss
  788. I0819 14:26:06.135409 12127 net.cpp:262] Setting up loss
  789. I0819 14:26:06.135419 12127 net.cpp:269] TEST Top shape for layer 11 'loss' (1)
  790. I0819 14:26:06.135423 12127 net.cpp:273] with loss weight 1
  791. I0819 14:26:06.135430 12127 net.cpp:338] loss needs backward computation.
  792. I0819 14:26:06.135437 12127 net.cpp:340] accuracy does not need backward computation.
  793. I0819 14:26:06.135442 12127 net.cpp:338] ip2_ip2_0_split needs backward computation.
  794. I0819 14:26:06.135447 12127 net.cpp:338] ip2 needs backward computation.
  795. I0819 14:26:06.135452 12127 net.cpp:338] relu1 needs backward computation.
  796. I0819 14:26:06.135455 12127 net.cpp:338] ip1 needs backward computation.
  797. I0819 14:26:06.135460 12127 net.cpp:338] pool2 needs backward computation.
  798. I0819 14:26:06.135464 12127 net.cpp:338] conv2 needs backward computation.
  799. I0819 14:26:06.135468 12127 net.cpp:338] pool1 needs backward computation.
  800. I0819 14:26:06.135473 12127 net.cpp:338] conv1 needs backward computation.
  801. I0819 14:26:06.135478 12127 net.cpp:340] label_mnist_1_split does not need backward computation.
  802. I0819 14:26:06.135483 12127 net.cpp:340] mnist does not need backward computation.
  803. I0819 14:26:06.135488 12127 net.cpp:382] This network produces output accuracy
  804. I0819 14:26:06.135493 12127 net.cpp:382] This network produces output loss
  805. I0819 14:26:06.135517 12127 net.cpp:405] Top memory (TEST) required for data: 525656 diff: 546068
  806. I0819 14:26:06.135524 12127 net.cpp:408] Bottom memory (TEST) required for data: 525648 diff: 546060
  807. I0819 14:26:06.135527 12127 net.cpp:411] Shared (in-place) memory (TEST) by data: 13000 diff: 13000
  808. I0819 14:26:06.135532 12127 net.cpp:414] Parameters memory (TEST) required for data: 862160 diff: 862160
  809. I0819 14:26:06.135550 12127 net.cpp:417] Parameters shared memory (TEST) by data: 0 diff: 0
  810. I0819 14:26:06.135553 12127 net.cpp:423] Network initialization done.
  811. I0819 14:26:06.135634 12127 solver.cpp:55] Solver scaffolding done.
  812. I0819 14:26:06.204330 12127 parallel.cpp:95] Starting sync 0 (of total 8), {0.8}
  813. I0819 14:26:06.204382 12127 parallel.cpp:169] [0 - 0] P2PSync adding callback
  814. I0819 14:26:06.204388 12127 parallel.cpp:95] Starting sync 1 (of total 8), {1.8}
  815. I0819 14:26:06.204393 12127 parallel.cpp:169] [1 - 1] P2PSync adding callback
  816. I0819 14:26:06.204397 12127 parallel.cpp:95] Starting sync 2 (of total 8), {2.8}
  817. I0819 14:26:06.204406 12127 parallel.cpp:169] [2 - 2] P2PSync adding callback
  818. I0819 14:26:06.204409 12127 parallel.cpp:95] Starting sync 3 (of total 8), {3.8}
  819. I0819 14:26:06.204414 12127 parallel.cpp:169] [3 - 3] P2PSync adding callback
  820. I0819 14:26:06.204418 12127 parallel.cpp:95] Starting sync 4 (of total 8), {4.8}
  821. I0819 14:26:06.204423 12127 parallel.cpp:169] [4 - 4] P2PSync adding callback
  822. I0819 14:26:06.204427 12127 parallel.cpp:95] Starting sync 5 (of total 8), {5.8}
  823. I0819 14:26:06.204432 12127 parallel.cpp:169] [5 - 5] P2PSync adding callback
  824. I0819 14:26:06.204435 12127 parallel.cpp:95] Starting sync 6 (of total 8), {6.8}
  825. I0819 14:26:06.204450 12127 parallel.cpp:169] [6 - 6] P2PSync adding callback
  826. I0819 14:26:06.204453 12127 parallel.cpp:95] Starting sync 7 (of total 8), {7.8}
  827. I0819 14:26:06.204458 12127 parallel.cpp:169] [7 - 7] P2PSync adding callback
  828. I0819 14:26:06.204463 12127 parallel.cpp:101] Starting Optimization
  829. I0819 14:26:06.204466 12127 internal_thread.cpp:18] Starting 1 internal thread(s) on device 0
  830. I0819 14:26:06.204527 12127 internal_thread.cpp:18] Starting 1 internal thread(s) on device 1
  831. I0819 14:26:06.204569 12127 internal_thread.cpp:18] Starting 1 internal thread(s) on device 2
  832. I0819 14:26:06.204617 12127 internal_thread.cpp:18] Starting 1 internal thread(s) on device 3
  833. I0819 14:26:06.204741 12127 internal_thread.cpp:18] Starting 1 internal thread(s) on device 4
  834. I0819 14:26:06.205221 12127 internal_thread.cpp:18] Starting 1 internal thread(s) on device 5
  835. I0819 14:26:06.205379 12127 internal_thread.cpp:18] Starting 1 internal thread(s) on device 6
  836. I0819 14:26:06.205521 12127 internal_thread.cpp:18] Starting 1 internal thread(s) on device 7
  837. I0819 14:26:06.226933 12180 common.cpp:533] NVML initialized, thread 12180
  838. I0819 14:26:06.367383 12180 common.cpp:555] {0} NVML succeeded to set CPU affinity
  839. I0819 14:26:06.518798 12181 common.cpp:555] {1} NVML succeeded to set CPU affinity
  840. I0819 14:26:06.812333 12182 common.cpp:555] {2} NVML succeeded to set CPU affinity
  841. I0819 14:26:07.146512 12183 common.cpp:555] {3} NVML succeeded to set CPU affinity
  842. I0819 14:26:07.524495 12184 common.cpp:555] {4} NVML succeeded to set CPU affinity
  843. I0819 14:26:07.940737 12185 common.cpp:555] {5} NVML succeeded to set CPU affinity
  844. I0819 14:26:08.378716 12186 common.cpp:555] {6} NVML succeeded to set CPU affinity
  845. I0819 14:26:08.804412 12181 solver.cpp:40] Solver data type: FLOAT16
  846. I0819 14:26:08.804412 12182 solver.cpp:40] Solver data type: FLOAT16
  847. I0819 14:26:08.804430 12183 solver.cpp:40] Solver data type: FLOAT16
  848. I0819 14:26:08.804477 12184 solver.cpp:40] Solver data type: FLOAT16
  849. I0819 14:26:08.804525 12185 solver.cpp:40] Solver data type: FLOAT16
  850. I0819 14:26:08.804567 12186 solver.cpp:40] Solver data type: FLOAT16
  851. I0819 14:26:08.807494 12182 internal_thread.cpp:18] Starting 1 internal thread(s) on device 2
  852. I0819 14:26:08.807727 12182 data_reader.cpp:59] Sample Data Reader threads: 1, out queues: 1, depth: 8
  853. I0819 14:26:08.809671 12182 internal_thread.cpp:18] Starting 1 internal thread(s) on device 2
  854. I0819 14:26:08.811395 12205 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_train_lmdb
  855. I0819 14:26:08.811578 12187 common.cpp:555] {7} NVML succeeded to set CPU affinity
  856. I0819 14:26:08.811616 12187 solver.cpp:40] Solver data type: FLOAT16
  857. I0819 14:26:08.811836 12184 internal_thread.cpp:18] Starting 1 internal thread(s) on device 4
  858. I0819 14:26:08.812060 12184 data_reader.cpp:59] Sample Data Reader threads: 1, out queues: 1, depth: 8
  859. I0819 14:26:08.815555 12184 internal_thread.cpp:18] Starting 1 internal thread(s) on device 4
  860. I0819 14:26:08.820106 12207 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_train_lmdb
  861. I0819 14:26:08.820359 12183 internal_thread.cpp:18] Starting 1 internal thread(s) on device 3
  862. I0819 14:26:08.820652 12183 data_reader.cpp:59] Sample Data Reader threads: 1, out queues: 1, depth: 8
  863. I0819 14:26:08.823220 12183 internal_thread.cpp:18] Starting 1 internal thread(s) on device 3
  864. I0819 14:26:08.826602 12209 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_train_lmdb
  865. I0819 14:26:08.827469 12181 internal_thread.cpp:18] Starting 1 internal thread(s) on device 1
  866. I0819 14:26:08.830896 12181 data_reader.cpp:59] Sample Data Reader threads: 1, out queues: 1, depth: 8
  867. I0819 14:26:08.830940 12181 internal_thread.cpp:18] Starting 1 internal thread(s) on device 1
  868. I0819 14:26:08.832684 12211 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_train_lmdb
  869. I0819 14:26:09.112213 12183 data_layer.cpp:200] [n0.d3.r3] Output data size: 8, 1, 28, 28
  870. I0819 14:26:09.112282 12183 internal_thread.cpp:18] Starting 1 internal thread(s) on device 3
  871. I0819 14:26:09.113519 12182 data_layer.cpp:200] [n0.d2.r2] Output data size: 8, 1, 28, 28
  872. I0819 14:26:09.114099 12182 internal_thread.cpp:18] Starting 1 internal thread(s) on device 2
  873. I0819 14:26:09.114202 12184 data_layer.cpp:200] [n0.d4.r4] Output data size: 8, 1, 28, 28
  874. I0819 14:26:09.114372 12184 internal_thread.cpp:18] Starting 1 internal thread(s) on device 4
  875. I0819 14:26:09.117770 12181 data_layer.cpp:200] [n0.d1.r1] Output data size: 8, 1, 28, 28
  876. I0819 14:26:09.118423 12181 internal_thread.cpp:18] Starting 1 internal thread(s) on device 1
  877. I0819 14:26:09.118736 12185 internal_thread.cpp:18] Starting 1 internal thread(s) on device 5
  878. I0819 14:26:09.123558 12185 data_reader.cpp:59] Sample Data Reader threads: 1, out queues: 1, depth: 8
  879. I0819 14:26:09.123598 12185 internal_thread.cpp:18] Starting 1 internal thread(s) on device 5
  880. I0819 14:26:09.125273 12217 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_train_lmdb
  881. I0819 14:26:09.125365 12186 internal_thread.cpp:18] Starting 1 internal thread(s) on device 6
  882. I0819 14:26:09.125687 12186 data_reader.cpp:59] Sample Data Reader threads: 1, out queues: 1, depth: 8
  883. I0819 14:26:09.128406 12186 internal_thread.cpp:18] Starting 1 internal thread(s) on device 6
  884. I0819 14:26:09.128469 12187 internal_thread.cpp:18] Starting 1 internal thread(s) on device 7
  885. I0819 14:26:09.130543 12219 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_train_lmdb
  886. I0819 14:26:09.132122 12187 data_reader.cpp:59] Sample Data Reader threads: 1, out queues: 1, depth: 8
  887. I0819 14:26:09.132172 12187 internal_thread.cpp:18] Starting 1 internal thread(s) on device 7
  888. I0819 14:26:09.136530 12221 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_train_lmdb
  889. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  890. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  891. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  892. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  893. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  894. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  895. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  896. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  897. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  898. in math_function.cpp, line 730, tmp_value = 1048576
  899. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  900. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  901. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  902. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  903. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  904. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  905. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  906. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  907. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  908. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  909. in math_function.cpp, line 730, tmp_value = 1048576
  910. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  911. definition of type is half.
  912. gspt r[i] = 0.0625 (Q_frac=24
  913. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  914. In blob.cpp, blob.Q_frac= 24
  915. definition of type is half.
  916. gspt r[i] = 0.0625 (Q_frac=24
  917. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  918. In blob.cpp, blob.Q_frac= 24
  919. I0819 14:26:09.164611 12185 data_layer.cpp:200] [n0.d5.r5] Output data size: 8, 1, 28, 28
  920. I0819 14:26:09.164680 12185 internal_thread.cpp:18] Starting 1 internal thread(s) on device 5
  921. I0819 14:26:09.165598 12186 data_layer.cpp:200] [n0.d6.r6] Output data size: 8, 1, 28, 28
  922. I0819 14:26:09.166170 12187 data_layer.cpp:200] [n0.d7.r7] Output data size: 8, 1, 28, 28
  923. I0819 14:26:09.166437 12187 internal_thread.cpp:18] Starting 1 internal thread(s) on device 7
  924. I0819 14:26:09.166457 12186 internal_thread.cpp:18] Starting 1 internal thread(s) on device 6
  925. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  926. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  927. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  928. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  929. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  930. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  931. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  932. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  933. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  934. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  935. in math_function.cpp, line 1227, tmp_value = 0
  936. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  937. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  938. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  939. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  940. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  941. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  942. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  943. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  944. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  945. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  946. in math_function.cpp, line 1227, tmp_value = 0
  947. definition of type is half [FILL ZERO].
  948. gspt r[i] = 0 (Q_frac=22
  949. In blob.cpp, blob.Q_frac= 22
  950. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  951. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  952. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  953. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  954. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  955. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  956. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  957. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  958. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  959. in math_function.cpp, line 730, tmp_value = 1048576
  960. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  961. definition of type is half [FILL ZERO].
  962. gspt r[i] = 0 (Q_frac=22
  963. In blob.cpp, blob.Q_frac= 22
  964. definition of type is half.
  965. gspt r[i] = 0.0625 (Q_frac=24
  966. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  967. In blob.cpp, blob.Q_frac= 24
  968. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  969. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  970. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  971. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  972. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  973. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  974. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  975. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  976. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  977. in math_function.cpp, line 730, tmp_value = 1048576
  978. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  979. definition of type is half.
  980. gspt r[i] = 0.0625 (Q_frac=24
  981. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  982. In blob.cpp, blob.Q_frac= 24
  983. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  984. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  985. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  986. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  987. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  988. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  989. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  990. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  991. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  992. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  993. in math_function.cpp, line 1227, tmp_value = 0
  994. definition of type is half [FILL ZERO].
  995. gspt r[i] = 0 (Q_frac=22
  996. In blob.cpp, blob.Q_frac= 22
  997. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  998. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  999. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  1000. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  1001. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  1002. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  1003. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  1004. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  1005. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  1006. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  1007. in math_function.cpp, line 1227, tmp_value = 0
  1008. definition of type is half [FILL ZERO].
  1009. gspt r[i] = 0 (Q_frac=22
  1010. In blob.cpp, blob.Q_frac= 22
  1011. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  1012. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  1013. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  1014. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  1015. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  1016. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  1017. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  1018. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  1019. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  1020. in math_function.cpp, line 730, tmp_value = 1048576
  1021. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  1022. definition of type is half.
  1023. gspt r[i] = 0.0625 (Q_frac=24
  1024. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  1025. In blob.cpp, blob.Q_frac= 24
  1026. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  1027. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  1028. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  1029. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  1030. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  1031. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  1032. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  1033. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  1034. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  1035. in math_function.cpp, line 730, tmp_value = 1048576
  1036. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  1037. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  1038. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  1039. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  1040. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  1041. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  1042. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  1043. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  1044. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  1045. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  1046. in math_function.cpp, line 730, tmp_value = 1048576
  1047. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  1048. definition of type is half.
  1049. gspt r[i] = 0.0625 (Q_frac=24
  1050. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  1051. In blob.cpp, blob.Q_frac= 24
  1052. definition of type is half.
  1053. gspt r[i] = 0.0625 (Q_frac=24
  1054. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  1055. In blob.cpp, blob.Q_frac= 24
  1056. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  1057. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  1058. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  1059. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  1060. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  1061. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  1062. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  1063. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  1064. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  1065. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  1066. in math_function.cpp, line 1227, tmp_value = 0
  1067. definition of type is half [FILL ZERO].
  1068. gspt r[i] = 0 (Q_frac=22
  1069. In blob.cpp, blob.Q_frac= 22
  1070. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  1071. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  1072. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  1073. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  1074. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  1075. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  1076. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  1077. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  1078. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  1079. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  1080. in math_function.cpp, line 1227, tmp_value = 0
  1081. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  1082. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  1083. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  1084. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  1085. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  1086. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  1087. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  1088. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  1089. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  1090. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  1091. in math_function.cpp, line 1227, tmp_value = 0
  1092. definition of type is half [FILL ZERO].
  1093. gspt r[i] = 0 (Q_frac=22
  1094. In blob.cpp, blob.Q_frac= 22
  1095. definition of type is half [FILL ZERO].
  1096. gspt r[i] = 0 (Q_frac=22
  1097. In blob.cpp, blob.Q_frac= 22
  1098. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  1099. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  1100. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  1101. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  1102. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  1103. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  1104. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  1105. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  1106. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  1107. in math_function.cpp, line 730, tmp_value = 1048576
  1108. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  1109. definition of type is half.
  1110. gspt r[i] = 0.015625 (Q_frac=26
  1111. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  1112. In blob.cpp, blob.Q_frac= 26
  1113. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  1114. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  1115. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  1116. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  1117. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  1118. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  1119. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  1120. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  1121. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  1122. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  1123. in math_function.cpp, line 1227, tmp_value = 0
  1124. definition of type is half [FILL ZERO].
  1125. gspt r[i] = 0 (Q_frac=22
  1126. In blob.cpp, blob.Q_frac= 22
  1127. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  1128. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  1129. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  1130. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  1131. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  1132. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  1133. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  1134. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  1135. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  1136. in math_function.cpp, line 730, tmp_value = 1048576
  1137. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  1138. definition of type is half.
  1139. gspt r[i] = 0.0078125 (Q_frac=27
  1140. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  1141. In blob.cpp, blob.Q_frac= 27
  1142. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  1143. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  1144. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  1145. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  1146. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  1147. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  1148. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  1149. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  1150. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  1151. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  1152. in math_function.cpp, line 1227, tmp_value = 0
  1153. definition of type is half [FILL ZERO].
  1154. gspt r[i] = 0 (Q_frac=22
  1155. In blob.cpp, blob.Q_frac= 22
  1156. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  1157. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  1158. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  1159. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  1160. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  1161. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  1162. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  1163. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  1164. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  1165. in math_function.cpp, line 730, tmp_value = 1048576
  1166. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  1167. definition of type is half.
  1168. gspt r[i] = 0.015625 (Q_frac=26
  1169. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  1170. In blob.cpp, blob.Q_frac= 26
  1171. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  1172. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  1173. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  1174. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  1175. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  1176. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  1177. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  1178. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  1179. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  1180. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  1181. in math_function.cpp, line 1227, tmp_value = 0
  1182. definition of type is half [FILL ZERO].
  1183. gspt r[i] = 0 (Q_frac=22
  1184. In blob.cpp, blob.Q_frac= 22
  1185. I0819 14:26:10.327457 12183 solver.cpp:174] Creating test net (#0) specified by net file: examples/mnist/lenet_train_test_fp16.prototxt
  1186. W0819 14:26:10.327579 12183 parallel.cpp:319] Batch size must be divisible by the number of solvers (GPUs): it's been adjusted from 100 to 104
  1187. I0819 14:26:10.327780 12183 internal_thread.cpp:18] Starting 1 internal thread(s) on device 3
  1188. I0819 14:26:10.328166 12183 data_reader.cpp:59] Data Reader threads: 1, out queues: 1, depth: 13
  1189. I0819 14:26:10.329828 12183 internal_thread.cpp:18] Starting 1 internal thread(s) on device 3
  1190. I0819 14:26:10.331306 12226 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_test_lmdb
  1191. I0819 14:26:10.355618 12183 data_layer.cpp:200] (n0.d3.r3) Output data size: 13, 1, 28, 28
  1192. I0819 14:26:10.355700 12183 internal_thread.cpp:18] Starting 1 internal thread(s) on device 3
  1193. I0819 14:26:10.359419 12227 data_layer.cpp:105] (n0.d3.r3) Parser threads: 1
  1194. I0819 14:26:10.359443 12227 data_layer.cpp:107] (n0.d3.r3) Transformer threads: 1
  1195. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  1196. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  1197. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  1198. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  1199. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  1200. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  1201. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  1202. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  1203. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  1204. in math_function.cpp, line 730, tmp_value = 1048576
  1205. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  1206. definition of type is half.
  1207. gspt r[i] = 0.0625 (Q_frac=24
  1208. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  1209. In blob.cpp, blob.Q_frac= 24
  1210. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  1211. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  1212. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  1213. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  1214. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  1215. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  1216. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  1217. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  1218. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  1219. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  1220. in math_function.cpp, line 1227, tmp_value = 0
  1221. definition of type is half [FILL ZERO].
  1222. gspt r[i] = 0 (Q_frac=22
  1223. In blob.cpp, blob.Q_frac= 22
  1224. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  1225. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  1226. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  1227. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  1228. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  1229. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  1230. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  1231. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  1232. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  1233. in math_function.cpp, line 730, tmp_value = 1048576
  1234. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  1235. definition of type is half.
  1236. gspt r[i] = 0.015625 (Q_frac=26
  1237. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  1238. In blob.cpp, blob.Q_frac= 26
  1239. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  1240. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  1241. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  1242. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  1243. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  1244. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  1245. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  1246. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  1247. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  1248. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  1249. in math_function.cpp, line 1227, tmp_value = 0
  1250. definition of type is half [FILL ZERO].
  1251. gspt r[i] = 0 (Q_frac=22
  1252. In blob.cpp, blob.Q_frac= 22
  1253. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  1254. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  1255. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  1256. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  1257. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  1258. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  1259. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  1260. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  1261. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  1262. in math_function.cpp, line 730, tmp_value = 1048576
  1263. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  1264. definition of type is half.
  1265. gspt r[i] = 0.0078125 (Q_frac=27
  1266. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  1267. In blob.cpp, blob.Q_frac= 27
  1268. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  1269. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  1270. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  1271. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  1272. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  1273. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  1274. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  1275. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  1276. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  1277. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  1278. in math_function.cpp, line 1227, tmp_value = 0
  1279. definition of type is half [FILL ZERO].
  1280. gspt r[i] = 0 (Q_frac=22
  1281. In blob.cpp, blob.Q_frac= 22
  1282. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  1283. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  1284. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  1285. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  1286. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  1287. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  1288. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  1289. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  1290. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  1291. in math_function.cpp, line 730, tmp_value = 1048576
  1292. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  1293. definition of type is half.
  1294. gspt r[i] = 0.015625 (Q_frac=26
  1295. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  1296. In blob.cpp, blob.Q_frac= 26
  1297. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  1298. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  1299. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  1300. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  1301. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  1302. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  1303. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  1304. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  1305. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  1306. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  1307. in math_function.cpp, line 1227, tmp_value = 0
  1308. definition of type is half [FILL ZERO].
  1309. gspt r[i] = 0 (Q_frac=22
  1310. In blob.cpp, blob.Q_frac= 22
  1311. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  1312. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  1313. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  1314. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  1315. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  1316. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  1317. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  1318. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  1319. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  1320. in math_function.cpp, line 730, tmp_value = 1048576
  1321. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  1322. definition of type is half.
  1323. gspt r[i] = 0.015625 (Q_frac=26
  1324. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  1325. In blob.cpp, blob.Q_frac= 26
  1326. I0819 14:26:10.835765 12183 solver.cpp:55] Solver scaffolding done.
  1327. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  1328. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  1329. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  1330. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  1331. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  1332. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  1333. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  1334. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  1335. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  1336. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  1337. in math_function.cpp, line 1227, tmp_value = 0
  1338. definition of type is half [FILL ZERO].
  1339. gspt r[i] = 0 (Q_frac=22
  1340. In blob.cpp, blob.Q_frac= 22
  1341. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  1342. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  1343. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  1344. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  1345. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  1346. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  1347. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  1348. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  1349. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  1350. in math_function.cpp, line 730, tmp_value = 1048576
  1351. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  1352. definition of type is half.
  1353. gspt r[i] = 0.0078125 (Q_frac=27
  1354. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  1355. In blob.cpp, blob.Q_frac= 27
  1356. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  1357. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  1358. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  1359. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  1360. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  1361. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  1362. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  1363. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  1364. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  1365. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  1366. in math_function.cpp, line 1227, tmp_value = 0
  1367. definition of type is half [FILL ZERO].
  1368. gspt r[i] = 0 (Q_frac=22
  1369. In blob.cpp, blob.Q_frac= 22
  1370. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  1371. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  1372. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  1373. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  1374. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  1375. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  1376. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  1377. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  1378. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  1379. in math_function.cpp, line 730, tmp_value = 1048576
  1380. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  1381. definition of type is half.
  1382. gspt r[i] = 0.015625 (Q_frac=26
  1383. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  1384. In blob.cpp, blob.Q_frac= 26
  1385. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  1386. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  1387. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  1388. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  1389. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  1390. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  1391. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  1392. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  1393. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  1394. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  1395. in math_function.cpp, line 1227, tmp_value = 0
  1396. definition of type is half [FILL ZERO].
  1397. gspt r[i] = 0 (Q_frac=22
  1398. In blob.cpp, blob.Q_frac= 22
  1399. I0819 14:26:11.257170 12182 solver.cpp:174] Creating test net (#0) specified by net file: examples/mnist/lenet_train_test_fp16.prototxt
  1400. W0819 14:26:11.257272 12182 parallel.cpp:319] Batch size must be divisible by the number of solvers (GPUs): it's been adjusted from 100 to 104
  1401. I0819 14:26:11.257467 12182 internal_thread.cpp:18] Starting 1 internal thread(s) on device 2
  1402. I0819 14:26:11.258131 12182 data_reader.cpp:59] Data Reader threads: 1, out queues: 1, depth: 13
  1403. I0819 14:26:11.259642 12182 internal_thread.cpp:18] Starting 1 internal thread(s) on device 2
  1404. I0819 14:26:11.261420 12229 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_test_lmdb
  1405. I0819 14:26:11.277657 12182 data_layer.cpp:200] (n0.d2.r2) Output data size: 13, 1, 28, 28
  1406. I0819 14:26:11.277732 12182 internal_thread.cpp:18] Starting 1 internal thread(s) on device 2
  1407. I0819 14:26:11.279595 12230 data_layer.cpp:105] (n0.d2.r2) Parser threads: 1
  1408. I0819 14:26:11.279628 12230 data_layer.cpp:107] (n0.d2.r2) Transformer threads: 1
  1409. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  1410. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  1411. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  1412. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  1413. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  1414. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  1415. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  1416. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  1417. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  1418. in math_function.cpp, line 730, tmp_value = 1048576
  1419. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  1420. definition of type is half.
  1421. gspt r[i] = 0.0625 (Q_frac=24
  1422. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  1423. In blob.cpp, blob.Q_frac= 24
  1424. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  1425. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  1426. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  1427. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  1428. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  1429. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  1430. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  1431. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  1432. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  1433. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  1434. in math_function.cpp, line 1227, tmp_value = 0
  1435. definition of type is half [FILL ZERO].
  1436. gspt r[i] = 0 (Q_frac=22
  1437. In blob.cpp, blob.Q_frac= 22
  1438. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  1439. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  1440. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  1441. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  1442. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  1443. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  1444. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  1445. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  1446. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  1447. in math_function.cpp, line 730, tmp_value = 1048576
  1448. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  1449. definition of type is half.
  1450. gspt r[i] = 0.015625 (Q_frac=26
  1451. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  1452. In blob.cpp, blob.Q_frac= 26
  1453. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  1454. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  1455. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  1456. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  1457. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  1458. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  1459. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  1460. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  1461. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  1462. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  1463. in math_function.cpp, line 1227, tmp_value = 0
  1464. definition of type is half [FILL ZERO].
  1465. gspt r[i] = 0 (Q_frac=22
  1466. In blob.cpp, blob.Q_frac= 22
  1467. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  1468. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  1469. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  1470. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  1471. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  1472. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  1473. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  1474. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  1475. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  1476. in math_function.cpp, line 730, tmp_value = 1048576
  1477. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  1478. definition of type is half.
  1479. gspt r[i] = 0.0078125 (Q_frac=27
  1480. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  1481. In blob.cpp, blob.Q_frac= 27
  1482. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  1483. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  1484. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  1485. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  1486. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  1487. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  1488. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  1489. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  1490. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  1491. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  1492. in math_function.cpp, line 1227, tmp_value = 0
  1493. definition of type is half [FILL ZERO].
  1494. gspt r[i] = 0 (Q_frac=22
  1495. In blob.cpp, blob.Q_frac= 22
  1496. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  1497. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  1498. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  1499. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  1500. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  1501. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  1502. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  1503. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  1504. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  1505. in math_function.cpp, line 730, tmp_value = 1048576
  1506. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  1507. definition of type is half.
  1508. gspt r[i] = 0.015625 (Q_frac=26
  1509. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  1510. In blob.cpp, blob.Q_frac= 26
  1511. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  1512. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  1513. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  1514. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  1515. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  1516. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  1517. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  1518. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  1519. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  1520. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  1521. in math_function.cpp, line 1227, tmp_value = 0
  1522. definition of type is half [FILL ZERO].
  1523. gspt r[i] = 0 (Q_frac=22
  1524. In blob.cpp, blob.Q_frac= 22
  1525. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  1526. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  1527. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  1528. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  1529. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  1530. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  1531. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  1532. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  1533. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  1534. in math_function.cpp, line 730, tmp_value = 1048576
  1535. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  1536. definition of type is half.
  1537. gspt r[i] = 0.015625 (Q_frac=26
  1538. I0819 14:26:11.719116 12182 solver.cpp:55] Solver scaffolding done.
  1539. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  1540. In blob.cpp, blob.Q_frac= 26
  1541. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  1542. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  1543. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  1544. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  1545. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  1546. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  1547. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  1548. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  1549. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  1550. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  1551. in math_function.cpp, line 1227, tmp_value = 0
  1552. definition of type is half [FILL ZERO].
  1553. gspt r[i] = 0 (Q_frac=22
  1554. In blob.cpp, blob.Q_frac= 22
  1555. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  1556. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  1557. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  1558. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  1559. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  1560. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  1561. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  1562. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  1563. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  1564. in math_function.cpp, line 730, tmp_value = 1048576
  1565. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  1566. definition of type is half.
  1567. gspt r[i] = 0.0078125 (Q_frac=27
  1568. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  1569. In blob.cpp, blob.Q_frac= 27
  1570. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  1571. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  1572. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  1573. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  1574. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  1575. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  1576. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  1577. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  1578. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  1579. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  1580. in math_function.cpp, line 1227, tmp_value = 0
  1581. definition of type is half [FILL ZERO].
  1582. gspt r[i] = 0 (Q_frac=22
  1583. In blob.cpp, blob.Q_frac= 22
  1584. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  1585. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  1586. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  1587. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  1588. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  1589. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  1590. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  1591. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  1592. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  1593. in math_function.cpp, line 730, tmp_value = 1048576
  1594. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  1595. definition of type is half.
  1596. gspt r[i] = 0.015625 (Q_frac=26
  1597. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  1598. In blob.cpp, blob.Q_frac= 26
  1599. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  1600. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  1601. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  1602. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  1603. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  1604. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  1605. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  1606. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  1607. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  1608. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  1609. in math_function.cpp, line 1227, tmp_value = 0
  1610. definition of type is half [FILL ZERO].
  1611. gspt r[i] = 0 (Q_frac=22
  1612. In blob.cpp, blob.Q_frac= 22
  1613. I0819 14:26:12.137696 12181 solver.cpp:174] Creating test net (#0) specified by net file: examples/mnist/lenet_train_test_fp16.prototxt
  1614. W0819 14:26:12.137830 12181 parallel.cpp:319] Batch size must be divisible by the number of solvers (GPUs): it's been adjusted from 100 to 104
  1615. I0819 14:26:12.138058 12181 internal_thread.cpp:18] Starting 1 internal thread(s) on device 1
  1616. I0819 14:26:12.138795 12181 data_reader.cpp:59] Data Reader threads: 1, out queues: 1, depth: 13
  1617. I0819 14:26:12.141125 12181 internal_thread.cpp:18] Starting 1 internal thread(s) on device 1
  1618. I0819 14:26:12.143594 12232 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_test_lmdb
  1619. I0819 14:26:12.259518 12181 data_layer.cpp:200] (n0.d1.r1) Output data size: 13, 1, 28, 28
  1620. I0819 14:26:12.259595 12181 internal_thread.cpp:18] Starting 1 internal thread(s) on device 1
  1621. I0819 14:26:12.261369 12233 data_layer.cpp:105] (n0.d1.r1) Parser threads: 1
  1622. I0819 14:26:12.261397 12233 data_layer.cpp:107] (n0.d1.r1) Transformer threads: 1
  1623. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  1624. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  1625. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  1626. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  1627. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  1628. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  1629. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  1630. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  1631. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  1632. in math_function.cpp, line 730, tmp_value = 1048576
  1633. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  1634. definition of type is half.
  1635. gspt r[i] = 0.0625 (Q_frac=24
  1636. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  1637. In blob.cpp, blob.Q_frac= 24
  1638. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  1639. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  1640. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  1641. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  1642. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  1643. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  1644. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  1645. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  1646. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  1647. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  1648. in math_function.cpp, line 1227, tmp_value = 0
  1649. definition of type is half [FILL ZERO].
  1650. gspt r[i] = 0 (Q_frac=22
  1651. In blob.cpp, blob.Q_frac= 22
  1652. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  1653. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  1654. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  1655. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  1656. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  1657. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  1658. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  1659. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  1660. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  1661. in math_function.cpp, line 730, tmp_value = 1048576
  1662. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  1663. definition of type is half.
  1664. gspt r[i] = 0.015625 (Q_frac=26
  1665. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  1666. In blob.cpp, blob.Q_frac= 26
  1667. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  1668. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  1669. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  1670. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  1671. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  1672. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  1673. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  1674. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  1675. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  1676. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  1677. in math_function.cpp, line 1227, tmp_value = 0
  1678. definition of type is half [FILL ZERO].
  1679. gspt r[i] = 0 (Q_frac=22
  1680. In blob.cpp, blob.Q_frac= 22
  1681. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  1682. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  1683. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  1684. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  1685. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  1686. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  1687. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  1688. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  1689. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  1690. in math_function.cpp, line 730, tmp_value = 1048576
  1691. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  1692. definition of type is half.
  1693. gspt r[i] = 0.015625 (Q_frac=26
  1694. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  1695. In blob.cpp, blob.Q_frac= 26
  1696. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  1697. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  1698. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  1699. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  1700. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  1701. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  1702. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  1703. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  1704. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  1705. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  1706. in math_function.cpp, line 1227, tmp_value = 0
  1707. definition of type is half [FILL ZERO].
  1708. gspt r[i] = 0 (Q_frac=22
  1709. In blob.cpp, blob.Q_frac= 22
  1710. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  1711. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  1712. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  1713. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  1714. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  1715. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  1716. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  1717. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  1718. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  1719. in math_function.cpp, line 730, tmp_value = 1048576
  1720. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  1721. definition of type is half.
  1722. gspt r[i] = 0.0078125 (Q_frac=27
  1723. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  1724. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  1725. in math_function.cpp line 565, grp_num= 1, rand_index= 4
  1726. in math_function.cpp line 565, grp_num= 2, rand_index= 5
  1727. in math_function.cpp line 565, grp_num= 3, rand_index= 2
  1728. in math_function.cpp line 565, grp_num= 4, rand_index= 1
  1729. in math_function.cpp line 565, grp_num= 5, rand_index= 2
  1730. in math_function.cpp line 565, grp_num= 6, rand_index= 4
  1731. in math_function.cpp line 565, grp_num= 7, rand_index= 3
  1732. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  1733. in math_function.cpp, line 730, tmp_value = 262144
  1734. in math_function.cpp line 656, gspt_info_1[i]= 9609891, gspt_info_2[i]= 0
  1735. definition of type is half.
  1736. gspt r[i] = 0.00195312 (Q_frac=27
  1737. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  1738. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  1739. in math_function.cpp line 565, grp_num= 1, rand_index= 0
  1740. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  1741. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  1742. in math_function.cpp line 565, grp_num= 4, rand_index= 2
  1743. in math_function.cpp line 565, grp_num= 5, rand_index= 1
  1744. in math_function.cpp line 565, grp_num= 6, rand_index= 1
  1745. in math_function.cpp line 565, grp_num= 7, rand_index= 1
  1746. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  1747. in math_function.cpp, line 730, tmp_value = -1179648
  1748. in math_function.cpp line 656, gspt_info_1[i]= 8389705, gspt_info_2[i]= 0
  1749. definition of type is half.
  1750. gspt r[i] = -0.0175781 (Q_frac=26
  1751. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  1752. In blob.cpp, blob.Q_frac= 27
  1753. In blob.cpp, blob.Q_frac= 26
  1754. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  1755. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  1756. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  1757. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  1758. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  1759. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  1760. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  1761. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  1762. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  1763. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  1764. in math_function.cpp, line 1227, tmp_value = 0
  1765. definition of type is half [FILL ZERO].
  1766. gspt r[i] = 0 (Q_frac=22
  1767. In blob.cpp, blob.Q_frac= 22
  1768. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  1769. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  1770. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  1771. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  1772. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  1773. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  1774. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  1775. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  1776. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  1777. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  1778. in math_function.cpp, line 1227, tmp_value = 0
  1779. definition of type is half [FILL ZERO].
  1780. gspt r[i] = 0 (Q_frac=22
  1781. In blob.cpp, blob.Q_frac= 22
  1782. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  1783. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  1784. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  1785. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  1786. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  1787. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  1788. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  1789. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  1790. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  1791. in math_function.cpp, line 730, tmp_value = 1048576
  1792. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  1793. definition of type is half.
  1794. gspt r[i] = 0.015625 (Q_frac=26
  1795. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  1796. In blob.cpp, blob.Q_frac= 26
  1797. In blob.cpp, blob.Q_frac= 27
  1798. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  1799. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  1800. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  1801. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  1802. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  1803. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  1804. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  1805. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  1806. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  1807. in math_function.cpp, line 730, tmp_value = 1048576
  1808. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  1809. definition of type is half.
  1810. gspt r[i] = 0.0078125 (Q_frac=27
  1811. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  1812. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  1813. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  1814. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  1815. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  1816. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  1817. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  1818. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  1819. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  1820. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  1821. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  1822. in math_function.cpp, line 1227, tmp_value = 0
  1823. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  1824. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  1825. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  1826. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  1827. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  1828. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  1829. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  1830. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  1831. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  1832. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  1833. in math_function.cpp, line 1227, tmp_value = 0
  1834. definition of type is half [FILL ZERO].
  1835. gspt r[i] = 0 (Q_frac=22
  1836. In blob.cpp, blob.Q_frac= 22
  1837. definition of type is half [FILL ZERO].
  1838. gspt r[i] = 0 (Q_frac=22
  1839. In blob.cpp, blob.Q_frac= 22
  1840. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  1841. in math_function.cpp line 565, grp_num= 1, rand_index= 4
  1842. in math_function.cpp line 565, grp_num= 2, rand_index= 3
  1843. in math_function.cpp line 565, grp_num= 3, rand_index= 3
  1844. in math_function.cpp line 565, grp_num= 4, rand_index= 3
  1845. in math_function.cpp line 565, grp_num= 5, rand_index= 5
  1846. in math_function.cpp line 565, grp_num= 6, rand_index= 1
  1847. in math_function.cpp line 565, grp_num= 7, rand_index= 0
  1848. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  1849. in math_function.cpp, line 730, tmp_value = -2228224
  1850. in math_function.cpp line 656, gspt_info_1[i]= 9549640, gspt_info_2[i]= 0
  1851. definition of type is half.
  1852. gspt r[i] = -0.0332031 (Q_frac=26
  1853. I0819 14:26:13.597795 12181 solver.cpp:55] Solver scaffolding done.
  1854. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  1855. In blob.cpp, blob.Q_frac= 26
  1856. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  1857. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  1858. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  1859. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  1860. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  1861. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  1862. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  1863. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  1864. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  1865. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  1866. in math_function.cpp, line 1227, tmp_value = 0
  1867. definition of type is half [FILL ZERO].
  1868. gspt r[i] = 0 (Q_frac=22
  1869. In blob.cpp, blob.Q_frac= 22
  1870. In blob.cpp, blob.Q_frac= 27
  1871. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  1872. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  1873. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  1874. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  1875. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  1876. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  1877. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  1878. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  1879. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  1880. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  1881. in math_function.cpp, line 1227, tmp_value = 0
  1882. definition of type is half [FILL ZERO].
  1883. gspt r[i] = 0 (Q_frac=22
  1884. In blob.cpp, blob.Q_frac= 22
  1885. I0819 14:26:13.756623 12184 solver.cpp:174] Creating test net (#0) specified by net file: examples/mnist/lenet_train_test_fp16.prototxt
  1886. W0819 14:26:13.756768 12184 parallel.cpp:319] Batch size must be divisible by the number of solvers (GPUs): it's been adjusted from 100 to 104
  1887. I0819 14:26:13.757032 12184 internal_thread.cpp:18] Starting 1 internal thread(s) on device 4
  1888. I0819 14:26:13.757282 12184 data_reader.cpp:59] Data Reader threads: 1, out queues: 1, depth: 13
  1889. I0819 14:26:13.759277 12184 internal_thread.cpp:18] Starting 1 internal thread(s) on device 4
  1890. I0819 14:26:13.761255 12235 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_test_lmdb
  1891. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  1892. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  1893. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  1894. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  1895. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  1896. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  1897. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  1898. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  1899. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  1900. in math_function.cpp, line 730, tmp_value = 1048576
  1901. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  1902. definition of type is half.
  1903. gspt r[i] = 0.015625 (Q_frac=26
  1904. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  1905. In blob.cpp, blob.Q_frac= 26
  1906. I0819 14:26:13.864593 12184 data_layer.cpp:200] (n0.d4.r4) Output data size: 13, 1, 28, 28
  1907. I0819 14:26:13.864696 12184 internal_thread.cpp:18] Starting 1 internal thread(s) on device 4
  1908. I0819 14:26:13.867190 12236 data_layer.cpp:105] (n0.d4.r4) Parser threads: 1
  1909. I0819 14:26:13.867213 12236 data_layer.cpp:107] (n0.d4.r4) Transformer threads: 1
  1910. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  1911. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  1912. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  1913. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  1914. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  1915. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  1916. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  1917. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  1918. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  1919. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  1920. in math_function.cpp, line 1227, tmp_value = 0
  1921. definition of type is half [FILL ZERO].
  1922. gspt r[i] = 0 (Q_frac=22
  1923. In blob.cpp, blob.Q_frac= 22
  1924. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  1925. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  1926. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  1927. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  1928. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  1929. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  1930. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  1931. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  1932. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  1933. in math_function.cpp, line 730, tmp_value = 1048576
  1934. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  1935. definition of type is half.
  1936. gspt r[i] = 0.0625 (Q_frac=24
  1937. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  1938. In blob.cpp, blob.Q_frac= 24
  1939. I0819 14:26:13.878717 12187 solver.cpp:174] Creating test net (#0) specified by net file: examples/mnist/lenet_train_test_fp16.prototxt
  1940. W0819 14:26:13.878849 12187 parallel.cpp:319] Batch size must be divisible by the number of solvers (GPUs): it's been adjusted from 100 to 104
  1941. I0819 14:26:13.879073 12187 internal_thread.cpp:18] Starting 1 internal thread(s) on device 7
  1942. I0819 14:26:13.879226 12187 data_reader.cpp:59] Data Reader threads: 1, out queues: 1, depth: 13
  1943. I0819 14:26:13.880722 12187 internal_thread.cpp:18] Starting 1 internal thread(s) on device 7
  1944. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  1945. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  1946. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  1947. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  1948. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  1949. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  1950. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  1951. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  1952. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  1953. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  1954. in math_function.cpp, line 1227, tmp_value = 0
  1955. definition of type is half [FILL ZERO].
  1956. gspt r[i] = 0 (Q_frac=22
  1957. In blob.cpp, blob.Q_frac= 22
  1958. I0819 14:26:13.882527 12238 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_test_lmdb
  1959. I0819 14:26:13.887152 12187 data_layer.cpp:200] (n0.d7.r7) Output data size: 13, 1, 28, 28
  1960. I0819 14:26:13.887217 12187 internal_thread.cpp:18] Starting 1 internal thread(s) on device 7
  1961. I0819 14:26:13.889994 12239 data_layer.cpp:105] (n0.d7.r7) Parser threads: 1
  1962. I0819 14:26:13.890019 12239 data_layer.cpp:107] (n0.d7.r7) Transformer threads: 1
  1963. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  1964. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  1965. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  1966. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  1967. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  1968. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  1969. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  1970. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  1971. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  1972. in math_function.cpp, line 730, tmp_value = 1048576
  1973. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  1974. definition of type is half.
  1975. gspt r[i] = 0.0625 (Q_frac=24
  1976. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  1977. In blob.cpp, blob.Q_frac= 24
  1978. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  1979. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  1980. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  1981. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  1982. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  1983. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  1984. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  1985. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  1986. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  1987. in math_function.cpp, line 730, tmp_value = 1048576
  1988. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  1989. definition of type is half.
  1990. gspt r[i] = 0.015625 (Q_frac=26
  1991. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  1992. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  1993. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  1994. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  1995. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  1996. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  1997. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  1998. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  1999. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  2000. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  2001. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  2002. in math_function.cpp, line 1227, tmp_value = 0
  2003. definition of type is half [FILL ZERO].
  2004. gspt r[i] = 0 (Q_frac=22
  2005. In blob.cpp, blob.Q_frac= 22
  2006. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  2007. in math_function.cpp line 565, grp_num= 1, rand_index= 1
  2008. in math_function.cpp line 565, grp_num= 2, rand_index= 6
  2009. in math_function.cpp line 565, grp_num= 3, rand_index= 4
  2010. in math_function.cpp line 565, grp_num= 4, rand_index= 2
  2011. in math_function.cpp line 565, grp_num= 5, rand_index= 6
  2012. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  2013. in math_function.cpp line 565, grp_num= 7, rand_index= 4
  2014. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  2015. in math_function.cpp, line 730, tmp_value = 524288
  2016. in math_function.cpp line 656, gspt_info_1[i]= 8865172, gspt_info_2[i]= 0
  2017. definition of type is half.
  2018. gspt r[i] = 0.0078125 (Q_frac=26
  2019. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  2020. In blob.cpp, blob.Q_frac= 26
  2021. In blob.cpp, blob.Q_frac= 26
  2022. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  2023. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  2024. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  2025. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  2026. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  2027. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  2028. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  2029. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  2030. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  2031. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  2032. in math_function.cpp, line 1227, tmp_value = 0
  2033. definition of type is half [FILL ZERO].
  2034. gspt r[i] = 0 (Q_frac=22
  2035. In blob.cpp, blob.Q_frac= 22
  2036. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  2037. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  2038. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  2039. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  2040. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  2041. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  2042. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  2043. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  2044. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  2045. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  2046. in math_function.cpp, line 1227, tmp_value = 0
  2047. definition of type is half [FILL ZERO].
  2048. gspt r[i] = 0 (Q_frac=22
  2049. In blob.cpp, blob.Q_frac= 22
  2050. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  2051. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  2052. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  2053. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  2054. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  2055. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  2056. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  2057. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  2058. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  2059. in math_function.cpp, line 730, tmp_value = 1048576
  2060. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  2061. definition of type is half.
  2062. gspt r[i] = 0.0078125 (Q_frac=27
  2063. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  2064. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  2065. in math_function.cpp line 565, grp_num= 1, rand_index= 2
  2066. in math_function.cpp line 565, grp_num= 2, rand_index= 1
  2067. in math_function.cpp line 565, grp_num= 3, rand_index= 1
  2068. in math_function.cpp line 565, grp_num= 4, rand_index= 2
  2069. in math_function.cpp line 565, grp_num= 5, rand_index= 5
  2070. in math_function.cpp line 565, grp_num= 6, rand_index= 4
  2071. in math_function.cpp line 565, grp_num= 7, rand_index= 0
  2072. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  2073. in math_function.cpp, line 730, tmp_value = -1835008
  2074. in math_function.cpp line 656, gspt_info_1[i]= 8951136, gspt_info_2[i]= 0
  2075. definition of type is half.
  2076. gspt r[i] = -0.0136719 (Q_frac=27
  2077. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  2078. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  2079. in math_function.cpp line 565, grp_num= 1, rand_index= 2
  2080. in math_function.cpp line 565, grp_num= 2, rand_index= 6
  2081. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  2082. in math_function.cpp line 565, grp_num= 4, rand_index= 2
  2083. in math_function.cpp line 565, grp_num= 5, rand_index= 6
  2084. in math_function.cpp line 565, grp_num= 6, rand_index= 3
  2085. in math_function.cpp line 565, grp_num= 7, rand_index= 4
  2086. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  2087. in math_function.cpp, line 730, tmp_value = 655360
  2088. in math_function.cpp line 656, gspt_info_1[i]= 9110940, gspt_info_2[i]= 0
  2089. definition of type is half.
  2090. gspt r[i] = 0.00976562 (Q_frac=26
  2091. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  2092. In blob.cpp, blob.Q_frac= 26
  2093. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  2094. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  2095. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  2096. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  2097. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  2098. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  2099. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  2100. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  2101. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  2102. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  2103. in math_function.cpp, line 1227, tmp_value = 0
  2104. definition of type is half [FILL ZERO].
  2105. gspt r[i] = 0 (Q_frac=22
  2106. In blob.cpp, blob.Q_frac= 22
  2107. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  2108. in math_function.cpp line 565, grp_num= 1, rand_index= 6
  2109. in math_function.cpp line 565, grp_num= 2, rand_index= 1
  2110. in math_function.cpp line 565, grp_num= 3, rand_index= 3
  2111. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  2112. in math_function.cpp line 565, grp_num= 5, rand_index= 2
  2113. in math_function.cpp line 565, grp_num= 6, rand_index= 3
  2114. in math_function.cpp line 565, grp_num= 7, rand_index= 3
  2115. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  2116. in math_function.cpp, line 730, tmp_value = 131072
  2117. in math_function.cpp line 656, gspt_info_1[i]= 10009755, gspt_info_2[i]= 0
  2118. definition of type is half.
  2119. gspt r[i] = 0.000976562 (Q_frac=27
  2120. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  2121. in math_function.cpp line 565, grp_num= 0, rand_index= 5
  2122. in math_function.cpp line 565, grp_num= 1, rand_index= 0
  2123. in math_function.cpp line 565, grp_num= 2, rand_index= 6
  2124. in math_function.cpp line 565, grp_num= 3, rand_index= 3
  2125. in math_function.cpp line 565, grp_num= 4, rand_index= 0
  2126. in math_function.cpp line 565, grp_num= 5, rand_index= 3
  2127. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  2128. in math_function.cpp line 565, grp_num= 7, rand_index= 0
  2129. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  2130. in math_function.cpp, line 730, tmp_value = -2097152
  2131. in math_function.cpp line 656, gspt_info_1[i]= 10694864, gspt_info_2[i]= 0
  2132. definition of type is half.
  2133. gspt r[i] = -0.03125 (Q_frac=26
  2134. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  2135. In blob.cpp, blob.Q_frac= 26
  2136. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  2137. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  2138. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  2139. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  2140. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  2141. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  2142. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  2143. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  2144. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  2145. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  2146. in math_function.cpp, line 1227, tmp_value = 0
  2147. definition of type is half [FILL ZERO].
  2148. gspt r[i] = 0 (Q_frac=22
  2149. In blob.cpp, blob.Q_frac= 22
  2150. in math_function.cpp line 565, grp_num= 0, rand_index= 5
  2151. in math_function.cpp line 565, grp_num= 1, rand_index= 3
  2152. in math_function.cpp line 565, grp_num= 2, rand_index= 2
  2153. in math_function.cpp line 565, grp_num= 3, rand_index= 2
  2154. in math_function.cpp line 565, grp_num= 4, rand_index= 1
  2155. in math_function.cpp line 565, grp_num= 5, rand_index= 1
  2156. in math_function.cpp line 565, grp_num= 6, rand_index= 1
  2157. in math_function.cpp line 565, grp_num= 7, rand_index= 0
  2158. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  2159. in math_function.cpp, line 730, tmp_value = -2228224
  2160. in math_function.cpp line 656, gspt_info_1[i]= 11346504, gspt_info_2[i]= 0
  2161. definition of type is half.
  2162. gspt r[i] = -0.0166016 (Q_frac=27
  2163. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  2164. In blob.cpp, blob.Q_frac= 27
  2165. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  2166. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  2167. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  2168. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  2169. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  2170. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  2171. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  2172. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  2173. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  2174. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  2175. in math_function.cpp, line 1227, tmp_value = 0
  2176. definition of type is half [FILL ZERO].
  2177. gspt r[i] = 0 (Q_frac=22
  2178. In blob.cpp, blob.Q_frac= 22
  2179. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  2180. in math_function.cpp line 565, grp_num= 1, rand_index= 0
  2181. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  2182. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  2183. in math_function.cpp line 565, grp_num= 4, rand_index= 1
  2184. in math_function.cpp line 565, grp_num= 5, rand_index= 4
  2185. in math_function.cpp line 565, grp_num= 6, rand_index= 1
  2186. in math_function.cpp line 565, grp_num= 7, rand_index= 3
  2187. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  2188. in math_function.cpp, line 730, tmp_value = -131072
  2189. in math_function.cpp line 656, gspt_info_1[i]= 8389387, gspt_info_2[i]= 0
  2190. definition of type is half.
  2191. gspt r[i] = -0.00195312 (Q_frac=26
  2192. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  2193. In blob.cpp, blob.Q_frac= 27
  2194. In blob.cpp, blob.Q_frac= 26
  2195. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  2196. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  2197. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  2198. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  2199. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  2200. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  2201. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  2202. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  2203. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  2204. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  2205. in math_function.cpp, line 1227, tmp_value = 0
  2206. definition of type is half [FILL ZERO].
  2207. gspt r[i] = 0 (Q_frac=22
  2208. In blob.cpp, blob.Q_frac= 22
  2209. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  2210. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  2211. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  2212. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  2213. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  2214. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  2215. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  2216. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  2217. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  2218. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  2219. in math_function.cpp, line 1227, tmp_value = 0
  2220. definition of type is half [FILL ZERO].
  2221. gspt r[i] = 0 (Q_frac=22
  2222. In blob.cpp, blob.Q_frac= 22
  2223. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  2224. in math_function.cpp line 565, grp_num= 1, rand_index= 0
  2225. in math_function.cpp line 565, grp_num= 2, rand_index= 6
  2226. in math_function.cpp line 565, grp_num= 3, rand_index= 1
  2227. in math_function.cpp line 565, grp_num= 4, rand_index= 0
  2228. in math_function.cpp line 565, grp_num= 5, rand_index= 4
  2229. in math_function.cpp line 565, grp_num= 6, rand_index= 3
  2230. in math_function.cpp line 565, grp_num= 7, rand_index= 2
  2231. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  2232. in math_function.cpp, line 730, tmp_value = -393216
  2233. in math_function.cpp line 656, gspt_info_1[i]= 8589594, gspt_info_2[i]= 0
  2234. definition of type is half.
  2235. gspt r[i] = -0.00585938 (Q_frac=26
  2236. I0819 14:26:15.608752 12184 solver.cpp:55] Solver scaffolding done.
  2237. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  2238. In blob.cpp, blob.Q_frac= 26
  2239. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  2240. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  2241. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  2242. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  2243. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  2244. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  2245. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  2246. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  2247. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  2248. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  2249. in math_function.cpp, line 1227, tmp_value = 0
  2250. definition of type is half [FILL ZERO].
  2251. gspt r[i] = 0 (Q_frac=22
  2252. In blob.cpp, blob.Q_frac= 22
  2253. I0819 14:26:15.621356 12187 solver.cpp:55] Solver scaffolding done.
  2254. In blob.cpp, blob.Q_frac= 27
  2255. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  2256. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  2257. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  2258. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  2259. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  2260. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  2261. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  2262. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  2263. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  2264. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  2265. in math_function.cpp, line 1227, tmp_value = 0
  2266. definition of type is half [FILL ZERO].
  2267. gspt r[i] = 0 (Q_frac=22
  2268. In blob.cpp, blob.Q_frac= 22
  2269. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  2270. in math_function.cpp line 565, grp_num= 1, rand_index= 0
  2271. in math_function.cpp line 565, grp_num= 2, rand_index= 6
  2272. in math_function.cpp line 565, grp_num= 3, rand_index= 1
  2273. in math_function.cpp line 565, grp_num= 4, rand_index= 5
  2274. in math_function.cpp line 565, grp_num= 5, rand_index= 1
  2275. in math_function.cpp line 565, grp_num= 6, rand_index= 1
  2276. in math_function.cpp line 565, grp_num= 7, rand_index= 2
  2277. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  2278. in math_function.cpp, line 730, tmp_value = -655360
  2279. in math_function.cpp line 656, gspt_info_1[i]= 8591946, gspt_info_2[i]= 0
  2280. definition of type is half.
  2281. gspt r[i] = -0.00976562 (Q_frac=26
  2282. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  2283. In blob.cpp, blob.Q_frac= 26
  2284. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  2285. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  2286. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  2287. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  2288. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  2289. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  2290. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  2291. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  2292. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  2293. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  2294. in math_function.cpp, line 1227, tmp_value = 0
  2295. definition of type is half [FILL ZERO].
  2296. gspt r[i] = 0 (Q_frac=22
  2297. In blob.cpp, blob.Q_frac= 22
  2298. I0819 14:26:16.119555 12185 solver.cpp:174] Creating test net (#0) specified by net file: examples/mnist/lenet_train_test_fp16.prototxt
  2299. W0819 14:26:16.119645 12185 parallel.cpp:319] Batch size must be divisible by the number of solvers (GPUs): it's been adjusted from 100 to 104
  2300. I0819 14:26:16.119815 12185 internal_thread.cpp:18] Starting 1 internal thread(s) on device 5
  2301. I0819 14:26:16.119961 12185 data_reader.cpp:59] Data Reader threads: 1, out queues: 1, depth: 13
  2302. I0819 14:26:16.121404 12185 internal_thread.cpp:18] Starting 1 internal thread(s) on device 5
  2303. I0819 14:26:16.122711 12241 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_test_lmdb
  2304. I0819 14:26:16.123720 12185 data_layer.cpp:200] (n0.d5.r5) Output data size: 13, 1, 28, 28
  2305. I0819 14:26:16.123762 12185 internal_thread.cpp:18] Starting 1 internal thread(s) on device 5
  2306. I0819 14:26:16.125058 12242 data_layer.cpp:105] (n0.d5.r5) Parser threads: 1
  2307. I0819 14:26:16.125072 12242 data_layer.cpp:107] (n0.d5.r5) Transformer threads: 1
  2308. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  2309. in math_function.cpp line 565, grp_num= 1, rand_index= 6
  2310. in math_function.cpp line 565, grp_num= 2, rand_index= 2
  2311. in math_function.cpp line 565, grp_num= 3, rand_index= 2
  2312. in math_function.cpp line 565, grp_num= 4, rand_index= 3
  2313. in math_function.cpp line 565, grp_num= 5, rand_index= 2
  2314. in math_function.cpp line 565, grp_num= 6, rand_index= 1
  2315. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  2316. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  2317. in math_function.cpp, line 730, tmp_value = 917504
  2318. in math_function.cpp line 656, gspt_info_1[i]= 10036877, gspt_info_2[i]= 0
  2319. definition of type is half.
  2320. gspt r[i] = 0.0546875 (Q_frac=24
  2321. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  2322. In blob.cpp, blob.Q_frac= 24
  2323. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  2324. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  2325. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  2326. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  2327. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  2328. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  2329. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  2330. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  2331. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  2332. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  2333. in math_function.cpp, line 1227, tmp_value = 0
  2334. definition of type is half [FILL ZERO].
  2335. gspt r[i] = 0 (Q_frac=22
  2336. In blob.cpp, blob.Q_frac= 22
  2337. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  2338. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  2339. in math_function.cpp line 565, grp_num= 2, rand_index= 2
  2340. in math_function.cpp line 565, grp_num= 3, rand_index= 1
  2341. in math_function.cpp line 565, grp_num= 4, rand_index= 2
  2342. in math_function.cpp line 565, grp_num= 5, rand_index= 2
  2343. in math_function.cpp line 565, grp_num= 6, rand_index= 3
  2344. in math_function.cpp line 565, grp_num= 7, rand_index= 2
  2345. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  2346. in math_function.cpp, line 730, tmp_value = -393216
  2347. in math_function.cpp line 656, gspt_info_1[i]= 9770138, gspt_info_2[i]= 0
  2348. definition of type is half.
  2349. gspt r[i] = -0.00585938 (Q_frac=26
  2350. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  2351. In blob.cpp, blob.Q_frac= 26
  2352. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  2353. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  2354. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  2355. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  2356. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  2357. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  2358. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  2359. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  2360. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  2361. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  2362. in math_function.cpp, line 1227, tmp_value = 0
  2363. definition of type is half [FILL ZERO].
  2364. gspt r[i] = 0 (Q_frac=22
  2365. In blob.cpp, blob.Q_frac= 22
  2366. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  2367. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  2368. in math_function.cpp line 565, grp_num= 2, rand_index= 6
  2369. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  2370. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  2371. in math_function.cpp line 565, grp_num= 5, rand_index= 3
  2372. in math_function.cpp line 565, grp_num= 6, rand_index= 3
  2373. in math_function.cpp line 565, grp_num= 7, rand_index= 4
  2374. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  2375. in math_function.cpp, line 730, tmp_value = 655360
  2376. in math_function.cpp line 656, gspt_info_1[i]= 9899228, gspt_info_2[i]= 0
  2377. definition of type is half.
  2378. gspt r[i] = 0.00488281 (Q_frac=27
  2379. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  2380. In blob.cpp, blob.Q_frac= 27
  2381. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  2382. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  2383. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  2384. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  2385. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  2386. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  2387. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  2388. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  2389. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  2390. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  2391. in math_function.cpp, line 1227, tmp_value = 0
  2392. definition of type is half [FILL ZERO].
  2393. gspt r[i] = 0 (Q_frac=22
  2394. In blob.cpp, blob.Q_frac= 22
  2395. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  2396. in math_function.cpp line 565, grp_num= 1, rand_index= 0
  2397. in math_function.cpp line 565, grp_num= 2, rand_index= 5
  2398. in math_function.cpp line 565, grp_num= 3, rand_index= 4
  2399. in math_function.cpp line 565, grp_num= 4, rand_index= 2
  2400. in math_function.cpp line 565, grp_num= 5, rand_index= 4
  2401. in math_function.cpp line 565, grp_num= 6, rand_index= 1
  2402. in math_function.cpp line 565, grp_num= 7, rand_index= 4
  2403. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  2404. in math_function.cpp, line 730, tmp_value = 393216
  2405. in math_function.cpp line 656, gspt_info_1[i]= 8570124, gspt_info_2[i]= 0
  2406. definition of type is half.
  2407. gspt r[i] = 0.00585938 (Q_frac=26
  2408. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  2409. In blob.cpp, blob.Q_frac= 26
  2410. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  2411. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  2412. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  2413. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  2414. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  2415. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  2416. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  2417. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  2418. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  2419. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  2420. in math_function.cpp, line 1227, tmp_value = 0
  2421. definition of type is half [FILL ZERO].
  2422. gspt r[i] = 0 (Q_frac=22
  2423. In blob.cpp, blob.Q_frac= 22
  2424. I0819 14:26:16.677002 12186 solver.cpp:174] Creating test net (#0) specified by net file: examples/mnist/lenet_train_test_fp16.prototxt
  2425. W0819 14:26:16.677161 12186 parallel.cpp:319] Batch size must be divisible by the number of solvers (GPUs): it's been adjusted from 100 to 104
  2426. I0819 14:26:16.677362 12186 internal_thread.cpp:18] Starting 1 internal thread(s) on device 6
  2427. I0819 14:26:16.677531 12186 data_reader.cpp:59] Data Reader threads: 1, out queues: 1, depth: 13
  2428. I0819 14:26:16.679579 12186 internal_thread.cpp:18] Starting 1 internal thread(s) on device 6
  2429. I0819 14:26:16.681725 12244 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_test_lmdb
  2430. I0819 14:26:16.682727 12186 data_layer.cpp:200] (n0.d6.r6) Output data size: 13, 1, 28, 28
  2431. I0819 14:26:16.682757 12186 internal_thread.cpp:18] Starting 1 internal thread(s) on device 6
  2432. I0819 14:26:16.684469 12245 data_layer.cpp:105] (n0.d6.r6) Parser threads: 1
  2433. I0819 14:26:16.684499 12245 data_layer.cpp:107] (n0.d6.r6) Transformer threads: 1
  2434. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  2435. in math_function.cpp line 565, grp_num= 1, rand_index= 4
  2436. in math_function.cpp line 565, grp_num= 2, rand_index= 6
  2437. in math_function.cpp line 565, grp_num= 3, rand_index= 2
  2438. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  2439. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  2440. in math_function.cpp line 565, grp_num= 6, rand_index= 4
  2441. in math_function.cpp line 565, grp_num= 7, rand_index= 3
  2442. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  2443. in math_function.cpp, line 730, tmp_value = 262144
  2444. in math_function.cpp line 656, gspt_info_1[i]= 9645091, gspt_info_2[i]= 0
  2445. definition of type is half.
  2446. gspt r[i] = 0.015625 (Q_frac=24
  2447. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  2448. In blob.cpp, blob.Q_frac= 24
  2449. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  2450. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  2451. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  2452. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  2453. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  2454. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  2455. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  2456. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  2457. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  2458. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  2459. in math_function.cpp, line 1227, tmp_value = 0
  2460. definition of type is half [FILL ZERO].
  2461. gspt r[i] = 0 (Q_frac=22
  2462. In blob.cpp, blob.Q_frac= 22
  2463. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  2464. in math_function.cpp line 565, grp_num= 1, rand_index= 6
  2465. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  2466. in math_function.cpp line 565, grp_num= 3, rand_index= 1
  2467. in math_function.cpp line 565, grp_num= 4, rand_index= 4
  2468. in math_function.cpp line 565, grp_num= 5, rand_index= 5
  2469. in math_function.cpp line 565, grp_num= 6, rand_index= 3
  2470. in math_function.cpp line 565, grp_num= 7, rand_index= 4
  2471. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  2472. in math_function.cpp, line 730, tmp_value = 655360
  2473. in math_function.cpp line 656, gspt_info_1[i]= 9967964, gspt_info_2[i]= 0
  2474. definition of type is half.
  2475. gspt r[i] = 0.00976562 (Q_frac=26
  2476. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  2477. In blob.cpp, blob.Q_frac= 26
  2478. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  2479. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  2480. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  2481. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  2482. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  2483. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  2484. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  2485. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  2486. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  2487. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  2488. in math_function.cpp, line 1227, tmp_value = 0
  2489. definition of type is half [FILL ZERO].
  2490. gspt r[i] = 0 (Q_frac=22
  2491. In blob.cpp, blob.Q_frac= 22
  2492. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  2493. in math_function.cpp line 565, grp_num= 1, rand_index= 1
  2494. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  2495. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  2496. in math_function.cpp line 565, grp_num= 4, rand_index= 4
  2497. in math_function.cpp line 565, grp_num= 5, rand_index= 1
  2498. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  2499. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  2500. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  2501. in math_function.cpp, line 730, tmp_value = 1048576
  2502. in math_function.cpp line 656, gspt_info_1[i]= 8652885, gspt_info_2[i]= 0
  2503. definition of type is half.
  2504. gspt r[i] = 0.0078125 (Q_frac=27
  2505. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  2506. In blob.cpp, blob.Q_frac= 27
  2507. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  2508. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  2509. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  2510. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  2511. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  2512. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  2513. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  2514. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  2515. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  2516. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  2517. in math_function.cpp, line 1227, tmp_value = 0
  2518. definition of type is half [FILL ZERO].
  2519. gspt r[i] = 0 (Q_frac=22
  2520. In blob.cpp, blob.Q_frac= 22
  2521. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  2522. in math_function.cpp line 565, grp_num= 1, rand_index= 0
  2523. in math_function.cpp line 565, grp_num= 2, rand_index= 6
  2524. in math_function.cpp line 565, grp_num= 3, rand_index= 2
  2525. in math_function.cpp line 565, grp_num= 4, rand_index= 1
  2526. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  2527. in math_function.cpp line 565, grp_num= 6, rand_index= 3
  2528. in math_function.cpp line 565, grp_num= 7, rand_index= 6
  2529. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  2530. in math_function.cpp, line 730, tmp_value = 2228224
  2531. in math_function.cpp line 656, gspt_info_1[i]= 8593950, gspt_info_2[i]= 0
  2532. definition of type is half.
  2533. gspt r[i] = 0.0332031 (Q_frac=26
  2534. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  2535. In blob.cpp, blob.Q_frac= 26
  2536. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  2537. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  2538. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  2539. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  2540. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  2541. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  2542. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  2543. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  2544. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  2545. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  2546. in math_function.cpp, line 1227, tmp_value = 0
  2547. definition of type is half [FILL ZERO].
  2548. gspt r[i] = 0 (Q_frac=22
  2549. In blob.cpp, blob.Q_frac= 22
  2550. I0819 14:26:17.502923 12185 solver.cpp:55] Solver scaffolding done.
  2551. In blob.cpp, blob.Q_frac= 27
  2552. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  2553. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  2554. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  2555. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  2556. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  2557. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  2558. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  2559. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  2560. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  2561. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  2562. in math_function.cpp, line 1227, tmp_value = 0
  2563. definition of type is half [FILL ZERO].
  2564. gspt r[i] = 0 (Q_frac=22
  2565. In blob.cpp, blob.Q_frac= 22
  2566. in math_function.cpp line 565, grp_num= 0, rand_index= 4
  2567. in math_function.cpp line 565, grp_num= 1, rand_index= 5
  2568. in math_function.cpp line 565, grp_num= 2, rand_index= 0
  2569. in math_function.cpp line 565, grp_num= 3, rand_index= 0
  2570. in math_function.cpp line 565, grp_num= 4, rand_index= 6
  2571. in math_function.cpp line 565, grp_num= 5, rand_index= 0
  2572. in math_function.cpp line 565, grp_num= 6, rand_index= 2
  2573. in math_function.cpp line 565, grp_num= 7, rand_index= 5
  2574. in math_function.cpp line 565, grp_num= 8, rand_index= 0
  2575. in math_function.cpp, line 730, tmp_value = 1048576
  2576. in math_function.cpp line 656, gspt_info_1[i]= 9702421, gspt_info_2[i]= 0
  2577. definition of type is half.
  2578. gspt r[i] = 0.015625 (Q_frac=26
  2579. In fp32 2^(16):65536.00000000000000000000, in fp16 2^(16):inf
  2580. In blob.cpp, blob.Q_frac= 26
  2581. in math_function.cpp line 934, grp_num= 0, rand_index= 3
  2582. in math_function.cpp line 934, grp_num= 1, rand_index= 3
  2583. in math_function.cpp line 934, grp_num= 2, rand_index= 3
  2584. in math_function.cpp line 934, grp_num= 3, rand_index= 2
  2585. in math_function.cpp line 934, grp_num= 4, rand_index= 3
  2586. in math_function.cpp line 934, grp_num= 5, rand_index= 3
  2587. in math_function.cpp line 934, grp_num= 6, rand_index= 2
  2588. in math_function.cpp line 934, grp_num= 7, rand_index= 3
  2589. in math_function.cpp line 934, grp_num= 8, rand_index= 0
  2590. in math_function.cpp line 1223, gspt_info_1[i]= 7186131, gspt_info_2[i]= 0
  2591. in math_function.cpp, line 1227, tmp_value = 0
  2592. definition of type is half [FILL ZERO].
  2593. gspt r[i] = 0 (Q_frac=22
  2594. In blob.cpp, blob.Q_frac= 22
  2595. I0819 14:26:17.563305 12186 solver.cpp:55] Solver scaffolding done.
  2596. I0819 14:26:20.059680 12183 parallel.cpp:221] [3 - 3] P2PSync added callback
  2597. I0819 14:26:20.059725 12181 parallel.cpp:221] [1 - 1] P2PSync added callback
  2598. I0819 14:26:20.059744 12183 solver.cpp:418] [0.3] Solving LeNet-fp16 Learning Rate Policy: inv
  2599. I0819 14:26:20.059762 12181 solver.cpp:418] [0.1] Solving LeNet-fp16 Learning Rate Policy: inv
  2600. I0819 14:26:20.059836 12182 parallel.cpp:221] [2 - 2] P2PSync added callback
  2601. I0819 14:26:20.059872 12182 solver.cpp:418] [0.2] Solving LeNet-fp16 Learning Rate Policy: inv
  2602. I0819 14:26:20.059886 12185 parallel.cpp:221] [5 - 5] P2PSync added callback
  2603. I0819 14:26:20.059888 12180 parallel.cpp:221] [0 - 0] P2PSync added callback
  2604. I0819 14:26:20.059901 12187 parallel.cpp:221] [7 - 7] P2PSync added callback
  2605. I0819 14:26:20.059939 12186 parallel.cpp:221] [6 - 6] P2PSync added callback
  2606. I0819 14:26:20.059949 12185 solver.cpp:418] [0.5] Solving LeNet-fp16 Learning Rate Policy: inv
  2607. I0819 14:26:20.059950 12180 solver.cpp:418] [0.0] Solving LeNet-fp16 Learning Rate Policy: inv
  2608. I0819 14:26:20.059957 12187 solver.cpp:418] [0.7] Solving LeNet-fp16 Learning Rate Policy: inv
  2609. I0819 14:26:20.059945 12184 parallel.cpp:221] [4 - 4] P2PSync added callback
  2610. I0819 14:26:20.059979 12186 solver.cpp:418] [0.6] Solving LeNet-fp16 Learning Rate Policy: inv
  2611. I0819 14:26:20.060006 12184 solver.cpp:418] [0.4] Solving LeNet-fp16 Learning Rate Policy: inv
  2612. I0819 14:26:20.060168 12187 net.cpp:1441] [0.7] Reserving 862592 bytes of shared learnable space for type FLOAT16
  2613. I0819 14:26:20.060173 12181 net.cpp:1441] [0.1] Reserving 862592 bytes of shared learnable space for type FLOAT16
  2614. I0819 14:26:20.060178 12183 net.cpp:1441] [0.3] Reserving 862592 bytes of shared learnable space for type FLOAT16
  2615. I0819 14:26:20.060173 12184 net.cpp:1441] [0.4] Reserving 862592 bytes of shared learnable space for type FLOAT16
  2616. I0819 14:26:20.060189 12186 net.cpp:1441] [0.6] Reserving 862592 bytes of shared learnable space for type FLOAT16
  2617. I0819 14:26:20.060181 12180 net.cpp:1441] [0.0] Reserving 862592 bytes of shared learnable space for type FLOAT16
  2618. I0819 14:26:20.060190 12182 net.cpp:1441] [0.2] Reserving 862592 bytes of shared learnable space for type FLOAT16
  2619. I0819 14:26:20.060183 12185 net.cpp:1441] [0.5] Reserving 862592 bytes of shared learnable space for type FLOAT16
  2620. I0819 14:26:20.077126 12181 solver.cpp:220] [0.1] Starting Optimization on GPU 1
  2621. I0819 14:26:20.077139 12184 solver.cpp:220] [0.4] Starting Optimization on GPU 4
  2622. I0819 14:26:20.077128 12185 solver.cpp:220] [0.5] Starting Optimization on GPU 5
  2623. I0819 14:26:20.077129 12186 solver.cpp:220] [0.6] Starting Optimization on GPU 6
  2624. I0819 14:26:20.077154 12182 solver.cpp:220] [0.2] Starting Optimization on GPU 2
  2625. I0819 14:26:20.077143 12183 solver.cpp:220] [0.3] Starting Optimization on GPU 3
  2626. I0819 14:26:20.077142 12180 solver.cpp:220] [0.0] Starting Optimization on GPU 0
  2627. I0819 14:26:20.077142 12187 solver.cpp:220] [0.7] Starting Optimization on GPU 7
  2628. I0819 14:26:20.080057 12255 common.cpp:555] {4} NVML succeeded to set CPU affinity
  2629. I0819 14:26:20.080103 12180 solver.cpp:257] [MultiGPU] Initial Test started...
  2630. I0819 14:26:20.080142 12180 solver.cpp:501] Iteration 0, Testing net (#0)
  2631. I0819 14:26:20.080691 12257 common.cpp:555] {3} NVML succeeded to set CPU affinity
  2632. I0819 14:26:20.080940 12261 common.cpp:555] {7} NVML succeeded to set CPU affinity
  2633. I0819 14:26:20.081197 12259 common.cpp:555] {6} NVML succeeded to set CPU affinity
  2634. I0819 14:26:20.081463 12258 common.cpp:555] {1} NVML succeeded to set CPU affinity
  2635. I0819 14:26:20.081702 12256 common.cpp:555] {5} NVML succeeded to set CPU affinity
  2636. I0819 14:26:20.081957 12260 common.cpp:555] {0} NVML succeeded to set CPU affinity
  2637. I0819 14:26:20.082206 12262 common.cpp:555] {2} NVML succeeded to set CPU affinity
  2638. I0819 14:26:21.656210 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.0769043
  2639. I0819 14:26:21.656258 12180 solver.cpp:588] (0.0) Test net output #1: loss = 2.30273 (* 1 = 2.30273 loss)
  2640. I0819 14:26:21.656350 12180 solver.cpp:262] [MultiGPU] Initial Test completed in 1.57611s
  2641. I0819 14:26:21.676004 12212 internal_thread.cpp:42] Restarting 4 internal thread(s) on device 3
  2642. I0819 14:26:21.691151 12215 internal_thread.cpp:42] Restarting 4 internal thread(s) on device 1
  2643. I0819 14:26:21.707749 12213 internal_thread.cpp:42] Restarting 4 internal thread(s) on device 2
  2644. I0819 14:26:21.722822 12223 internal_thread.cpp:42] Restarting 4 internal thread(s) on device 6
  2645. I0819 14:26:21.736966 12224 internal_thread.cpp:42] Restarting 4 internal thread(s) on device 7
  2646. I0819 14:26:21.755342 12214 internal_thread.cpp:42] Restarting 4 internal thread(s) on device 4
  2647. I0819 14:26:21.771665 12175 internal_thread.cpp:42] Restarting 4 internal thread(s) on device 0
  2648. I0819 14:26:21.787470 12222 internal_thread.cpp:42] Restarting 4 internal thread(s) on device 5
  2649. I0819 14:26:21.787524 12212 internal_thread.cpp:18] Starting 1 internal thread(s) on device 3
  2650. I0819 14:26:21.787591 12213 internal_thread.cpp:18] Starting 1 internal thread(s) on device 2
  2651. I0819 14:26:21.787559 12215 internal_thread.cpp:18] Starting 1 internal thread(s) on device 1
  2652. I0819 14:26:21.787701 12223 internal_thread.cpp:18] Starting 1 internal thread(s) on device 6
  2653. I0819 14:26:21.788395 12224 internal_thread.cpp:18] Starting 1 internal thread(s) on device 7
  2654. I0819 14:26:21.788424 12175 internal_thread.cpp:18] Starting 1 internal thread(s) on device 0
  2655. I0819 14:26:21.788426 12214 internal_thread.cpp:18] Starting 1 internal thread(s) on device 4
  2656. I0819 14:26:21.792286 12212 data_reader.cpp:59] Data Reader threads: 3, out queues: 12, depth: 8
  2657. I0819 14:26:21.792323 12215 data_reader.cpp:59] Data Reader threads: 3, out queues: 12, depth: 8
  2658. I0819 14:26:21.792358 12304 common.cpp:555] {1} NVML succeeded to set CPU affinity
  2659. I0819 14:26:21.792376 12214 data_reader.cpp:59] Data Reader threads: 3, out queues: 12, depth: 8
  2660. I0819 14:26:21.792440 12175 data_reader.cpp:59] Data Reader threads: 3, out queues: 12, depth: 8
  2661. I0819 14:26:21.793954 12305 common.cpp:555] {3} NVML succeeded to set CPU affinity
  2662. I0819 14:26:21.795452 12306 common.cpp:555] {4} NVML succeeded to set CPU affinity
  2663. I0819 14:26:21.796964 12309 common.cpp:555] {2} NVML succeeded to set CPU affinity
  2664. I0819 14:26:21.798382 12310 common.cpp:555] {6} NVML succeeded to set CPU affinity
  2665. I0819 14:26:21.800101 12307 common.cpp:555] {0} NVML succeeded to set CPU affinity
  2666. I0819 14:26:21.801498 12222 internal_thread.cpp:18] Starting 1 internal thread(s) on device 5
  2667. I0819 14:26:21.801631 12308 common.cpp:555] {7} NVML succeeded to set CPU affinity
  2668. I0819 14:26:21.801648 12215 internal_thread.cpp:18] Starting 3 internal thread(s) on device 1
  2669. I0819 14:26:21.801681 12224 data_reader.cpp:59] Data Reader threads: 3, out queues: 12, depth: 8
  2670. I0819 14:26:21.801687 12214 internal_thread.cpp:18] Starting 3 internal thread(s) on device 4
  2671. I0819 14:26:21.801651 12212 internal_thread.cpp:18] Starting 3 internal thread(s) on device 3
  2672. I0819 14:26:21.801689 12175 internal_thread.cpp:18] Starting 3 internal thread(s) on device 0
  2673. I0819 14:26:21.801734 12213 data_reader.cpp:59] Data Reader threads: 3, out queues: 12, depth: 8
  2674. I0819 14:26:21.801728 12222 data_reader.cpp:59] Data Reader threads: 3, out queues: 12, depth: 8
  2675. I0819 14:26:21.801715 12223 data_reader.cpp:59] Data Reader threads: 3, out queues: 12, depth: 8
  2676. I0819 14:26:21.804661 12311 common.cpp:555] {5} NVML succeeded to set CPU affinity
  2677. I0819 14:26:21.805318 12313 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_train_lmdb
  2678. I0819 14:26:21.806767 12223 internal_thread.cpp:18] Starting 3 internal thread(s) on device 6
  2679. I0819 14:26:21.806782 12312 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_train_lmdb
  2680. I0819 14:26:21.806785 12222 internal_thread.cpp:18] Starting 3 internal thread(s) on device 5
  2681. I0819 14:26:21.808486 12224 internal_thread.cpp:18] Starting 3 internal thread(s) on device 7
  2682. I0819 14:26:21.808490 12316 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_train_lmdb
  2683. I0819 14:26:21.809957 12315 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_train_lmdb
  2684. I0819 14:26:21.811379 12317 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_train_lmdb
  2685. I0819 14:26:21.811403 12213 internal_thread.cpp:18] Starting 3 internal thread(s) on device 2
  2686. I0819 14:26:21.812842 12314 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_train_lmdb
  2687. I0819 14:26:21.814517 12318 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_train_lmdb
  2688. I0819 14:26:21.815894 12320 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_train_lmdb
  2689. I0819 14:26:21.817451 12321 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_train_lmdb
  2690. I0819 14:26:21.819092 12319 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_train_lmdb
  2691. I0819 14:26:21.820793 12322 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_train_lmdb
  2692. I0819 14:26:21.824617 12323 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_train_lmdb
  2693. I0819 14:26:21.825839 12324 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_train_lmdb
  2694. I0819 14:26:21.827411 12325 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_train_lmdb
  2695. I0819 14:26:21.828995 12327 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_train_lmdb
  2696. I0819 14:26:21.830147 12326 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_train_lmdb
  2697. I0819 14:26:21.831641 12328 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_train_lmdb
  2698. I0819 14:26:21.833058 12329 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_train_lmdb
  2699. I0819 14:26:21.834400 12332 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_train_lmdb
  2700. I0819 14:26:21.835896 12330 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_train_lmdb
  2701. I0819 14:26:21.837077 12331 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_train_lmdb
  2702. I0819 14:26:21.838599 12333 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_train_lmdb
  2703. I0819 14:26:21.840051 12334 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_train_lmdb
  2704. I0819 14:26:21.841423 12335 db_lmdb.cpp:36] Opened lmdb examples/mnist/mnist_train_lmdb
  2705. I0819 14:26:21.864646 12215 data_layer.cpp:200] [n0.d1.r1] Output data size: 8, 1, 28, 28
  2706. I0819 14:26:21.864681 12215 data_layer.cpp:105] [n0.d1.r1] Parser threads: 3 (auto)
  2707. I0819 14:26:21.864693 12215 data_layer.cpp:107] [n0.d1.r1] Transformer threads: 4 (auto)
  2708. I0819 14:26:21.865779 12214 data_layer.cpp:200] [n0.d4.r4] Output data size: 8, 1, 28, 28
  2709. I0819 14:26:21.865823 12214 data_layer.cpp:105] [n0.d4.r4] Parser threads: 3 (auto)
  2710. I0819 14:26:21.865835 12214 data_layer.cpp:107] [n0.d4.r4] Transformer threads: 4 (auto)
  2711. I0819 14:26:21.866470 12212 data_layer.cpp:200] [n0.d3.r3] Output data size: 8, 1, 28, 28
  2712. I0819 14:26:21.866511 12212 data_layer.cpp:105] [n0.d3.r3] Parser threads: 3 (auto)
  2713. I0819 14:26:21.866523 12212 data_layer.cpp:107] [n0.d3.r3] Transformer threads: 4 (auto)
  2714. I0819 14:26:21.867338 12175 data_layer.cpp:200] [n0.d0.r0] Output data size: 8, 1, 28, 28
  2715. I0819 14:26:21.867383 12175 data_layer.cpp:105] [n0.d0.r0] Parser threads: 3 (auto)
  2716. I0819 14:26:21.867393 12175 data_layer.cpp:107] [n0.d0.r0] Transformer threads: 4 (auto)
  2717. I0819 14:26:21.870844 12222 data_layer.cpp:200] [n0.d5.r5] Output data size: 8, 1, 28, 28
  2718. I0819 14:26:21.870885 12222 data_layer.cpp:105] [n0.d5.r5] Parser threads: 3 (auto)
  2719. I0819 14:26:21.870895 12222 data_layer.cpp:107] [n0.d5.r5] Transformer threads: 4 (auto)
  2720. I0819 14:26:21.871150 12223 data_layer.cpp:200] [n0.d6.r6] Output data size: 8, 1, 28, 28
  2721. I0819 14:26:21.871199 12223 data_layer.cpp:105] [n0.d6.r6] Parser threads: 3 (auto)
  2722. I0819 14:26:21.871244 12223 data_layer.cpp:107] [n0.d6.r6] Transformer threads: 4 (auto)
  2723. I0819 14:26:21.871914 12224 data_layer.cpp:200] [n0.d7.r7] Output data size: 8, 1, 28, 28
  2724. I0819 14:26:21.871960 12224 data_layer.cpp:105] [n0.d7.r7] Parser threads: 3 (auto)
  2725. I0819 14:26:21.871973 12224 data_layer.cpp:107] [n0.d7.r7] Transformer threads: 4 (auto)
  2726. I0819 14:26:21.877614 12213 data_layer.cpp:200] [n0.d2.r2] Output data size: 8, 1, 28, 28
  2727. I0819 14:26:21.877653 12213 data_layer.cpp:105] [n0.d2.r2] Parser threads: 3 (auto)
  2728. I0819 14:26:21.877663 12213 data_layer.cpp:107] [n0.d2.r2] Transformer threads: 4 (auto)
  2729. I0819 14:26:22.008081 12180 solver.cpp:341] [0.0] Iteration 0 (0.351633 s), loss = 2.30273
  2730. I0819 14:26:22.009574 12180 solver.cpp:361] [0.0] Train net output #0: loss = 2.30273 (* 1 = 2.30273 loss)
  2731. I0819 14:26:22.009629 12180 sgd_solver.cpp:180] [0.0] Iteration 0, lr = 0.006, m = 0.9, lrm = 0.06, wd = 0.0005, gs = 1
  2732. I0819 14:26:22.028228 12180 solver.cpp:341] [0.0] Iteration 1 (0.0201338 s), loss = 2.30273
  2733. I0819 14:26:22.028343 12180 solver.cpp:361] [0.0] Train net output #0: loss = 2.30273 (* 1 = 2.30273 loss)
  2734. I0819 14:26:22.175338 12187 cudnn_conv_layer.cpp:849] [n0.d7.r7] Conv Algos (F,BD,BF): 'conv1' with space 6.91K 1/1 0p 1 0p (avail 11.35G, req 2.06M) t: 0 0 0.1
  2735. I0819 14:26:22.256990 12183 cudnn_conv_layer.cpp:849] [n0.d3.r3] Conv Algos (F,BD,BF): 'conv1' with space 2.06M 1/1 0p 1 0p (avail 11.35G, req 2.06M) t: 0 0 0.1
  2736. I0819 14:26:22.359421 12185 cudnn_conv_layer.cpp:849] [n0.d5.r5] Conv Algos (F,BD,BF): 'conv1' with space 2.06M 1/1 1p 1 5p (avail 11.35G, req 2.06M) t: 0 0 0.11
  2737. I0819 14:26:22.361574 12184 cudnn_conv_layer.cpp:849] [n0.d4.r4] Conv Algos (F,BD,BF): 'conv1' with space 2.06M 1/1 0p 1 0p (avail 11.35G, req 2.06M) t: 0 0 0.1
  2738. I0819 14:26:22.458999 12186 cudnn_conv_layer.cpp:849] [n0.d6.r6] Conv Algos (F,BD,BF): 'conv1' with space 2.06M 1/1 0p 1 0p (avail 11.35G, req 2.06M) t: 0 0 0.11
  2739. I0819 14:26:22.461419 12182 cudnn_conv_layer.cpp:849] [n0.d2.r2] Conv Algos (F,BD,BF): 'conv1' with space 2.06M 1/1 0p 1 0p (avail 11.35G, req 2.06M) t: 0 0 0.11
  2740. I0819 14:26:22.464751 12181 cudnn_conv_layer.cpp:849] [n0.d1.r1] Conv Algos (F,BD,BF): 'conv1' with space 2.06M 1/1 0p 1 5p (avail 11.35G, req 2.06M) t: 0 0 0.11
  2741. I0819 14:26:22.473542 12180 cudnn_conv_layer.cpp:849] [n0.d0.r0] Conv Algos (F,BD,BF): 'conv1' with space 2.06M 1/1 1p 1 0p (avail 11.09G, req 2.06M) t: 0 0 0.11
  2742. I0819 14:26:22.581921 12183 cudnn_conv_layer.cpp:849] [n0.d3.r3] Conv Algos (F,BD,BF): 'conv2' with space 2.06M 20/1 7p 0p 0p (avail 11.35G, req 2.06M) t: 0 0.07 0.06
  2743. I0819 14:26:22.589903 12184 cudnn_conv_layer.cpp:849] [n0.d4.r4] Conv Algos (F,BD,BF): 'conv2' with space 2.06M 20/1 7p 0p 0p (avail 11.35G, req 2.06M) t: 0 0.07 0.06
  2744. I0819 14:26:22.591792 12187 cudnn_conv_layer.cpp:849] [n0.d7.r7] Conv Algos (F,BD,BF): 'conv2' with space 2.06M 20/1 7p 0p 0p (avail 11.35G, req 2.06M) t: 0 0.06 0.06
  2745. I0819 14:26:22.609736 12185 cudnn_conv_layer.cpp:849] [n0.d5.r5] Conv Algos (F,BD,BF): 'conv2' with space 2.06M 20/1 7p 0p 0p (avail 11.35G, req 2.06M) t: 0 0.07 0.06
  2746. I0819 14:26:22.613392 12186 cudnn_conv_layer.cpp:849] [n0.d6.r6] Conv Algos (F,BD,BF): 'conv2' with space 2.06M 20/1 7p 0p 0p (avail 11.35G, req 2.06M) t: 0 0.06 0.06
  2747. I0819 14:26:22.636795 12182 cudnn_conv_layer.cpp:849] [n0.d2.r2] Conv Algos (F,BD,BF): 'conv2' with space 2.06M 20/1 7p 0p 0p (avail 11.35G, req 2.06M) t: 0 0.06 0.05
  2748. I0819 14:26:22.650578 12181 cudnn_conv_layer.cpp:849] [n0.d1.r1] Conv Algos (F,BD,BF): 'conv2' with space 2.06M 20/1 7p 0p 0p (avail 11.35G, req 2.06M) t: 0 0.06 0.05
  2749. I0819 14:26:22.682121 12180 cudnn_conv_layer.cpp:849] [n0.d0.r0] Conv Algos (F,BD,BF): 'conv2' with space 2.06M 20/1 7p 0p 0p (avail 11.09G, req 2.06M) t: 0 0.06 0.06
  2750. I0819 14:26:22.686460 12180 solver.cpp:341] [0.0] Iteration 2 (0.658181 s), loss = 2.30273
  2751. I0819 14:26:22.686547 12180 solver.cpp:361] [0.0] Train net output #0: loss = 2.30273 (* 1 = 2.30273 loss)
  2752. I0819 14:26:23.101030 12180 solver.cpp:333] [0.0] Iteration 100 (236.412 iter/s, 0.414531s/98 iter), loss = 2.3125
  2753. I0819 14:26:23.101125 12180 solver.cpp:361] [0.0] Train net output #0: loss = 2.3125 (* 1 = 2.3125 loss)
  2754. I0819 14:26:23.101152 12180 sgd_solver.cpp:180] [0.0] Iteration 100, lr = 0.00595539, m = 0.9, lrm = 0.0595539, wd = 0.0005, gs = 1
  2755. I0819 14:26:23.484020 12180 solver.cpp:333] [0.0] Iteration 200 (261.139 iter/s, 0.382937s/100 iter), loss = 2.35156
  2756. I0819 14:26:23.484200 12180 solver.cpp:361] [0.0] Train net output #0: loss = 2.35156 (* 1 = 2.35156 loss)
  2757. I0819 14:26:23.484244 12180 sgd_solver.cpp:180] [0.0] Iteration 200, lr = 0.00591155, m = 0.9, lrm = 0.0591155, wd = 0.0005, gs = 1
  2758. I0819 14:26:23.861603 12180 solver.cpp:333] [0.0] Iteration 300 (264.859 iter/s, 0.37756s/100 iter), loss = 2.32617
  2759. I0819 14:26:23.861706 12180 solver.cpp:361] [0.0] Train net output #0: loss = 2.32617 (* 1 = 2.32617 loss)
  2760. I0819 14:26:23.861726 12180 sgd_solver.cpp:180] [0.0] Iteration 300, lr = 0.00586845, m = 0.9, lrm = 0.0586845, wd = 0.0005, gs = 1
  2761. I0819 14:26:24.242758 12180 solver.cpp:333] [0.0] Iteration 400 (262.38 iter/s, 0.381127s/100 iter), loss = 2.29492
  2762. I0819 14:26:24.242908 12180 solver.cpp:361] [0.0] Train net output #0: loss = 2.29492 (* 1 = 2.29492 loss)
  2763. I0819 14:26:24.242923 12180 sgd_solver.cpp:180] [0.0] Iteration 400, lr = 0.00582608, m = 0.9, lrm = 0.0582608, wd = 0.0005, gs = 1
  2764. I0819 14:26:24.618224 12180 solver.cpp:501] Iteration 500, Testing net (#0)
  2765. I0819 14:26:24.923928 12177 data_reader.cpp:321] Restarting data pre-fetching
  2766. I0819 14:26:24.957980 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.110742
  2767. I0819 14:26:24.958009 12180 solver.cpp:588] (0.0) Test net output #1: loss = 2.30234 (* 1 = 2.30234 loss)
  2768. I0819 14:26:24.958096 12180 solver.cpp:271] [MultiGPU] Tests completed in 0.715304s
  2769. I0819 14:26:24.961958 12180 solver.cpp:333] [0.0] Iteration 500 (139.801 iter/s, 0.715304s/100 iter), loss = 2.29492
  2770. I0819 14:26:24.962043 12180 solver.cpp:361] [0.0] Train net output #0: loss = 2.29492 (* 1 = 2.29492 loss)
  2771. I0819 14:26:24.962056 12180 sgd_solver.cpp:180] [0.0] Iteration 500, lr = 0.00578441, m = 0.9, lrm = 0.0578441, wd = 0.0005, gs = 1
  2772. I0819 14:26:25.341384 12180 solver.cpp:333] [0.0] Iteration 600 (263.582 iter/s, 0.379389s/100 iter), loss = 2.31836
  2773. I0819 14:26:25.341576 12180 solver.cpp:361] [0.0] Train net output #0: loss = 2.31836 (* 1 = 2.31836 loss)
  2774. I0819 14:26:25.341591 12180 sgd_solver.cpp:180] [0.0] Iteration 600, lr = 0.00574344, m = 0.9, lrm = 0.0574344, wd = 0.0005, gs = 1
  2775. I0819 14:26:25.718746 12180 solver.cpp:333] [0.0] Iteration 700 (265.02 iter/s, 0.377331s/100 iter), loss = 2.34766
  2776. I0819 14:26:25.718859 12180 solver.cpp:361] [0.0] Train net output #0: loss = 2.34766 (* 1 = 2.34766 loss)
  2777. I0819 14:26:25.718874 12180 sgd_solver.cpp:180] [0.0] Iteration 700, lr = 0.00570313, m = 0.9, lrm = 0.0570313, wd = 0.0005, gs = 1
  2778. I0819 14:26:26.091969 12180 solver.cpp:333] [0.0] Iteration 800 (267.958 iter/s, 0.373193s/100 iter), loss = 2.32031
  2779. I0819 14:26:26.092077 12180 solver.cpp:361] [0.0] Train net output #0: loss = 2.32031 (* 1 = 2.32031 loss)
  2780. I0819 14:26:26.092092 12180 sgd_solver.cpp:180] [0.0] Iteration 800, lr = 0.00566348, m = 0.9, lrm = 0.0566348, wd = 0.0005, gs = 1
  2781. I0819 14:26:26.462466 12180 solver.cpp:333] [0.0] Iteration 900 (269.93 iter/s, 0.370466s/100 iter), loss = 2.30469
  2782. I0819 14:26:26.462560 12180 solver.cpp:361] [0.0] Train net output #0: loss = 2.30469 (* 1 = 2.30469 loss)
  2783. I0819 14:26:26.462589 12180 sgd_solver.cpp:180] [0.0] Iteration 900, lr = 0.00562447, m = 0.9, lrm = 0.0562447, wd = 0.0005, gs = 1
  2784. I0819 14:26:26.500018 12318 data_reader.cpp:321] Restarting data pre-fetching
  2785. I0819 14:26:26.846081 12180 solver.cpp:501] Iteration 1000, Testing net (#0)
  2786. I0819 14:26:27.138350 12177 data_reader.cpp:321] Restarting data pre-fetching
  2787. I0819 14:26:27.185463 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.108435
  2788. I0819 14:26:27.185499 12180 solver.cpp:588] (0.0) Test net output #1: loss = 2.30098 (* 1 = 2.30098 loss)
  2789. I0819 14:26:27.185529 12180 solver.cpp:271] [MultiGPU] Tests completed in 0.72303s
  2790. I0819 14:26:27.189757 12180 solver.cpp:333] [0.0] Iteration 1000 (138.307 iter/s, 0.72303s/100 iter), 1.1/21.3ep, loss = 2.26758
  2791. I0819 14:26:27.189815 12180 solver.cpp:361] [0.0] Train net output #0: loss = 2.26758 (* 1 = 2.26758 loss)
  2792. I0819 14:26:27.189831 12180 sgd_solver.cpp:180] [0.0] Iteration 1000, lr = 0.00558607, m = 0.9, lrm = 0.0558607, wd = 0.0005, gs = 1
  2793. I0819 14:26:27.570098 12180 solver.cpp:333] [0.0] Iteration 1100 (262.946 iter/s, 0.380306s/100 iter), 1.2/21.3ep, loss = 2.31445
  2794. I0819 14:26:27.570201 12180 solver.cpp:361] [0.0] Train net output #0: loss = 2.31445 (* 1 = 2.31445 loss)
  2795. I0819 14:26:27.570243 12180 sgd_solver.cpp:180] [0.0] Iteration 1100, lr = 0.00554829, m = 0.9, lrm = 0.0554829, wd = 0.0005, gs = 1
  2796. I0819 14:26:27.948230 12180 solver.cpp:333] [0.0] Iteration 1200 (264.486 iter/s, 0.378093s/100 iter), 1.3/21.3ep, loss = 2.33398
  2797. I0819 14:26:27.948328 12180 solver.cpp:361] [0.0] Train net output #0: loss = 2.33398 (* 1 = 2.33398 loss)
  2798. I0819 14:26:27.948377 12180 sgd_solver.cpp:180] [0.0] Iteration 1200, lr = 0.00551109, m = 0.9, lrm = 0.0551109, wd = 0.0005, gs = 1
  2799. I0819 14:26:28.326474 12180 solver.cpp:333] [0.0] Iteration 1300 (264.401 iter/s, 0.378213s/100 iter), 1.4/21.3ep, loss = 2.3418
  2800. I0819 14:26:28.326596 12180 solver.cpp:361] [0.0] Train net output #0: loss = 2.3418 (* 1 = 2.3418 loss)
  2801. I0819 14:26:28.326632 12180 sgd_solver.cpp:180] [0.0] Iteration 1300, lr = 0.00547447, m = 0.9, lrm = 0.0547447, wd = 0.0005, gs = 1
  2802. I0819 14:26:28.703507 12180 solver.cpp:333] [0.0] Iteration 1400 (265.243 iter/s, 0.377012s/100 iter), 1.5/21.3ep, loss = 2.34375
  2803. I0819 14:26:28.703608 12180 solver.cpp:361] [0.0] Train net output #0: loss = 2.34375 (* 1 = 2.34375 loss)
  2804. I0819 14:26:28.703624 12180 sgd_solver.cpp:180] [0.0] Iteration 1400, lr = 0.00543842, m = 0.9, lrm = 0.0543842, wd = 0.0005, gs = 1
  2805. I0819 14:26:29.077296 12180 solver.cpp:501] Iteration 1500, Testing net (#0)
  2806. I0819 14:26:29.356602 12177 data_reader.cpp:321] Restarting data pre-fetching
  2807. I0819 14:26:29.425398 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.116895
  2808. I0819 14:26:29.425431 12180 solver.cpp:588] (0.0) Test net output #1: loss = 2.30863 (* 1 = 2.30863 loss)
  2809. I0819 14:26:29.425524 12180 solver.cpp:271] [MultiGPU] Tests completed in 0.721988s
  2810. I0819 14:26:29.430272 12180 solver.cpp:333] [0.0] Iteration 1500 (138.507 iter/s, 0.721988s/100 iter), 1.6/21.3ep, loss = 2.28711
  2811. I0819 14:26:29.430392 12180 solver.cpp:361] [0.0] Train net output #0: loss = 2.28711 (* 1 = 2.28711 loss)
  2812. I0819 14:26:29.430408 12180 sgd_solver.cpp:180] [0.0] Iteration 1500, lr = 0.00540291, m = 0.9, lrm = 0.0540291, wd = 0.0005, gs = 1
  2813. I0819 14:26:29.816975 12180 solver.cpp:333] [0.0] Iteration 1600 (258.62 iter/s, 0.386668s/100 iter), 1.7/21.3ep, loss = 2.30859
  2814. I0819 14:26:29.817056 12180 solver.cpp:361] [0.0] Train net output #0: loss = 2.30859 (* 1 = 2.30859 loss)
  2815. I0819 14:26:29.817080 12180 sgd_solver.cpp:180] [0.0] Iteration 1600, lr = 0.00536794, m = 0.9, lrm = 0.0536794, wd = 0.0005, gs = 1
  2816. I0819 14:26:30.191833 12180 solver.cpp:333] [0.0] Iteration 1700 (266.789 iter/s, 0.374828s/100 iter), 1.8/21.3ep, loss = 2.31055
  2817. I0819 14:26:30.191944 12180 solver.cpp:361] [0.0] Train net output #0: loss = 2.31055 (* 1 = 2.31055 loss)
  2818. I0819 14:26:30.191958 12180 sgd_solver.cpp:180] [0.0] Iteration 1700, lr = 0.00533349, m = 0.9, lrm = 0.0533349, wd = 0.0005, gs = 1
  2819. I0819 14:26:30.557349 12180 solver.cpp:333] [0.0] Iteration 1800 (273.608 iter/s, 0.365486s/100 iter), 1.9/21.3ep, loss = 0.461182
  2820. I0819 14:26:30.557466 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.461182 (* 1 = 0.461182 loss)
  2821. I0819 14:26:30.557484 12180 sgd_solver.cpp:180] [0.0] Iteration 1800, lr = 0.00529956, m = 0.9, lrm = 0.0529956, wd = 0.0005, gs = 1
  2822. I0819 14:26:30.721966 12318 data_reader.cpp:321] Restarting data pre-fetching
  2823. I0819 14:26:30.923511 12180 solver.cpp:333] [0.0] Iteration 1900 (273.13 iter/s, 0.366126s/100 iter), 2/21.3ep, loss = 0.404297
  2824. I0819 14:26:30.923636 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.404297 (* 1 = 0.404297 loss)
  2825. I0819 14:26:30.923694 12180 sgd_solver.cpp:180] [0.0] Iteration 1900, lr = 0.00526612, m = 0.9, lrm = 0.0526612, wd = 0.0005, gs = 1
  2826. I0819 14:26:31.294430 12180 solver.cpp:501] Iteration 2000, Testing net (#0)
  2827. I0819 14:26:31.559896 12177 data_reader.cpp:321] Restarting data pre-fetching
  2828. I0819 14:26:31.634709 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.937607
  2829. I0819 14:26:31.634754 12180 solver.cpp:588] (0.0) Test net output #1: loss = 0.210998 (* 1 = 0.210998 loss)
  2830. I0819 14:26:31.634786 12180 solver.cpp:271] [MultiGPU] Tests completed in 0.711246s
  2831. I0819 14:26:31.638682 12180 solver.cpp:333] [0.0] Iteration 2000 (140.598 iter/s, 0.711246s/100 iter), 2.1/21.3ep, loss = 0.00550461
  2832. I0819 14:26:31.638777 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00550461 (* 1 = 0.00550461 loss)
  2833. I0819 14:26:31.638789 12180 sgd_solver.cpp:180] [0.0] Iteration 2000, lr = 0.00523318, m = 0.9, lrm = 0.0523317, wd = 0.0005, gs = 1
  2834. I0819 14:26:32.012344 12180 solver.cpp:333] [0.0] Iteration 2100 (267.655 iter/s, 0.373615s/100 iter), 2.2/21.3ep, loss = 0.273926
  2835. I0819 14:26:32.012411 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.273926 (* 1 = 0.273926 loss)
  2836. I0819 14:26:32.012425 12180 sgd_solver.cpp:180] [0.0] Iteration 2100, lr = 0.00520071, m = 0.9, lrm = 0.052007, wd = 0.0005, gs = 1
  2837. I0819 14:26:32.390202 12180 solver.cpp:333] [0.0] Iteration 2200 (264.684 iter/s, 0.377809s/100 iter), 2.3/21.3ep, loss = 0.747559
  2838. I0819 14:26:32.390300 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.747559 (* 1 = 0.747559 loss)
  2839. I0819 14:26:32.390316 12180 sgd_solver.cpp:180] [0.0] Iteration 2200, lr = 0.0051687, m = 0.9, lrm = 0.051687, wd = 0.0005, gs = 1
  2840. I0819 14:26:32.768738 12180 solver.cpp:333] [0.0] Iteration 2300 (264.201 iter/s, 0.378499s/100 iter), 2.5/21.3ep, loss = 0.0458069
  2841. I0819 14:26:32.768821 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.0458069 (* 1 = 0.0458069 loss)
  2842. I0819 14:26:32.768837 12180 sgd_solver.cpp:180] [0.0] Iteration 2300, lr = 0.00513715, m = 0.9, lrm = 0.0513715, wd = 0.0005, gs = 1
  2843. I0819 14:26:33.147343 12180 solver.cpp:333] [0.0] Iteration 2400 (264.152 iter/s, 0.37857s/100 iter), 2.6/21.3ep, loss = 0.000305653
  2844. I0819 14:26:33.147423 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000305653 (* 1 = 0.000305653 loss)
  2845. I0819 14:26:33.147435 12180 sgd_solver.cpp:180] [0.0] Iteration 2400, lr = 0.00510605, m = 0.9, lrm = 0.0510605, wd = 0.0005, gs = 1
  2846. I0819 14:26:33.518666 12180 solver.cpp:501] Iteration 2500, Testing net (#0)
  2847. I0819 14:26:33.772068 12177 data_reader.cpp:321] Restarting data pre-fetching
  2848. I0819 14:26:33.858614 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.971484
  2849. I0819 14:26:33.858675 12180 solver.cpp:588] (0.0) Test net output #1: loss = 0.0847127 (* 1 = 0.0847127 loss)
  2850. I0819 14:26:33.858703 12180 solver.cpp:271] [MultiGPU] Tests completed in 0.711336s
  2851. I0819 14:26:33.862964 12180 solver.cpp:333] [0.0] Iteration 2500 (140.581 iter/s, 0.711336s/100 iter), 2.7/21.3ep, loss = 0.0294952
  2852. I0819 14:26:33.863178 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.0294952 (* 1 = 0.0294952 loss)
  2853. I0819 14:26:33.863195 12180 sgd_solver.cpp:180] [0.0] Iteration 2500, lr = 0.00507538, m = 0.9, lrm = 0.0507538, wd = 0.0005, gs = 1
  2854. I0819 14:26:34.237495 12180 solver.cpp:333] [0.0] Iteration 2600 (267.034 iter/s, 0.374484s/100 iter), 2.8/21.3ep, loss = 0.0422363
  2855. I0819 14:26:34.237598 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.0422363 (* 1 = 0.0422363 loss)
  2856. I0819 14:26:34.237634 12180 sgd_solver.cpp:180] [0.0] Iteration 2600, lr = 0.00504514, m = 0.9, lrm = 0.0504514, wd = 0.0005, gs = 1
  2857. I0819 14:26:34.616835 12180 solver.cpp:333] [0.0] Iteration 2700 (263.637 iter/s, 0.379309s/100 iter), 2.9/21.3ep, loss = 0.00308228
  2858. I0819 14:26:34.616940 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00308228 (* 1 = 0.00308228 loss)
  2859. I0819 14:26:34.616959 12180 sgd_solver.cpp:180] [0.0] Iteration 2700, lr = 0.00501532, m = 0.9, lrm = 0.0501532, wd = 0.0005, gs = 1
  2860. I0819 14:26:34.939465 12318 data_reader.cpp:321] Restarting data pre-fetching
  2861. I0819 14:26:35.003248 12180 solver.cpp:333] [0.0] Iteration 2800 (258.811 iter/s, 0.386382s/100 iter), 3/21.3ep, loss = 6.10352e-05
  2862. I0819 14:26:35.003343 12180 solver.cpp:361] [0.0] Train net output #0: loss = 6.10352e-05 (* 1 = 6.10352e-05 loss)
  2863. I0819 14:26:35.003365 12180 sgd_solver.cpp:180] [0.0] Iteration 2800, lr = 0.0049859, m = 0.9, lrm = 0.049859, wd = 0.0005, gs = 1
  2864. I0819 14:26:35.385121 12180 solver.cpp:333] [0.0] Iteration 2900 (261.887 iter/s, 0.381843s/100 iter), 3.1/21.3ep, loss = 0.0159302
  2865. I0819 14:26:35.385216 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.0159302 (* 1 = 0.0159302 loss)
  2866. I0819 14:26:35.385231 12180 sgd_solver.cpp:180] [0.0] Iteration 2900, lr = 0.00495689, m = 0.9, lrm = 0.0495689, wd = 0.0005, gs = 1
  2867. I0819 14:26:35.763062 12180 solver.cpp:501] Iteration 3000, Testing net (#0)
  2868. I0819 14:26:36.004958 12177 data_reader.cpp:321] Restarting data pre-fetching
  2869. I0819 14:26:36.104571 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.983032
  2870. I0819 14:26:36.104607 12180 solver.cpp:588] (0.0) Test net output #1: loss = 0.0602388 (* 1 = 0.0602388 loss)
  2871. I0819 14:26:36.104666 12180 solver.cpp:271] [MultiGPU] Tests completed in 0.719494s
  2872. I0819 14:26:36.109220 12180 solver.cpp:333] [0.0] Iteration 3000 (138.987 iter/s, 0.719494s/100 iter), 3.2/21.3ep, loss = 0.00344086
  2873. I0819 14:26:36.109329 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00344086 (* 1 = 0.00344086 loss)
  2874. I0819 14:26:36.109360 12180 sgd_solver.cpp:180] [0.0] Iteration 3000, lr = 0.00492826, m = 0.9, lrm = 0.0492826, wd = 0.0005, gs = 1
  2875. I0819 14:26:36.488287 12180 solver.cpp:333] [0.0] Iteration 3100 (263.83 iter/s, 0.379033s/100 iter), 3.3/21.3ep, loss = 0.0142212
  2876. I0819 14:26:36.488363 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.0142212 (* 1 = 0.0142212 loss)
  2877. I0819 14:26:36.488376 12180 sgd_solver.cpp:180] [0.0] Iteration 3100, lr = 0.00490002, m = 0.9, lrm = 0.0490002, wd = 0.0005, gs = 1
  2878. I0819 14:26:36.871675 12180 solver.cpp:333] [0.0] Iteration 3200 (260.857 iter/s, 0.383352s/100 iter), 3.4/21.3ep, loss = 0.0726318
  2879. I0819 14:26:36.871765 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.0726318 (* 1 = 0.0726318 loss)
  2880. I0819 14:26:36.871778 12180 sgd_solver.cpp:180] [0.0] Iteration 3200, lr = 0.00487215, m = 0.9, lrm = 0.0487215, wd = 0.0005, gs = 1
  2881. I0819 14:26:37.262538 12180 solver.cpp:333] [0.0] Iteration 3300 (255.872 iter/s, 0.390821s/100 iter), 3.5/21.3ep, loss = 0.0606079
  2882. I0819 14:26:37.262645 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.0606079 (* 1 = 0.0606079 loss)
  2883. I0819 14:26:37.262689 12180 sgd_solver.cpp:180] [0.0] Iteration 3300, lr = 0.00484465, m = 0.9, lrm = 0.0484465, wd = 0.0005, gs = 1
  2884. I0819 14:26:37.661991 12180 solver.cpp:333] [0.0] Iteration 3400 (250.363 iter/s, 0.399419s/100 iter), 3.6/21.3ep, loss = 0.0738525
  2885. I0819 14:26:37.662087 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.0738525 (* 1 = 0.0738525 loss)
  2886. I0819 14:26:37.662102 12180 sgd_solver.cpp:180] [0.0] Iteration 3400, lr = 0.00481751, m = 0.9, lrm = 0.0481751, wd = 0.0005, gs = 1
  2887. I0819 14:26:38.071043 12180 solver.cpp:501] Iteration 3500, Testing net (#0)
  2888. I0819 14:26:38.304494 12177 data_reader.cpp:321] Restarting data pre-fetching
  2889. I0819 14:26:38.414863 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.98458
  2890. I0819 14:26:38.414906 12180 solver.cpp:588] (0.0) Test net output #1: loss = 0.0620035 (* 1 = 0.0620035 loss)
  2891. I0819 14:26:38.414933 12180 solver.cpp:271] [MultiGPU] Tests completed in 0.752913s
  2892. I0819 14:26:38.419159 12180 solver.cpp:333] [0.0] Iteration 3500 (132.817 iter/s, 0.752913s/100 iter), 3.7/21.3ep, loss = 0.00721359
  2893. I0819 14:26:38.419271 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00721359 (* 1 = 0.00721359 loss)
  2894. I0819 14:26:38.419294 12180 sgd_solver.cpp:180] [0.0] Iteration 3500, lr = 0.00479072, m = 0.9, lrm = 0.0479072, wd = 0.0005, gs = 1
  2895. I0819 14:26:38.804241 12180 solver.cpp:333] [0.0] Iteration 3600 (259.708 iter/s, 0.385048s/100 iter), 3.8/21.3ep, loss = 0.0017128
  2896. I0819 14:26:38.804404 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.0017128 (* 1 = 0.0017128 loss)
  2897. I0819 14:26:38.804424 12180 sgd_solver.cpp:180] [0.0] Iteration 3600, lr = 0.00476428, m = 0.9, lrm = 0.0476428, wd = 0.0005, gs = 1
  2898. I0819 14:26:39.195200 12180 solver.cpp:333] [0.0] Iteration 3700 (255.824 iter/s, 0.390894s/100 iter), 3.9/21.3ep, loss = 0.000611782
  2899. I0819 14:26:39.195310 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000611782 (* 1 = 0.000611782 loss)
  2900. I0819 14:26:39.195330 12180 sgd_solver.cpp:180] [0.0] Iteration 3700, lr = 0.00473817, m = 0.9, lrm = 0.0473817, wd = 0.0005, gs = 1
  2901. I0819 14:26:39.275179 12318 data_reader.cpp:321] Restarting data pre-fetching
  2902. I0819 14:26:39.588418 12180 solver.cpp:333] [0.0] Iteration 3800 (254.338 iter/s, 0.393178s/100 iter), 4.1/21.3ep, loss = 0.0181732
  2903. I0819 14:26:39.588510 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.0181732 (* 1 = 0.0181732 loss)
  2904. I0819 14:26:39.588526 12180 sgd_solver.cpp:180] [0.0] Iteration 3800, lr = 0.0047124, m = 0.9, lrm = 0.047124, wd = 0.0005, gs = 1
  2905. I0819 14:26:39.973237 12180 solver.cpp:333] [0.0] Iteration 3900 (259.877 iter/s, 0.384798s/100 iter), 4.2/21.3ep, loss = 0.00190639
  2906. I0819 14:26:39.973321 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00190639 (* 1 = 0.00190639 loss)
  2907. I0819 14:26:39.973356 12180 sgd_solver.cpp:180] [0.0] Iteration 3900, lr = 0.00468695, m = 0.9, lrm = 0.0468695, wd = 0.0005, gs = 1
  2908. I0819 14:26:40.355121 12180 solver.cpp:501] Iteration 4000, Testing net (#0)
  2909. I0819 14:26:40.572141 12177 data_reader.cpp:321] Restarting data pre-fetching
  2910. I0819 14:26:40.698192 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.989976
  2911. I0819 14:26:40.698222 12180 solver.cpp:588] (0.0) Test net output #1: loss = 0.0351509 (* 1 = 0.0351509 loss)
  2912. I0819 14:26:40.698256 12180 solver.cpp:271] [MultiGPU] Tests completed in 0.724985s
  2913. I0819 14:26:40.702327 12180 solver.cpp:333] [0.0] Iteration 4000 (137.934 iter/s, 0.724985s/100 iter), 4.3/21.3ep, loss = 0.000366688
  2914. I0819 14:26:40.702389 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000366688 (* 1 = 0.000366688 loss)
  2915. I0819 14:26:40.702417 12180 sgd_solver.cpp:180] [0.0] Iteration 4000, lr = 0.00466182, m = 0.9, lrm = 0.0466182, wd = 0.0005, gs = 1
  2916. I0819 14:26:41.102905 12180 solver.cpp:333] [0.0] Iteration 4100 (249.671 iter/s, 0.400527s/100 iter), 4.4/21.3ep, loss = 0.00313187
  2917. I0819 14:26:41.103052 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00313187 (* 1 = 0.00313187 loss)
  2918. I0819 14:26:41.103081 12180 sgd_solver.cpp:180] [0.0] Iteration 4100, lr = 0.004637, m = 0.9, lrm = 0.04637, wd = 0.0005, gs = 1
  2919. I0819 14:26:41.513613 12180 solver.cpp:333] [0.0] Iteration 4200 (243.506 iter/s, 0.410668s/100 iter), 4.5/21.3ep, loss = 0.00296211
  2920. I0819 14:26:41.513764 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00296211 (* 1 = 0.00296211 loss)
  2921. I0819 14:26:41.513788 12180 sgd_solver.cpp:180] [0.0] Iteration 4200, lr = 0.00461249, m = 0.9, lrm = 0.0461248, wd = 0.0005, gs = 1
  2922. I0819 14:26:41.919658 12180 solver.cpp:333] [0.0] Iteration 4300 (246.325 iter/s, 0.405968s/100 iter), 4.6/21.3ep, loss = 0.0287476
  2923. I0819 14:26:41.919812 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.0287476 (* 1 = 0.0287476 loss)
  2924. I0819 14:26:41.919832 12180 sgd_solver.cpp:180] [0.0] Iteration 4300, lr = 0.00458827, m = 0.9, lrm = 0.0458827, wd = 0.0005, gs = 1
  2925. I0819 14:26:42.304530 12180 solver.cpp:333] [0.0] Iteration 4400 (259.831 iter/s, 0.384866s/100 iter), 4.7/21.3ep, loss = 0.000183225
  2926. I0819 14:26:42.304636 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000183225 (* 1 = 0.000183225 loss)
  2927. I0819 14:26:42.304654 12180 sgd_solver.cpp:180] [0.0] Iteration 4400, lr = 0.00456435, m = 0.9, lrm = 0.0456435, wd = 0.0005, gs = 1
  2928. I0819 14:26:42.688249 12180 solver.cpp:501] Iteration 4500, Testing net (#0)
  2929. I0819 14:26:42.894536 12177 data_reader.cpp:321] Restarting data pre-fetching
  2930. I0819 14:26:43.027454 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.989971
  2931. I0819 14:26:43.027487 12180 solver.cpp:588] (0.0) Test net output #1: loss = 0.0380082 (* 1 = 0.0380082 loss)
  2932. I0819 14:26:43.027531 12180 solver.cpp:271] [MultiGPU] Tests completed in 0.722957s
  2933. I0819 14:26:43.031596 12180 solver.cpp:333] [0.0] Iteration 4500 (138.321 iter/s, 0.722957s/100 iter), 4.8/21.3ep, loss = 0.357666
  2934. I0819 14:26:43.031718 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.357666 (* 1 = 0.357666 loss)
  2935. I0819 14:26:43.031742 12180 sgd_solver.cpp:180] [0.0] Iteration 4500, lr = 0.00454073, m = 0.9, lrm = 0.0454072, wd = 0.0005, gs = 1
  2936. I0819 14:26:43.411060 12180 solver.cpp:333] [0.0] Iteration 4600 (263.558 iter/s, 0.379423s/100 iter), 4.9/21.3ep, loss = 0
  2937. I0819 14:26:43.411164 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0 (* 1 = 0 loss)
  2938. I0819 14:26:43.411185 12180 sgd_solver.cpp:180] [0.0] Iteration 4600, lr = 0.00451738, m = 0.9, lrm = 0.0451738, wd = 0.0005, gs = 1
  2939. I0819 14:26:43.642176 12318 data_reader.cpp:321] Restarting data pre-fetching
  2940. I0819 14:26:43.803308 12180 solver.cpp:333] [0.0] Iteration 4700 (254.968 iter/s, 0.392206s/100 iter), 5/21.3ep, loss = 0.00122356
  2941. I0819 14:26:43.803382 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00122356 (* 1 = 0.00122356 loss)
  2942. I0819 14:26:43.803405 12180 sgd_solver.cpp:180] [0.0] Iteration 4700, lr = 0.00449431, m = 0.9, lrm = 0.0449431, wd = 0.0005, gs = 1
  2943. I0819 14:26:44.195493 12180 solver.cpp:333] [0.0] Iteration 4800 (255.012 iter/s, 0.392139s/100 iter), 5.1/21.3ep, loss = 0.000305414
  2944. I0819 14:26:44.195591 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000305414 (* 1 = 0.000305414 loss)
  2945. I0819 14:26:44.195611 12180 sgd_solver.cpp:180] [0.0] Iteration 4800, lr = 0.00447152, m = 0.9, lrm = 0.0447152, wd = 0.0005, gs = 1
  2946. I0819 14:26:44.582638 12180 solver.cpp:333] [0.0] Iteration 4900 (258.326 iter/s, 0.387108s/100 iter), 5.2/21.3ep, loss = 0.000427723
  2947. I0819 14:26:44.582801 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000427723 (* 1 = 0.000427723 loss)
  2948. I0819 14:26:44.582828 12180 sgd_solver.cpp:180] [0.0] Iteration 4900, lr = 0.00444899, m = 0.9, lrm = 0.0444899, wd = 0.0005, gs = 1
  2949. I0819 14:26:44.971208 12180 solver.cpp:501] Iteration 5000, Testing net (#0)
  2950. I0819 14:26:45.163486 12177 data_reader.cpp:321] Restarting data pre-fetching
  2951. I0819 14:26:45.314141 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.989209
  2952. I0819 14:26:45.314182 12180 solver.cpp:588] (0.0) Test net output #1: loss = 0.0298161 (* 1 = 0.0298161 loss)
  2953. I0819 14:26:45.314218 12180 solver.cpp:271] [MultiGPU] Tests completed in 0.731553s
  2954. I0819 14:26:45.318676 12180 solver.cpp:333] [0.0] Iteration 5000 (136.696 iter/s, 0.731553s/100 iter), 5.3/21.3ep, loss = 0.00227928
  2955. I0819 14:26:45.318773 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00227928 (* 1 = 0.00227928 loss)
  2956. I0819 14:26:45.318789 12180 sgd_solver.cpp:180] [0.0] Iteration 5000, lr = 0.00442673, m = 0.9, lrm = 0.0442673, wd = 0.0005, gs = 1
  2957. I0819 14:26:45.712577 12180 solver.cpp:333] [0.0] Iteration 5100 (253.928 iter/s, 0.393812s/100 iter), 5.4/21.3ep, loss = 0.000244379
  2958. I0819 14:26:45.712747 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000244379 (* 1 = 0.000244379 loss)
  2959. I0819 14:26:45.712774 12180 sgd_solver.cpp:180] [0.0] Iteration 5100, lr = 0.00440472, m = 0.9, lrm = 0.0440472, wd = 0.0005, gs = 1
  2960. I0819 14:26:46.098834 12180 solver.cpp:333] [0.0] Iteration 5200 (258.883 iter/s, 0.386274s/100 iter), 5.5/21.3ep, loss = 0.00012207
  2961. I0819 14:26:46.098973 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00012207 (* 1 = 0.00012207 loss)
  2962. I0819 14:26:46.098989 12180 sgd_solver.cpp:180] [0.0] Iteration 5200, lr = 0.00438297, m = 0.9, lrm = 0.0438297, wd = 0.0005, gs = 1
  2963. I0819 14:26:46.494845 12180 solver.cpp:333] [0.0] Iteration 5300 (252.542 iter/s, 0.395974s/100 iter), 5.7/21.3ep, loss = 0.00325394
  2964. I0819 14:26:46.494944 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00325394 (* 1 = 0.00325394 loss)
  2965. I0819 14:26:46.495009 12180 sgd_solver.cpp:180] [0.0] Iteration 5300, lr = 0.00436147, m = 0.9, lrm = 0.0436147, wd = 0.0005, gs = 1
  2966. I0819 14:26:46.878840 12180 solver.cpp:333] [0.0] Iteration 5400 (260.441 iter/s, 0.383964s/100 iter), 5.8/21.3ep, loss = 0.000795364
  2967. I0819 14:26:46.878952 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000795364 (* 1 = 0.000795364 loss)
  2968. I0819 14:26:46.878974 12180 sgd_solver.cpp:180] [0.0] Iteration 5400, lr = 0.00434021, m = 0.9, lrm = 0.0434021, wd = 0.0005, gs = 1
  2969. I0819 14:26:47.253578 12180 solver.cpp:501] Iteration 5500, Testing net (#0)
  2970. I0819 14:26:47.432548 12177 data_reader.cpp:321] Restarting data pre-fetching
  2971. I0819 14:26:47.594924 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.993828
  2972. I0819 14:26:47.594957 12180 solver.cpp:588] (0.0) Test net output #1: loss = 0.0290734 (* 1 = 0.0290734 loss)
  2973. I0819 14:26:47.594997 12180 solver.cpp:271] [MultiGPU] Tests completed in 0.716128s
  2974. I0819 14:26:47.598889 12180 solver.cpp:333] [0.0] Iteration 5500 (139.64 iter/s, 0.716128s/100 iter), 5.9/21.3ep, loss = 0.000734806
  2975. I0819 14:26:47.598986 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000734806 (* 1 = 0.000734806 loss)
  2976. I0819 14:26:47.599012 12180 sgd_solver.cpp:180] [0.0] Iteration 5500, lr = 0.00431919, m = 0.9, lrm = 0.0431919, wd = 0.0005, gs = 1
  2977. I0819 14:26:47.964301 12318 data_reader.cpp:321] Restarting data pre-fetching
  2978. I0819 14:26:47.984242 12180 solver.cpp:333] [0.0] Iteration 5600 (259.546 iter/s, 0.385288s/100 iter), 6/21.3ep, loss = 0.00482178
  2979. I0819 14:26:47.984391 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00482178 (* 1 = 0.00482178 loss)
  2980. I0819 14:26:47.984429 12180 sgd_solver.cpp:180] [0.0] Iteration 5600, lr = 0.00429841, m = 0.9, lrm = 0.0429841, wd = 0.0005, gs = 1
  2981. I0819 14:26:48.372097 12180 solver.cpp:333] [0.0] Iteration 5700 (257.849 iter/s, 0.387824s/100 iter), 6.1/21.3ep, loss = 0.000855923
  2982. I0819 14:26:48.372200 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000855923 (* 1 = 0.000855923 loss)
  2983. I0819 14:26:48.372222 12180 sgd_solver.cpp:180] [0.0] Iteration 5700, lr = 0.00427786, m = 0.9, lrm = 0.0427786, wd = 0.0005, gs = 1
  2984. I0819 14:26:48.762524 12180 solver.cpp:333] [0.0] Iteration 5800 (256.143 iter/s, 0.390406s/100 iter), 6.2/21.3ep, loss = 0.000366688
  2985. I0819 14:26:48.762622 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000366688 (* 1 = 0.000366688 loss)
  2986. I0819 14:26:48.762637 12180 sgd_solver.cpp:180] [0.0] Iteration 5800, lr = 0.00425754, m = 0.9, lrm = 0.0425754, wd = 0.0005, gs = 1
  2987. I0819 14:26:49.154209 12180 solver.cpp:333] [0.0] Iteration 5900 (255.327 iter/s, 0.391654s/100 iter), 6.3/21.3ep, loss = 0.0144882
  2988. I0819 14:26:49.154320 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.0144882 (* 1 = 0.0144882 loss)
  2989. I0819 14:26:49.154371 12180 sgd_solver.cpp:180] [0.0] Iteration 5900, lr = 0.00423744, m = 0.9, lrm = 0.0423744, wd = 0.0005, gs = 1
  2990. I0819 14:26:49.542238 12180 solver.cpp:501] Iteration 6000, Testing net (#0)
  2991. I0819 14:26:49.707733 12177 data_reader.cpp:321] Restarting data pre-fetching
  2992. I0819 14:26:49.888193 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.989976
  2993. I0819 14:26:49.888240 12180 solver.cpp:588] (0.0) Test net output #1: loss = 0.0260334 (* 1 = 0.0260334 loss)
  2994. I0819 14:26:49.888269 12180 solver.cpp:271] [MultiGPU] Tests completed in 0.734035s
  2995. I0819 14:26:49.892340 12180 solver.cpp:333] [0.0] Iteration 6000 (136.233 iter/s, 0.734035s/100 iter), 6.4/21.3ep, loss = 0.00012207
  2996. I0819 14:26:49.892462 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00012207 (* 1 = 0.00012207 loss)
  2997. I0819 14:26:49.892477 12180 sgd_solver.cpp:180] [0.0] Iteration 6000, lr = 0.00421756, m = 0.9, lrm = 0.0421756, wd = 0.0005, gs = 1
  2998. I0819 14:26:50.281786 12180 solver.cpp:333] [0.0] Iteration 6100 (256.803 iter/s, 0.389404s/100 iter), 6.5/21.3ep, loss = 0.000305653
  2999. I0819 14:26:50.281940 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000305653 (* 1 = 0.000305653 loss)
  3000. I0819 14:26:50.281965 12180 sgd_solver.cpp:180] [0.0] Iteration 6100, lr = 0.0041979, m = 0.9, lrm = 0.041979, wd = 0.0005, gs = 1
  3001. I0819 14:26:50.666496 12180 solver.cpp:333] [0.0] Iteration 6200 (259.956 iter/s, 0.38468s/100 iter), 6.6/21.3ep, loss = 0.00270271
  3002. I0819 14:26:50.666636 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00270271 (* 1 = 0.00270271 loss)
  3003. I0819 14:26:50.666682 12180 sgd_solver.cpp:180] [0.0] Iteration 6200, lr = 0.00417845, m = 0.9, lrm = 0.0417845, wd = 0.0005, gs = 1
  3004. I0819 14:26:51.070538 12180 solver.cpp:333] [0.0] Iteration 6300 (247.521 iter/s, 0.404006s/100 iter), 6.7/21.3ep, loss = 0.00104237
  3005. I0819 14:26:51.070636 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00104237 (* 1 = 0.00104237 loss)
  3006. I0819 14:26:51.070677 12180 sgd_solver.cpp:180] [0.0] Iteration 6300, lr = 0.00415921, m = 0.9, lrm = 0.0415921, wd = 0.0005, gs = 1
  3007. I0819 14:26:51.463475 12180 solver.cpp:333] [0.0] Iteration 6400 (254.513 iter/s, 0.392907s/100 iter), 6.8/21.3ep, loss = 0.00371742
  3008. I0819 14:26:51.463573 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00371742 (* 1 = 0.00371742 loss)
  3009. I0819 14:26:51.463642 12180 sgd_solver.cpp:180] [0.0] Iteration 6400, lr = 0.00414017, m = 0.9, lrm = 0.0414017, wd = 0.0005, gs = 1
  3010. I0819 14:26:51.844105 12180 solver.cpp:501] Iteration 6500, Testing net (#0)
  3011. I0819 14:26:51.994576 12225 blocking_queue.cpp:40] Data layer prefetch queue empty
  3012. I0819 14:26:51.997326 12177 data_reader.cpp:321] Restarting data pre-fetching
  3013. I0819 14:26:52.187436 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.98689
  3014. I0819 14:26:52.187467 12180 solver.cpp:588] (0.0) Test net output #1: loss = 0.0386341 (* 1 = 0.0386341 loss)
  3015. I0819 14:26:52.187510 12180 solver.cpp:271] [MultiGPU] Tests completed in 0.724005s
  3016. I0819 14:26:52.191495 12180 solver.cpp:333] [0.0] Iteration 6500 (138.121 iter/s, 0.724005s/100 iter), 6.9/21.3ep, loss = 0.0087204
  3017. I0819 14:26:52.191601 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.0087204 (* 1 = 0.0087204 loss)
  3018. I0819 14:26:52.191622 12180 sgd_solver.cpp:180] [0.0] Iteration 6500, lr = 0.00412134, m = 0.9, lrm = 0.0412134, wd = 0.0005, gs = 1
  3019. I0819 14:26:52.322299 12318 data_reader.cpp:321] Restarting data pre-fetching
  3020. I0819 14:26:52.588073 12180 solver.cpp:333] [0.0] Iteration 6600 (252.193 iter/s, 0.396522s/100 iter), 7/21.3ep, loss = 0.000611782
  3021. I0819 14:26:52.588196 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000611782 (* 1 = 0.000611782 loss)
  3022. I0819 14:26:52.588223 12180 sgd_solver.cpp:180] [0.0] Iteration 6600, lr = 0.0041027, m = 0.9, lrm = 0.041027, wd = 0.0005, gs = 1
  3023. I0819 14:26:52.991333 12180 solver.cpp:333] [0.0] Iteration 6700 (247.991 iter/s, 0.40324s/100 iter), 7.1/21.3ep, loss = 0.00110149
  3024. I0819 14:26:52.991420 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00110149 (* 1 = 0.00110149 loss)
  3025. I0819 14:26:52.991436 12180 sgd_solver.cpp:180] [0.0] Iteration 6700, lr = 0.00408426, m = 0.9, lrm = 0.0408426, wd = 0.0005, gs = 1
  3026. I0819 14:26:53.389897 12180 solver.cpp:333] [0.0] Iteration 6800 (250.916 iter/s, 0.39854s/100 iter), 7.3/21.3ep, loss = 6.10352e-05
  3027. I0819 14:26:53.389988 12180 solver.cpp:361] [0.0] Train net output #0: loss = 6.10352e-05 (* 1 = 6.10352e-05 loss)
  3028. I0819 14:26:53.390013 12180 sgd_solver.cpp:180] [0.0] Iteration 6800, lr = 0.00406602, m = 0.9, lrm = 0.0406602, wd = 0.0005, gs = 1
  3029. I0819 14:26:53.784565 12180 solver.cpp:333] [0.0] Iteration 6900 (253.402 iter/s, 0.39463s/100 iter), 7.4/21.3ep, loss = 0
  3030. I0819 14:26:53.784647 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0 (* 1 = 0 loss)
  3031. I0819 14:26:53.784677 12180 sgd_solver.cpp:180] [0.0] Iteration 6900, lr = 0.00404796, m = 0.9, lrm = 0.0404796, wd = 0.0005, gs = 1
  3032. I0819 14:26:54.168697 12180 solver.cpp:501] Iteration 7000, Testing net (#0)
  3033. I0819 14:26:54.309862 12177 data_reader.cpp:321] Restarting data pre-fetching
  3034. I0819 14:26:54.509017 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.993057
  3035. I0819 14:26:54.509055 12180 solver.cpp:588] (0.0) Test net output #1: loss = 0.0355603 (* 1 = 0.0355603 loss)
  3036. I0819 14:26:54.509093 12180 solver.cpp:271] [MultiGPU] Tests completed in 0.72449s
  3037. I0819 14:26:54.513242 12180 solver.cpp:333] [0.0] Iteration 7000 (138.028 iter/s, 0.72449s/100 iter), 7.5/21.3ep, loss = 6.10352e-05
  3038. I0819 14:26:54.513398 12180 solver.cpp:361] [0.0] Train net output #0: loss = 6.10352e-05 (* 1 = 6.10352e-05 loss)
  3039. I0819 14:26:54.513414 12180 sgd_solver.cpp:180] [0.0] Iteration 7000, lr = 0.00403009, m = 0.9, lrm = 0.0403009, wd = 0.0005, gs = 1
  3040. I0819 14:26:54.905464 12180 solver.cpp:333] [0.0] Iteration 7100 (254.986 iter/s, 0.392179s/100 iter), 7.6/21.3ep, loss = 0
  3041. I0819 14:26:54.905551 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0 (* 1 = 0 loss)
  3042. I0819 14:26:54.905565 12180 sgd_solver.cpp:180] [0.0] Iteration 7100, lr = 0.0040124, m = 0.9, lrm = 0.040124, wd = 0.0005, gs = 1
  3043. I0819 14:26:55.298621 12180 solver.cpp:333] [0.0] Iteration 7200 (254.368 iter/s, 0.393131s/100 iter), 7.7/21.3ep, loss = 0.000734329
  3044. I0819 14:26:55.298749 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000734329 (* 1 = 0.000734329 loss)
  3045. I0819 14:26:55.298769 12180 sgd_solver.cpp:180] [0.0] Iteration 7200, lr = 0.00399489, m = 0.9, lrm = 0.0399489, wd = 0.0005, gs = 1
  3046. I0819 14:26:55.693562 12180 solver.cpp:333] [0.0] Iteration 7300 (253.234 iter/s, 0.394892s/100 iter), 7.8/21.3ep, loss = 0.0197296
  3047. I0819 14:26:55.693650 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.0197296 (* 1 = 0.0197296 loss)
  3048. I0819 14:26:55.693665 12180 sgd_solver.cpp:180] [0.0] Iteration 7300, lr = 0.00397756, m = 0.9, lrm = 0.0397756, wd = 0.0005, gs = 1
  3049. I0819 14:26:56.084816 12180 solver.cpp:333] [0.0] Iteration 7400 (255.604 iter/s, 0.39123s/100 iter), 7.9/21.3ep, loss = 0.0277405
  3050. I0819 14:26:56.084923 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.0277405 (* 1 = 0.0277405 loss)
  3051. I0819 14:26:56.084939 12180 sgd_solver.cpp:180] [0.0] Iteration 7400, lr = 0.0039604, m = 0.9, lrm = 0.039604, wd = 0.0005, gs = 1
  3052. I0819 14:26:56.358763 12318 data_reader.cpp:321] Restarting data pre-fetching
  3053. I0819 14:26:56.475492 12180 solver.cpp:501] Iteration 7500, Testing net (#0)
  3054. I0819 14:26:56.604173 12177 data_reader.cpp:321] Restarting data pre-fetching
  3055. I0819 14:26:56.816341 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.993828
  3056. I0819 14:26:56.816376 12180 solver.cpp:588] (0.0) Test net output #1: loss = 0.0310044 (* 1 = 0.0310044 loss)
  3057. I0819 14:26:56.816414 12180 solver.cpp:271] [MultiGPU] Tests completed in 0.731554s
  3058. I0819 14:26:56.820881 12180 solver.cpp:333] [0.0] Iteration 7500 (136.695 iter/s, 0.731554s/100 iter), 8/21.3ep, loss = 0
  3059. I0819 14:26:56.820950 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0 (* 1 = 0 loss)
  3060. I0819 14:26:56.820981 12180 sgd_solver.cpp:180] [0.0] Iteration 7500, lr = 0.00394342, m = 0.9, lrm = 0.0394342, wd = 0.0005, gs = 1
  3061. I0819 14:26:57.207087 12180 solver.cpp:333] [0.0] Iteration 7600 (258.951 iter/s, 0.386174s/100 iter), 8.1/21.3ep, loss = 6.10352e-05
  3062. I0819 14:26:57.207180 12180 solver.cpp:361] [0.0] Train net output #0: loss = 6.10352e-05 (* 1 = 6.10352e-05 loss)
  3063. I0819 14:26:57.207208 12180 sgd_solver.cpp:180] [0.0] Iteration 7600, lr = 0.0039266, m = 0.9, lrm = 0.039266, wd = 0.0005, gs = 1
  3064. I0819 14:26:57.592334 12180 solver.cpp:333] [0.0] Iteration 7700 (259.598 iter/s, 0.385211s/100 iter), 8.2/21.3ep, loss = 0.000550747
  3065. I0819 14:26:57.592437 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000550747 (* 1 = 0.000550747 loss)
  3066. I0819 14:26:57.592453 12180 sgd_solver.cpp:180] [0.0] Iteration 7700, lr = 0.00390995, m = 0.9, lrm = 0.0390995, wd = 0.0005, gs = 1
  3067. I0819 14:26:57.978354 12180 solver.cpp:333] [0.0] Iteration 7800 (259.068 iter/s, 0.385999s/100 iter), 8.3/21.3ep, loss = 0
  3068. I0819 14:26:57.978477 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0 (* 1 = 0 loss)
  3069. I0819 14:26:57.978518 12180 sgd_solver.cpp:180] [0.0] Iteration 7800, lr = 0.00389346, m = 0.9, lrm = 0.0389346, wd = 0.0005, gs = 1
  3070. I0819 14:26:58.365398 12180 solver.cpp:333] [0.0] Iteration 7900 (258.386 iter/s, 0.387018s/100 iter), 8.4/21.3ep, loss = 0
  3071. I0819 14:26:58.365538 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0 (* 1 = 0 loss)
  3072. I0819 14:26:58.365553 12180 sgd_solver.cpp:180] [0.0] Iteration 7900, lr = 0.00387714, m = 0.9, lrm = 0.0387714, wd = 0.0005, gs = 1
  3073. I0819 14:26:58.748659 12180 solver.cpp:501] Iteration 8000, Testing net (#0)
  3074. I0819 14:26:58.862890 12177 data_reader.cpp:321] Restarting data pre-fetching
  3075. I0819 14:26:59.090896 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.991514
  3076. I0819 14:26:59.090925 12180 solver.cpp:588] (0.0) Test net output #1: loss = 0.0295883 (* 1 = 0.0295883 loss)
  3077. I0819 14:26:59.090956 12180 solver.cpp:271] [MultiGPU] Tests completed in 0.725529s
  3078. I0819 14:26:59.095119 12180 solver.cpp:333] [0.0] Iteration 8000 (137.831 iter/s, 0.725529s/100 iter), 8.5/21.3ep, loss = 0.00171947
  3079. I0819 14:26:59.095240 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00171947 (* 1 = 0.00171947 loss)
  3080. I0819 14:26:59.095261 12180 sgd_solver.cpp:180] [0.0] Iteration 8000, lr = 0.00386097, m = 0.9, lrm = 0.0386097, wd = 0.0005, gs = 1
  3081. I0819 14:26:59.482128 12180 solver.cpp:333] [0.0] Iteration 8100 (258.422 iter/s, 0.386963s/100 iter), 8.6/21.3ep, loss = 0.000488758
  3082. I0819 14:26:59.482215 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000488758 (* 1 = 0.000488758 loss)
  3083. I0819 14:26:59.482231 12180 sgd_solver.cpp:180] [0.0] Iteration 8100, lr = 0.00384496, m = 0.9, lrm = 0.0384496, wd = 0.0005, gs = 1
  3084. I0819 14:26:59.869750 12180 solver.cpp:333] [0.0] Iteration 8200 (258.005 iter/s, 0.387589s/100 iter), 8.7/21.3ep, loss = 0.000671864
  3085. I0819 14:26:59.869866 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000671864 (* 1 = 0.000671864 loss)
  3086. I0819 14:26:59.869880 12180 sgd_solver.cpp:180] [0.0] Iteration 8200, lr = 0.00382911, m = 0.9, lrm = 0.0382911, wd = 0.0005, gs = 1
  3087. I0819 14:27:00.257104 12180 solver.cpp:333] [0.0] Iteration 8300 (258.185 iter/s, 0.387319s/100 iter), 8.9/21.3ep, loss = 0
  3088. I0819 14:27:00.257232 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0 (* 1 = 0 loss)
  3089. I0819 14:27:00.257248 12180 sgd_solver.cpp:180] [0.0] Iteration 8300, lr = 0.00381341, m = 0.9, lrm = 0.038134, wd = 0.0005, gs = 1
  3090. I0819 14:27:00.655155 12180 solver.cpp:333] [0.0] Iteration 8400 (251.247 iter/s, 0.398015s/100 iter), 9/21.3ep, loss = 0.00271416
  3091. I0819 14:27:00.655246 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00271416 (* 1 = 0.00271416 loss)
  3092. I0819 14:27:00.655261 12180 sgd_solver.cpp:180] [0.0] Iteration 8400, lr = 0.00379785, m = 0.9, lrm = 0.0379785, wd = 0.0005, gs = 1
  3093. I0819 14:27:00.691988 12318 data_reader.cpp:321] Restarting data pre-fetching
  3094. I0819 14:27:01.037777 12180 solver.cpp:501] Iteration 8500, Testing net (#0)
  3095. I0819 14:27:01.139003 12177 data_reader.cpp:321] Restarting data pre-fetching
  3096. I0819 14:27:01.378856 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.991514
  3097. I0819 14:27:01.378890 12180 solver.cpp:588] (0.0) Test net output #1: loss = 0.0338354 (* 1 = 0.0338354 loss)
  3098. I0819 14:27:01.378931 12180 solver.cpp:271] [MultiGPU] Tests completed in 0.723752s
  3099. I0819 14:27:01.383092 12180 solver.cpp:333] [0.0] Iteration 8500 (138.169 iter/s, 0.723752s/100 iter), 9.1/21.3ep, loss = 0.00012207
  3100. I0819 14:27:01.383271 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00012207 (* 1 = 0.00012207 loss)
  3101. I0819 14:27:01.383285 12180 sgd_solver.cpp:180] [0.0] Iteration 8500, lr = 0.00378244, m = 0.9, lrm = 0.0378244, wd = 0.0005, gs = 1
  3102. I0819 14:27:01.770723 12180 solver.cpp:333] [0.0] Iteration 8600 (258.006 iter/s, 0.387588s/100 iter), 9.2/21.3ep, loss = 6.10352e-05
  3103. I0819 14:27:01.770825 12180 solver.cpp:361] [0.0] Train net output #0: loss = 6.10352e-05 (* 1 = 6.10352e-05 loss)
  3104. I0819 14:27:01.770874 12180 sgd_solver.cpp:180] [0.0] Iteration 8600, lr = 0.00376718, m = 0.9, lrm = 0.0376718, wd = 0.0005, gs = 1
  3105. I0819 14:27:02.161785 12180 solver.cpp:333] [0.0] Iteration 8700 (255.733 iter/s, 0.391033s/100 iter), 9.3/21.3ep, loss = 0.00344086
  3106. I0819 14:27:02.161886 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00344086 (* 1 = 0.00344086 loss)
  3107. I0819 14:27:02.161898 12180 sgd_solver.cpp:180] [0.0] Iteration 8700, lr = 0.00375206, m = 0.9, lrm = 0.0375206, wd = 0.0005, gs = 1
  3108. I0819 14:27:02.553359 12180 solver.cpp:333] [0.0] Iteration 8800 (255.407 iter/s, 0.391533s/100 iter), 9.4/21.3ep, loss = 0.00012219
  3109. I0819 14:27:02.553467 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00012219 (* 1 = 0.00012219 loss)
  3110. I0819 14:27:02.553489 12180 sgd_solver.cpp:180] [0.0] Iteration 8800, lr = 0.00373708, m = 0.9, lrm = 0.0373708, wd = 0.0005, gs = 1
  3111. I0819 14:27:02.945791 12180 solver.cpp:333] [0.0] Iteration 8900 (254.861 iter/s, 0.392371s/100 iter), 9.5/21.3ep, loss = 0.0030899
  3112. I0819 14:27:02.945888 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.0030899 (* 1 = 0.0030899 loss)
  3113. I0819 14:27:02.945902 12180 sgd_solver.cpp:180] [0.0] Iteration 8900, lr = 0.00372224, m = 0.9, lrm = 0.0372224, wd = 0.0005, gs = 1
  3114. I0819 14:27:03.326556 12180 solver.cpp:501] Iteration 9000, Testing net (#0)
  3115. I0819 14:27:03.414835 12177 data_reader.cpp:321] Restarting data pre-fetching
  3116. I0819 14:27:03.667979 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.992285
  3117. I0819 14:27:03.668007 12180 solver.cpp:588] (0.0) Test net output #1: loss = 0.031059 (* 1 = 0.031059 loss)
  3118. I0819 14:27:03.668042 12180 solver.cpp:271] [MultiGPU] Tests completed in 0.722252s
  3119. I0819 14:27:03.671975 12180 solver.cpp:333] [0.0] Iteration 9000 (138.456 iter/s, 0.722252s/100 iter), 9.6/21.3ep, loss = 0.201294
  3120. I0819 14:27:03.672065 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.201294 (* 1 = 0.201294 loss)
  3121. I0819 14:27:03.672085 12180 sgd_solver.cpp:180] [0.0] Iteration 9000, lr = 0.00370754, m = 0.9, lrm = 0.0370754, wd = 0.0005, gs = 1
  3122. I0819 14:27:04.061833 12180 solver.cpp:333] [0.0] Iteration 9100 (256.53 iter/s, 0.389818s/100 iter), 9.7/21.3ep, loss = 0.204102
  3123. I0819 14:27:04.061956 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.204102 (* 1 = 0.204102 loss)
  3124. I0819 14:27:04.062000 12180 sgd_solver.cpp:180] [0.0] Iteration 9100, lr = 0.00369297, m = 0.9, lrm = 0.0369297, wd = 0.0005, gs = 1
  3125. I0819 14:27:04.448829 12180 solver.cpp:333] [0.0] Iteration 9200 (258.421 iter/s, 0.386966s/100 iter), 9.8/21.3ep, loss = 0
  3126. I0819 14:27:04.448938 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0 (* 1 = 0 loss)
  3127. I0819 14:27:04.448961 12180 sgd_solver.cpp:180] [0.0] Iteration 9200, lr = 0.00367854, m = 0.9, lrm = 0.0367854, wd = 0.0005, gs = 1
  3128. I0819 14:27:04.842108 12180 solver.cpp:333] [0.0] Iteration 9300 (254.293 iter/s, 0.393248s/100 iter), 9.9/21.3ep, loss = 0
  3129. I0819 14:27:04.842198 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0 (* 1 = 0 loss)
  3130. I0819 14:27:04.842217 12180 sgd_solver.cpp:180] [0.0] Iteration 9300, lr = 0.00366423, m = 0.9, lrm = 0.0366423, wd = 0.0005, gs = 1
  3131. I0819 14:27:05.016857 12318 data_reader.cpp:321] Restarting data pre-fetching
  3132. I0819 14:27:05.229874 12180 solver.cpp:333] [0.0] Iteration 9400 (257.909 iter/s, 0.387734s/100 iter), 10/21.3ep, loss = 0.000734329
  3133. I0819 14:27:05.229962 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000734329 (* 1 = 0.000734329 loss)
  3134. I0819 14:27:05.229982 12180 sgd_solver.cpp:180] [0.0] Iteration 9400, lr = 0.00365006, m = 0.9, lrm = 0.0365006, wd = 0.0005, gs = 1
  3135. I0819 14:27:05.620606 12180 solver.cpp:501] Iteration 9500, Testing net (#0)
  3136. I0819 14:27:05.695765 12177 data_reader.cpp:321] Restarting data pre-fetching
  3137. I0819 14:27:05.961028 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.98689
  3138. I0819 14:27:05.961060 12180 solver.cpp:588] (0.0) Test net output #1: loss = 0.0439001 (* 1 = 0.0439001 loss)
  3139. I0819 14:27:05.961089 12180 solver.cpp:271] [MultiGPU] Tests completed in 0.731181s
  3140. I0819 14:27:05.965332 12180 solver.cpp:333] [0.0] Iteration 9500 (136.765 iter/s, 0.731181s/100 iter), 10.1/21.3ep, loss = 0
  3141. I0819 14:27:05.965427 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0 (* 1 = 0 loss)
  3142. I0819 14:27:05.965452 12180 sgd_solver.cpp:180] [0.0] Iteration 9500, lr = 0.00363601, m = 0.9, lrm = 0.0363601, wd = 0.0005, gs = 1
  3143. I0819 14:27:06.356194 12180 solver.cpp:333] [0.0] Iteration 9600 (255.871 iter/s, 0.390822s/100 iter), 10.2/21.3ep, loss = 6.10352e-05
  3144. I0819 14:27:06.356318 12180 solver.cpp:361] [0.0] Train net output #0: loss = 6.10352e-05 (* 1 = 6.10352e-05 loss)
  3145. I0819 14:27:06.356350 12180 sgd_solver.cpp:180] [0.0] Iteration 9600, lr = 0.00362209, m = 0.9, lrm = 0.0362209, wd = 0.0005, gs = 1
  3146. I0819 14:27:06.743566 12180 solver.cpp:333] [0.0] Iteration 9700 (258.18 iter/s, 0.387326s/100 iter), 10.3/21.3ep, loss = 0.000305653
  3147. I0819 14:27:06.743641 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000305653 (* 1 = 0.000305653 loss)
  3148. I0819 14:27:06.743657 12180 sgd_solver.cpp:180] [0.0] Iteration 9700, lr = 0.00360829, m = 0.9, lrm = 0.0360829, wd = 0.0005, gs = 1
  3149. I0819 14:27:07.134608 12180 solver.cpp:333] [0.0] Iteration 9800 (255.739 iter/s, 0.391023s/100 iter), 10.5/21.3ep, loss = 0.000855923
  3150. I0819 14:27:07.134759 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000855923 (* 1 = 0.000855923 loss)
  3151. I0819 14:27:07.134781 12180 sgd_solver.cpp:180] [0.0] Iteration 9800, lr = 0.00359461, m = 0.9, lrm = 0.0359461, wd = 0.0005, gs = 1
  3152. I0819 14:27:07.526099 12180 solver.cpp:333] [0.0] Iteration 9900 (255.467 iter/s, 0.39144s/100 iter), 10.6/21.3ep, loss = 0
  3153. I0819 14:27:07.526212 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0 (* 1 = 0 loss)
  3154. I0819 14:27:07.526252 12180 sgd_solver.cpp:180] [0.0] Iteration 9900, lr = 0.00358106, m = 0.9, lrm = 0.0358106, wd = 0.0005, gs = 1
  3155. I0819 14:27:07.906834 12180 solver.cpp:501] Iteration 10000, Testing net (#0)
  3156. I0819 14:27:07.972169 12177 data_reader.cpp:321] Restarting data pre-fetching
  3157. I0819 14:27:08.248234 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.987661
  3158. I0819 14:27:08.248268 12180 solver.cpp:588] (0.0) Test net output #1: loss = 0.0476012 (* 1 = 0.0476012 loss)
  3159. I0819 14:27:08.248308 12180 solver.cpp:271] [MultiGPU] Tests completed in 0.722188s
  3160. I0819 14:27:08.252460 12180 solver.cpp:333] [0.0] Iteration 10000 (138.468 iter/s, 0.722188s/100 iter), 10.7/21.3ep, loss = 0.00012219
  3161. I0819 14:27:08.252575 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00012219 (* 1 = 0.00012219 loss)
  3162. I0819 14:27:08.252589 12180 sgd_solver.cpp:180] [0.0] Iteration 10000, lr = 0.00356762, m = 0.9, lrm = 0.0356762, wd = 0.0005, gs = 1
  3163. I0819 14:27:08.641191 12180 solver.cpp:333] [0.0] Iteration 10100 (257.263 iter/s, 0.388707s/100 iter), 10.8/21.3ep, loss = 0.0248413
  3164. I0819 14:27:08.641288 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.0248413 (* 1 = 0.0248413 loss)
  3165. I0819 14:27:08.641307 12180 sgd_solver.cpp:180] [0.0] Iteration 10100, lr = 0.0035543, m = 0.9, lrm = 0.035543, wd = 0.0005, gs = 1
  3166. I0819 14:27:09.031965 12180 solver.cpp:333] [0.0] Iteration 10200 (255.974 iter/s, 0.390664s/100 iter), 10.9/21.3ep, loss = 0
  3167. I0819 14:27:09.032181 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0 (* 1 = 0 loss)
  3168. I0819 14:27:09.032232 12180 sgd_solver.cpp:180] [0.0] Iteration 10200, lr = 0.0035411, m = 0.9, lrm = 0.035411, wd = 0.0005, gs = 1
  3169. I0819 14:27:09.378978 12318 data_reader.cpp:321] Restarting data pre-fetching
  3170. I0819 14:27:09.448051 12180 solver.cpp:333] [0.0] Iteration 10300 (240.311 iter/s, 0.416127s/100 iter), 11/21.3ep, loss = 0
  3171. I0819 14:27:09.448151 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0 (* 1 = 0 loss)
  3172. I0819 14:27:09.448179 12180 sgd_solver.cpp:180] [0.0] Iteration 10300, lr = 0.00352801, m = 0.9, lrm = 0.03528, wd = 0.0005, gs = 1
  3173. I0819 14:27:09.858690 12180 solver.cpp:333] [0.0] Iteration 10400 (243.553 iter/s, 0.410588s/100 iter), 11.1/21.3ep, loss = 0.00159168
  3174. I0819 14:27:09.858820 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00159168 (* 1 = 0.00159168 loss)
  3175. I0819 14:27:09.858844 12180 sgd_solver.cpp:180] [0.0] Iteration 10400, lr = 0.00351503, m = 0.9, lrm = 0.0351503, wd = 0.0005, gs = 1
  3176. I0819 14:27:10.248284 12180 solver.cpp:501] Iteration 10500, Testing net (#0)
  3177. I0819 14:27:10.301270 12177 data_reader.cpp:321] Restarting data pre-fetching
  3178. I0819 14:27:10.592121 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.989976
  3179. I0819 14:27:10.592262 12180 solver.cpp:588] (0.0) Test net output #1: loss = 0.0490732 (* 1 = 0.0490732 loss)
  3180. I0819 14:27:10.592314 12180 solver.cpp:271] [MultiGPU] Tests completed in 0.733582s
  3181. I0819 14:27:10.596913 12180 solver.cpp:333] [0.0] Iteration 10500 (136.317 iter/s, 0.733582s/100 iter), 11.2/21.3ep, loss = 0.00503159
  3182. I0819 14:27:10.597009 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00503159 (* 1 = 0.00503159 loss)
  3183. I0819 14:27:10.597033 12180 sgd_solver.cpp:180] [0.0] Iteration 10500, lr = 0.00350216, m = 0.9, lrm = 0.0350216, wd = 0.0005, gs = 1
  3184. I0819 14:27:10.991122 12180 solver.cpp:333] [0.0] Iteration 10600 (253.708 iter/s, 0.394154s/100 iter), 11.3/21.3ep, loss = 0.00012219
  3185. I0819 14:27:10.991216 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00012219 (* 1 = 0.00012219 loss)
  3186. I0819 14:27:10.991238 12180 sgd_solver.cpp:180] [0.0] Iteration 10600, lr = 0.0034894, m = 0.9, lrm = 0.034894, wd = 0.0005, gs = 1
  3187. I0819 14:27:11.382978 12180 solver.cpp:333] [0.0] Iteration 10700 (255.209 iter/s, 0.391836s/100 iter), 11.4/21.3ep, loss = 0.0326843
  3188. I0819 14:27:11.383086 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.0326843 (* 1 = 0.0326843 loss)
  3189. I0819 14:27:11.383101 12180 sgd_solver.cpp:180] [0.0] Iteration 10700, lr = 0.00347675, m = 0.9, lrm = 0.0347675, wd = 0.0005, gs = 1
  3190. I0819 14:27:11.769176 12180 solver.cpp:333] [0.0] Iteration 10800 (258.96 iter/s, 0.386161s/100 iter), 11.5/21.3ep, loss = 0.0303955
  3191. I0819 14:27:11.769268 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.0303955 (* 1 = 0.0303955 loss)
  3192. I0819 14:27:11.769285 12180 sgd_solver.cpp:180] [0.0] Iteration 10800, lr = 0.00346421, m = 0.9, lrm = 0.0346421, wd = 0.0005, gs = 1
  3193. I0819 14:27:12.156428 12180 solver.cpp:333] [0.0] Iteration 10900 (258.253 iter/s, 0.387217s/100 iter), 11.6/21.3ep, loss = 0.0141907
  3194. I0819 14:27:12.156536 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.0141907 (* 1 = 0.0141907 loss)
  3195. I0819 14:27:12.156559 12180 sgd_solver.cpp:180] [0.0] Iteration 10900, lr = 0.00345177, m = 0.9, lrm = 0.0345177, wd = 0.0005, gs = 1
  3196. I0819 14:27:12.551945 12180 solver.cpp:501] Iteration 11000, Testing net (#0)
  3197. I0819 14:27:12.591337 12177 data_reader.cpp:321] Restarting data pre-fetching
  3198. I0819 14:27:12.897469 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.993057
  3199. I0819 14:27:12.897501 12180 solver.cpp:588] (0.0) Test net output #1: loss = 0.0169954 (* 1 = 0.0169954 loss)
  3200. I0819 14:27:12.897538 12180 solver.cpp:271] [MultiGPU] Tests completed in 0.741069s
  3201. I0819 14:27:12.901711 12180 solver.cpp:333] [0.0] Iteration 11000 (134.94 iter/s, 0.741069s/100 iter), 11.7/21.3ep, loss = 6.10352e-05
  3202. I0819 14:27:12.901855 12180 solver.cpp:361] [0.0] Train net output #0: loss = 6.10352e-05 (* 1 = 6.10352e-05 loss)
  3203. I0819 14:27:12.901872 12180 sgd_solver.cpp:180] [0.0] Iteration 11000, lr = 0.00343943, m = 0.9, lrm = 0.0343943, wd = 0.0005, gs = 1
  3204. I0819 14:27:13.294100 12180 solver.cpp:333] [0.0] Iteration 11100 (254.879 iter/s, 0.392343s/100 iter), 11.8/21.3ep, loss = 0.000244379
  3205. I0819 14:27:13.294205 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000244379 (* 1 = 0.000244379 loss)
  3206. I0819 14:27:13.294224 12180 sgd_solver.cpp:180] [0.0] Iteration 11100, lr = 0.0034272, m = 0.9, lrm = 0.034272, wd = 0.0005, gs = 1
  3207. I0819 14:27:13.687636 12180 solver.cpp:333] [0.0] Iteration 11200 (254.121 iter/s, 0.393513s/100 iter), 11.9/21.3ep, loss = 6.10352e-05
  3208. I0819 14:27:13.687731 12180 solver.cpp:361] [0.0] Train net output #0: loss = 6.10352e-05 (* 1 = 6.10352e-05 loss)
  3209. I0819 14:27:13.687752 12180 sgd_solver.cpp:180] [0.0] Iteration 11200, lr = 0.00341507, m = 0.9, lrm = 0.0341507, wd = 0.0005, gs = 1
  3210. I0819 14:27:13.767225 12318 data_reader.cpp:321] Restarting data pre-fetching
  3211. I0819 14:27:14.082840 12180 solver.cpp:333] [0.0] Iteration 11300 (253.052 iter/s, 0.395175s/100 iter), 12.1/21.3ep, loss = 0.0273438
  3212. I0819 14:27:14.082963 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.0273438 (* 1 = 0.0273438 loss)
  3213. I0819 14:27:14.082983 12180 sgd_solver.cpp:180] [0.0] Iteration 11300, lr = 0.00340304, m = 0.9, lrm = 0.0340304, wd = 0.0005, gs = 1
  3214. I0819 14:27:14.470371 12180 solver.cpp:333] [0.0] Iteration 11400 (258.067 iter/s, 0.387496s/100 iter), 12.2/21.3ep, loss = 0
  3215. I0819 14:27:14.470458 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0 (* 1 = 0 loss)
  3216. I0819 14:27:14.470472 12180 sgd_solver.cpp:180] [0.0] Iteration 11400, lr = 0.0033911, m = 0.9, lrm = 0.033911, wd = 0.0005, gs = 1
  3217. I0819 14:27:14.855865 12180 solver.cpp:501] Iteration 11500, Testing net (#0)
  3218. I0819 14:27:14.880189 12177 data_reader.cpp:321] Restarting data pre-fetching
  3219. I0819 14:27:15.204100 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.99229
  3220. I0819 14:27:15.204133 12180 solver.cpp:588] (0.0) Test net output #1: loss = 0.0275124 (* 1 = 0.0275124 loss)
  3221. I0819 14:27:15.204174 12180 solver.cpp:271] [MultiGPU] Tests completed in 0.733757s
  3222. I0819 14:27:15.208448 12180 solver.cpp:333] [0.0] Iteration 11500 (136.285 iter/s, 0.733757s/100 iter), 12.3/21.3ep, loss = 0
  3223. I0819 14:27:15.208532 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0 (* 1 = 0 loss)
  3224. I0819 14:27:15.208549 12180 sgd_solver.cpp:180] [0.0] Iteration 11500, lr = 0.00337927, m = 0.9, lrm = 0.0337927, wd = 0.0005, gs = 1
  3225. I0819 14:27:15.596454 12180 solver.cpp:333] [0.0] Iteration 11600 (257.752 iter/s, 0.38797s/100 iter), 12.4/21.3ep, loss = 0.0008564
  3226. I0819 14:27:15.596580 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.0008564 (* 1 = 0.0008564 loss)
  3227. I0819 14:27:15.596594 12180 sgd_solver.cpp:180] [0.0] Iteration 11600, lr = 0.00336753, m = 0.9, lrm = 0.0336753, wd = 0.0005, gs = 1
  3228. I0819 14:27:15.997716 12180 solver.cpp:333] [0.0] Iteration 11700 (249.234 iter/s, 0.40123s/100 iter), 12.5/21.3ep, loss = 6.10352e-05
  3229. I0819 14:27:15.997829 12180 solver.cpp:361] [0.0] Train net output #0: loss = 6.10352e-05 (* 1 = 6.10352e-05 loss)
  3230. I0819 14:27:15.997881 12180 sgd_solver.cpp:180] [0.0] Iteration 11700, lr = 0.00335588, m = 0.9, lrm = 0.0335588, wd = 0.0005, gs = 1
  3231. I0819 14:27:16.396939 12180 solver.cpp:333] [0.0] Iteration 11800 (250.512 iter/s, 0.399182s/100 iter), 12.6/21.3ep, loss = 0.00378036
  3232. I0819 14:27:16.397038 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00378036 (* 1 = 0.00378036 loss)
  3233. I0819 14:27:16.397061 12180 sgd_solver.cpp:180] [0.0] Iteration 11800, lr = 0.00334433, m = 0.9, lrm = 0.0334433, wd = 0.0005, gs = 1
  3234. I0819 14:27:16.805059 12180 solver.cpp:333] [0.0] Iteration 11900 (245.048 iter/s, 0.408083s/100 iter), 12.7/21.3ep, loss = 0
  3235. I0819 14:27:16.805160 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0 (* 1 = 0 loss)
  3236. I0819 14:27:16.805238 12180 sgd_solver.cpp:180] [0.0] Iteration 11900, lr = 0.00333287, m = 0.9, lrm = 0.0333287, wd = 0.0005, gs = 1
  3237. I0819 14:27:17.198304 12180 solver.cpp:501] Iteration 12000, Testing net (#0)
  3238. I0819 14:27:17.209579 12177 data_reader.cpp:321] Restarting data pre-fetching
  3239. I0819 14:27:17.530982 12177 data_reader.cpp:321] Restarting data pre-fetching
  3240. I0819 14:27:17.541715 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.98458
  3241. I0819 14:27:17.541761 12180 solver.cpp:588] (0.0) Test net output #1: loss = 0.0551621 (* 1 = 0.0551621 loss)
  3242. I0819 14:27:17.541786 12180 solver.cpp:271] [MultiGPU] Tests completed in 0.736701s
  3243. I0819 14:27:17.546231 12180 solver.cpp:333] [0.0] Iteration 12000 (135.74 iter/s, 0.736701s/100 iter), 12.8/21.3ep, loss = 0.0225525
  3244. I0819 14:27:17.546320 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.0225525 (* 1 = 0.0225525 loss)
  3245. I0819 14:27:17.546344 12180 sgd_solver.cpp:180] [0.0] Iteration 12000, lr = 0.0033215, m = 0.9, lrm = 0.033215, wd = 0.0005, gs = 1
  3246. I0819 14:27:17.933728 12180 solver.cpp:333] [0.0] Iteration 12100 (258.091 iter/s, 0.38746s/100 iter), 12.9/21.3ep, loss = 0
  3247. I0819 14:27:17.933846 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0 (* 1 = 0 loss)
  3248. I0819 14:27:17.933861 12180 sgd_solver.cpp:180] [0.0] Iteration 12100, lr = 0.00331022, m = 0.9, lrm = 0.0331022, wd = 0.0005, gs = 1
  3249. I0819 14:27:18.163508 12318 data_reader.cpp:321] Restarting data pre-fetching
  3250. I0819 14:27:18.325758 12180 solver.cpp:333] [0.0] Iteration 12200 (255.103 iter/s, 0.391998s/100 iter), 13/21.3ep, loss = 0.000183225
  3251. I0819 14:27:18.325863 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000183225 (* 1 = 0.000183225 loss)
  3252. I0819 14:27:18.325882 12180 sgd_solver.cpp:180] [0.0] Iteration 12200, lr = 0.00329903, m = 0.9, lrm = 0.0329903, wd = 0.0005, gs = 1
  3253. I0819 14:27:18.715947 12180 solver.cpp:333] [0.0] Iteration 12300 (256.309 iter/s, 0.390153s/100 iter), 13.1/21.3ep, loss = 0.00012207
  3254. I0819 14:27:18.716033 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00012207 (* 1 = 0.00012207 loss)
  3255. I0819 14:27:18.716050 12180 sgd_solver.cpp:180] [0.0] Iteration 12300, lr = 0.00328793, m = 0.9, lrm = 0.0328793, wd = 0.0005, gs = 1
  3256. I0819 14:27:19.105150 12180 solver.cpp:333] [0.0] Iteration 12400 (256.959 iter/s, 0.389167s/100 iter), 13.2/21.3ep, loss = 0.0015974
  3257. I0819 14:27:19.105255 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.0015974 (* 1 = 0.0015974 loss)
  3258. I0819 14:27:19.105278 12180 sgd_solver.cpp:180] [0.0] Iteration 12400, lr = 0.00327692, m = 0.9, lrm = 0.0327691, wd = 0.0005, gs = 1
  3259. I0819 14:27:19.493669 12180 solver.cpp:501] Iteration 12500, Testing net (#0)
  3260. I0819 14:27:19.812252 12176 blocking_queue.cpp:40] Data layer prefetch queue empty
  3261. I0819 14:27:19.812331 12177 data_reader.cpp:321] Restarting data pre-fetching
  3262. I0819 14:27:19.834314 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.986123
  3263. I0819 14:27:19.834334 12180 solver.cpp:588] (0.0) Test net output #1: loss = 0.0434638 (* 1 = 0.0434638 loss)
  3264. I0819 14:27:19.834391 12180 solver.cpp:271] [MultiGPU] Tests completed in 0.729201s
  3265. I0819 14:27:19.838479 12180 solver.cpp:333] [0.0] Iteration 12500 (137.136 iter/s, 0.729201s/100 iter), 13.3/21.3ep, loss = 0.000183225
  3266. I0819 14:27:19.838559 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000183225 (* 1 = 0.000183225 loss)
  3267. I0819 14:27:19.838574 12180 sgd_solver.cpp:180] [0.0] Iteration 12500, lr = 0.00326599, m = 0.9, lrm = 0.0326599, wd = 0.0005, gs = 1
  3268. I0819 14:27:20.234576 12180 solver.cpp:333] [0.0] Iteration 12600 (252.492 iter/s, 0.396052s/100 iter), 13.4/21.3ep, loss = 0
  3269. I0819 14:27:20.234731 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0 (* 1 = 0 loss)
  3270. I0819 14:27:20.234755 12180 sgd_solver.cpp:180] [0.0] Iteration 12600, lr = 0.00325514, m = 0.9, lrm = 0.0325514, wd = 0.0005, gs = 1
  3271. I0819 14:27:20.628679 12180 solver.cpp:333] [0.0] Iteration 12700 (253.765 iter/s, 0.394065s/100 iter), 13.5/21.3ep, loss = 0.000183225
  3272. I0819 14:27:20.628859 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000183225 (* 1 = 0.000183225 loss)
  3273. I0819 14:27:20.628877 12180 sgd_solver.cpp:180] [0.0] Iteration 12700, lr = 0.00324438, m = 0.9, lrm = 0.0324438, wd = 0.0005, gs = 1
  3274. I0819 14:27:21.021441 12180 solver.cpp:333] [0.0] Iteration 12800 (254.622 iter/s, 0.392739s/100 iter), 13.7/21.3ep, loss = 0.00171375
  3275. I0819 14:27:21.021562 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00171375 (* 1 = 0.00171375 loss)
  3276. I0819 14:27:21.021598 12180 sgd_solver.cpp:180] [0.0] Iteration 12800, lr = 0.0032337, m = 0.9, lrm = 0.032337, wd = 0.0005, gs = 1
  3277. I0819 14:27:21.408782 12180 solver.cpp:333] [0.0] Iteration 12900 (258.194 iter/s, 0.387305s/100 iter), 13.8/21.3ep, loss = 0.000366688
  3278. I0819 14:27:21.408885 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000366688 (* 1 = 0.000366688 loss)
  3279. I0819 14:27:21.408906 12180 sgd_solver.cpp:180] [0.0] Iteration 12900, lr = 0.00322311, m = 0.9, lrm = 0.0322311, wd = 0.0005, gs = 1
  3280. I0819 14:27:21.800763 12180 solver.cpp:501] Iteration 13000, Testing net (#0)
  3281. I0819 14:27:22.109992 12177 data_reader.cpp:321] Restarting data pre-fetching
  3282. I0819 14:27:22.141007 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.984575
  3283. I0819 14:27:22.141039 12180 solver.cpp:588] (0.0) Test net output #1: loss = 0.0443439 (* 1 = 0.0443439 loss)
  3284. I0819 14:27:22.141146 12180 solver.cpp:271] [MultiGPU] Tests completed in 0.732302s
  3285. I0819 14:27:22.145115 12180 solver.cpp:333] [0.0] Iteration 13000 (136.556 iter/s, 0.732302s/100 iter), 13.9/21.3ep, loss = 0.00271416
  3286. I0819 14:27:22.145195 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00271416 (* 1 = 0.00271416 loss)
  3287. I0819 14:27:22.145211 12180 sgd_solver.cpp:180] [0.0] Iteration 13000, lr = 0.00321259, m = 0.9, lrm = 0.0321259, wd = 0.0005, gs = 1
  3288. I0819 14:27:22.510828 12318 data_reader.cpp:321] Restarting data pre-fetching
  3289. I0819 14:27:22.530239 12180 solver.cpp:333] [0.0] Iteration 13100 (259.687 iter/s, 0.385079s/100 iter), 14/21.3ep, loss = 0.00246048
  3290. I0819 14:27:22.530345 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00246048 (* 1 = 0.00246048 loss)
  3291. I0819 14:27:22.530367 12180 sgd_solver.cpp:180] [0.0] Iteration 13100, lr = 0.00320215, m = 0.9, lrm = 0.0320215, wd = 0.0005, gs = 1
  3292. I0819 14:27:22.929361 12180 solver.cpp:333] [0.0] Iteration 13200 (250.568 iter/s, 0.399094s/100 iter), 14.1/21.3ep, loss = 0.000489235
  3293. I0819 14:27:22.929467 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000489235 (* 1 = 0.000489235 loss)
  3294. I0819 14:27:22.929481 12180 sgd_solver.cpp:180] [0.0] Iteration 13200, lr = 0.0031918, m = 0.9, lrm = 0.031918, wd = 0.0005, gs = 1
  3295. I0819 14:27:23.326866 12180 solver.cpp:333] [0.0] Iteration 13300 (251.597 iter/s, 0.397462s/100 iter), 14.2/21.3ep, loss = 6.10352e-05
  3296. I0819 14:27:23.327001 12180 solver.cpp:361] [0.0] Train net output #0: loss = 6.10352e-05 (* 1 = 6.10352e-05 loss)
  3297. I0819 14:27:23.327023 12180 sgd_solver.cpp:180] [0.0] Iteration 13300, lr = 0.00318152, m = 0.9, lrm = 0.0318152, wd = 0.0005, gs = 1
  3298. I0819 14:27:23.726469 12180 solver.cpp:333] [0.0] Iteration 13400 (250.268 iter/s, 0.399572s/100 iter), 14.3/21.3ep, loss = 0.00177479
  3299. I0819 14:27:23.726600 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00177479 (* 1 = 0.00177479 loss)
  3300. I0819 14:27:23.726621 12180 sgd_solver.cpp:180] [0.0] Iteration 13400, lr = 0.00317131, m = 0.9, lrm = 0.0317131, wd = 0.0005, gs = 1
  3301. I0819 14:27:24.119738 12180 solver.cpp:501] Iteration 13500, Testing net (#0)
  3302. I0819 14:27:24.413442 12177 data_reader.cpp:321] Restarting data pre-fetching
  3303. I0819 14:27:24.459936 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.985347
  3304. I0819 14:27:24.459985 12180 solver.cpp:588] (0.0) Test net output #1: loss = 0.0556239 (* 1 = 0.0556239 loss)
  3305. I0819 14:27:24.460084 12180 solver.cpp:271] [MultiGPU] Tests completed in 0.733583s
  3306. I0819 14:27:24.464371 12180 solver.cpp:333] [0.0] Iteration 13500 (136.317 iter/s, 0.733583s/100 iter), 14.4/21.3ep, loss = 6.10352e-05
  3307. I0819 14:27:24.464541 12180 solver.cpp:361] [0.0] Train net output #0: loss = 6.10352e-05 (* 1 = 6.10352e-05 loss)
  3308. I0819 14:27:24.464561 12180 sgd_solver.cpp:180] [0.0] Iteration 13500, lr = 0.00316119, m = 0.9, lrm = 0.0316119, wd = 0.0005, gs = 1
  3309. I0819 14:27:24.852434 12180 solver.cpp:333] [0.0] Iteration 13600 (257.729 iter/s, 0.388005s/100 iter), 14.5/21.3ep, loss = 0.000366688
  3310. I0819 14:27:24.852527 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000366688 (* 1 = 0.000366688 loss)
  3311. I0819 14:27:24.852540 12180 sgd_solver.cpp:180] [0.0] Iteration 13600, lr = 0.00315114, m = 0.9, lrm = 0.0315114, wd = 0.0005, gs = 1
  3312. I0819 14:27:25.244174 12180 solver.cpp:333] [0.0] Iteration 13700 (255.285 iter/s, 0.391719s/100 iter), 14.6/21.3ep, loss = 0.00476837
  3313. I0819 14:27:25.244303 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00476837 (* 1 = 0.00476837 loss)
  3314. I0819 14:27:25.244325 12180 sgd_solver.cpp:180] [0.0] Iteration 13700, lr = 0.00314116, m = 0.9, lrm = 0.0314116, wd = 0.0005, gs = 1
  3315. I0819 14:27:25.637338 12180 solver.cpp:333] [0.0] Iteration 13800 (254.368 iter/s, 0.393131s/100 iter), 14.7/21.3ep, loss = 0.00390625
  3316. I0819 14:27:25.637475 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00390625 (* 1 = 0.00390625 loss)
  3317. I0819 14:27:25.637491 12180 sgd_solver.cpp:180] [0.0] Iteration 13800, lr = 0.00313126, m = 0.9, lrm = 0.0313125, wd = 0.0005, gs = 1
  3318. I0819 14:27:26.029569 12180 solver.cpp:333] [0.0] Iteration 13900 (254.991 iter/s, 0.392171s/100 iter), 14.8/21.3ep, loss = 0.000183225
  3319. I0819 14:27:26.029652 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000183225 (* 1 = 0.000183225 loss)
  3320. I0819 14:27:26.029667 12180 sgd_solver.cpp:180] [0.0] Iteration 13900, lr = 0.00312142, m = 0.9, lrm = 0.0312142, wd = 0.0005, gs = 1
  3321. I0819 14:27:26.419548 12180 solver.cpp:501] Iteration 14000, Testing net (#0)
  3322. I0819 14:27:26.700500 12177 data_reader.cpp:321] Restarting data pre-fetching
  3323. I0819 14:27:26.759594 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.987666
  3324. I0819 14:27:26.759631 12180 solver.cpp:588] (0.0) Test net output #1: loss = 0.0496093 (* 1 = 0.0496093 loss)
  3325. I0819 14:27:26.759672 12180 solver.cpp:271] [MultiGPU] Tests completed in 0.730102s
  3326. I0819 14:27:26.763726 12180 solver.cpp:333] [0.0] Iteration 14000 (136.967 iter/s, 0.730102s/100 iter), 14.9/21.3ep, loss = 0.000366688
  3327. I0819 14:27:26.763816 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000366688 (* 1 = 0.000366688 loss)
  3328. I0819 14:27:26.763836 12180 sgd_solver.cpp:180] [0.0] Iteration 14000, lr = 0.00311166, m = 0.9, lrm = 0.0311166, wd = 0.0005, gs = 1
  3329. I0819 14:27:26.896312 12318 data_reader.cpp:321] Restarting data pre-fetching
  3330. I0819 14:27:27.151182 12180 solver.cpp:333] [0.0] Iteration 14100 (258.12 iter/s, 0.387416s/100 iter), 15/21.3ep, loss = 0.000427961
  3331. I0819 14:27:27.151305 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000427961 (* 1 = 0.000427961 loss)
  3332. I0819 14:27:27.151338 12180 sgd_solver.cpp:180] [0.0] Iteration 14100, lr = 0.00310198, m = 0.9, lrm = 0.0310198, wd = 0.0005, gs = 1
  3333. I0819 14:27:27.542544 12180 solver.cpp:333] [0.0] Iteration 14200 (255.548 iter/s, 0.391316s/100 iter), 15.1/21.3ep, loss = 6.10352e-05
  3334. I0819 14:27:27.542711 12180 solver.cpp:361] [0.0] Train net output #0: loss = 6.10352e-05 (* 1 = 6.10352e-05 loss)
  3335. I0819 14:27:27.542734 12180 sgd_solver.cpp:180] [0.0] Iteration 14200, lr = 0.00309236, m = 0.9, lrm = 0.0309236, wd = 0.0005, gs = 1
  3336. I0819 14:27:27.934154 12180 solver.cpp:333] [0.0] Iteration 14300 (255.369 iter/s, 0.39159s/100 iter), 15.3/21.3ep, loss = 6.10352e-05
  3337. I0819 14:27:27.934242 12180 solver.cpp:361] [0.0] Train net output #0: loss = 6.10352e-05 (* 1 = 6.10352e-05 loss)
  3338. I0819 14:27:27.934255 12180 sgd_solver.cpp:180] [0.0] Iteration 14300, lr = 0.00308281, m = 0.9, lrm = 0.0308281, wd = 0.0005, gs = 1
  3339. I0819 14:27:28.328300 12180 solver.cpp:333] [0.0] Iteration 14400 (253.73 iter/s, 0.394119s/100 iter), 15.4/21.3ep, loss = 0.000183225
  3340. I0819 14:27:28.328442 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000183225 (* 1 = 0.000183225 loss)
  3341. I0819 14:27:28.328456 12180 sgd_solver.cpp:180] [0.0] Iteration 14400, lr = 0.00307333, m = 0.9, lrm = 0.0307333, wd = 0.0005, gs = 1
  3342. I0819 14:27:28.712457 12180 solver.cpp:501] Iteration 14500, Testing net (#0)
  3343. I0819 14:27:28.980823 12177 data_reader.cpp:321] Restarting data pre-fetching
  3344. I0819 14:27:29.051789 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.986123
  3345. I0819 14:27:29.051827 12180 solver.cpp:588] (0.0) Test net output #1: loss = 0.0394769 (* 1 = 0.0394769 loss)
  3346. I0819 14:27:29.051851 12180 solver.cpp:271] [MultiGPU] Tests completed in 0.723519s
  3347. I0819 14:27:29.056363 12180 solver.cpp:333] [0.0] Iteration 14500 (138.213 iter/s, 0.723519s/100 iter), 15.5/21.3ep, loss = 0.000244379
  3348. I0819 14:27:29.056437 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000244379 (* 1 = 0.000244379 loss)
  3349. I0819 14:27:29.056452 12180 sgd_solver.cpp:180] [0.0] Iteration 14500, lr = 0.00306392, m = 0.9, lrm = 0.0306391, wd = 0.0005, gs = 1
  3350. I0819 14:27:29.445569 12180 solver.cpp:333] [0.0] Iteration 14600 (256.951 iter/s, 0.389179s/100 iter), 15.6/21.3ep, loss = 0
  3351. I0819 14:27:29.445713 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0 (* 1 = 0 loss)
  3352. I0819 14:27:29.445734 12180 sgd_solver.cpp:180] [0.0] Iteration 14600, lr = 0.00305457, m = 0.9, lrm = 0.0305457, wd = 0.0005, gs = 1
  3353. I0819 14:27:29.842738 12180 solver.cpp:333] [0.0] Iteration 14700 (251.802 iter/s, 0.397138s/100 iter), 15.7/21.3ep, loss = 0.00141144
  3354. I0819 14:27:29.842880 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00141144 (* 1 = 0.00141144 loss)
  3355. I0819 14:27:29.842911 12180 sgd_solver.cpp:180] [0.0] Iteration 14700, lr = 0.00304529, m = 0.9, lrm = 0.0304529, wd = 0.0005, gs = 1
  3356. I0819 14:27:30.244244 12180 solver.cpp:333] [0.0] Iteration 14800 (249.096 iter/s, 0.401452s/100 iter), 15.8/21.3ep, loss = 0.019577
  3357. I0819 14:27:30.244328 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.019577 (* 1 = 0.019577 loss)
  3358. I0819 14:27:30.244345 12180 sgd_solver.cpp:180] [0.0] Iteration 14800, lr = 0.00303608, m = 0.9, lrm = 0.0303607, wd = 0.0005, gs = 1
  3359. I0819 14:27:30.639144 12180 solver.cpp:333] [0.0] Iteration 14900 (253.251 iter/s, 0.394865s/100 iter), 15.9/21.3ep, loss = 0.00944519
  3360. I0819 14:27:30.639266 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00944519 (* 1 = 0.00944519 loss)
  3361. I0819 14:27:30.639307 12180 sgd_solver.cpp:180] [0.0] Iteration 14900, lr = 0.00302693, m = 0.9, lrm = 0.0302692, wd = 0.0005, gs = 1
  3362. I0819 14:27:30.914341 12318 data_reader.cpp:321] Restarting data pre-fetching
  3363. I0819 14:27:31.028600 12180 solver.cpp:501] Iteration 15000, Testing net (#0)
  3364. I0819 14:27:31.283694 12177 data_reader.cpp:321] Restarting data pre-fetching
  3365. I0819 14:27:31.368924 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.986123
  3366. I0819 14:27:31.368963 12180 solver.cpp:588] (0.0) Test net output #1: loss = 0.040018 (* 1 = 0.040018 loss)
  3367. I0819 14:27:31.369006 12180 solver.cpp:271] [MultiGPU] Tests completed in 0.729852s
  3368. I0819 14:27:31.373312 12180 solver.cpp:333] [0.0] Iteration 15000 (137.014 iter/s, 0.729852s/100 iter), 16/21.3ep, loss = 0
  3369. I0819 14:27:31.373409 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0 (* 1 = 0 loss)
  3370. I0819 14:27:31.373426 12180 sgd_solver.cpp:180] [0.0] Iteration 15000, lr = 0.00301784, m = 0.9, lrm = 0.0301784, wd = 0.0005, gs = 1
  3371. I0819 14:27:31.769130 12180 solver.cpp:333] [0.0] Iteration 15100 (252.664 iter/s, 0.395782s/100 iter), 16.1/21.3ep, loss = 6.10352e-05
  3372. I0819 14:27:31.769268 12180 solver.cpp:361] [0.0] Train net output #0: loss = 6.10352e-05 (* 1 = 6.10352e-05 loss)
  3373. I0819 14:27:31.769290 12180 sgd_solver.cpp:180] [0.0] Iteration 15100, lr = 0.00300882, m = 0.9, lrm = 0.0300882, wd = 0.0005, gs = 1
  3374. I0819 14:27:32.165801 12180 solver.cpp:333] [0.0] Iteration 15200 (252.128 iter/s, 0.396624s/100 iter), 16.2/21.3ep, loss = 0.00485229
  3375. I0819 14:27:32.165938 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00485229 (* 1 = 0.00485229 loss)
  3376. I0819 14:27:32.165967 12180 sgd_solver.cpp:180] [0.0] Iteration 15200, lr = 0.00299986, m = 0.9, lrm = 0.0299986, wd = 0.0005, gs = 1
  3377. I0819 14:27:32.557323 12180 solver.cpp:333] [0.0] Iteration 15300 (255.426 iter/s, 0.391503s/100 iter), 16.3/21.3ep, loss = 0.000672817
  3378. I0819 14:27:32.557482 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000672817 (* 1 = 0.000672817 loss)
  3379. I0819 14:27:32.557500 12180 sgd_solver.cpp:180] [0.0] Iteration 15300, lr = 0.00299096, m = 0.9, lrm = 0.0299096, wd = 0.0005, gs = 1
  3380. I0819 14:27:32.945071 12180 solver.cpp:333] [0.0] Iteration 15400 (257.92 iter/s, 0.387717s/100 iter), 16.4/21.3ep, loss = 0
  3381. I0819 14:27:32.945174 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0 (* 1 = 0 loss)
  3382. I0819 14:27:32.945188 12180 sgd_solver.cpp:180] [0.0] Iteration 15400, lr = 0.00298213, m = 0.9, lrm = 0.0298213, wd = 0.0005, gs = 1
  3383. I0819 14:27:33.330049 12180 solver.cpp:501] Iteration 15500, Testing net (#0)
  3384. I0819 14:27:33.572710 12177 data_reader.cpp:321] Restarting data pre-fetching
  3385. I0819 14:27:33.672628 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.989199
  3386. I0819 14:27:33.672662 12180 solver.cpp:588] (0.0) Test net output #1: loss = 0.0357635 (* 1 = 0.0357635 loss)
  3387. I0819 14:27:33.672703 12180 solver.cpp:271] [MultiGPU] Tests completed in 0.727585s
  3388. I0819 14:27:33.676717 12180 solver.cpp:333] [0.0] Iteration 15500 (137.441 iter/s, 0.727585s/100 iter), 16.5/21.3ep, loss = 0.0188446
  3389. I0819 14:27:33.676795 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.0188446 (* 1 = 0.0188446 loss)
  3390. I0819 14:27:33.676815 12180 sgd_solver.cpp:180] [0.0] Iteration 15500, lr = 0.00297335, m = 0.9, lrm = 0.0297335, wd = 0.0005, gs = 1
  3391. I0819 14:27:34.072019 12180 solver.cpp:333] [0.0] Iteration 15600 (252.996 iter/s, 0.395264s/100 iter), 16.6/21.3ep, loss = 0.000979424
  3392. I0819 14:27:34.072121 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000979424 (* 1 = 0.000979424 loss)
  3393. I0819 14:27:34.072139 12180 sgd_solver.cpp:180] [0.0] Iteration 15600, lr = 0.00296464, m = 0.9, lrm = 0.0296463, wd = 0.0005, gs = 1
  3394. I0819 14:27:34.463007 12180 solver.cpp:333] [0.0] Iteration 15700 (255.794 iter/s, 0.39094s/100 iter), 16.7/21.3ep, loss = 0
  3395. I0819 14:27:34.463106 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0 (* 1 = 0 loss)
  3396. I0819 14:27:34.463129 12180 sgd_solver.cpp:180] [0.0] Iteration 15700, lr = 0.00295598, m = 0.9, lrm = 0.0295598, wd = 0.0005, gs = 1
  3397. I0819 14:27:34.856030 12180 solver.cpp:333] [0.0] Iteration 15800 (254.451 iter/s, 0.393003s/100 iter), 16.9/21.3ep, loss = 0
  3398. I0819 14:27:34.856140 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0 (* 1 = 0 loss)
  3399. I0819 14:27:34.856155 12180 sgd_solver.cpp:180] [0.0] Iteration 15800, lr = 0.00294738, m = 0.9, lrm = 0.0294738, wd = 0.0005, gs = 1
  3400. I0819 14:27:35.244522 12180 solver.cpp:333] [0.0] Iteration 15900 (257.423 iter/s, 0.388466s/100 iter), 17/21.3ep, loss = 0.00122643
  3401. I0819 14:27:35.244832 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00122643 (* 1 = 0.00122643 loss)
  3402. I0819 14:27:35.244850 12180 sgd_solver.cpp:180] [0.0] Iteration 15900, lr = 0.00293884, m = 0.9, lrm = 0.0293884, wd = 0.0005, gs = 1
  3403. I0819 14:27:35.280970 12318 data_reader.cpp:321] Restarting data pre-fetching
  3404. I0819 14:27:35.635706 12180 solver.cpp:501] Iteration 16000, Testing net (#0)
  3405. I0819 14:27:35.868583 12177 data_reader.cpp:321] Restarting data pre-fetching
  3406. I0819 14:27:35.980756 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.988433
  3407. I0819 14:27:35.980792 12180 solver.cpp:588] (0.0) Test net output #1: loss = 0.0517566 (* 1 = 0.0517566 loss)
  3408. I0819 14:27:35.980839 12180 solver.cpp:271] [MultiGPU] Tests completed in 0.736285s
  3409. I0819 14:27:35.985051 12180 solver.cpp:333] [0.0] Iteration 16000 (135.817 iter/s, 0.736285s/100 iter), 17.1/21.3ep, loss = 0
  3410. I0819 14:27:35.985157 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0 (* 1 = 0 loss)
  3411. I0819 14:27:35.985183 12180 sgd_solver.cpp:180] [0.0] Iteration 16000, lr = 0.00293036, m = 0.9, lrm = 0.0293036, wd = 0.0005, gs = 1
  3412. I0819 14:27:36.380599 12180 solver.cpp:333] [0.0] Iteration 16100 (252.837 iter/s, 0.395512s/100 iter), 17.2/21.3ep, loss = 0
  3413. I0819 14:27:36.380712 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0 (* 1 = 0 loss)
  3414. I0819 14:27:36.380733 12180 sgd_solver.cpp:180] [0.0] Iteration 16100, lr = 0.00292194, m = 0.9, lrm = 0.0292194, wd = 0.0005, gs = 1
  3415. I0819 14:27:36.777509 12180 solver.cpp:333] [0.0] Iteration 16200 (251.965 iter/s, 0.39688s/100 iter), 17.3/21.3ep, loss = 0.00430679
  3416. I0819 14:27:36.777609 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00430679 (* 1 = 0.00430679 loss)
  3417. I0819 14:27:36.777647 12180 sgd_solver.cpp:180] [0.0] Iteration 16200, lr = 0.00291357, m = 0.9, lrm = 0.0291357, wd = 0.0005, gs = 1
  3418. I0819 14:27:37.174001 12180 solver.cpp:333] [0.0] Iteration 16300 (252.233 iter/s, 0.396459s/100 iter), 17.4/21.3ep, loss = 0
  3419. I0819 14:27:37.174110 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0 (* 1 = 0 loss)
  3420. I0819 14:27:37.174132 12180 sgd_solver.cpp:180] [0.0] Iteration 16300, lr = 0.00290526, m = 0.9, lrm = 0.0290526, wd = 0.0005, gs = 1
  3421. I0819 14:27:37.562978 12180 solver.cpp:333] [0.0] Iteration 16400 (257.115 iter/s, 0.38893s/100 iter), 17.5/21.3ep, loss = 0.00067234
  3422. I0819 14:27:37.563056 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00067234 (* 1 = 0.00067234 loss)
  3423. I0819 14:27:37.563078 12180 sgd_solver.cpp:180] [0.0] Iteration 16400, lr = 0.002897, m = 0.9, lrm = 0.02897, wd = 0.0005, gs = 1
  3424. I0819 14:27:37.955337 12180 solver.cpp:501] Iteration 16500, Testing net (#0)
  3425. I0819 14:27:38.174645 12177 data_reader.cpp:321] Restarting data pre-fetching
  3426. I0819 14:27:38.311553 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.988433
  3427. I0819 14:27:38.311589 12180 solver.cpp:588] (0.0) Test net output #1: loss = 0.0518357 (* 1 = 0.0518357 loss)
  3428. I0819 14:27:38.311628 12180 solver.cpp:271] [MultiGPU] Tests completed in 0.748634s
  3429. I0819 14:27:38.315997 12180 solver.cpp:333] [0.0] Iteration 16500 (133.577 iter/s, 0.748634s/100 iter), 17.6/21.3ep, loss = 0.232056
  3430. I0819 14:27:38.316052 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.232056 (* 1 = 0.232056 loss)
  3431. I0819 14:27:38.316118 12180 sgd_solver.cpp:180] [0.0] Iteration 16500, lr = 0.0028888, m = 0.9, lrm = 0.0288879, wd = 0.0005, gs = 1
  3432. I0819 14:27:38.713359 12180 solver.cpp:333] [0.0] Iteration 16600 (251.682 iter/s, 0.397326s/100 iter), 17.7/21.3ep, loss = 0.0216827
  3433. I0819 14:27:38.713481 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.0216827 (* 1 = 0.0216827 loss)
  3434. I0819 14:27:38.713501 12180 sgd_solver.cpp:180] [0.0] Iteration 16600, lr = 0.00288065, m = 0.9, lrm = 0.0288065, wd = 0.0005, gs = 1
  3435. I0819 14:27:39.104097 12180 solver.cpp:333] [0.0] Iteration 16700 (255.96 iter/s, 0.390686s/100 iter), 17.8/21.3ep, loss = 0
  3436. I0819 14:27:39.104202 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0 (* 1 = 0 loss)
  3437. I0819 14:27:39.104259 12180 sgd_solver.cpp:180] [0.0] Iteration 16700, lr = 0.00287255, m = 0.9, lrm = 0.0287255, wd = 0.0005, gs = 1
  3438. I0819 14:27:39.495709 12180 solver.cpp:333] [0.0] Iteration 16800 (255.37 iter/s, 0.391589s/100 iter), 17.9/21.3ep, loss = 0
  3439. I0819 14:27:39.495846 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0 (* 1 = 0 loss)
  3440. I0819 14:27:39.495867 12180 sgd_solver.cpp:180] [0.0] Iteration 16800, lr = 0.00286451, m = 0.9, lrm = 0.0286451, wd = 0.0005, gs = 1
  3441. I0819 14:27:39.675549 12318 data_reader.cpp:321] Restarting data pre-fetching
  3442. I0819 14:27:39.895102 12180 solver.cpp:333] [0.0] Iteration 16900 (250.401 iter/s, 0.399359s/100 iter), 18/21.3ep, loss = 0.00159168
  3443. I0819 14:27:39.895221 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00159168 (* 1 = 0.00159168 loss)
  3444. I0819 14:27:39.895236 12180 sgd_solver.cpp:180] [0.0] Iteration 16900, lr = 0.00285652, m = 0.9, lrm = 0.0285652, wd = 0.0005, gs = 1
  3445. I0819 14:27:40.282938 12180 solver.cpp:501] Iteration 17000, Testing net (#0)
  3446. I0819 14:27:40.491072 12177 data_reader.cpp:321] Restarting data pre-fetching
  3447. I0819 14:27:40.624936 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.990747
  3448. I0819 14:27:40.624969 12180 solver.cpp:588] (0.0) Test net output #1: loss = 0.0361191 (* 1 = 0.0361191 loss)
  3449. I0819 14:27:40.625003 12180 solver.cpp:271] [MultiGPU] Tests completed in 0.729877s
  3450. I0819 14:27:40.629124 12180 solver.cpp:333] [0.0] Iteration 17000 (137.009 iter/s, 0.729877s/100 iter), 18.1/21.3ep, loss = 6.10352e-05
  3451. I0819 14:27:40.629197 12180 solver.cpp:361] [0.0] Train net output #0: loss = 6.10352e-05 (* 1 = 6.10352e-05 loss)
  3452. I0819 14:27:40.629211 12180 sgd_solver.cpp:180] [0.0] Iteration 17000, lr = 0.00284858, m = 0.9, lrm = 0.0284858, wd = 0.0005, gs = 1
  3453. I0819 14:27:41.020599 12180 solver.cpp:333] [0.0] Iteration 17100 (255.456 iter/s, 0.391457s/100 iter), 18.2/21.3ep, loss = 6.10352e-05
  3454. I0819 14:27:41.020704 12180 solver.cpp:361] [0.0] Train net output #0: loss = 6.10352e-05 (* 1 = 6.10352e-05 loss)
  3455. I0819 14:27:41.020720 12180 sgd_solver.cpp:180] [0.0] Iteration 17100, lr = 0.00284069, m = 0.9, lrm = 0.0284069, wd = 0.0005, gs = 1
  3456. I0819 14:27:41.407433 12180 solver.cpp:333] [0.0] Iteration 17200 (258.535 iter/s, 0.386796s/100 iter), 18.3/21.3ep, loss = 0.000244379
  3457. I0819 14:27:41.407536 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000244379 (* 1 = 0.000244379 loss)
  3458. I0819 14:27:41.407567 12180 sgd_solver.cpp:180] [0.0] Iteration 17200, lr = 0.00283286, m = 0.9, lrm = 0.0283286, wd = 0.0005, gs = 1
  3459. I0819 14:27:41.800132 12180 solver.cpp:333] [0.0] Iteration 17300 (254.67 iter/s, 0.392665s/100 iter), 18.5/21.3ep, loss = 0.0199738
  3460. I0819 14:27:41.800249 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.0199738 (* 1 = 0.0199738 loss)
  3461. I0819 14:27:41.800274 12180 sgd_solver.cpp:180] [0.0] Iteration 17300, lr = 0.00282507, m = 0.9, lrm = 0.0282507, wd = 0.0005, gs = 1
  3462. I0819 14:27:42.191341 12180 solver.cpp:333] [0.0] Iteration 17400 (255.639 iter/s, 0.391177s/100 iter), 18.6/21.3ep, loss = 0.000183225
  3463. I0819 14:27:42.191417 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000183225 (* 1 = 0.000183225 loss)
  3464. I0819 14:27:42.191434 12180 sgd_solver.cpp:180] [0.0] Iteration 17400, lr = 0.00281733, m = 0.9, lrm = 0.0281733, wd = 0.0005, gs = 1
  3465. I0819 14:27:42.576354 12180 solver.cpp:501] Iteration 17500, Testing net (#0)
  3466. I0819 14:27:42.769623 12177 data_reader.cpp:321] Restarting data pre-fetching
  3467. I0819 14:27:42.918392 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.987666
  3468. I0819 14:27:42.918422 12180 solver.cpp:588] (0.0) Test net output #1: loss = 0.0419272 (* 1 = 0.0419272 loss)
  3469. I0819 14:27:42.918457 12180 solver.cpp:271] [MultiGPU] Tests completed in 0.727082s
  3470. I0819 14:27:42.922796 12180 solver.cpp:333] [0.0] Iteration 17500 (137.536 iter/s, 0.727082s/100 iter), 18.7/21.3ep, loss = 6.10352e-05
  3471. I0819 14:27:42.922881 12180 solver.cpp:361] [0.0] Train net output #0: loss = 6.10352e-05 (* 1 = 6.10352e-05 loss)
  3472. I0819 14:27:42.922974 12180 sgd_solver.cpp:180] [0.0] Iteration 17500, lr = 0.00280965, m = 0.9, lrm = 0.0280965, wd = 0.0005, gs = 1
  3473. I0819 14:27:43.326612 12180 solver.cpp:333] [0.0] Iteration 17600 (247.677 iter/s, 0.403752s/100 iter), 18.8/21.3ep, loss = 0.117798
  3474. I0819 14:27:43.326740 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.117798 (* 1 = 0.117798 loss)
  3475. I0819 14:27:43.326763 12180 sgd_solver.cpp:180] [0.0] Iteration 17600, lr = 0.00280201, m = 0.9, lrm = 0.0280201, wd = 0.0005, gs = 1
  3476. I0819 14:27:43.715689 12180 solver.cpp:333] [0.0] Iteration 17700 (257.025 iter/s, 0.389068s/100 iter), 18.9/21.3ep, loss = 0.000733852
  3477. I0819 14:27:43.715785 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000733852 (* 1 = 0.000733852 loss)
  3478. I0819 14:27:43.715813 12180 sgd_solver.cpp:180] [0.0] Iteration 17700, lr = 0.00279442, m = 0.9, lrm = 0.0279442, wd = 0.0005, gs = 1
  3479. I0819 14:27:44.042356 12318 data_reader.cpp:321] Restarting data pre-fetching
  3480. I0819 14:27:44.106453 12180 solver.cpp:333] [0.0] Iteration 17800 (255.93 iter/s, 0.390732s/100 iter), 19/21.3ep, loss = 6.10352e-05
  3481. I0819 14:27:44.106566 12180 solver.cpp:361] [0.0] Train net output #0: loss = 6.10352e-05 (* 1 = 6.10352e-05 loss)
  3482. I0819 14:27:44.106595 12180 sgd_solver.cpp:180] [0.0] Iteration 17800, lr = 0.00278688, m = 0.9, lrm = 0.0278688, wd = 0.0005, gs = 1
  3483. I0819 14:27:44.504736 12180 solver.cpp:333] [0.0] Iteration 17900 (251.104 iter/s, 0.398241s/100 iter), 19.1/21.3ep, loss = 0.000366688
  3484. I0819 14:27:44.504832 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000366688 (* 1 = 0.000366688 loss)
  3485. I0819 14:27:44.504860 12180 sgd_solver.cpp:180] [0.0] Iteration 17900, lr = 0.00277938, m = 0.9, lrm = 0.0277938, wd = 0.0005, gs = 1
  3486. I0819 14:27:44.896610 12180 solver.cpp:501] Iteration 18000, Testing net (#0)
  3487. I0819 14:27:45.080484 12177 data_reader.cpp:321] Restarting data pre-fetching
  3488. I0819 14:27:45.239715 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.988433
  3489. I0819 14:27:45.239766 12180 solver.cpp:588] (0.0) Test net output #1: loss = 0.0333037 (* 1 = 0.0333037 loss)
  3490. I0819 14:27:45.239790 12180 solver.cpp:271] [MultiGPU] Tests completed in 0.735032s
  3491. I0819 14:27:45.244056 12180 solver.cpp:333] [0.0] Iteration 18000 (136.048 iter/s, 0.735032s/100 iter), 19.2/21.3ep, loss = 0.000366688
  3492. I0819 14:27:45.244135 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000366688 (* 1 = 0.000366688 loss)
  3493. I0819 14:27:45.244149 12180 sgd_solver.cpp:180] [0.0] Iteration 18000, lr = 0.00277193, m = 0.9, lrm = 0.0277193, wd = 0.0005, gs = 1
  3494. I0819 14:27:45.644682 12180 solver.cpp:333] [0.0] Iteration 18100 (249.639 iter/s, 0.400579s/100 iter), 19.3/21.3ep, loss = 0.0153732
  3495. I0819 14:27:45.644776 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.0153732 (* 1 = 0.0153732 loss)
  3496. I0819 14:27:45.644798 12180 sgd_solver.cpp:180] [0.0] Iteration 18100, lr = 0.00276453, m = 0.9, lrm = 0.0276453, wd = 0.0005, gs = 1
  3497. I0819 14:27:46.047291 12180 solver.cpp:333] [0.0] Iteration 18200 (248.4 iter/s, 0.402576s/100 iter), 19.4/21.3ep, loss = 0.000183225
  3498. I0819 14:27:46.047417 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000183225 (* 1 = 0.000183225 loss)
  3499. I0819 14:27:46.047441 12180 sgd_solver.cpp:180] [0.0] Iteration 18200, lr = 0.00275718, m = 0.9, lrm = 0.0275717, wd = 0.0005, gs = 1
  3500. I0819 14:27:46.446501 12180 solver.cpp:333] [0.0] Iteration 18300 (250.518 iter/s, 0.399173s/100 iter), 19.5/21.3ep, loss = 0.000183225
  3501. I0819 14:27:46.446692 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000183225 (* 1 = 0.000183225 loss)
  3502. I0819 14:27:46.446728 12180 sgd_solver.cpp:180] [0.0] Iteration 18300, lr = 0.00274987, m = 0.9, lrm = 0.0274986, wd = 0.0005, gs = 1
  3503. I0819 14:27:46.839052 12180 solver.cpp:333] [0.0] Iteration 18400 (254.776 iter/s, 0.392501s/100 iter), 19.6/21.3ep, loss = 0.000183225
  3504. I0819 14:27:46.839157 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000183225 (* 1 = 0.000183225 loss)
  3505. I0819 14:27:46.839222 12180 sgd_solver.cpp:180] [0.0] Iteration 18400, lr = 0.0027426, m = 0.9, lrm = 0.027426, wd = 0.0005, gs = 1
  3506. I0819 14:27:47.228477 12180 solver.cpp:501] Iteration 18500, Testing net (#0)
  3507. I0819 14:27:47.395493 12177 data_reader.cpp:321] Restarting data pre-fetching
  3508. I0819 14:27:47.571609 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.986904
  3509. I0819 14:27:47.571638 12180 solver.cpp:588] (0.0) Test net output #1: loss = 0.0460434 (* 1 = 0.0460434 loss)
  3510. I0819 14:27:47.571682 12180 solver.cpp:271] [MultiGPU] Tests completed in 0.73261s
  3511. I0819 14:27:47.575708 12180 solver.cpp:333] [0.0] Iteration 18500 (136.498 iter/s, 0.73261s/100 iter), 19.7/21.3ep, loss = 0.00116253
  3512. I0819 14:27:47.575798 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00116253 (* 1 = 0.00116253 loss)
  3513. I0819 14:27:47.575822 12180 sgd_solver.cpp:180] [0.0] Iteration 18500, lr = 0.00273538, m = 0.9, lrm = 0.0273538, wd = 0.0005, gs = 1
  3514. I0819 14:27:47.966687 12180 solver.cpp:333] [0.0] Iteration 18600 (255.824 iter/s, 0.390894s/100 iter), 19.8/21.3ep, loss = 0.0254974
  3515. I0819 14:27:47.966786 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.0254974 (* 1 = 0.0254974 loss)
  3516. I0819 14:27:47.966800 12180 sgd_solver.cpp:180] [0.0] Iteration 18600, lr = 0.0027282, m = 0.9, lrm = 0.027282, wd = 0.0005, gs = 1
  3517. I0819 14:27:48.361131 12180 solver.cpp:333] [0.0] Iteration 18700 (253.518 iter/s, 0.394449s/100 iter), 19.9/21.3ep, loss = 0.00012219
  3518. I0819 14:27:48.361234 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00012219 (* 1 = 0.00012219 loss)
  3519. I0819 14:27:48.361249 12180 sgd_solver.cpp:180] [0.0] Iteration 18700, lr = 0.00272107, m = 0.9, lrm = 0.0272107, wd = 0.0005, gs = 1
  3520. I0819 14:27:48.442080 12318 data_reader.cpp:321] Restarting data pre-fetching
  3521. I0819 14:27:48.759543 12180 solver.cpp:333] [0.0] Iteration 18800 (251.032 iter/s, 0.398356s/100 iter), 20.1/21.3ep, loss = 0.0012846
  3522. I0819 14:27:48.759675 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.0012846 (* 1 = 0.0012846 loss)
  3523. I0819 14:27:48.759693 12180 sgd_solver.cpp:180] [0.0] Iteration 18800, lr = 0.00271398, m = 0.9, lrm = 0.0271398, wd = 0.0005, gs = 1
  3524. I0819 14:27:49.157918 12180 solver.cpp:333] [0.0] Iteration 18900 (251.031 iter/s, 0.398356s/100 iter), 20.2/21.3ep, loss = 6.10352e-05
  3525. I0819 14:27:49.158027 12180 solver.cpp:361] [0.0] Train net output #0: loss = 6.10352e-05 (* 1 = 6.10352e-05 loss)
  3526. I0819 14:27:49.158057 12180 sgd_solver.cpp:180] [0.0] Iteration 18900, lr = 0.00270694, m = 0.9, lrm = 0.0270693, wd = 0.0005, gs = 1
  3527. I0819 14:27:49.545063 12180 solver.cpp:501] Iteration 19000, Testing net (#0)
  3528. I0819 14:27:49.697620 12177 data_reader.cpp:321] Restarting data pre-fetching
  3529. I0819 14:27:49.698765 12228 blocking_queue.cpp:40] Data layer prefetch queue empty
  3530. I0819 14:27:49.888763 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.986118
  3531. I0819 14:27:49.888818 12180 solver.cpp:588] (0.0) Test net output #1: loss = 0.037638 (* 1 = 0.037638 loss)
  3532. I0819 14:27:49.888841 12180 solver.cpp:271] [MultiGPU] Tests completed in 0.730892s
  3533. I0819 14:27:49.893033 12180 solver.cpp:333] [0.0] Iteration 19000 (136.819 iter/s, 0.730892s/100 iter), 20.3/21.3ep, loss = 0
  3534. I0819 14:27:49.893095 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0 (* 1 = 0 loss)
  3535. I0819 14:27:49.893110 12180 sgd_solver.cpp:180] [0.0] Iteration 19000, lr = 0.00269993, m = 0.9, lrm = 0.0269993, wd = 0.0005, gs = 1
  3536. I0819 14:27:50.287739 12180 solver.cpp:333] [0.0] Iteration 19100 (253.381 iter/s, 0.394663s/100 iter), 20.4/21.3ep, loss = 0
  3537. I0819 14:27:50.287842 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0 (* 1 = 0 loss)
  3538. I0819 14:27:50.287858 12180 sgd_solver.cpp:180] [0.0] Iteration 19100, lr = 0.00269297, m = 0.9, lrm = 0.0269297, wd = 0.0005, gs = 1
  3539. I0819 14:27:50.689630 12180 solver.cpp:333] [0.0] Iteration 19200 (248.855 iter/s, 0.40184s/100 iter), 20.5/21.3ep, loss = 0
  3540. I0819 14:27:50.689730 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0 (* 1 = 0 loss)
  3541. I0819 14:27:50.689788 12180 sgd_solver.cpp:180] [0.0] Iteration 19200, lr = 0.00268605, m = 0.9, lrm = 0.0268605, wd = 0.0005, gs = 1
  3542. I0819 14:27:51.086681 12180 solver.cpp:333] [0.0] Iteration 19300 (251.881 iter/s, 0.397013s/100 iter), 20.6/21.3ep, loss = 0.000427723
  3543. I0819 14:27:51.086788 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000427723 (* 1 = 0.000427723 loss)
  3544. I0819 14:27:51.086802 12180 sgd_solver.cpp:180] [0.0] Iteration 19300, lr = 0.00267917, m = 0.9, lrm = 0.0267917, wd = 0.0005, gs = 1
  3545. I0819 14:27:51.485642 12180 solver.cpp:333] [0.0] Iteration 19400 (250.665 iter/s, 0.398938s/100 iter), 20.7/21.3ep, loss = 0
  3546. I0819 14:27:51.485769 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0 (* 1 = 0 loss)
  3547. I0819 14:27:51.485783 12180 sgd_solver.cpp:180] [0.0] Iteration 19400, lr = 0.00267233, m = 0.9, lrm = 0.0267233, wd = 0.0005, gs = 1
  3548. I0819 14:27:51.878471 12180 solver.cpp:501] Iteration 19500, Testing net (#0)
  3549. I0819 14:27:52.025992 12177 data_reader.cpp:321] Restarting data pre-fetching
  3550. I0819 14:27:52.228219 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.986118
  3551. I0819 14:27:52.228269 12180 solver.cpp:588] (0.0) Test net output #1: loss = 0.0535487 (* 1 = 0.0535487 loss)
  3552. I0819 14:27:52.228309 12180 solver.cpp:271] [MultiGPU] Tests completed in 0.742619s
  3553. I0819 14:27:52.232753 12180 solver.cpp:333] [0.0] Iteration 19500 (134.659 iter/s, 0.742619s/100 iter), 20.8/21.3ep, loss = 0.00878143
  3554. I0819 14:27:52.232823 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.00878143 (* 1 = 0.00878143 loss)
  3555. I0819 14:27:52.232838 12180 sgd_solver.cpp:180] [0.0] Iteration 19500, lr = 0.00266554, m = 0.9, lrm = 0.0266554, wd = 0.0005, gs = 1
  3556. I0819 14:27:52.641575 12180 solver.cpp:333] [0.0] Iteration 19600 (244.628 iter/s, 0.408784s/100 iter), 20.9/21.3ep, loss = 6.10352e-05
  3557. I0819 14:27:52.641687 12180 solver.cpp:361] [0.0] Train net output #0: loss = 6.10352e-05 (* 1 = 6.10352e-05 loss)
  3558. I0819 14:27:52.641707 12180 sgd_solver.cpp:180] [0.0] Iteration 19600, lr = 0.00265878, m = 0.9, lrm = 0.0265878, wd = 0.0005, gs = 1
  3559. I0819 14:27:52.885428 12318 data_reader.cpp:321] Restarting data pre-fetching
  3560. I0819 14:27:53.055436 12180 solver.cpp:333] [0.0] Iteration 19700 (241.658 iter/s, 0.413809s/100 iter), 21/21.3ep, loss = 6.10352e-05
  3561. I0819 14:27:53.055524 12180 solver.cpp:361] [0.0] Train net output #0: loss = 6.10352e-05 (* 1 = 6.10352e-05 loss)
  3562. I0819 14:27:53.055538 12180 sgd_solver.cpp:180] [0.0] Iteration 19700, lr = 0.00265206, m = 0.9, lrm = 0.0265206, wd = 0.0005, gs = 1
  3563. I0819 14:27:53.459993 12180 solver.cpp:333] [0.0] Iteration 19800 (247.192 iter/s, 0.404544s/100 iter), 21.1/21.3ep, loss = 0
  3564. I0819 14:27:53.460112 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0 (* 1 = 0 loss)
  3565. I0819 14:27:53.460125 12180 sgd_solver.cpp:180] [0.0] Iteration 19800, lr = 0.00264539, m = 0.9, lrm = 0.0264539, wd = 0.0005, gs = 1
  3566. I0819 14:27:53.862141 12180 solver.cpp:333] [0.0] Iteration 19900 (248.689 iter/s, 0.402108s/100 iter), 21.2/21.3ep, loss = 0.000550747
  3567. I0819 14:27:53.862237 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0.000550747 (* 1 = 0.000550747 loss)
  3568. I0819 14:27:53.862257 12180 sgd_solver.cpp:180] [0.0] Iteration 19900, lr = 0.00263875, m = 0.9, lrm = 0.0263875, wd = 0.0005, gs = 1
  3569. I0819 14:27:54.252153 12180 solver.cpp:333] [0.0] Iteration 20000 (253.862 iter/s, 0.389976s/99 iter), 21.3/21.3ep, loss = 0
  3570. I0819 14:27:54.252243 12180 solver.cpp:361] [0.0] Train net output #0: loss = 0 (* 1 = 0 loss)
  3571. I0819 14:27:54.256789 12180 solver.cpp:769] Snapshotting to binary proto file _iter_20000.caffemodel
  3572. I0819 14:27:54.265588 12180 sgd_solver.cpp:448] Snapshotting solver state to binary proto file _iter_20000.solverstate
  3573. I0819 14:27:54.267252 12180 solver.cpp:466] Iteration 20000, loss = 0.0498047
  3574. I0819 14:27:54.267278 12180 solver.cpp:501] Iteration 20000, Testing net (#0)
  3575. I0819 14:27:54.396373 12177 data_reader.cpp:321] Restarting data pre-fetching
  3576. I0819 14:27:54.607734 12180 solver.cpp:588] (0.0) Test net output #0: accuracy = 0.992285
  3577. I0819 14:27:54.607769 12180 solver.cpp:588] (0.0) Test net output #1: loss = 0.0369105 (* 1 = 0.0369105 loss)
  3578. I0819 14:27:54.608280 12127 parallel.cpp:118] Root Solver performance on device 0: 218.8 * 8 = 1750 img/sec (20000 itr in 91.39 sec)
  3579. I0819 14:27:54.608325 12127 parallel.cpp:128] Solver performance on device 1: 218.8 * 8 = 1750 img/sec (20000 itr in 91.4 sec)
  3580. I0819 14:27:54.608333 12127 parallel.cpp:128] Solver performance on device 2: 218.8 * 8 = 1750 img/sec (20000 itr in 91.4 sec)
  3581. I0819 14:27:54.608340 12127 parallel.cpp:128] Solver performance on device 3: 218.8 * 8 = 1750 img/sec (20000 itr in 91.4 sec)
  3582. I0819 14:27:54.608347 12127 parallel.cpp:128] Solver performance on device 4: 218.8 * 8 = 1750 img/sec (20000 itr in 91.4 sec)
  3583. I0819 14:27:54.608355 12127 parallel.cpp:128] Solver performance on device 5: 218.8 * 8 = 1750 img/sec (20000 itr in 91.4 sec)
  3584. I0819 14:27:54.608378 12127 parallel.cpp:128] Solver performance on device 6: 218.8 * 8 = 1750 img/sec (20000 itr in 91.4 sec)
  3585. I0819 14:27:54.608386 12127 parallel.cpp:128] Solver performance on device 7: 218.8 * 8 = 1750 img/sec (20000 itr in 91.4 sec)
  3586. I0819 14:27:54.608388 12127 parallel.cpp:143] Overall multi-GPU performance: 14003.7 img/sec
  3587. I0819 14:27:55.038269 12127 caffe.cpp:269] Optimization Done in 1m 51s
  3588. Entering ReduceAndUpdate thread ........................
  3589. Entering ReduceAndUpdate thread ........................
  3590. Entering ReduceAndUpdate thread ........................
  3591. Entering ReduceAndUpdate thread ........................
  3592. Entering ReduceAndUpdate thread ........................
  3593. Entering ReduceAndUpdate thread ........................
  3594. Entering ReduceAndUpdate thread ........................
  3595. Entering ReduceAndUpdate thread ........................
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!