API tools faq
paste
Guest User

Untitled

a guest
Nov 14th, 2019
233
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 66.36 KB | None | 0 0
raw download clone embed print report
  1. 2019-11-14 14:38:14.028064: I tensorflow/stream_executor/cuda/cuda_driver.cc:831] failed to allocate 7.60G (8160437760 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
  2. 2019-11-14 14:38:14.028311: I tensorflow/stream_executor/cuda/cuda_driver.cc:831] failed to allocate 6.84G (7344393728 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
  3. WARNING:tensorflow:From c:\python\lib\site-packages\tensorflow_core\python\training\saver.py:1069: get_checkpoint_mtimes (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
  4. Instructions for updating:
  5. Use standard file utilities to get mtimes.
  6. W1114 14:38:14.551839 9104 deprecation.py:323] From c:\python\lib\site-packages\tensorflow_core\python\training\saver.py:1069: get_checkpoint_mtimes (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
  7. Instructions for updating:
  8. Use standard file utilities to get mtimes.
  9. INFO:tensorflow:Running local_init_op.
  10. I1114 14:38:14.811158 9104 session_manager.py:500] Running local_init_op.
  11. INFO:tensorflow:Done running local_init_op.
  12. I1114 14:38:14.944813 9104 session_manager.py:502] Done running local_init_op.
  13. INFO:tensorflow:Saving checkpoints for 0 into C:\Users\conspiracy2\Documents\music comp\out\11-14 set 2.0\checkpts\model.ckpt.
  14. I1114 14:38:17.920329 9104 basic_session_run_hooks.py:606] Saving checkpoints for 0 into C:\Users\conspiracy2\Documents\music comp\out\11-14 set 2.0\checkpts\model.ckpt.
  15. 2019-11-14 14:38:21.598678: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_100.dll
  16. 2019-11-14 14:38:22.574418: I tensorflow/stream_executor/cuda/cuda_driver.cc:831] failed to allocate 1.44G (1550483456 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
  17. 2019-11-14 14:38:22.574642: I tensorflow/stream_executor/cuda/cuda_driver.cc:831] failed to allocate 1.44G (1550483456 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
  18. 2019-11-14 14:38:32.575117: I tensorflow/stream_executor/cuda/cuda_driver.cc:831] failed to allocate 1.44G (1550483456 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
  19. 2019-11-14 14:38:32.575322: I tensorflow/stream_executor/cuda/cuda_driver.cc:831] failed to allocate 1.44G (1550483456 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
  20. 2019-11-14 14:38:32.575478: W tensorflow/core/common_runtime/bfc_allocator.cc:419] Allocator (GPU_0_bfc) ran out of memory trying to allocate 576.00MiB (rounded to 603979776). Current allocation summary follows.
  21. 2019-11-14 14:38:32.575683: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (256): Total Chunks: 75, Chunks in use: 69. 18.8KiB allocated for chunks. 17.3KiB in use in bin. 304B client-requested in use in bin.
  22. 2019-11-14 14:38:32.575871: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (512): Total Chunks: 1, Chunks in use: 0. 512B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
  23. 2019-11-14 14:38:32.576033: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (1024): Total Chunks: 2, Chunks in use: 2. 2.3KiB allocated for chunks. 2.3KiB in use in bin. 2.0KiB client-requested in use in bin.
  24. 2019-11-14 14:38:32.576206: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (2048): Total Chunks: 96, Chunks in use: 96. 192.0KiB allocated for chunks. 192.0KiB in use in bin. 192.0KiB client-requested in use in bin.
  25. 2019-11-14 14:38:32.576406: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (4096): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
  26. 2019-11-14 14:38:32.576604: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (8192): Total Chunks: 19, Chunks in use: 18. 158.8KiB allocated for chunks. 144.0KiB in use in bin. 144.0KiB client-requested in use in bin.
  27. 2019-11-14 14:38:32.576926: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (16384): Total Chunks: 13, Chunks in use: 12. 208.0KiB allocated for chunks. 192.0KiB in use in bin. 192.0KiB client-requested in use in bin.
  28. 2019-11-14 14:38:32.577128: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (32768): Total Chunks: 48, Chunks in use: 48. 1.82MiB allocated for chunks. 1.82MiB in use in bin. 1.82MiB client-requested in use in bin.
  29. 2019-11-14 14:38:32.577355: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (65536): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
  30. 2019-11-14 14:38:32.577566: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (131072): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
  31. 2019-11-14 14:38:32.577770: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (262144): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
  32. 2019-11-14 14:38:32.577973: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (524288): Total Chunks: 1, Chunks in use: 1. 620.0KiB allocated for chunks. 620.0KiB in use in bin. 620.0KiB client-requested in use in bin.
  33. 2019-11-14 14:38:32.578238: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (1048576): Total Chunks: 72, Chunks in use: 72. 72.00MiB allocated for chunks. 72.00MiB in use in bin. 72.00MiB client-requested in use in bin.
  34. 2019-11-14 14:38:32.578395: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (2097152): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
  35. 2019-11-14 14:38:32.578561: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (4194304): Total Chunks: 37, Chunks in use: 36. 151.84MiB allocated for chunks. 144.00MiB in use in bin. 144.00MiB client-requested in use in bin.
  36. 2019-11-14 14:38:32.578834: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (8388608): Total Chunks: 38, Chunks in use: 37. 304.00MiB allocated for chunks. 296.00MiB in use in bin. 296.00MiB client-requested in use in bin.
  37. 2019-11-14 14:38:32.579017: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
  38. 2019-11-14 14:38:32.579203: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (33554432): Total Chunks: 6, Chunks in use: 6. 192.00MiB allocated for chunks. 192.00MiB in use in bin. 192.00MiB client-requested in use in bin.
  39. 2019-11-14 14:38:32.579489: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (67108864): Total Chunks: 2, Chunks in use: 1. 160.00MiB allocated for chunks. 64.00MiB in use in bin. 64.00MiB client-requested in use in bin.
  40. 2019-11-14 14:38:32.579704: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
  41. 2019-11-14 14:38:32.579998: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (268435456): Total Chunks: 11, Chunks in use: 10. 5.29GiB allocated for chunks. 5.00GiB in use in bin. 5.00GiB client-requested in use in bin.
  42. 2019-11-14 14:38:32.580279: I tensorflow/core/common_runtime/bfc_allocator.cc:885] Bin for 576.00MiB was 256.00MiB, Chunk State:
  43. 2019-11-14 14:38:32.580407: I tensorflow/core/common_runtime/bfc_allocator.cc:891] Size: 300.92MiB | Requested Size: 0B | in_use: 0 | bin_num: 20, prev: Size: 512.00MiB | Requested Size: 512.00MiB | in_use: 1 | bin_num: -1
  44. 2019-11-14 14:38:32.580626: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 6609954304
  45. 2019-11-14 14:38:32.580743: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070BE00000 next 1 of size 1280
  46. 2019-11-14 14:38:32.580875: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070BE00500 next 2 of size 256
  47. 2019-11-14 14:38:32.580950: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070BE00600 next 3 of size 256
  48. 2019-11-14 14:38:32.581009: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070BE00700 next 4 of size 256
  49. 2019-11-14 14:38:32.581135: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070BE00800 next 5 of size 256
  50. 2019-11-14 14:38:32.581260: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070BE00900 next 6 of size 2048
  51. 2019-11-14 14:38:32.581380: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070BE01100 next 7 of size 2048
  52. 2019-11-14 14:38:32.581496: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070BE01900 next 8 of size 2048
  53. 2019-11-14 14:38:32.581613: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070BE02100 next 9 of size 2048
  54. 2019-11-14 14:38:32.581740: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070BE02900 next 10 of size 8192
  55. 2019-11-14 14:38:32.581858: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070BE04900 next 11 of size 1048576
  56. 2019-11-14 14:38:32.581993: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070BF04900 next 12 of size 8192
  57. 2019-11-14 14:38:32.582149: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070BF06900 next 13 of size 1048576
  58. 2019-11-14 14:38:32.582312: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070C006900 next 14 of size 1048576
  59. 2019-11-14 14:38:32.582446: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070C106900 next 15 of size 2048
  60. 2019-11-14 14:38:32.582638: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070C107100 next 16 of size 2048
  61. 2019-11-14 14:38:32.582824: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070C107900 next 17 of size 1048576
  62. 2019-11-14 14:38:32.583034: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070C207900 next 18 of size 2048
  63. 2019-11-14 14:38:32.583301: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070C208100 next 19 of size 1048576
  64. 2019-11-14 14:38:32.583452: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070C308100 next 20 of size 1048576
  65. 2019-11-14 14:38:32.583629: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070C408100 next 21 of size 1048576
  66. 2019-11-14 14:38:32.583800: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070C508100 next 22 of size 2048
  67. 2019-11-14 14:38:32.583953: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070C508900 next 23 of size 8192
  68. 2019-11-14 14:38:32.584095: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070C50A900 next 24 of size 4194304
  69. 2019-11-14 14:38:32.584174: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070C90A900 next 25 of size 2048
  70. 2019-11-14 14:38:32.584236: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070C90B100 next 26 of size 4194304
  71. 2019-11-14 14:38:32.584434: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070CD0B100 next 27 of size 1048576
  72. 2019-11-14 14:38:32.584563: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070CE0B100 next 28 of size 1048576
  73. 2019-11-14 14:38:32.584727: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070CF0B100 next 29 of size 2048
  74. 2019-11-14 14:38:32.584851: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070CF0B900 next 30 of size 1048576
  75. 2019-11-14 14:38:32.584977: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070D00B900 next 31 of size 2048
  76. 2019-11-14 14:38:32.585154: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070D00C100 next 32 of size 2048
  77. 2019-11-14 14:38:32.585311: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070D00C900 next 33 of size 1048576
  78. 2019-11-14 14:38:32.585467: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070D10C900 next 34 of size 2048
  79. 2019-11-14 14:38:32.585542: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070D10D100 next 35 of size 1048576
  80. 2019-11-14 14:38:32.585731: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070D20D100 next 36 of size 1048576
  81. 2019-11-14 14:38:32.585865: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070D30D100 next 37 of size 40960
  82. 2019-11-14 14:38:32.585988: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070D317100 next 38 of size 40960
  83. 2019-11-14 14:38:32.586134: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070D321100 next 39 of size 2048
  84. 2019-11-14 14:38:32.586279: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070D321900 next 40 of size 1048576
  85. 2019-11-14 14:38:32.586360: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070D421900 next 41 of size 2048
  86. 2019-11-14 14:38:32.586438: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070D422100 next 42 of size 40960
  87. 2019-11-14 14:38:32.586638: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070D42C100 next 43 of size 2048
  88. 2019-11-14 14:38:32.586743: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070D42C900 next 44 of size 2048
  89. 2019-11-14 14:38:32.586836: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070D42D100 next 45 of size 1048576
  90. 2019-11-14 14:38:32.587027: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070D52D100 next 46 of size 2048
  91. 2019-11-14 14:38:32.587155: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070D52D900 next 47 of size 1048576
  92. 2019-11-14 14:38:32.587238: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070D62D900 next 48 of size 2048
  93. 2019-11-14 14:38:32.587412: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070D62E100 next 49 of size 2048
  94. 2019-11-14 14:38:32.587540: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070D62E900 next 50 of size 1048576
  95. 2019-11-14 14:38:32.587619: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070D72E900 next 51 of size 2048
  96. 2019-11-14 14:38:32.587678: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070D72F100 next 52 of size 1048576
  97. 2019-11-14 14:38:32.587739: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070D82F100 next 53 of size 4194304
  98. 2019-11-14 14:38:32.587887: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070DC2F100 next 54 of size 38912
  99. 2019-11-14 14:38:32.588011: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070DC38900 next 55 of size 8192
  100. 2019-11-14 14:38:32.588127: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070DC3A900 next 56 of size 1048576
  101. 2019-11-14 14:38:32.588210: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070DD3A900 next 57 of size 1048576
  102. 2019-11-14 14:38:32.588273: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070DE3A900 next 58 of size 2048
  103. 2019-11-14 14:38:32.588448: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070DE3B100 next 59 of size 2048
  104. 2019-11-14 14:38:32.588629: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070DE3B900 next 60 of size 4194304
  105. 2019-11-14 14:38:32.588746: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070E23B900 next 61 of size 2048
  106. 2019-11-14 14:38:32.588868: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070E23C100 next 62 of size 1048576
  107. 2019-11-14 14:38:32.589006: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070E33C100 next 63 of size 4194304
  108. 2019-11-14 14:38:32.589131: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070E73C100 next 64 of size 2048
  109. 2019-11-14 14:38:32.589196: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070E73C900 next 65 of size 4194304
  110. 2019-11-14 14:38:32.589374: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070EB3C900 next 66 of size 2048
  111. 2019-11-14 14:38:32.589494: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070EB3D100 next 67 of size 2048
  112. 2019-11-14 14:38:32.589623: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070EB3D900 next 68 of size 2048
  113. 2019-11-14 14:38:32.589735: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070EB3E100 next 69 of size 38912
  114. 2019-11-14 14:38:32.589912: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070EB47900 next 70 of size 1048576
  115. 2019-11-14 14:38:32.590048: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070EC47900 next 71 of size 4194304
  116. 2019-11-14 14:38:32.590193: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070F047900 next 72 of size 2048
  117. 2019-11-14 14:38:32.590344: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070F048100 next 73 of size 1048576
  118. 2019-11-14 14:38:32.590531: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070F148100 next 74 of size 2048
  119. 2019-11-14 14:38:32.590696: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070F148900 next 75 of size 1048576
  120. 2019-11-14 14:38:32.590847: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070F248900 next 76 of size 2048
  121. 2019-11-14 14:38:32.591026: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070F249100 next 77 of size 1048576
  122. 2019-11-14 14:38:32.591163: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070F349100 next 78 of size 1048576
  123. 2019-11-14 14:38:32.591339: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070F449100 next 79 of size 38912
  124. 2019-11-14 14:38:32.591514: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070F452900 next 80 of size 2048
  125. 2019-11-14 14:38:32.591708: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070F453100 next 81 of size 8192
  126. 2019-11-14 14:38:32.591869: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070F455100 next 82 of size 2048
  127. 2019-11-14 14:38:32.592015: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070F455900 next 83 of size 1048576
  128. 2019-11-14 14:38:32.592123: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070F555900 next 84 of size 1048576
  129. 2019-11-14 14:38:32.592344: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070F655900 next 85 of size 1048576
  130. 2019-11-14 14:38:32.592496: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070F755900 next 86 of size 1048576
  131. 2019-11-14 14:38:32.592741: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070F855900 next 87 of size 40960
  132. 2019-11-14 14:38:32.592876: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070F85F900 next 88 of size 38912
  133. 2019-11-14 14:38:32.592951: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070F869100 next 89 of size 4194304
  134. 2019-11-14 14:38:32.593020: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070FC69100 next 90 of size 2048
  135. 2019-11-14 14:38:32.593121: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070FC69900 next 91 of size 1048576
  136. 2019-11-14 14:38:32.593367: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000070FD69900 next 92 of size 4194304
  137. 2019-11-14 14:38:32.593509: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000710169900 next 93 of size 1048576
  138. 2019-11-14 14:38:32.593653: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000710269900 next 94 of size 1048576
  139. 2019-11-14 14:38:32.593799: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000710369900 next 95 of size 1048576
  140. 2019-11-14 14:38:32.594025: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000710469900 next 96 of size 2048
  141. 2019-11-14 14:38:32.594209: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071046A100 next 97 of size 2048
  142. 2019-11-14 14:38:32.594310: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071046A900 next 98 of size 1048576
  143. 2019-11-14 14:38:32.594548: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071056A900 next 99 of size 2048
  144. 2019-11-14 14:38:32.594694: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071056B100 next 100 of size 2048
  145. 2019-11-14 14:38:32.594862: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071056B900 next 101 of size 1048576
  146. 2019-11-14 14:38:32.595019: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071066B900 next 102 of size 2048
  147. 2019-11-14 14:38:32.595238: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071066C100 next 103 of size 1048576
  148. 2019-11-14 14:38:32.595398: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071076C100 next 104 of size 1048576
  149. 2019-11-14 14:38:32.595624: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071086C100 next 105 of size 2048
  150. 2019-11-14 14:38:32.595768: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071086C900 next 106 of size 2048
  151. 2019-11-14 14:38:32.595915: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071086D100 next 107 of size 4194304
  152. 2019-11-14 14:38:32.596040: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000710C6D100 next 108 of size 2048
  153. 2019-11-14 14:38:32.596244: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000710C6D900 next 109 of size 1048576
  154. 2019-11-14 14:38:32.596380: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000710D6D900 next 110 of size 2048
  155. 2019-11-14 14:38:32.596508: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000710D6E100 next 111 of size 1048576
  156. 2019-11-14 14:38:32.596626: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000710E6E100 next 112 of size 256
  157. 2019-11-14 14:38:32.596748: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000710E6E200 next 113 of size 2048
  158. 2019-11-14 14:38:32.596880: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000710E6EA00 next 114 of size 1048576
  159. 2019-11-14 14:38:32.596955: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000710F6EA00 next 115 of size 2048
  160. 2019-11-14 14:38:32.597149: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000710F6F200 next 116 of size 2048
  161. 2019-11-14 14:38:32.597265: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000710F6FA00 next 117 of size 8192
  162. 2019-11-14 14:38:32.597452: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000710F71A00 next 118 of size 38912
  163. 2019-11-14 14:38:32.597672: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000710F7B200 next 119 of size 38912
  164. 2019-11-14 14:38:32.597836: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000710F84A00 next 120 of size 40960
  165. 2019-11-14 14:38:32.598050: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000710F8EA00 next 121 of size 38912
  166. 2019-11-14 14:38:32.598185: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000710F98200 next 122 of size 4194304
  167. 2019-11-14 14:38:32.598354: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000711398200 next 123 of size 38912
  168. 2019-11-14 14:38:32.598474: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007113A1A00 next 124 of size 2048
  169. 2019-11-14 14:38:32.598588: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007113A2200 next 125 of size 2048
  170. 2019-11-14 14:38:32.598684: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007113A2A00 next 126 of size 2048
  171. 2019-11-14 14:38:32.598889: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007113A3200 next 127 of size 4194304
  172. 2019-11-14 14:38:32.599013: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007117A3200 next 128 of size 1048576
  173. 2019-11-14 14:38:32.599187: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007118A3200 next 129 of size 38912
  174. 2019-11-14 14:38:32.599319: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007118ACA00 next 130 of size 1048576
  175. 2019-11-14 14:38:32.599501: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007119ACA00 next 131 of size 2048
  176. 2019-11-14 14:38:32.599648: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007119AD200 next 132 of size 38912
  177. 2019-11-14 14:38:32.599812: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007119B6A00 next 133 of size 2048
  178. 2019-11-14 14:38:32.599982: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007119B7200 next 134 of size 2048
  179. 2019-11-14 14:38:32.600149: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007119B7A00 next 135 of size 38912
  180. 2019-11-14 14:38:32.600286: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007119C1200 next 136 of size 4194304
  181. 2019-11-14 14:38:32.600436: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000711DC1200 next 137 of size 38912
  182. 2019-11-14 14:38:32.600578: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000711DCAA00 next 138 of size 2048
  183. 2019-11-14 14:38:32.600720: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000711DCB200 next 139 of size 1048576
  184. 2019-11-14 14:38:32.600900: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000711ECB200 next 140 of size 2048
  185. 2019-11-14 14:38:32.601047: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000711ECBA00 next 141 of size 2048
  186. 2019-11-14 14:38:32.601192: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000711ECC200 next 142 of size 1048576
  187. 2019-11-14 14:38:32.601358: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000711FCC200 next 143 of size 2048
  188. 2019-11-14 14:38:32.601473: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000711FCCA00 next 144 of size 1048576
  189. 2019-11-14 14:38:32.601681: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007120CCA00 next 145 of size 4194304
  190. 2019-11-14 14:38:32.601834: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007124CCA00 next 146 of size 2048
  191. 2019-11-14 14:38:32.601981: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007124CD200 next 147 of size 1048576
  192. 2019-11-14 14:38:32.602152: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007125CD200 next 148 of size 1048576
  193. 2019-11-14 14:38:32.602295: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007126CD200 next 149 of size 4194304
  194. 2019-11-14 14:38:32.602462: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000712ACD200 next 150 of size 40960
  195. 2019-11-14 14:38:32.602608: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000712AD7200 next 151 of size 40960
  196. 2019-11-14 14:38:32.602753: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000712AE1200 next 152 of size 38912
  197. 2019-11-14 14:38:32.602895: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000712AEAA00 next 153 of size 1048576
  198. 2019-11-14 14:38:32.603094: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000712BEAA00 next 154 of size 1048576
  199. 2019-11-14 14:38:32.603312: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000712CEAA00 next 155 of size 1048576
  200. 2019-11-14 14:38:32.603457: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000712DEAA00 next 156 of size 2048
  201. 2019-11-14 14:38:32.603587: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000712DEB200 next 157 of size 4194304
  202. 2019-11-14 14:38:32.603745: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007131EB200 next 158 of size 2048
  203. 2019-11-14 14:38:32.603869: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007131EBA00 next 159 of size 40960
  204. 2019-11-14 14:38:32.603955: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007131F5A00 next 160 of size 38912
  205. 2019-11-14 14:38:32.604106: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007131FF200 next 161 of size 38912
  206. 2019-11-14 14:38:32.604245: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000713208A00 next 162 of size 2048
  207. 2019-11-14 14:38:32.604395: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000713209200 next 163 of size 8192
  208. 2019-11-14 14:38:32.604543: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071320B200 next 164 of size 2048
  209. 2019-11-14 14:38:32.604699: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071320BA00 next 165 of size 4194304
  210. 2019-11-14 14:38:32.604847: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071360BA00 next 166 of size 1048576
  211. 2019-11-14 14:38:32.604994: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071370BA00 next 167 of size 2048
  212. 2019-11-14 14:38:32.605123: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071370C200 next 168 of size 2048
  213. 2019-11-14 14:38:32.605196: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071370CA00 next 169 of size 2048
  214. 2019-11-14 14:38:32.605398: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071370D200 next 170 of size 1048576
  215. 2019-11-14 14:38:32.605562: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071380D200 next 171 of size 40960
  216. 2019-11-14 14:38:32.605700: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000713817200 next 172 of size 38912
  217. 2019-11-14 14:38:32.605868: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000713820A00 next 173 of size 2048
  218. 2019-11-14 14:38:32.605946: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000713821200 next 174 of size 1048576
  219. 2019-11-14 14:38:32.606017: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000713921200 next 175 of size 4194304
  220. 2019-11-14 14:38:32.606220: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000713D21200 next 176 of size 1048576
  221. 2019-11-14 14:38:32.606431: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000713E21200 next 177 of size 2048
  222. 2019-11-14 14:38:32.606561: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000713E21A00 next 178 of size 2048
  223. 2019-11-14 14:38:32.606743: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000713E22200 next 179 of size 2048
  224. 2019-11-14 14:38:32.606882: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000713E22A00 next 180 of size 40960
  225. 2019-11-14 14:38:32.607026: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000713E2CA00 next 181 of size 2048
  226. 2019-11-14 14:38:32.607167: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000713E2D200 next 182 of size 40960
  227. 2019-11-14 14:38:32.607338: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000713E37200 next 183 of size 4194304
  228. 2019-11-14 14:38:32.607435: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000714237200 next 184 of size 40960
  229. 2019-11-14 14:38:32.607589: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000714241200 next 185 of size 38912
  230. 2019-11-14 14:38:32.607738: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071424AA00 next 186 of size 4194304
  231. 2019-11-14 14:38:32.607877: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071464AA00 next 187 of size 2048
  232. 2019-11-14 14:38:32.608029: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071464B200 next 188 of size 8192
  233. 2019-11-14 14:38:32.608173: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071464D200 next 189 of size 8192
  234. 2019-11-14 14:38:32.608273: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071464F200 next 190 of size 1048576
  235. 2019-11-14 14:38:32.608427: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071474F200 next 191 of size 2048
  236. 2019-11-14 14:38:32.608586: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071474FA00 next 192 of size 40960
  237. 2019-11-14 14:38:32.608732: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000714759A00 next 193 of size 38912
  238. 2019-11-14 14:38:32.608855: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000714763200 next 194 of size 38912
  239. 2019-11-14 14:38:32.609016: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071476CA00 next 195 of size 40960
  240. 2019-11-14 14:38:32.609105: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000714776A00 next 196 of size 8192
  241. 2019-11-14 14:38:32.609293: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000714778A00 next 197 of size 256
  242. 2019-11-14 14:38:32.609434: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000714778B00 next 198 of size 2048
  243. 2019-11-14 14:38:32.609579: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000714779300 next 199 of size 8192
  244. 2019-11-14 14:38:32.609732: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071477B300 next 200 of size 4194304
  245. 2019-11-14 14:38:32.609872: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000714B7B300 next 201 of size 4194304
  246. 2019-11-14 14:38:32.610009: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000714F7B300 next 202 of size 2048
  247. 2019-11-14 14:38:32.610136: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000714F7BB00 next 203 of size 1048576
  248. 2019-11-14 14:38:32.610278: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071507BB00 next 204 of size 1048576
  249. 2019-11-14 14:38:32.610429: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071517BB00 next 205 of size 38912
  250. 2019-11-14 14:38:32.610571: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000715185300 next 206 of size 40960
  251. 2019-11-14 14:38:32.610699: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071518F300 next 207 of size 38912
  252. 2019-11-14 14:38:32.610861: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000715198B00 next 208 of size 2048
  253. 2019-11-14 14:38:32.611014: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000715199300 next 209 of size 38912
  254. 2019-11-14 14:38:32.611150: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007151A2B00 next 210 of size 2048
  255. 2019-11-14 14:38:32.611281: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007151A3300 next 211 of size 1048576
  256. 2019-11-14 14:38:32.611478: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007152A3300 next 212 of size 1048576
  257. 2019-11-14 14:38:32.611590: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007153A3300 next 213 of size 2048
  258. 2019-11-14 14:38:32.611735: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007153A3B00 next 214 of size 8192
  259. 2019-11-14 14:38:32.611876: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007153A5B00 next 215 of size 1048576
  260. 2019-11-14 14:38:32.611999: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007154A5B00 next 216 of size 1048576
  261. 2019-11-14 14:38:32.612074: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007155A5B00 next 217 of size 38912
  262. 2019-11-14 14:38:32.612248: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007155AF300 next 218 of size 2048
  263. 2019-11-14 14:38:32.612378: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007155AFB00 next 219 of size 2048
  264. 2019-11-14 14:38:32.612478: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007155B0300 next 220 of size 40960
  265. 2019-11-14 14:38:32.612623: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007155BA300 next 221 of size 4194304
  266. 2019-11-14 14:38:32.612756: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007159BA300 next 222 of size 8192
  267. 2019-11-14 14:38:32.612877: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007159BC300 next 223 of size 1048576
  268. 2019-11-14 14:38:32.613046: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000715ABC300 next 224 of size 38912
  269. 2019-11-14 14:38:32.613184: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000715AC5B00 next 225 of size 4194304
  270. 2019-11-14 14:38:32.613308: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000715EC5B00 next 226 of size 2048
  271. 2019-11-14 14:38:32.613480: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000715EC6300 next 227 of size 4194304
  272. 2019-11-14 14:38:32.613630: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007162C6300 next 228 of size 8192
  273. 2019-11-14 14:38:32.613747: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007162C8300 next 229 of size 4194304
  274. 2019-11-14 14:38:32.613854: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007166C8300 next 230 of size 8192
  275. 2019-11-14 14:38:32.613973: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007166CA300 next 231 of size 4194304
  276. 2019-11-14 14:38:32.614177: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000716ACA300 next 232 of size 2048
  277. 2019-11-14 14:38:32.614333: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000716ACAB00 next 233 of size 1048576
  278. 2019-11-14 14:38:32.614470: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000716BCAB00 next 234 of size 38912
  279. 2019-11-14 14:38:32.614567: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000716BD4300 next 235 of size 40960
  280. 2019-11-14 14:38:32.614791: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000716BDE300 next 236 of size 4194304
  281. 2019-11-14 14:38:32.614924: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000716FDE300 next 237 of size 4194304
  282. 2019-11-14 14:38:32.615081: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007173DE300 next 238 of size 2048
  283. 2019-11-14 14:38:32.615229: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007173DEB00 next 239 of size 1048576
  284. 2019-11-14 14:38:32.615368: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007174DEB00 next 240 of size 4194304
  285. 2019-11-14 14:38:32.615509: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007178DEB00 next 241 of size 2048
  286. 2019-11-14 14:38:32.615645: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007178DF300 next 242 of size 4194304
  287. 2019-11-14 14:38:32.615736: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000717CDF300 next 243 of size 4194304
  288. 2019-11-14 14:38:32.615935: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007180DF300 next 244 of size 8192
  289. 2019-11-14 14:38:32.616165: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007180E1300 next 245 of size 38912
  290. 2019-11-14 14:38:32.616338: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007180EAB00 next 246 of size 2048
  291. 2019-11-14 14:38:32.616474: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007180EB300 next 247 of size 1048576
  292. 2019-11-14 14:38:32.616608: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007181EB300 next 248 of size 2048
  293. 2019-11-14 14:38:32.616705: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007181EBB00 next 249 of size 8192
  294. 2019-11-14 14:38:32.616774: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007181EDB00 next 250 of size 4194304
  295. 2019-11-14 14:38:32.616946: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007185EDB00 next 251 of size 38912
  296. 2019-11-14 14:38:32.617118: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007185F7300 next 252 of size 40960
  297. 2019-11-14 14:38:32.617255: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000718601300 next 253 of size 38912
  298. 2019-11-14 14:38:32.617503: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071860AB00 next 254 of size 2048
  299. 2019-11-14 14:38:32.617677: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071860B300 next 255 of size 1048576
  300. 2019-11-14 14:38:32.617798: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071870B300 next 256 of size 2048
  301. 2019-11-14 14:38:32.617950: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071870BB00 next 257 of size 1048576
  302. 2019-11-14 14:38:32.618167: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071880BB00 next 258 of size 2048
  303. 2019-11-14 14:38:32.618328: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071880C300 next 259 of size 1048576
  304. 2019-11-14 14:38:32.618463: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071890C300 next 260 of size 1048576
  305. 2019-11-14 14:38:32.618630: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000718A0C300 next 261 of size 2048
  306. 2019-11-14 14:38:32.618730: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000718A0CB00 next 262 of size 2048
  307. 2019-11-14 14:38:32.618907: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000718A0D300 next 263 of size 4194304
  308. 2019-11-14 14:38:32.619016: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000718E0D300 next 264 of size 1048576
  309. 2019-11-14 14:38:32.619180: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000718F0D300 next 265 of size 4194304
  310. 2019-11-14 14:38:32.619317: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071930D300 next 266 of size 8192
  311. 2019-11-14 14:38:32.619451: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071930F300 next 267 of size 38912
  312. 2019-11-14 14:38:32.619642: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719318B00 next 268 of size 38912
  313. 2019-11-14 14:38:32.619801: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719322300 next 269 of size 2048
  314. 2019-11-14 14:38:32.619898: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719322B00 next 270 of size 4194304
  315. 2019-11-14 14:38:32.619987: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719722B00 next 271 of size 2048
  316. 2019-11-14 14:38:32.620177: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719723300 next 272 of size 2048
  317. 2019-11-14 14:38:32.620307: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719723B00 next 273 of size 2048
  318. 2019-11-14 14:38:32.620440: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719724300 next 274 of size 1048576
  319. 2019-11-14 14:38:32.620587: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719824300 next 275 of size 2048
  320. 2019-11-14 14:38:32.620746: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719824B00 next 276 of size 2048
  321. 2019-11-14 14:38:32.620866: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719825300 next 277 of size 2048
  322. 2019-11-14 14:38:32.620960: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719825B00 next 278 of size 256
  323. 2019-11-14 14:38:32.621118: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719825C00 next 279 of size 256
  324. 2019-11-14 14:38:32.621189: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719825D00 next 280 of size 256
  325. 2019-11-14 14:38:32.621378: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719825E00 next 281 of size 256
  326. 2019-11-14 14:38:32.621518: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719825F00 next 282 of size 256
  327. 2019-11-14 14:38:32.621670: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719826000 next 283 of size 256
  328. 2019-11-14 14:38:32.621804: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719826100 next 284 of size 256
  329. 2019-11-14 14:38:32.621946: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719826200 next 285 of size 1024
  330. 2019-11-14 14:38:32.622075: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719826600 next 286 of size 256
  331. 2019-11-14 14:38:32.622196: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719826700 next 287 of size 256
  332. 2019-11-14 14:38:32.622388: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719826800 next 288 of size 256
  333. 2019-11-14 14:38:32.622527: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719826900 next 289 of size 256
  334. 2019-11-14 14:38:32.622669: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719826A00 next 290 of size 256
  335. 2019-11-14 14:38:32.622826: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719826B00 next 291 of size 256
  336. 2019-11-14 14:38:32.622923: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719826C00 next 292 of size 256
  337. 2019-11-14 14:38:32.623113: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719826D00 next 293 of size 256
  338. 2019-11-14 14:38:32.623252: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719826E00 next 294 of size 256
  339. 2019-11-14 14:38:32.623384: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719826F00 next 295 of size 256
  340. 2019-11-14 14:38:32.623515: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719827000 next 296 of size 256
  341. 2019-11-14 14:38:32.623647: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719827100 next 297 of size 256
  342. 2019-11-14 14:38:32.623772: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719827200 next 298 of size 256
  343. 2019-11-14 14:38:32.623895: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719827300 next 299 of size 256
  344. 2019-11-14 14:38:32.624093: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719827400 next 300 of size 256
  345. 2019-11-14 14:38:32.624224: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719827500 next 301 of size 256
  346. 2019-11-14 14:38:32.624395: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719827600 next 302 of size 256
  347. 2019-11-14 14:38:32.624569: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719827700 next 303 of size 256
  348. 2019-11-14 14:38:32.624659: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719827800 next 304 of size 256
  349. 2019-11-14 14:38:32.624831: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719827900 next 305 of size 256
  350. 2019-11-14 14:38:32.624939: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719827A00 next 306 of size 256
  351. 2019-11-14 14:38:32.625009: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719827B00 next 307 of size 256
  352. 2019-11-14 14:38:32.625155: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719827C00 next 308 of size 256
  353. 2019-11-14 14:38:32.625305: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719827D00 next 309 of size 256
  354. 2019-11-14 14:38:32.625402: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719827E00 next 310 of size 256
  355. 2019-11-14 14:38:32.625595: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719827F00 next 311 of size 256
  356. 2019-11-14 14:38:32.625742: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719828000 next 312 of size 256
  357. 2019-11-14 14:38:32.625923: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719828100 next 313 of size 256
  358. 2019-11-14 14:38:32.626093: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719828200 next 314 of size 256
  359. 2019-11-14 14:38:32.626272: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719828300 next 315 of size 256
  360. 2019-11-14 14:38:32.626467: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719828400 next 316 of size 256
  361. 2019-11-14 14:38:32.626626: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719828500 next 317 of size 256
  362. 2019-11-14 14:38:32.626801: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719828600 next 318 of size 256
  363. 2019-11-14 14:38:32.626893: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0000000719828700 next 319 of size 634880
  364. 2019-11-14 14:38:32.627136: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007198C3700 next 320 of size 256
  365. 2019-11-14 14:38:32.627299: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007198C3800 next 321 of size 256
  366. 2019-11-14 14:38:32.627434: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007198C3900 next 322 of size 256
  367. 2019-11-14 14:38:32.627583: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007198C3A00 next 323 of size 256
  368. 2019-11-14 14:38:32.627675: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 00000007198C3B00 next 324 of size 256
  369. 2019-11-14 14:38:32.627851: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007198C3C00 next 325 of size 256
  370. 2019-11-14 14:38:32.627992: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007198C3D00 next 326 of size 256
  371. 2019-11-14 14:38:32.628143: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 00000007198C3E00 next 327 of size 256
  372. 2019-11-14 14:38:32.628259: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007198C3F00 next 328 of size 256
  373. 2019-11-14 14:38:32.628435: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007198C4000 next 329 of size 256
  374. 2019-11-14 14:38:32.628581: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 00000007198C4100 next 330 of size 256
  375. 2019-11-14 14:38:32.628671: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007198C4200 next 331 of size 256
  376. 2019-11-14 14:38:32.628810: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007198C4300 next 332 of size 256
  377. 2019-11-14 14:38:32.628926: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007198C4400 next 333 of size 256
  378. 2019-11-14 14:38:32.629086: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007198C4500 next 334 of size 256
  379. 2019-11-14 14:38:32.629226: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 00000007198C4600 next 335 of size 256
  380. 2019-11-14 14:38:32.629369: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007198C4700 next 336 of size 256
  381. 2019-11-14 14:38:32.629519: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007198C4800 next 337 of size 256
  382. 2019-11-14 14:38:32.629666: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 00000007198C4900 next 338 of size 256
  383. 2019-11-14 14:38:32.629806: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007198C4A00 next 339 of size 256
  384. 2019-11-14 14:38:32.629951: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007198C4B00 next 340 of size 256
  385. 2019-11-14 14:38:32.630084: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007198C4C00 next 341 of size 256
  386. 2019-11-14 14:38:32.630178: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007198C4D00 next 342 of size 256
  387. 2019-11-14 14:38:32.630355: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007198C4E00 next 343 of size 256
  388. 2019-11-14 14:38:32.630500: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007198C4F00 next 344 of size 256
  389. 2019-11-14 14:38:32.630613: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007198C5000 next 345 of size 16384
  390. 2019-11-14 14:38:32.630684: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007198C9000 next 349 of size 16384
  391. 2019-11-14 14:38:32.630752: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007198CD000 next 354 of size 16384
  392. 2019-11-14 14:38:32.630938: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007198D1000 next 355 of size 256
  393. 2019-11-14 14:38:32.631110: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007198D1100 next 367 of size 16384
  394. 2019-11-14 14:38:32.631262: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007198D5100 next 370 of size 16384
  395. 2019-11-14 14:38:32.631400: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007198D9100 next 371 of size 16384
  396. 2019-11-14 14:38:32.631543: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007198DD100 next 380 of size 16384
  397. 2019-11-14 14:38:32.631681: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007198E1100 next 381 of size 16384
  398. 2019-11-14 14:38:32.631773: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 00000007198E5100 next 410 of size 256
  399. 2019-11-14 14:38:32.631909: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007198E5200 next 411 of size 256
  400. 2019-11-14 14:38:32.632044: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 00000007198E5300 next 400 of size 512
  401. 2019-11-14 14:38:32.632227: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007198E5500 next 401 of size 256
  402. 2019-11-14 14:38:32.632364: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 00000007198E5600 next 394 of size 15104
  403. 2019-11-14 14:38:32.632493: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007198E9100 next 395 of size 16384
  404. 2019-11-14 14:38:32.632627: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007198ED100 next 404 of size 16384
  405. 2019-11-14 14:38:32.632793: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 00000007198F1100 next 407 of size 16384
  406. 2019-11-14 14:38:32.632963: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007198F5100 next 408 of size 16384
  407. 2019-11-14 14:38:32.633111: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 00000007198F9100 next 351 of size 8216832
  408. 2019-11-14 14:38:32.633248: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071A0CF200 next 352 of size 16384
  409. 2019-11-14 14:38:32.633382: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071A0D3200 next 347 of size 67108864
  410. 2019-11-14 14:38:32.633530: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071E0D3200 next 348 of size 8388608
  411. 2019-11-14 14:38:32.633678: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071E8D3200 next 346 of size 8388608
  412. 2019-11-14 14:38:32.633797: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071F0D3200 next 353 of size 8388608
  413. 2019-11-14 14:38:32.633920: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000071F8D3200 next 350 of size 8388608
  414. 2019-11-14 14:38:32.634060: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007200D3200 next 356 of size 8388608
  415. 2019-11-14 14:38:32.634204: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007208D3200 next 357 of size 8388608
  416. 2019-11-14 14:38:32.634327: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007210D3200 next 358 of size 8388608
  417. 2019-11-14 14:38:32.634469: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007218D3200 next 359 of size 8388608
  418. 2019-11-14 14:38:32.634623: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007220D3200 next 360 of size 536870912
  419. 2019-11-14 14:38:32.634792: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007420D3200 next 361 of size 536870912
  420. 2019-11-14 14:38:32.634933: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007620D3200 next 362 of size 536870912
  421. 2019-11-14 14:38:32.635123: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007820D3200 next 363 of size 8388608
  422. 2019-11-14 14:38:32.635319: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007828D3200 next 366 of size 8388608
  423. 2019-11-14 14:38:32.635457: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007830D3200 next 368 of size 8388608
  424. 2019-11-14 14:38:32.635595: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007838D3200 next 369 of size 8388608
  425. 2019-11-14 14:38:32.635724: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007840D3200 next 372 of size 8388608
  426. 2019-11-14 14:38:32.635888: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007848D3200 next 373 of size 33554432
  427. 2019-11-14 14:38:32.636028: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007868D3200 next 374 of size 33554432
  428. 2019-11-14 14:38:32.636250: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007888D3200 next 375 of size 33554432
  429. 2019-11-14 14:38:32.636394: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000078A8D3200 next 376 of size 8388608
  430. 2019-11-14 14:38:32.636512: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000078B0D3200 next 377 of size 8388608
  431. 2019-11-14 14:38:32.636699: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000078B8D3200 next 378 of size 8388608
  432. 2019-11-14 14:38:32.636831: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000078C0D3200 next 379 of size 8388608
  433. 2019-11-14 14:38:32.637001: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000078C8D3200 next 382 of size 8388608
  434. 2019-11-14 14:38:32.637126: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000078D0D3200 next 385 of size 8388608
  435. 2019-11-14 14:38:32.637262: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000078D8D3200 next 386 of size 8388608
  436. 2019-11-14 14:38:32.637398: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000078E0D3200 next 384 of size 8388608
  437. 2019-11-14 14:38:32.637534: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000078E8D3200 next 390 of size 8388608
  438. 2019-11-14 14:38:32.637670: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000078F0D3200 next 391 of size 8388608
  439. 2019-11-14 14:38:32.637810: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000078F8D3200 next 389 of size 8388608
  440. 2019-11-14 14:38:32.637895: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007900D3200 next 392 of size 8388608
  441. 2019-11-14 14:38:32.637967: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007908D3200 next 393 of size 8388608
  442. 2019-11-14 14:38:32.638202: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007910D3200 next 396 of size 8388608
  443. 2019-11-14 14:38:32.638316: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007918D3200 next 402 of size 33554432
  444. 2019-11-14 14:38:32.638475: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007938D3200 next 397 of size 33554432
  445. 2019-11-14 14:38:32.638603: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007958D3200 next 399 of size 33554432
  446. 2019-11-14 14:38:32.638800: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007978D3200 next 398 of size 8388608
  447. 2019-11-14 14:38:32.638938: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007980D3200 next 403 of size 8388608
  448. 2019-11-14 14:38:32.639119: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007988D3200 next 405 of size 8388608
  449. 2019-11-14 14:38:32.639210: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007990D3200 next 406 of size 8388608
  450. 2019-11-14 14:38:32.639407: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007998D3200 next 409 of size 8388608
  451. 2019-11-14 14:38:32.639562: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000079A0D3200 next 412 of size 8388608
  452. 2019-11-14 14:38:32.639713: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000079A8D3200 next 413 of size 8388608
  453. 2019-11-14 14:38:32.639848: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000079B0D3200 next 414 of size 8388608
  454. 2019-11-14 14:38:32.639967: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 000000079B8D3200 next 419 of size 8388608
  455. 2019-11-14 14:38:32.640181: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 000000079C0D3200 next 420 of size 8388608
  456. 2019-11-14 14:38:32.640333: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 000000079C8D3200 next 364 of size 100663296
  457. 2019-11-14 14:38:32.640473: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007A28D3200 next 365 of size 8388608
  458. 2019-11-14 14:38:32.640614: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007A30D3200 next 383 of size 536870912
  459. 2019-11-14 14:38:32.640776: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007C30D3200 next 387 of size 536870912
  460. 2019-11-14 14:38:32.640940: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000007E30D3200 next 388 of size 536870912
  461. 2019-11-14 14:38:32.641111: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000008030D3200 next 415 of size 536870912
  462. 2019-11-14 14:38:32.641210: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000008230D3200 next 416 of size 536870912
  463. 2019-11-14 14:38:32.641405: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000008430D3200 next 417 of size 536870912
  464. 2019-11-14 14:38:32.641528: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 00000008630D3200 next 418 of size 536870912
  465. 2019-11-14 14:38:32.641669: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 00000008830D3200 next 18446744073709551615 of size 315536384
  466. 2019-11-14 14:38:32.641825: I tensorflow/core/common_runtime/bfc_allocator.cc:914] Summary of in-use Chunks by size:
  467. 2019-11-14 14:38:32.641955: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 69 Chunks of size 256 totalling 17.3KiB
  468. 2019-11-14 14:38:32.642105: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 1024 totalling 1.0KiB
  469. 2019-11-14 14:38:32.642236: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 1280 totalling 1.3KiB
  470. 2019-11-14 14:38:32.642363: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 96 Chunks of size 2048 totalling 192.0KiB
  471. 2019-11-14 14:38:32.642494: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 18 Chunks of size 8192 totalling 144.0KiB
  472. 2019-11-14 14:38:32.642635: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 12 Chunks of size 16384 totalling 192.0KiB
  473. 2019-11-14 14:38:32.642775: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 30 Chunks of size 38912 totalling 1.11MiB
  474. 2019-11-14 14:38:32.642908: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 18 Chunks of size 40960 totalling 720.0KiB
  475. 2019-11-14 14:38:32.643041: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 634880 totalling 620.0KiB
  476. 2019-11-14 14:38:32.643173: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 72 Chunks of size 1048576 totalling 72.00MiB
  477. 2019-11-14 14:38:32.643307: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 36 Chunks of size 4194304 totalling 144.00MiB
  478. 2019-11-14 14:38:32.643448: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 37 Chunks of size 8388608 totalling 296.00MiB
  479. 2019-11-14 14:38:32.643590: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 6 Chunks of size 33554432 totalling 192.00MiB
  480. 2019-11-14 14:38:32.643706: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 67108864 totalling 64.00MiB
  481. 2019-11-14 14:38:32.643831: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 10 Chunks of size 536870912 totalling 5.00GiB
  482. 2019-11-14 14:38:32.643932: I tensorflow/core/common_runtime/bfc_allocator.cc:921] Sum Total of in-use chunks: 5.75GiB
  483. 2019-11-14 14:38:32.644132: I tensorflow/core/common_runtime/bfc_allocator.cc:923] total_region_allocated_bytes_: 6609954304 memory_limit_: 8160437862 available bytes: 1550483558 curr_region_allocation_bytes_: 16320876032
  484. 2019-11-14 14:38:32.644377: I tensorflow/core/common_runtime/bfc_allocator.cc:929] Stats:
  485. Limit: 8160437862
  486. InUse: 6177115648
  487. MaxInUse: 6185504256
  488. NumAllocs: 611
  489. MaxAllocSize: 603979776
  490.  
  491. 2019-11-14 14:38:32.644686: W tensorflow/core/common_runtime/bfc_allocator.cc:424] *************************************_**********************************************************____
  492. 2019-11-14 14:38:32.644868: W tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES failed at pad_op.cc:122 : Resource exhausted: OOM when allocating tensor with shape[16777216,9] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
  493. Traceback (most recent call last):
  494. File "c:\python\lib\site-packages\tensorflow_core\python\client\session.py", line 1365, in _do_call
  495. return fn(*args)
  496. File "c:\python\lib\site-packages\tensorflow_core\python\client\session.py", line 1350, in _run_fn
  497. target_list, run_metadata)
  498. File "c:\python\lib\site-packages\tensorflow_core\python\client\session.py", line 1443, in _call_tf_sessionrun
  499. run_metadata)
  500. tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
  501. (0) Resource exhausted: OOM when allocating tensor with shape[16777216,9] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
  502. [[{{node transformer/parallel_0_4/transformer/transformer/body/decoder/layer_2/self_attention/multihead_attention/dot_product_attention/Pad}}]]
  503. Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!