API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Apr 12th, 2018
444
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 157.31 KB | None | 0 0
raw download clone embed print report
  1. C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\h5py\__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  2. from ._conv import register_converters as _register_converters
  3. Using TensorFlow backend.
  4. Model A Directory: C:\OpenFaceSwap\alignA
  5. Model B Directory: C:\OpenFaceSwap\alignB
  6. Training data directory: C:\OpenFaceSwap\model
  7. Loading data, this may take a while...
  8. Using live preview
  9. Loading Model from Model_LowMem plugin...
  10. WARNING:tensorflow:From C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\keras\backend\tensorflow_backend.py:1264: calling reduce_prod (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
  11. Instructions for updating:
  12. keep_dims is deprecated, use keepdims instead
  13. WARNING:tensorflow:From C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\keras\backend\tensorflow_backend.py:1349: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
  14. Instructions for updating:
  15. keep_dims is deprecated, use keepdims instead
  16. Failed loading existing training data.
  17. Unable to open file (unable to open file: name = 'C:\OpenFaceSwap\model\encoder.h5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
  18. Loading Trainer from Model_LowMem plugin...
  19. Starting. Press "Enter" to stop training and save model
  20. 2018-04-12 12:33:50.445670: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1344] Found device 0 with properties:
  21. name: GeForce GTX 1060 3GB major: 6 minor: 1 memoryClockRate(GHz): 1.797
  22. pciBusID: 0000:01:00.0
  23. totalMemory: 3.00GiB freeMemory: 2.33GiB
  24. 2018-04-12 12:33:50.451623: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1423] Adding visible gpu devices: 0
  25. 2018-04-12 12:33:51.138831: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
  26. 2018-04-12 12:33:51.142522: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:917] 0
  27. 2018-04-12 12:33:51.144819: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:930] 0: N
  28. 2018-04-12 12:33:51.147693: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2044 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 3GB, pci bus id: 0000:01:00.0, compute capability: 6.1)
  29. 2018-04-12 12:33:54.339590: W T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.14GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
  30. 2018-04-12 12:33:54.353425: W T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.07GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
  31. 2018-04-12 12:33:54.365032: W T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.04GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
  32. 2018-04-12 12:33:54.404519: W T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.04GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
  33. 2018-04-12 12:33:54.422120: W T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.03GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
  34. 2018-04-12 12:33:54.433277: W T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.07GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
  35. 2018-04-12 12:33:54.454873: W T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.06GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
  36. 2018-04-12 12:33:54.468705: W T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.14GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
  37. 2018-04-12 12:33:54.477042: W T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.13GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
  38. 2018-04-12 12:33:54.491854: W T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.13GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
  39. 2018-04-12 12:34:07.320671: W T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 72.00MiB. Current allocation summary follows.
  40. 2018-04-12 12:34:07.324766: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (256):
  41. Total Chunks: 59, Chunks in use: 55. 14.8KiB allocated for chunks. 13.8KiB in use in bin. 300B client-requested in use in bin.
  42. 2018-04-12 12:34:07.330622: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (512):
  43. Total Chunks: 10, Chunks in use: 9. 5.0KiB allocated for chunks. 4.5KiB in use in bin. 4.5KiB client-requested in use in bin.
  44. 2018-04-12 12:34:07.336119: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (1024): Total Chunks: 17, Chunks in use: 16. 17.5KiB allocated for chunks. 16.3KiB in use in bin. 16.0KiB client-requested in use in bin.
  45. 2018-04-12 12:34:07.342008: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (2048): Total Chunks: 23, Chunks in use: 23. 46.0KiB allocated for chunks. 46.0KiB in use in bin. 46.0KiB client-requested in use in bin.
  46. 2018-04-12 12:34:07.349516: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (4096): Total Chunks: 9, Chunks in use: 8. 36.3KiB allocated for chunks. 32.0KiB in use in bin. 32.0KiB client-requested in use in bin.
  47. 2018-04-12 12:34:07.354982: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (8192): Total Chunks: 11, Chunks in use: 10. 91.5KiB allocated for chunks. 80.0KiB in use in bin. 80.0KiB client-requested in use in bin.
  48. 2018-04-12 12:34:07.361512: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (16384): Total Chunks: 8, Chunks in use: 8. 151.5KiB allocated for chunks. 151.5KiB in use in bin. 150.0KiB client-requested in use in bin.
  49. 2018-04-12 12:34:07.368817: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (32768): Total Chunks: 12, Chunks in use: 10. 485.5KiB allocated for chunks. 397.3KiB in use in bin. 385.5KiB client-requested in use in bin.
  50. 2018-04-12 12:34:07.374237: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (65536): Total Chunks: 13, Chunks in use: 11. 843.8KiB allocated for chunks. 715.8KiB in use in bin. 704.0KiB client-requested in use in bin.
  51. 2018-04-12 12:34:07.380500: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (131072): Total Chunks: 4, Chunks in use: 4. 574.8KiB allocated for chunks. 574.8KiB in use in bin. 512.0KiB client-requested in use in bin.
  52. 2018-04-12 12:34:07.387291: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (262144): Total Chunks: 3, Chunks in use: 3. 797.3KiB allocated for chunks. 797.3KiB in use in bin. 768.0KiB client-requested in use in bin.
  53. 2018-04-12 12:34:07.393290: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (524288): Total Chunks: 3, Chunks in use: 3. 1.57MiB allocated for chunks. 1.57MiB in use in bin. 1.50MiB client-requested in use in bin.
  54. 2018-04-12 12:34:07.399971: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (1048576): Total Chunks: 7, Chunks in use: 7. 7.88MiB allocated for chunks. 7.88MiB in use in bin. 7.88MiB client-requested in use in bin.
  55. 2018-04-12 12:34:07.406346: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (2097152): Total Chunks: 10, Chunks in use: 9. 30.58MiB allocated for chunks. 27.00MiB in use in bin. 26.13MiB client-requested in use in bin.
  56. 2018-04-12 12:34:07.416768: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (4194304): Total Chunks: 9, Chunks in use: 8. 40.50MiB allocated for chunks. 36.00MiB in use in bin. 34.63MiB client-requested in use in bin.
  57. 2018-04-12 12:34:07.423822: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (8388608): Total Chunks: 9, Chunks in use: 9. 108.00MiB allocated for chunks. 108.00MiB in use in bin. 104.50MiB client-requested in use in bin.
  58. 2018-04-12 12:34:07.429292: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (16777216): Total Chunks: 9, Chunks in use: 8. 162.00MiB allocated for chunks. 144.00MiB in use in bin. 138.50MiB client-requested in use in bin.
  59. 2018-04-12 12:34:07.436206: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (33554432): Total Chunks: 10, Chunks in use: 10. 320.00MiB allocated for chunks. 320.00MiB in use in bin. 306.00MiB client-requested in use in bin.
  60. 2018-04-12 12:34:07.442918: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (67108864): Total Chunks: 18, Chunks in use: 18. 1.20GiB allocated for chunks. 1.20GiB in use in bin. 1.20GiB client-requested in use in bin.
  61. 2018-04-12 12:34:07.449091: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (134217728): Total Chunks: 1, Chunks in use: 1. 138.74MiB allocated for chunks. 138.74MiB in use in bin. 72.00MiB client-requested in use in bin.
  62. 2018-04-12 12:34:07.455301: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (268435456): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
  63. 2018-04-12 12:34:07.461210: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:646] Bin for 72.00MiB was 64.00MiB, Chunk State:
  64. 2018-04-12 12:34:07.464541: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706600000 of size 1280
  65. 2018-04-12 12:34:07.469631: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706600500 of size 256
  66. 2018-04-12 12:34:07.473097: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706600600 of size 256
  67. 2018-04-12 12:34:07.477458: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706600700 of size 512
  68. 2018-04-12 12:34:07.480730: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706600900 of size 256
  69. 2018-04-12 12:34:07.485044: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706600A00 of size 256
  70. 2018-04-12 12:34:07.489707: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706600B00 of size 1024
  71. 2018-04-12 12:34:07.492856: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706600F00 of size 256
  72. 2018-04-12 12:34:07.496406: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706601000 of size 256
  73. 2018-04-12 12:34:07.500100: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706601100 of size 2048
  74. 2018-04-12 12:34:07.503195: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706601900 of size 256
  75. 2018-04-12 12:34:07.508171: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706601A00 of size 256
  76. 2018-04-12 12:34:07.511175: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706601B00 of size 256
  77. 2018-04-12 12:34:07.515984: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706601C00 of size 256
  78. 2018-04-12 12:34:07.519609: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706601D00 of size 65536
  79. 2018-04-12 12:34:07.523323: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706611D00 of size 256
  80. 2018-04-12 12:34:07.526768: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706611E00 of size 256
  81. 2018-04-12 12:34:07.529867: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706611F00 of size 8192
  82. 2018-04-12 12:34:07.534200: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706613F00 of size 256
  83. 2018-04-12 12:34:07.537217: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706614000 of size 256
  84. 2018-04-12 12:34:07.540344: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706614100 of size 4096
  85. 2018-04-12 12:34:07.544269: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706615100 of size 256
  86. 2018-04-12 12:34:07.547275: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706615200 of size 256
  87. 2018-04-12 12:34:07.552225: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706615300 of size 256
  88. 2018-04-12 12:34:07.555309: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706615400 of size 256
  89. 2018-04-12 12:34:07.559113: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706615500 of size 256
  90. 2018-04-12 12:34:07.562226: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706615600 of size 256
  91. 2018-04-12 12:34:07.566746: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706615700 of size 256
  92. 2018-04-12 12:34:07.570280: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706615800 of size 256
  93. 2018-04-12 12:34:07.573671: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706615900 of size 256
  94. 2018-04-12 12:34:07.577557: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706615A00 of size 256
  95. 2018-04-12 12:34:07.581184: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706615B00 of size 256
  96. 2018-04-12 12:34:07.584881: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706615C00 of size 256
  97. 2018-04-12 12:34:07.588588: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706615D00 of size 38400
  98. 2018-04-12 12:34:07.591674: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000070661F300 of size 3276800
  99. 2018-04-12 12:34:07.594802: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000070693F300 of size 13107200
  100. 2018-04-12 12:34:07.598761: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007075BF300 of size 67108864
  101. 2018-04-12 12:34:07.603631: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000070B5BF300 of size 33554432
  102. 2018-04-12 12:34:07.609132: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000070D5BF300 of size 75497472
  103. 2018-04-12 12:34:07.612893: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000711DBF300 of size 18874368
  104. 2018-04-12 12:34:07.617540: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000712FBF300 of size 4718592
  105. 2018-04-12 12:34:07.621126: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071343F300 of size 1179648
  106. 2018-04-12 12:34:07.624876: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071355F300 of size 19200
  107. 2018-04-12 12:34:07.628118: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000713563E00 of size 19200
  108. 2018-04-12 12:34:07.631361: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000713568900 of size 19200
  109. 2018-04-12 12:34:07.636063: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071356D400 of size 512
  110. 2018-04-12 12:34:07.639411: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071356D600 of size 1179648
  111. 2018-04-12 12:34:07.643151: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071368D600 of size 2097152
  112. 2018-04-12 12:34:07.646480: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071388D600 of size 1024
  113. 2018-04-12 12:34:07.650282: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071388DA00 of size 4718592
  114. 2018-04-12 12:34:07.654322: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000713D0DA00 of size 8388608
  115. 2018-04-12 12:34:07.657373: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071450DA00 of size 2048
  116. 2018-04-12 12:34:07.661227: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071450E200 of size 33554432
  117. 2018-04-12 12:34:07.664640: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071650E200 of size 33554432
  118. 2018-04-12 12:34:07.668776: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071850E200 of size 2048
  119. 2018-04-12 12:34:07.672492: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071850EA00 of size 33554432
  120. 2018-04-12 12:34:07.676078: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071A50EA00 of size 65536
  121. 2018-04-12 12:34:07.679877: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071A51EA00 of size 75497472
  122. 2018-04-12 12:34:07.684059: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071ED1EA00 of size 8192
  123. 2018-04-12 12:34:07.687341: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071ED20A00 of size 18874368
  124. 2018-04-12 12:34:07.691212: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071FF20A00 of size 18874368
  125. 2018-04-12 12:34:07.694264: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721120A00 of size 4096
  126. 2018-04-12 12:34:07.698492: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721121A00 of size 4718592
  127. 2018-04-12 12:34:07.702216: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007215A1A00 of size 4718592
  128. 2018-04-12 12:34:07.705390: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721A21A00 of size 2048
  129. 2018-04-12 12:34:07.709091: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721A22200 of size 2048
  130. 2018-04-12 12:34:07.712353: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721A22A00 of size 1179648
  131. 2018-04-12 12:34:07.716542: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B42A00 of size 38400
  132. 2018-04-12 12:34:07.719553: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B4C000 of size 65536
  133. 2018-04-12 12:34:07.722776: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B5C000 of size 256
  134. 2018-04-12 12:34:07.726835: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B5C100 of size 38400
  135. 2018-04-12 12:34:07.730299: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B65700 of size 512
  136. 2018-04-12 12:34:07.734493: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B65900 of size 1024
  137. 2018-04-12 12:34:07.737636: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B65D00 of size 2048
  138. 2018-04-12 12:34:07.740603: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B66500 of size 2048
  139. 2018-04-12 12:34:07.744717: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B66D00 of size 65536
  140. 2018-04-12 12:34:07.747866: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B76D00 of size 8192
  141. 2018-04-12 12:34:07.751382: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B78D00 of size 4096
  142. 2018-04-12 12:34:07.755898: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B79D00 of size 2048
  143. 2018-04-12 12:34:07.759392: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B7A500 of size 4096
  144. 2018-04-12 12:34:07.763244: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B7B500 of size 1024
  145. 2018-04-12 12:34:07.767279: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B7B900 of size 19200
  146. 2018-04-12 12:34:07.772442: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B80400 of size 19200
  147. 2018-04-12 12:34:07.776242: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B84F00 of size 256
  148. 2018-04-12 12:34:07.781378: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B85000 of size 38400
  149. 2018-04-12 12:34:07.785624: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B8E600 of size 512
  150. 2018-04-12 12:34:07.790136: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B8E800 of size 1024
  151. 2018-04-12 12:34:07.793425: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B8EC00 of size 2048
  152. 2018-04-12 12:34:07.796496: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B8F400 of size 2048
  153. 2018-04-12 12:34:07.801136: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B8FC00 of size 2048
  154. 2018-04-12 12:34:07.805049: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B90400 of size 256
  155. 2018-04-12 12:34:07.809669: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B90500 of size 65536
  156. 2018-04-12 12:34:07.813546: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BA0500 of size 1024
  157. 2018-04-12 12:34:07.818030: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BA0900 of size 8192
  158. 2018-04-12 12:34:07.821312: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BA2900 of size 256
  159. 2018-04-12 12:34:07.824704: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BA2A00 of size 256
  160. 2018-04-12 12:34:07.828735: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BA2B00 of size 256
  161. 2018-04-12 12:34:07.831922: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BA2C00 of size 256
  162. 2018-04-12 12:34:07.836166: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BA2D00 of size 256
  163. 2018-04-12 12:34:07.839252: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BA2E00 of size 256
  164. 2018-04-12 12:34:07.842478: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BA2F00 of size 256
  165. 2018-04-12 12:34:07.846227: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 0000000721BA3000 of size 49408
  166. 2018-04-12 12:34:07.850020: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BAF100 of size 8192
  167. 2018-04-12 12:34:07.854254: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 0000000721BB1100 of size 40960
  168. 2018-04-12 12:34:07.857738: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BBB100 of size 256
  169. 2018-04-12 12:34:07.861157: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 0000000721BBB200 of size 512
  170. 2018-04-12 12:34:07.866037: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BBB400 of size 38400
  171. 2018-04-12 12:34:07.870103: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BC4A00 of size 49152
  172. 2018-04-12 12:34:07.874475: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BD0A00 of size 256
  173. 2018-04-12 12:34:07.878349: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BD0B00 of size 597760
  174. 2018-04-12 12:34:07.884553: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C62A00 of size 1024
  175. 2018-04-12 12:34:07.888159: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C62E00 of size 256
  176. 2018-04-12 12:34:07.892181: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C62F00 of size 256
  177. 2018-04-12 12:34:07.895739: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C63000 of size 256
  178. 2018-04-12 12:34:07.900960: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C63100 of size 256
  179. 2018-04-12 12:34:07.904346: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C63200 of size 256
  180. 2018-04-12 12:34:07.907660: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C63300 of size 256
  181. 2018-04-12 12:34:07.911959: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C63400 of size 256
  182. 2018-04-12 12:34:07.915423: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C63500 of size 512
  183. 2018-04-12 12:34:07.919284: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C63700 of size 1024
  184. 2018-04-12 12:34:07.923068: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C63B00 of size 2048
  185. 2018-04-12 12:34:07.926528: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C64300 of size 8192
  186. 2018-04-12 12:34:07.930587: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C66300 of size 4096
  187. 2018-04-12 12:34:07.934563: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C67300 of size 20736
  188. 2018-04-12 12:34:07.939348: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C6C400 of size 256
  189. 2018-04-12 12:34:07.942843: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C6C500 of size 2048
  190. 2018-04-12 12:34:07.947464: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C6CD00 of size 4096
  191. 2018-04-12 12:34:07.952065: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C6DD00 of size 2048
  192. 2018-04-12 12:34:07.956174: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C6E500 of size 2048
  193. 2018-04-12 12:34:07.959563: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C6ED00 of size 256
  194. 2018-04-12 12:34:07.963345: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C6EE00 of size 1024
  195. 2018-04-12 12:34:07.967870: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C6F200 of size 65536
  196. 2018-04-12 12:34:07.973621: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C7F200 of size 256
  197. 2018-04-12 12:34:07.977123: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C7F300 of size 256
  198. 2018-04-12 12:34:07.980579: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C7F400 of size 256
  199. 2018-04-12 12:34:07.985623: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C7F500 of size 256
  200. 2018-04-12 12:34:07.989178: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C7F600 of size 256
  201. 2018-04-12 12:34:07.993178: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C7F700 of size 256
  202. 2018-04-12 12:34:07.996533: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C7F800 of size 38400
  203. 2018-04-12 12:34:08.001766: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C88E00 of size 512
  204. 2018-04-12 12:34:08.005285: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C89000 of size 1024
  205. 2018-04-12 12:34:08.008830: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C89400 of size 3276800
  206. 2018-04-12 12:34:08.013176: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721FA9400 of size 1024
  207. 2018-04-12 12:34:08.017135: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721FA9800 of size 13107200
  208. 2018-04-12 12:34:08.021916: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000722C29800 of size 13107200
  209. 2018-04-12 12:34:08.025197: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007238A9800 of size 2048
  210. 2018-04-12 12:34:08.029068: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007238AA000 of size 8192
  211. 2018-04-12 12:34:08.033004: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007238AC000 of size 67108864
  212. 2018-04-12 12:34:08.036462: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007278AC000 of size 67108864
  213. 2018-04-12 12:34:08.041173: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000072B8AC000 of size 2048
  214. 2018-04-12 12:34:08.045234: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000072B8AC800 of size 33554432
  215. 2018-04-12 12:34:08.050742: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000072D8AC800 of size 33554432
  216. 2018-04-12 12:34:08.053986: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000072F8AC800 of size 65536
  217. 2018-04-12 12:34:08.058053: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000072F8BC800 of size 75497472
  218. 2018-04-12 12:34:08.061563: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007340BC800 of size 75497472
  219. 2018-04-12 12:34:08.066501: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007388BC800 of size 8192
  220. 2018-04-12 12:34:08.070139: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007388BE800 of size 18874368
  221. 2018-04-12 12:34:08.073413: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000739ABE800 of size 18874368
  222. 2018-04-12 12:34:08.079246: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073ACBE800 of size 4096
  223. 2018-04-12 12:34:08.083246: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073ACBF800 of size 4718592
  224. 2018-04-12 12:34:08.087284: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073B13F800 of size 2048
  225. 2018-04-12 12:34:08.090458: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073B140000 of size 1179648
  226. 2018-04-12 12:34:08.094539: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073B260000 of size 1179648
  227. 2018-04-12 12:34:08.098017: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073B380000 of size 1024
  228. 2018-04-12 12:34:08.101420: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073B380400 of size 4096
  229. 2018-04-12 12:34:08.105308: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073B381400 of size 19200
  230. 2018-04-12 12:34:08.108298: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073B385F00 of size 256
  231. 2018-04-12 12:34:08.112168: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073B386000 of size 19200
  232. 2018-04-12 12:34:08.115362: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073B38AB00 of size 38400
  233. 2018-04-12 12:34:08.118649: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073B394100 of size 512
  234. 2018-04-12 12:34:08.122562: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073B394300 of size 4718592
  235. 2018-04-12 12:34:08.125473: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073B814300 of size 3276800
  236. 2018-04-12 12:34:08.128697: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073BB34300 of size 1024
  237. 2018-04-12 12:34:08.133049: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073BB34700 of size 38400
  238. 2018-04-12 12:34:08.136628: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073BB3DD00 of size 3276800
  239. 2018-04-12 12:34:08.140207: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073BE5DD00 of size 13107200
  240. 2018-04-12 12:34:08.143330: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073CADDD00 of size 67108864
  241. 2018-04-12 12:34:08.146706: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000740ADDD00 of size 75497472
  242. 2018-04-12 12:34:08.151069: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007452DDD00 of size 75497472
  243. 2018-04-12 12:34:08.155084: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000749ADDD00 of size 18874368
  244. 2018-04-12 12:34:08.158699: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000074ACDDD00 of size 3276800
  245. 2018-04-12 12:34:08.162047: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000074AFFDD00 of size 13107200
  246. 2018-04-12 12:34:08.165271: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000074BC7DD00 of size 67108864
  247. 2018-04-12 12:34:08.169893: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000074FC7DD00 of size 33554432
  248. 2018-04-12 12:34:08.173127: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000751C7DD00 of size 75497472
  249. 2018-04-12 12:34:08.177043: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000075647DD00 of size 18874368
  250. 2018-04-12 12:34:08.180432: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000075767DD00 of size 4718592
  251. 2018-04-12 12:34:08.185340: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000757AFDD00 of size 1179648
  252. 2018-04-12 12:34:08.188555: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000757C1DD00 of size 4718592
  253. 2018-04-12 12:34:08.191893: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000075809DD00 of size 3276800
  254. 2018-04-12 12:34:08.195572: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007583BDD00 of size 13107200
  255. 2018-04-12 12:34:08.198785: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000075903DD00 of size 67108864
  256. 2018-04-12 12:34:08.202281: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000075D03DD00 of size 1179648
  257. 2018-04-12 12:34:08.206115: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000075D15DD00 of size 33554432
  258. 2018-04-12 12:34:08.209411: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000075F15DD00 of size 75497472
  259. 2018-04-12 12:34:08.213379: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000076395DD00 of size 18874368
  260. 2018-04-12 12:34:08.218205: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 0000000764B5DD00 of size 256
  261. 2018-04-12 12:34:08.222496: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000764B5DE00 of size 256
  262. 2018-04-12 12:34:08.225743: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000764B5DF00 of size 50432
  263. 2018-04-12 12:34:08.229086: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000764B6A400 of size 512
  264. 2018-04-12 12:34:08.233495: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000764B6A600 of size 524288
  265. 2018-04-12 12:34:08.236609: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 0000000764BEA600 of size 256
  266. 2018-04-12 12:34:08.240832: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000764BEA700 of size 3276800
  267. 2018-04-12 12:34:08.243967: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000764F0A700 of size 1024
  268. 2018-04-12 12:34:08.247144: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 0000000764F0AB00 of size 256
  269. 2018-04-12 12:34:08.251779: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000764F0AC00 of size 13107200
  270. 2018-04-12 12:34:08.254891: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000765B8AC00 of size 2048
  271. 2018-04-12 12:34:08.257931: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 0000000765B8B400 of size 4352
  272. 2018-04-12 12:34:08.262079: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000765B8C500 of size 2048
  273. 2018-04-12 12:34:08.265369: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000765B8CD00 of size 67108864
  274. 2018-04-12 12:34:08.270063: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000769B8CD00 of size 33554432
  275. 2018-04-12 12:34:08.274269: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000076BB8CD00 of size 65536
  276. 2018-04-12 12:34:08.278191: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000076BB9CD00 of size 75497472
  277. 2018-04-12 12:34:08.281762: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000077039CD00 of size 524288
  278. 2018-04-12 12:34:08.285489: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 000000077041CD00 of size 18874368
  279. 2018-04-12 12:34:08.289365: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000077161CD00 of size 262144
  280. 2018-04-12 12:34:08.292514: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000077165CD00 of size 1024
  281. 2018-04-12 12:34:08.295733: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000077165D100 of size 2048
  282. 2018-04-12 12:34:08.300119: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000077165D900 of size 65536
  283. 2018-04-12 12:34:08.303325: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 000000077166D900 of size 1280
  284. 2018-04-12 12:34:08.308113: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000077166DE00 of size 195328
  285. 2018-04-12 12:34:08.311053: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 000000077169D900 of size 256
  286. 2018-04-12 12:34:08.314771: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000077169DA00 of size 292096
  287. 2018-04-12 12:34:08.318202: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 00000007716E4F00 of size 4718592
  288. 2018-04-12 12:34:08.321613: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000771B64F00 of size 3276800
  289. 2018-04-12 12:34:08.325560: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000771E84F00 of size 262144
  290. 2018-04-12 12:34:08.328772: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000771EC4F00 of size 2048
  291. 2018-04-12 12:34:08.332320: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000771EC5700 of size 8192
  292. 2018-04-12 12:34:08.336278: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 0000000771EC7700 of size 11776
  293. 2018-04-12 12:34:08.339278: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000771ECA500 of size 131072
  294. 2018-04-12 12:34:08.343172: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000771EEA500 of size 2048
  295. 2018-04-12 12:34:08.346710: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 0000000771EEAD00 of size 65536
  296. 2018-04-12 12:34:08.350396: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000771EFAD00 of size 77568
  297. 2018-04-12 12:34:08.354451: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000771F0DC00 of size 8192
  298. 2018-04-12 12:34:08.357793: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000771F0FC00 of size 131072
  299. 2018-04-12 12:34:08.361705: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000771F2FC00 of size 65536
  300. 2018-04-12 12:34:08.365147: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 0000000771F3FC00 of size 65536
  301. 2018-04-12 12:34:08.368929: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000771F4FC00 of size 131072
  302. 2018-04-12 12:34:08.372833: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 0000000771F6FC00 of size 3756800
  303. 2018-04-12 12:34:08.376135: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000772304F00 of size 512
  304. 2018-04-12 12:34:08.379820: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000772305100 of size 13107200
  305. 2018-04-12 12:34:08.383496: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000772F85100 of size 67108864
  306. 2018-04-12 12:34:08.386551: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000776F85100 of size 33554432
  307. 2018-04-12 12:34:08.390601: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000778F85100 of size 75497472
  308. 2018-04-12 12:34:08.393779: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000077D785100 of size 145482240
  309. 2018-04-12 12:34:08.397696: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:671] Summary of in-use Chunks by size:
  310. 2018-04-12 12:34:08.401328: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 55 Chunks of size 256 totalling 13.8KiB
  311. 2018-04-12 12:34:08.404456: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 9 Chunks of size 512 totalling 4.5KiB
  312. 2018-04-12 12:34:08.408114: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 15 Chunks of size 1024 totalling 15.0KiB
  313. 2018-04-12 12:34:08.411268: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 1280 totalling 1.3KiB
  314. 2018-04-12 12:34:08.417808: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 23 Chunks of size 2048 totalling 46.0KiB
  315. 2018-04-12 12:34:08.421151: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 8 Chunks of size 4096 totalling 32.0KiB
  316. 2018-04-12 12:34:08.424957: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 10 Chunks of size 8192 totalling 80.0KiB
  317. 2018-04-12 12:34:08.428200: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 7 Chunks of size 19200 totalling 131.3KiB
  318. 2018-04-12 12:34:08.431710: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 20736 totalling 20.3KiB
  319. 2018-04-12 12:34:08.436129: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 8 Chunks of size 38400 totalling 300.0KiB
  320. 2018-04-12 12:34:08.439623: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 49152 totalling 48.0KiB
  321. 2018-04-12 12:34:08.443358: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 50432 totalling 49.3KiB
  322. 2018-04-12 12:34:08.446502: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 10 Chunks of size 65536 totalling 640.0KiB
  323. 2018-04-12 12:34:08.450028: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 77568 totalling 75.8KiB
  324. 2018-04-12 12:34:08.454472: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 3 Chunks of size 131072 totalling 384.0KiB
  325. 2018-04-12 12:34:08.457952: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 195328 totalling 190.8KiB
  326. 2018-04-12 12:34:08.462556: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 2 Chunks of size 262144 totalling 512.0KiB
  327. 2018-04-12 12:34:08.467502: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 292096 totalling 285.3KiB
  328. 2018-04-12 12:34:08.471371: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 2 Chunks of size 524288 totalling 1.00MiB
  329. 2018-04-12 12:34:08.475180: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 597760 totalling 583.8KiB
  330. 2018-04-12 12:34:08.479605: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 7 Chunks of size 1179648 totalling 7.88MiB
  331. 2018-04-12 12:34:08.482906: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 2097152 totalling 2.00MiB
  332. 2018-04-12 12:34:08.486070: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 8 Chunks of size 3276800 totalling 25.00MiB
  333. 2018-04-12 12:34:08.490346: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 8 Chunks of size 4718592 totalling 36.00MiB
  334. 2018-04-12 12:34:08.493533: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 8388608 totalling 8.00MiB
  335. 2018-04-12 12:34:08.496624: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 8 Chunks of size 13107200 totalling 100.00MiB
  336. 2018-04-12 12:34:08.500668: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 8 Chunks of size 18874368 totalling 144.00MiB
  337. 2018-04-12 12:34:08.503926: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 10 Chunks of size 33554432 totalling 320.00MiB
  338. 2018-04-12 12:34:08.508025: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 8 Chunks of size 67108864 totalling 512.00MiB
  339. 2018-04-12 12:34:08.511353: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 10 Chunks of size 75497472 totalling 720.00MiB
  340. 2018-04-12 12:34:08.515198: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 145482240 totalling 138.74MiB
  341. 2018-04-12 12:34:08.520444: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:678] Sum Total of in-use chunks: 1.97GiB
  342. 2018-04-12 12:34:08.524006: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:680] Stats:
  343. Limit: 2143564595
  344. InUse: 2115974400
  345. MaxInUse: 2128993024
  346. NumAllocs: 845
  347. MaxAllocSize: 573792256
  348.  
  349. 2018-04-12 12:34:08.532925: W T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:279] *************************************************************************************************xxx
  350. 2018-04-12 12:34:08.538003: W T:\src\github\tensorflow\tensorflow\core\framework\op_kernel.cc:1273] OP_REQUIRES failed at conv_grad_filter_ops.cc:834 : Resource exhausted: OOM when allocating tensor with shape[2048,1024,3,3] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
  351. 2018-04-12 12:34:18.545133: W T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 32.00MiB. Current allocation summary follows.
  352. 2018-04-12 12:34:18.550109: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (256):
  353. Total Chunks: 59, Chunks in use: 55. 14.8KiB allocated for chunks. 13.8KiB in use in bin. 300B client-requested in use in bin.
  354. 2018-04-12 12:34:18.557623: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (512):
  355. Total Chunks: 10, Chunks in use: 9. 5.0KiB allocated for chunks. 4.5KiB in use in bin. 4.5KiB client-requested in use in bin.
  356. 2018-04-12 12:34:18.564907: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (1024): Total Chunks: 17, Chunks in use: 16. 17.5KiB allocated for chunks. 16.3KiB in use in bin. 16.0KiB client-requested in use in bin.
  357. 2018-04-12 12:34:18.571051: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (2048): Total Chunks: 24, Chunks in use: 23. 48.3KiB allocated for chunks. 46.3KiB in use in bin. 46.0KiB client-requested in use in bin.
  358. 2018-04-12 12:34:18.578271: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (4096): Total Chunks: 8, Chunks in use: 8. 32.0KiB allocated for chunks. 32.0KiB in use in bin. 32.0KiB client-requested in use in bin.
  359. 2018-04-12 12:34:18.587547: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (8192): Total Chunks: 9, Chunks in use: 9. 72.0KiB allocated for chunks. 72.0KiB in use in bin. 72.0KiB client-requested in use in bin.
  360. 2018-04-12 12:34:18.594697: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (16384): Total Chunks: 8, Chunks in use: 8. 151.5KiB allocated for chunks. 151.5KiB in use in bin. 150.0KiB client-requested in use in bin.
  361. 2018-04-12 12:34:18.602744: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (32768): Total Chunks: 12, Chunks in use: 10. 485.5KiB allocated for chunks. 397.3KiB in use in bin. 385.5KiB client-requested in use in bin.
  362. 2018-04-12 12:34:18.609858: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (65536): Total Chunks: 10, Chunks in use: 10. 651.8KiB allocated for chunks. 651.8KiB in use in bin. 640.0KiB client-requested in use in bin.
  363. 2018-04-12 12:34:18.617064: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (131072): Total Chunks: 4, Chunks in use: 3. 660.3KiB allocated for chunks. 446.8KiB in use in bin. 384.0KiB client-requested in use in bin.
  364. 2018-04-12 12:34:18.624480: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (262144): Total Chunks: 3, Chunks in use: 3. 797.3KiB allocated for chunks. 797.3KiB in use in bin. 768.0KiB client-requested in use in bin.
  365. 2018-04-12 12:34:18.633427: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (524288): Total Chunks: 3, Chunks in use: 3. 1.57MiB allocated for chunks. 1.57MiB in use in bin. 1.50MiB client-requested in use in bin.
  366. 2018-04-12 12:34:18.640708: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (1048576): Total Chunks: 7, Chunks in use: 7. 7.88MiB allocated for chunks. 7.88MiB in use in bin. 7.88MiB client-requested in use in bin.
  367. 2018-04-12 12:34:18.647934: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (2097152): Total Chunks: 10, Chunks in use: 9. 30.71MiB allocated for chunks. 27.00MiB in use in bin. 26.13MiB client-requested in use in bin.
  368. 2018-04-12 12:34:18.654225: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (4194304): Total Chunks: 9, Chunks in use: 8. 40.50MiB allocated for chunks. 36.00MiB in use in bin. 34.63MiB client-requested in use in bin.
  369. 2018-04-12 12:34:18.661187: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (8388608): Total Chunks: 9, Chunks in use: 9. 108.00MiB allocated for chunks. 108.00MiB in use in bin. 104.50MiB client-requested in use in bin.
  370. 2018-04-12 12:34:18.668719: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (16777216): Total Chunks: 9, Chunks in use: 8. 162.00MiB allocated for chunks. 144.00MiB in use in bin. 138.50MiB client-requested in use in bin.
  371. 2018-04-12 12:34:18.676222: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (33554432): Total Chunks: 11, Chunks in use: 11. 352.00MiB allocated for chunks. 352.00MiB in use in bin. 338.00MiB client-requested in use in bin.
  372. 2018-04-12 12:34:18.685098: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (67108864): Total Chunks: 19, Chunks in use: 19. 1.31GiB allocated for chunks. 1.31GiB in use in bin. 1.26GiB client-requested in use in bin.
  373. 2018-04-12 12:34:18.692751: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
  374. 2018-04-12 12:34:18.699169: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (268435456): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
  375. 2018-04-12 12:34:18.705117: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:646] Bin for 32.00MiB was 32.00MiB, Chunk State:
  376. 2018-04-12 12:34:18.709033: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706600000 of size 1280
  377. 2018-04-12 12:34:18.713250: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706600500 of size 256
  378. 2018-04-12 12:34:18.718125: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706600600 of size 256
  379. 2018-04-12 12:34:18.722825: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706600700 of size 512
  380. 2018-04-12 12:34:18.726363: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706600900 of size 256
  381. 2018-04-12 12:34:18.730451: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706600A00 of size 256
  382. 2018-04-12 12:34:18.734493: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706600B00 of size 1024
  383. 2018-04-12 12:34:18.739404: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706600F00 of size 256
  384. 2018-04-12 12:34:18.743195: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706601000 of size 256
  385. 2018-04-12 12:34:18.747195: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706601100 of size 2048
  386. 2018-04-12 12:34:18.750955: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706601900 of size 256
  387. 2018-04-12 12:34:18.754553: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706601A00 of size 256
  388. 2018-04-12 12:34:18.758805: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706601B00 of size 256
  389. 2018-04-12 12:34:18.762091: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706601C00 of size 256
  390. 2018-04-12 12:34:18.766840: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706601D00 of size 65536
  391. 2018-04-12 12:34:18.770663: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706611D00 of size 256
  392. 2018-04-12 12:34:18.775943: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706611E00 of size 256
  393. 2018-04-12 12:34:18.780770: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706611F00 of size 8192
  394. 2018-04-12 12:34:18.786718: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706613F00 of size 256
  395. 2018-04-12 12:34:18.790403: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706614000 of size 256
  396. 2018-04-12 12:34:18.794780: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706614100 of size 4096
  397. 2018-04-12 12:34:18.798748: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706615100 of size 256
  398. 2018-04-12 12:34:18.803138: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706615200 of size 256
  399. 2018-04-12 12:34:18.806861: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706615300 of size 256
  400. 2018-04-12 12:34:18.811326: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706615400 of size 256
  401. 2018-04-12 12:34:18.815990: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706615500 of size 256
  402. 2018-04-12 12:34:18.820169: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706615600 of size 256
  403. 2018-04-12 12:34:18.823687: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706615700 of size 256
  404. 2018-04-12 12:34:18.827234: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706615800 of size 256
  405. 2018-04-12 12:34:18.832369: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706615900 of size 256
  406. 2018-04-12 12:34:18.835904: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706615A00 of size 256
  407. 2018-04-12 12:34:18.839958: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706615B00 of size 256
  408. 2018-04-12 12:34:18.843572: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706615C00 of size 256
  409. 2018-04-12 12:34:18.848697: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706615D00 of size 38400
  410. 2018-04-12 12:34:18.852362: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000070661F300 of size 3276800
  411. 2018-04-12 12:34:18.856467: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000070693F300 of size 13107200
  412. 2018-04-12 12:34:18.860074: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007075BF300 of size 67108864
  413. 2018-04-12 12:34:18.863985: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000070B5BF300 of size 33554432
  414. 2018-04-12 12:34:18.869047: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000070D5BF300 of size 75497472
  415. 2018-04-12 12:34:18.872748: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000711DBF300 of size 18874368
  416. 2018-04-12 12:34:18.877125: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000712FBF300 of size 4718592
  417. 2018-04-12 12:34:18.880259: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071343F300 of size 1179648
  418. 2018-04-12 12:34:18.883913: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071355F300 of size 19200
  419. 2018-04-12 12:34:18.888095: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000713563E00 of size 19200
  420. 2018-04-12 12:34:18.891051: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000713568900 of size 19200
  421. 2018-04-12 12:34:18.894884: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071356D400 of size 512
  422. 2018-04-12 12:34:18.898481: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071356D600 of size 1179648
  423. 2018-04-12 12:34:18.901713: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071368D600 of size 2097152
  424. 2018-04-12 12:34:18.905757: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071388D600 of size 1024
  425. 2018-04-12 12:34:18.908940: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071388DA00 of size 4718592
  426. 2018-04-12 12:34:18.912927: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000713D0DA00 of size 8388608
  427. 2018-04-12 12:34:18.916699: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071450DA00 of size 2048
  428. 2018-04-12 12:34:18.920203: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071450E200 of size 33554432
  429. 2018-04-12 12:34:18.924342: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071650E200 of size 33554432
  430. 2018-04-12 12:34:18.928105: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071850E200 of size 2048
  431. 2018-04-12 12:34:18.932036: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071850EA00 of size 33554432
  432. 2018-04-12 12:34:18.935300: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071A50EA00 of size 65536
  433. 2018-04-12 12:34:18.938786: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071A51EA00 of size 75497472
  434. 2018-04-12 12:34:18.943253: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071ED1EA00 of size 8192
  435. 2018-04-12 12:34:18.946564: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071ED20A00 of size 18874368
  436. 2018-04-12 12:34:18.950928: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071FF20A00 of size 18874368
  437. 2018-04-12 12:34:18.954077: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721120A00 of size 4096
  438. 2018-04-12 12:34:18.958409: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721121A00 of size 4718592
  439. 2018-04-12 12:34:18.961583: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007215A1A00 of size 4718592
  440. 2018-04-12 12:34:18.965052: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721A21A00 of size 2048
  441. 2018-04-12 12:34:18.969373: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721A22200 of size 2048
  442. 2018-04-12 12:34:18.972817: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721A22A00 of size 1179648
  443. 2018-04-12 12:34:18.975942: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B42A00 of size 38400
  444. 2018-04-12 12:34:18.979971: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B4C000 of size 65536
  445. 2018-04-12 12:34:18.983527: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B5C000 of size 256
  446. 2018-04-12 12:34:18.987259: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B5C100 of size 38400
  447. 2018-04-12 12:34:18.990598: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B65700 of size 512
  448. 2018-04-12 12:34:18.993799: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B65900 of size 1024
  449. 2018-04-12 12:34:18.998021: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B65D00 of size 2048
  450. 2018-04-12 12:34:19.001561: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B66500 of size 2048
  451. 2018-04-12 12:34:19.005781: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B66D00 of size 65536
  452. 2018-04-12 12:34:19.009023: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B76D00 of size 8192
  453. 2018-04-12 12:34:19.012211: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B78D00 of size 4096
  454. 2018-04-12 12:34:19.016081: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B79D00 of size 2048
  455. 2018-04-12 12:34:19.019331: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B7A500 of size 4096
  456. 2018-04-12 12:34:19.023391: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B7B500 of size 1024
  457. 2018-04-12 12:34:19.026477: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B7B900 of size 19200
  458. 2018-04-12 12:34:19.029777: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B80400 of size 19200
  459. 2018-04-12 12:34:19.034951: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B84F00 of size 256
  460. 2018-04-12 12:34:19.038702: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B85000 of size 38400
  461. 2018-04-12 12:34:19.044814: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B8E600 of size 512
  462. 2018-04-12 12:34:19.050435: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B8E800 of size 1024
  463. 2018-04-12 12:34:19.053717: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B8EC00 of size 2048
  464. 2018-04-12 12:34:19.057004: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B8F400 of size 2048
  465. 2018-04-12 12:34:19.061872: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B8FC00 of size 2048
  466. 2018-04-12 12:34:19.067400: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B90400 of size 256
  467. 2018-04-12 12:34:19.072301: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B90500 of size 65536
  468. 2018-04-12 12:34:19.076066: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BA0500 of size 1024
  469. 2018-04-12 12:34:19.080907: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BA0900 of size 8192
  470. 2018-04-12 12:34:19.085126: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BA2900 of size 256
  471. 2018-04-12 12:34:19.089929: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BA2A00 of size 256
  472. 2018-04-12 12:34:19.094039: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BA2B00 of size 256
  473. 2018-04-12 12:34:19.098472: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BA2C00 of size 256
  474. 2018-04-12 12:34:19.102388: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BA2D00 of size 256
  475. 2018-04-12 12:34:19.107235: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BA2E00 of size 256
  476. 2018-04-12 12:34:19.111364: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BA2F00 of size 256
  477. 2018-04-12 12:34:19.115982: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 0000000721BA3000 of size 49408
  478. 2018-04-12 12:34:19.119807: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BAF100 of size 8192
  479. 2018-04-12 12:34:19.123980: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 0000000721BB1100 of size 40960
  480. 2018-04-12 12:34:19.127827: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BBB100 of size 256
  481. 2018-04-12 12:34:19.132993: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 0000000721BBB200 of size 512
  482. 2018-04-12 12:34:19.136998: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BBB400 of size 38400
  483. 2018-04-12 12:34:19.140077: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BC4A00 of size 49152
  484. 2018-04-12 12:34:19.143966: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BD0A00 of size 256
  485. 2018-04-12 12:34:19.147265: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BD0B00 of size 597760
  486. 2018-04-12 12:34:19.152161: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C62A00 of size 1024
  487. 2018-04-12 12:34:19.155383: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C62E00 of size 256
  488. 2018-04-12 12:34:19.158515: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C62F00 of size 256
  489. 2018-04-12 12:34:19.162631: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C63000 of size 256
  490. 2018-04-12 12:34:19.165943: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C63100 of size 256
  491. 2018-04-12 12:34:19.170181: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C63200 of size 256
  492. 2018-04-12 12:34:19.173456: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C63300 of size 256
  493. 2018-04-12 12:34:19.176868: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C63400 of size 256
  494. 2018-04-12 12:34:19.182344: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C63500 of size 512
  495. 2018-04-12 12:34:19.185509: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C63700 of size 1024
  496. 2018-04-12 12:34:19.189416: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C63B00 of size 2048
  497. 2018-04-12 12:34:19.192865: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C64300 of size 8192
  498. 2018-04-12 12:34:19.195858: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C66300 of size 4096
  499. 2018-04-12 12:34:19.200136: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C67300 of size 20736
  500. 2018-04-12 12:34:19.203274: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C6C400 of size 256
  501. 2018-04-12 12:34:19.207188: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C6C500 of size 2048
  502. 2018-04-12 12:34:19.210436: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C6CD00 of size 4096
  503. 2018-04-12 12:34:19.213558: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C6DD00 of size 2048
  504. 2018-04-12 12:34:19.217925: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C6E500 of size 2048
  505. 2018-04-12 12:34:19.220981: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C6ED00 of size 256
  506. 2018-04-12 12:34:19.224088: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C6EE00 of size 1024
  507. 2018-04-12 12:34:19.228002: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C6F200 of size 65536
  508. 2018-04-12 12:34:19.231129: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C7F200 of size 256
  509. 2018-04-12 12:34:19.235221: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C7F300 of size 256
  510. 2018-04-12 12:34:19.238377: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C7F400 of size 256
  511. 2018-04-12 12:34:19.241709: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C7F500 of size 256
  512. 2018-04-12 12:34:19.245970: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C7F600 of size 256
  513. 2018-04-12 12:34:19.249380: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C7F700 of size 256
  514. 2018-04-12 12:34:19.253103: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C7F800 of size 38400
  515. 2018-04-12 12:34:19.256520: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C88E00 of size 512
  516. 2018-04-12 12:34:19.259639: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C89000 of size 1024
  517. 2018-04-12 12:34:19.263516: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C89400 of size 3276800
  518. 2018-04-12 12:34:19.267276: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721FA9400 of size 1024
  519. 2018-04-12 12:34:19.271618: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721FA9800 of size 13107200
  520. 2018-04-12 12:34:19.274749: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000722C29800 of size 13107200
  521. 2018-04-12 12:34:19.278085: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007238A9800 of size 2048
  522. 2018-04-12 12:34:19.282411: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007238AA000 of size 8192
  523. 2018-04-12 12:34:19.286340: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007238AC000 of size 67108864
  524. 2018-04-12 12:34:19.290311: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007278AC000 of size 67108864
  525. 2018-04-12 12:34:19.293457: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000072B8AC000 of size 2048
  526. 2018-04-12 12:34:19.296425: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000072B8AC800 of size 33554432
  527. 2018-04-12 12:34:19.301127: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000072D8AC800 of size 33554432
  528. 2018-04-12 12:34:19.304521: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000072F8AC800 of size 65536
  529. 2018-04-12 12:34:19.308332: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000072F8BC800 of size 75497472
  530. 2018-04-12 12:34:19.311671: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007340BC800 of size 75497472
  531. 2018-04-12 12:34:19.315393: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007388BC800 of size 8192
  532. 2018-04-12 12:34:19.319242: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007388BE800 of size 18874368
  533. 2018-04-12 12:34:19.322479: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000739ABE800 of size 18874368
  534. 2018-04-12 12:34:19.325599: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073ACBE800 of size 4096
  535. 2018-04-12 12:34:19.329345: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073ACBF800 of size 4718592
  536. 2018-04-12 12:34:19.332713: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073B13F800 of size 2048
  537. 2018-04-12 12:34:19.336760: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073B140000 of size 1179648
  538. 2018-04-12 12:34:19.339978: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073B260000 of size 1179648
  539. 2018-04-12 12:34:19.343517: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073B380000 of size 1024
  540. 2018-04-12 12:34:19.347468: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073B380400 of size 4096
  541. 2018-04-12 12:34:19.351148: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073B381400 of size 19200
  542. 2018-04-12 12:34:19.355171: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073B385F00 of size 256
  543. 2018-04-12 12:34:19.358267: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073B386000 of size 19200
  544. 2018-04-12 12:34:19.361606: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073B38AB00 of size 38400
  545. 2018-04-12 12:34:19.366344: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073B394100 of size 512
  546. 2018-04-12 12:34:19.369824: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073B394300 of size 4718592
  547. 2018-04-12 12:34:19.373662: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073B814300 of size 3276800
  548. 2018-04-12 12:34:19.377098: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073BB34300 of size 1024
  549. 2018-04-12 12:34:19.381734: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073BB34700 of size 38400
  550. 2018-04-12 12:34:19.386501: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073BB3DD00 of size 3276800
  551. 2018-04-12 12:34:19.391870: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073BE5DD00 of size 13107200
  552. 2018-04-12 12:34:19.395563: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073CADDD00 of size 67108864
  553. 2018-04-12 12:34:19.402902: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000740ADDD00 of size 75497472
  554. 2018-04-12 12:34:19.406134: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007452DDD00 of size 75497472
  555. 2018-04-12 12:34:19.410428: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000749ADDD00 of size 18874368
  556. 2018-04-12 12:34:19.413937: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000074ACDDD00 of size 3276800
  557. 2018-04-12 12:34:19.419534: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000074AFFDD00 of size 13107200
  558. 2018-04-12 12:34:19.422875: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000074BC7DD00 of size 67108864
  559. 2018-04-12 12:34:19.427183: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000074FC7DD00 of size 33554432
  560. 2018-04-12 12:34:19.431070: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000751C7DD00 of size 75497472
  561. 2018-04-12 12:34:19.435106: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000075647DD00 of size 18874368
  562. 2018-04-12 12:34:19.440642: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000075767DD00 of size 4718592
  563. 2018-04-12 12:34:19.443769: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000757AFDD00 of size 1179648
  564. 2018-04-12 12:34:19.448055: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000757C1DD00 of size 4718592
  565. 2018-04-12 12:34:19.451935: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000075809DD00 of size 3276800
  566. 2018-04-12 12:34:19.455891: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007583BDD00 of size 13107200
  567. 2018-04-12 12:34:19.459242: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000075903DD00 of size 67108864
  568. 2018-04-12 12:34:19.463261: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000075D03DD00 of size 1179648
  569. 2018-04-12 12:34:19.468025: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000075D15DD00 of size 33554432
  570. 2018-04-12 12:34:19.472511: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000075F15DD00 of size 75497472
  571. 2018-04-12 12:34:19.476146: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000076395DD00 of size 18874368
  572. 2018-04-12 12:34:19.480039: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 0000000764B5DD00 of size 256
  573. 2018-04-12 12:34:19.484714: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000764B5DE00 of size 256
  574. 2018-04-12 12:34:19.489014: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000764B5DF00 of size 50432
  575. 2018-04-12 12:34:19.493482: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000764B6A400 of size 512
  576. 2018-04-12 12:34:19.497266: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000764B6A600 of size 524288
  577. 2018-04-12 12:34:19.502079: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 0000000764BEA600 of size 256
  578. 2018-04-12 12:34:19.506524: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000764BEA700 of size 3276800
  579. 2018-04-12 12:34:19.511050: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000764F0A700 of size 1024
  580. 2018-04-12 12:34:19.515738: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 0000000764F0AB00 of size 256
  581. 2018-04-12 12:34:19.520103: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000764F0AC00 of size 13107200
  582. 2018-04-12 12:34:19.524330: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000765B8AC00 of size 2048
  583. 2018-04-12 12:34:19.528424: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 0000000765B8B400 of size 2048
  584. 2018-04-12 12:34:19.532336: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000765B8BC00 of size 2304
  585. 2018-04-12 12:34:19.536366: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000765B8C500 of size 2048
  586. 2018-04-12 12:34:19.539492: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000765B8CD00 of size 67108864
  587. 2018-04-12 12:34:19.542542: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000769B8CD00 of size 33554432
  588. 2018-04-12 12:34:19.546546: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000076BB8CD00 of size 65536
  589. 2018-04-12 12:34:19.551479: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000076BB9CD00 of size 75497472
  590. 2018-04-12 12:34:19.555757: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000077039CD00 of size 524288
  591. 2018-04-12 12:34:19.558855: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 000000077041CD00 of size 18874368
  592. 2018-04-12 12:34:19.561971: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000077161CD00 of size 262144
  593. 2018-04-12 12:34:19.566260: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000077165CD00 of size 1024
  594. 2018-04-12 12:34:19.569508: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000077165D100 of size 2048
  595. 2018-04-12 12:34:19.573714: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000077165D900 of size 65536
  596. 2018-04-12 12:34:19.576954: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 000000077166D900 of size 1280
  597. 2018-04-12 12:34:19.580192: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000077166DE00 of size 195328
  598. 2018-04-12 12:34:19.586062: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 000000077169D900 of size 256
  599. 2018-04-12 12:34:19.589500: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000077169DA00 of size 292096
  600. 2018-04-12 12:34:19.593449: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 00000007716E4F00 of size 4718592
  601. 2018-04-12 12:34:19.596894: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000771B64F00 of size 3276800
  602. 2018-04-12 12:34:19.601737: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000771E84F00 of size 262144
  603. 2018-04-12 12:34:19.605183: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000771EC4F00 of size 2048
  604. 2018-04-12 12:34:19.608618: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 0000000771EC5700 of size 218624
  605. 2018-04-12 12:34:19.612801: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000771EFAD00 of size 77568
  606. 2018-04-12 12:34:19.616925: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000771F0DC00 of size 8192
  607. 2018-04-12 12:34:19.621076: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000771F0FC00 of size 131072
  608. 2018-04-12 12:34:19.624735: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000771F2FC00 of size 131072
  609. 2018-04-12 12:34:19.629113: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 0000000771F4FC00 of size 3887872
  610. 2018-04-12 12:34:19.633151: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000772304F00 of size 512
  611. 2018-04-12 12:34:19.636364: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000772305100 of size 13107200
  612. 2018-04-12 12:34:19.640668: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000772F85100 of size 67108864
  613. 2018-04-12 12:34:19.644883: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000776F85100 of size 33554432
  614. 2018-04-12 12:34:19.649094: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000778F85100 of size 75497472
  615. 2018-04-12 12:34:19.652376: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000077D785100 of size 33554432
  616. 2018-04-12 12:34:19.655428: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000077F785100 of size 111927808
  617. 2018-04-12 12:34:19.659412: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:671] Summary of in-use Chunks by size:
  618. 2018-04-12 12:34:19.662936: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 55 Chunks of size 256 totalling 13.8KiB
  619. 2018-04-12 12:34:19.667268: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 9 Chunks of size 512 totalling 4.5KiB
  620. 2018-04-12 12:34:19.670883: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 15 Chunks of size 1024 totalling 15.0KiB
  621. 2018-04-12 12:34:19.674480: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 1280 totalling 1.3KiB
  622. 2018-04-12 12:34:19.678481: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 22 Chunks of size 2048 totalling 44.0KiB
  623. 2018-04-12 12:34:19.681939: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 2304 totalling 2.3KiB
  624. 2018-04-12 12:34:19.685883: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 8 Chunks of size 4096 totalling 32.0KiB
  625. 2018-04-12 12:34:19.689264: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 9 Chunks of size 8192 totalling 72.0KiB
  626. 2018-04-12 12:34:19.692441: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 7 Chunks of size 19200 totalling 131.3KiB
  627. 2018-04-12 12:34:19.696236: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 20736 totalling 20.3KiB
  628. 2018-04-12 12:34:19.700118: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 8 Chunks of size 38400 totalling 300.0KiB
  629. 2018-04-12 12:34:19.703914: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 49152 totalling 48.0KiB
  630. 2018-04-12 12:34:19.707657: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 50432 totalling 49.3KiB
  631. 2018-04-12 12:34:19.710801: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 9 Chunks of size 65536 totalling 576.0KiB
  632. 2018-04-12 12:34:19.714660: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 77568 totalling 75.8KiB
  633. 2018-04-12 12:34:19.718592: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 2 Chunks of size 131072 totalling 256.0KiB
  634. 2018-04-12 12:34:19.722396: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 195328 totalling 190.8KiB
  635. 2018-04-12 12:34:19.725477: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 2 Chunks of size 262144 totalling 512.0KiB
  636. 2018-04-12 12:34:19.728744: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 292096 totalling 285.3KiB
  637. 2018-04-12 12:34:19.732963: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 2 Chunks of size 524288 totalling 1.00MiB
  638. 2018-04-12 12:34:19.736987: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 597760 totalling 583.8KiB
  639. 2018-04-12 12:34:19.741078: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 7 Chunks of size 1179648 totalling 7.88MiB
  640. 2018-04-12 12:34:19.744319: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 2097152 totalling 2.00MiB
  641. 2018-04-12 12:34:19.747597: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 8 Chunks of size 3276800 totalling 25.00MiB
  642. 2018-04-12 12:34:19.752030: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 8 Chunks of size 4718592 totalling 36.00MiB
  643. 2018-04-12 12:34:19.755261: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 8388608 totalling 8.00MiB
  644. 2018-04-12 12:34:19.758490: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 8 Chunks of size 13107200 totalling 100.00MiB
  645. 2018-04-12 12:34:19.762480: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 8 Chunks of size 18874368 totalling 144.00MiB
  646. 2018-04-12 12:34:19.767127: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 11 Chunks of size 33554432 totalling 352.00MiB
  647. 2018-04-12 12:34:19.772591: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 8 Chunks of size 67108864 totalling 512.00MiB
  648. 2018-04-12 12:34:19.776660: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 10 Chunks of size 75497472 totalling 720.00MiB
  649. 2018-04-12 12:34:19.781418: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 111927808 totalling 106.74MiB
  650. 2018-04-12 12:34:19.787385: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:678] Sum Total of in-use chunks: 1.97GiB
  651. 2018-04-12 12:34:19.790795: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:680] Stats:
  652. Limit: 2143564595
  653. InUse: 2115769856
  654. MaxInUse: 2128993024
  655. NumAllocs: 853
  656. MaxAllocSize: 573792256
  657.  
  658. 2018-04-12 12:34:19.801739: W T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:279] **************************************************************************************************xx
  659. 2018-04-12 12:34:19.806891: W T:\src\github\tensorflow\tensorflow\core\framework\op_kernel.cc:1273] OP_REQUIRES failed at cwise_ops_common.cc:70 : Resource exhausted: OOM when allocating tensor with shape[512,16384] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
  660. 2018-04-12 12:34:29.814113: W T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 64.00MiB. Current allocation summary follows.
  661. 2018-04-12 12:34:29.819343: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (256):
  662. Total Chunks: 59, Chunks in use: 55. 14.8KiB allocated for chunks. 13.8KiB in use in bin. 300B client-requested in use in bin.
  663. 2018-04-12 12:34:29.826990: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (512):
  664. Total Chunks: 10, Chunks in use: 9. 5.0KiB allocated for chunks. 4.5KiB in use in bin. 4.5KiB client-requested in use in bin.
  665. 2018-04-12 12:34:29.835376: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (1024): Total Chunks: 17, Chunks in use: 16. 17.5KiB allocated for chunks. 16.3KiB in use in bin. 16.0KiB client-requested in use in bin.
  666. 2018-04-12 12:34:29.841423: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (2048): Total Chunks: 24, Chunks in use: 23. 48.3KiB allocated for chunks. 46.3KiB in use in bin. 46.0KiB client-requested in use in bin.
  667. 2018-04-12 12:34:29.849790: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (4096): Total Chunks: 8, Chunks in use: 8. 32.0KiB allocated for chunks. 32.0KiB in use in bin. 32.0KiB client-requested in use in bin.
  668. 2018-04-12 12:34:29.857202: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (8192): Total Chunks: 9, Chunks in use: 9. 72.0KiB allocated for chunks. 72.0KiB in use in bin. 72.0KiB client-requested in use in bin.
  669. 2018-04-12 12:34:29.864552: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (16384): Total Chunks: 8, Chunks in use: 8. 151.5KiB allocated for chunks. 151.5KiB in use in bin. 150.0KiB client-requested in use in bin.
  670. 2018-04-12 12:34:29.873687: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (32768): Total Chunks: 12, Chunks in use: 10. 485.5KiB allocated for chunks. 397.3KiB in use in bin. 385.5KiB client-requested in use in bin.
  671. 2018-04-12 12:34:29.880403: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (65536): Total Chunks: 11, Chunks in use: 11. 715.8KiB allocated for chunks. 715.8KiB in use in bin. 704.0KiB client-requested in use in bin.
  672. 2018-04-12 12:34:29.886942: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (131072): Total Chunks: 4, Chunks in use: 3. 596.3KiB allocated for chunks. 446.8KiB in use in bin. 384.0KiB client-requested in use in bin.
  673. 2018-04-12 12:34:29.893796: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (262144): Total Chunks: 3, Chunks in use: 3. 797.3KiB allocated for chunks. 797.3KiB in use in bin. 768.0KiB client-requested in use in bin.
  674. 2018-04-12 12:34:29.901020: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (524288): Total Chunks: 3, Chunks in use: 3. 1.57MiB allocated for chunks. 1.57MiB in use in bin. 1.50MiB client-requested in use in bin.
  675. 2018-04-12 12:34:29.907678: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (1048576): Total Chunks: 7, Chunks in use: 7. 7.88MiB allocated for chunks. 7.88MiB in use in bin. 7.88MiB client-requested in use in bin.
  676. 2018-04-12 12:34:29.915777: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (2097152): Total Chunks: 10, Chunks in use: 9. 30.71MiB allocated for chunks. 27.00MiB in use in bin. 26.13MiB client-requested in use in bin.
  677. 2018-04-12 12:34:29.922556: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (4194304): Total Chunks: 9, Chunks in use: 8. 40.50MiB allocated for chunks. 36.00MiB in use in bin. 34.63MiB client-requested in use in bin.
  678. 2018-04-12 12:34:29.930154: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (8388608): Total Chunks: 9, Chunks in use: 9. 108.00MiB allocated for chunks. 108.00MiB in use in bin. 104.50MiB client-requested in use in bin.
  679. 2018-04-12 12:34:29.939019: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (16777216): Total Chunks: 9, Chunks in use: 8. 162.00MiB allocated for chunks. 144.00MiB in use in bin. 138.50MiB client-requested in use in bin.
  680. 2018-04-12 12:34:29.947684: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (33554432): Total Chunks: 11, Chunks in use: 11. 352.00MiB allocated for chunks. 352.00MiB in use in bin. 338.00MiB client-requested in use in bin.
  681. 2018-04-12 12:34:29.954715: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (67108864): Total Chunks: 19, Chunks in use: 19. 1.31GiB allocated for chunks. 1.31GiB in use in bin. 1.26GiB client-requested in use in bin.
  682. 2018-04-12 12:34:29.962038: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
  683. 2018-04-12 12:34:29.968871: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (268435456): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
  684. 2018-04-12 12:34:29.975021: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:646] Bin for 64.00MiB was 64.00MiB, Chunk State:
  685. 2018-04-12 12:34:29.979603: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706600000 of size 1280
  686. 2018-04-12 12:34:29.985065: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706600500 of size 256
  687. 2018-04-12 12:34:29.989848: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706600600 of size 256
  688. 2018-04-12 12:34:29.993885: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706600700 of size 512
  689. 2018-04-12 12:34:29.999807: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706600900 of size 256
  690. 2018-04-12 12:34:30.005079: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706600A00 of size 256
  691. 2018-04-12 12:34:30.010608: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706600B00 of size 1024
  692. 2018-04-12 12:34:30.015201: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706600F00 of size 256
  693. 2018-04-12 12:34:30.019937: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706601000 of size 256
  694. 2018-04-12 12:34:30.024153: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706601100 of size 2048
  695. 2018-04-12 12:34:30.029437: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706601900 of size 256
  696. 2018-04-12 12:34:30.034161: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706601A00 of size 256
  697. 2018-04-12 12:34:30.038869: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706601B00 of size 256
  698. 2018-04-12 12:34:30.045098: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706601C00 of size 256
  699. 2018-04-12 12:34:30.050479: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706601D00 of size 65536
  700. 2018-04-12 12:34:30.054713: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706611D00 of size 256
  701. 2018-04-12 12:34:30.058293: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706611E00 of size 256
  702. 2018-04-12 12:34:30.062021: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706611F00 of size 8192
  703. 2018-04-12 12:34:30.067642: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706613F00 of size 256
  704. 2018-04-12 12:34:30.071833: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706614000 of size 256
  705. 2018-04-12 12:34:30.076231: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706614100 of size 4096
  706. 2018-04-12 12:34:30.079383: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706615100 of size 256
  707. 2018-04-12 12:34:30.084076: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706615200 of size 256
  708. 2018-04-12 12:34:30.087561: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706615300 of size 256
  709. 2018-04-12 12:34:30.090763: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706615400 of size 256
  710. 2018-04-12 12:34:30.094566: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706615500 of size 256
  711. 2018-04-12 12:34:30.097488: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706615600 of size 256
  712. 2018-04-12 12:34:30.101547: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706615700 of size 256
  713. 2018-04-12 12:34:30.104457: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706615800 of size 256
  714. 2018-04-12 12:34:30.107903: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706615900 of size 256
  715. 2018-04-12 12:34:30.111609: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706615A00 of size 256
  716. 2018-04-12 12:34:30.114823: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706615B00 of size 256
  717. 2018-04-12 12:34:30.117728: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706615C00 of size 256
  718. 2018-04-12 12:34:30.121645: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000706615D00 of size 38400
  719. 2018-04-12 12:34:30.124592: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000070661F300 of size 3276800
  720. 2018-04-12 12:34:30.129120: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000070693F300 of size 13107200
  721. 2018-04-12 12:34:30.133234: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007075BF300 of size 67108864
  722. 2018-04-12 12:34:30.136778: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000070B5BF300 of size 33554432
  723. 2018-04-12 12:34:30.140485: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000070D5BF300 of size 75497472
  724. 2018-04-12 12:34:30.143600: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000711DBF300 of size 18874368
  725. 2018-04-12 12:34:30.147509: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000712FBF300 of size 4718592
  726. 2018-04-12 12:34:30.151400: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071343F300 of size 1179648
  727. 2018-04-12 12:34:30.154588: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071355F300 of size 19200
  728. 2018-04-12 12:34:30.158374: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000713563E00 of size 19200
  729. 2018-04-12 12:34:30.161339: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000713568900 of size 19200
  730. 2018-04-12 12:34:30.165244: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071356D400 of size 512
  731. 2018-04-12 12:34:30.168466: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071356D600 of size 1179648
  732. 2018-04-12 12:34:30.171478: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071368D600 of size 2097152
  733. 2018-04-12 12:34:30.175252: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071388D600 of size 1024
  734. 2018-04-12 12:34:30.178479: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071388DA00 of size 4718592
  735. 2018-04-12 12:34:30.183042: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000713D0DA00 of size 8388608
  736. 2018-04-12 12:34:30.186243: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071450DA00 of size 2048
  737. 2018-04-12 12:34:30.189823: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071450E200 of size 33554432
  738. 2018-04-12 12:34:30.194251: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071650E200 of size 33554432
  739. 2018-04-12 12:34:30.198306: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071850E200 of size 2048
  740. 2018-04-12 12:34:30.202913: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071850EA00 of size 33554432
  741. 2018-04-12 12:34:30.206365: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071A50EA00 of size 65536
  742. 2018-04-12 12:34:30.209328: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071A51EA00 of size 75497472
  743. 2018-04-12 12:34:30.213172: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071ED1EA00 of size 8192
  744. 2018-04-12 12:34:30.216654: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071ED20A00 of size 18874368
  745. 2018-04-12 12:34:30.220479: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000071FF20A00 of size 18874368
  746. 2018-04-12 12:34:30.223689: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721120A00 of size 4096
  747. 2018-04-12 12:34:30.226886: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721121A00 of size 4718592
  748. 2018-04-12 12:34:30.231445: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007215A1A00 of size 4718592
  749. 2018-04-12 12:34:30.234763: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721A21A00 of size 2048
  750. 2018-04-12 12:34:30.238668: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721A22200 of size 2048
  751. 2018-04-12 12:34:30.241683: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721A22A00 of size 1179648
  752. 2018-04-12 12:34:30.244936: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B42A00 of size 38400
  753. 2018-04-12 12:34:30.248950: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B4C000 of size 65536
  754. 2018-04-12 12:34:30.252274: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B5C000 of size 256
  755. 2018-04-12 12:34:30.255842: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B5C100 of size 38400
  756. 2018-04-12 12:34:30.259161: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B65700 of size 512
  757. 2018-04-12 12:34:30.262863: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B65900 of size 1024
  758. 2018-04-12 12:34:30.267776: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B65D00 of size 2048
  759. 2018-04-12 12:34:30.271656: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B66500 of size 2048
  760. 2018-04-12 12:34:30.276159: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B66D00 of size 65536
  761. 2018-04-12 12:34:30.279491: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B76D00 of size 8192
  762. 2018-04-12 12:34:30.283937: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B78D00 of size 4096
  763. 2018-04-12 12:34:30.287178: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B79D00 of size 2048
  764. 2018-04-12 12:34:30.290611: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B7A500 of size 4096
  765. 2018-04-12 12:34:30.294672: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B7B500 of size 1024
  766. 2018-04-12 12:34:30.298542: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B7B900 of size 19200
  767. 2018-04-12 12:34:30.302703: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B80400 of size 19200
  768. 2018-04-12 12:34:30.306037: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B84F00 of size 256
  769. 2018-04-12 12:34:30.310040: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B85000 of size 38400
  770. 2018-04-12 12:34:30.314519: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B8E600 of size 512
  771. 2018-04-12 12:34:30.317820: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B8E800 of size 1024
  772. 2018-04-12 12:34:30.321815: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B8EC00 of size 2048
  773. 2018-04-12 12:34:30.324922: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B8F400 of size 2048
  774. 2018-04-12 12:34:30.328721: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B8FC00 of size 2048
  775. 2018-04-12 12:34:30.332675: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B90400 of size 256
  776. 2018-04-12 12:34:30.335624: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721B90500 of size 65536
  777. 2018-04-12 12:34:30.339431: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BA0500 of size 1024
  778. 2018-04-12 12:34:30.342514: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BA0900 of size 8192
  779. 2018-04-12 12:34:30.345631: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BA2900 of size 256
  780. 2018-04-12 12:34:30.349818: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BA2A00 of size 256
  781. 2018-04-12 12:34:30.353590: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BA2B00 of size 256
  782. 2018-04-12 12:34:30.357308: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BA2C00 of size 256
  783. 2018-04-12 12:34:30.360622: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BA2D00 of size 256
  784. 2018-04-12 12:34:30.365420: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BA2E00 of size 256
  785. 2018-04-12 12:34:30.368630: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BA2F00 of size 256
  786. 2018-04-12 12:34:30.371689: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 0000000721BA3000 of size 49408
  787. 2018-04-12 12:34:30.375506: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BAF100 of size 8192
  788. 2018-04-12 12:34:30.378807: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 0000000721BB1100 of size 40960
  789. 2018-04-12 12:34:30.382387: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BBB100 of size 256
  790. 2018-04-12 12:34:30.387429: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 0000000721BBB200 of size 512
  791. 2018-04-12 12:34:30.390617: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BBB400 of size 38400
  792. 2018-04-12 12:34:30.394411: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BC4A00 of size 49152
  793. 2018-04-12 12:34:30.397571: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BD0A00 of size 256
  794. 2018-04-12 12:34:30.402302: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721BD0B00 of size 597760
  795. 2018-04-12 12:34:30.405347: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C62A00 of size 1024
  796. 2018-04-12 12:34:30.408614: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C62E00 of size 256
  797. 2018-04-12 12:34:30.412482: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C62F00 of size 256
  798. 2018-04-12 12:34:30.415984: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C63000 of size 256
  799. 2018-04-12 12:34:30.419935: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C63100 of size 256
  800. 2018-04-12 12:34:30.423046: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C63200 of size 256
  801. 2018-04-12 12:34:30.426388: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C63300 of size 256
  802. 2018-04-12 12:34:30.429984: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C63400 of size 256
  803. 2018-04-12 12:34:30.433816: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C63500 of size 512
  804. 2018-04-12 12:34:30.438106: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C63700 of size 1024
  805. 2018-04-12 12:34:30.441626: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C63B00 of size 2048
  806. 2018-04-12 12:34:30.445273: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C64300 of size 8192
  807. 2018-04-12 12:34:30.449416: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C66300 of size 4096
  808. 2018-04-12 12:34:30.452312: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C67300 of size 20736
  809. 2018-04-12 12:34:30.456298: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C6C400 of size 256
  810. 2018-04-12 12:34:30.459395: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C6C500 of size 2048
  811. 2018-04-12 12:34:30.462472: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C6CD00 of size 4096
  812. 2018-04-12 12:34:30.467626: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C6DD00 of size 2048
  813. 2018-04-12 12:34:30.471767: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C6E500 of size 2048
  814. 2018-04-12 12:34:30.475806: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C6ED00 of size 256
  815. 2018-04-12 12:34:30.479173: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C6EE00 of size 1024
  816. 2018-04-12 12:34:30.483543: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C6F200 of size 65536
  817. 2018-04-12 12:34:30.486767: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C7F200 of size 256
  818. 2018-04-12 12:34:30.490155: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C7F300 of size 256
  819. 2018-04-12 12:34:30.494580: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C7F400 of size 256
  820. 2018-04-12 12:34:30.497988: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C7F500 of size 256
  821. 2018-04-12 12:34:30.501835: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C7F600 of size 256
  822. 2018-04-12 12:34:30.505315: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C7F700 of size 256
  823. 2018-04-12 12:34:30.508624: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C7F800 of size 38400
  824. 2018-04-12 12:34:30.512706: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C88E00 of size 512
  825. 2018-04-12 12:34:30.516019: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C89000 of size 1024
  826. 2018-04-12 12:34:30.519872: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721C89400 of size 3276800
  827. 2018-04-12 12:34:30.523226: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721FA9400 of size 1024
  828. 2018-04-12 12:34:30.526620: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000721FA9800 of size 13107200
  829. 2018-04-12 12:34:30.530753: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000722C29800 of size 13107200
  830. 2018-04-12 12:34:30.534931: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007238A9800 of size 2048
  831. 2018-04-12 12:34:30.539192: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007238AA000 of size 8192
  832. 2018-04-12 12:34:30.542130: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007238AC000 of size 67108864
  833. 2018-04-12 12:34:30.545382: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007278AC000 of size 67108864
  834. 2018-04-12 12:34:30.549880: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000072B8AC000 of size 2048
  835. 2018-04-12 12:34:30.553028: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000072B8AC800 of size 33554432
  836. 2018-04-12 12:34:30.556902: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000072D8AC800 of size 33554432
  837. 2018-04-12 12:34:30.559927: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000072F8AC800 of size 65536
  838. 2018-04-12 12:34:30.563291: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000072F8BC800 of size 75497472
  839. 2018-04-12 12:34:30.568124: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007340BC800 of size 75497472
  840. 2018-04-12 12:34:30.571126: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007388BC800 of size 8192
  841. 2018-04-12 12:34:30.574970: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007388BE800 of size 18874368
  842. 2018-04-12 12:34:30.578103: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000739ABE800 of size 18874368
  843. 2018-04-12 12:34:30.581658: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073ACBE800 of size 4096
  844. 2018-04-12 12:34:30.585995: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073ACBF800 of size 4718592
  845. 2018-04-12 12:34:30.589360: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073B13F800 of size 2048
  846. 2018-04-12 12:34:30.592954: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073B140000 of size 1179648
  847. 2018-04-12 12:34:30.596343: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073B260000 of size 1179648
  848. 2018-04-12 12:34:30.600236: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073B380000 of size 1024
  849. 2018-04-12 12:34:30.603953: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073B380400 of size 4096
  850. 2018-04-12 12:34:30.607216: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073B381400 of size 19200
  851. 2018-04-12 12:34:30.610924: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073B385F00 of size 256
  852. 2018-04-12 12:34:30.614188: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073B386000 of size 19200
  853. 2018-04-12 12:34:30.617640: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073B38AB00 of size 38400
  854. 2018-04-12 12:34:30.621495: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073B394100 of size 512
  855. 2018-04-12 12:34:30.624528: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073B394300 of size 4718592
  856. 2018-04-12 12:34:30.627895: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073B814300 of size 3276800
  857. 2018-04-12 12:34:30.632681: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073BB34300 of size 1024
  858. 2018-04-12 12:34:30.636160: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073BB34700 of size 38400
  859. 2018-04-12 12:34:30.640015: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073BB3DD00 of size 3276800
  860. 2018-04-12 12:34:30.643229: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073BE5DD00 of size 13107200
  861. 2018-04-12 12:34:30.647210: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073CADDD00 of size 67108864
  862. 2018-04-12 12:34:30.650834: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000740ADDD00 of size 75497472
  863. 2018-04-12 12:34:30.654192: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007452DDD00 of size 75497472
  864. 2018-04-12 12:34:30.658117: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000749ADDD00 of size 18874368
  865. 2018-04-12 12:34:30.661619: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000074ACDDD00 of size 3276800
  866. 2018-04-12 12:34:30.665912: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000074AFFDD00 of size 13107200
  867. 2018-04-12 12:34:30.669015: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000074BC7DD00 of size 67108864
  868. 2018-04-12 12:34:30.672162: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000074FC7DD00 of size 33554432
  869. 2018-04-12 12:34:30.676091: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000751C7DD00 of size 75497472
  870. 2018-04-12 12:34:30.679296: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000075647DD00 of size 18874368
  871. 2018-04-12 12:34:30.682798: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000075767DD00 of size 4718592
  872. 2018-04-12 12:34:30.686801: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000757AFDD00 of size 1179648
  873. 2018-04-12 12:34:30.690065: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000757C1DD00 of size 4718592
  874. 2018-04-12 12:34:30.694215: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000075809DD00 of size 3276800
  875. 2018-04-12 12:34:30.697956: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007583BDD00 of size 13107200
  876. 2018-04-12 12:34:30.701452: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000075903DD00 of size 67108864
  877. 2018-04-12 12:34:30.705417: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000075D03DD00 of size 1179648
  878. 2018-04-12 12:34:30.708441: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000075D15DD00 of size 33554432
  879. 2018-04-12 12:34:30.712218: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000075F15DD00 of size 75497472
  880. 2018-04-12 12:34:30.715782: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000076395DD00 of size 18874368
  881. 2018-04-12 12:34:30.719883: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 0000000764B5DD00 of size 256
  882. 2018-04-12 12:34:30.723779: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000764B5DE00 of size 256
  883. 2018-04-12 12:34:30.726979: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000764B5DF00 of size 50432
  884. 2018-04-12 12:34:30.730790: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000764B6A400 of size 512
  885. 2018-04-12 12:34:30.735013: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000764B6A600 of size 524288
  886. 2018-04-12 12:34:30.738166: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 0000000764BEA600 of size 256
  887. 2018-04-12 12:34:30.741737: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000764BEA700 of size 3276800
  888. 2018-04-12 12:34:30.744806: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000764F0A700 of size 1024
  889. 2018-04-12 12:34:30.748173: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 0000000764F0AB00 of size 256
  890. 2018-04-12 12:34:30.752164: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000764F0AC00 of size 13107200
  891. 2018-04-12 12:34:30.755330: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000765B8AC00 of size 2048
  892. 2018-04-12 12:34:30.759476: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 0000000765B8B400 of size 2048
  893. 2018-04-12 12:34:30.762521: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000765B8BC00 of size 2304
  894. 2018-04-12 12:34:30.766447: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000765B8C500 of size 2048
  895. 2018-04-12 12:34:30.770232: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000765B8CD00 of size 67108864
  896. 2018-04-12 12:34:30.773598: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000769B8CD00 of size 33554432
  897. 2018-04-12 12:34:30.778176: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000076BB8CD00 of size 65536
  898. 2018-04-12 12:34:30.783245: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000076BB9CD00 of size 75497472
  899. 2018-04-12 12:34:30.788084: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000077039CD00 of size 524288
  900. 2018-04-12 12:34:30.792117: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 000000077041CD00 of size 18874368
  901. 2018-04-12 12:34:30.795381: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000077161CD00 of size 262144
  902. 2018-04-12 12:34:30.799508: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000077165CD00 of size 1024
  903. 2018-04-12 12:34:30.802775: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000077165D100 of size 2048
  904. 2018-04-12 12:34:30.806344: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000077165D900 of size 65536
  905. 2018-04-12 12:34:30.809446: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 000000077166D900 of size 1280
  906. 2018-04-12 12:34:30.812900: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000077166DE00 of size 195328
  907. 2018-04-12 12:34:30.817197: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 000000077169D900 of size 256
  908. 2018-04-12 12:34:30.820346: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000077169DA00 of size 292096
  909. 2018-04-12 12:34:30.823443: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 00000007716E4F00 of size 4718592
  910. 2018-04-12 12:34:30.827148: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000771B64F00 of size 3276800
  911. 2018-04-12 12:34:30.830241: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000771E84F00 of size 262144
  912. 2018-04-12 12:34:30.834934: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000771EC4F00 of size 2048
  913. 2018-04-12 12:34:30.837862: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000771EC5700 of size 65536
  914. 2018-04-12 12:34:30.841076: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 0000000771ED5700 of size 153088
  915. 2018-04-12 12:34:30.845546: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000771EFAD00 of size 77568
  916. 2018-04-12 12:34:30.849467: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000771F0DC00 of size 8192
  917. 2018-04-12 12:34:30.853894: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000771F0FC00 of size 131072
  918. 2018-04-12 12:34:30.857238: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000771F2FC00 of size 131072
  919. 2018-04-12 12:34:30.862009: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 0000000771F4FC00 of size 3887872
  920. 2018-04-12 12:34:30.866745: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000772304F00 of size 512
  921. 2018-04-12 12:34:30.870623: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000772305100 of size 13107200
  922. 2018-04-12 12:34:30.873797: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000772F85100 of size 67108864
  923. 2018-04-12 12:34:30.877473: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000776F85100 of size 33554432
  924. 2018-04-12 12:34:30.882544: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000778F85100 of size 75497472
  925. 2018-04-12 12:34:30.886634: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000077D785100 of size 33554432
  926. 2018-04-12 12:34:30.890358: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000077F785100 of size 111927808
  927. 2018-04-12 12:34:30.893683: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:671] Summary of in-use Chunks by size:
  928. 2018-04-12 12:34:30.898008: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 55 Chunks of size 256 totalling 13.8KiB
  929. 2018-04-12 12:34:30.901703: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 9 Chunks of size 512 totalling 4.5KiB
  930. 2018-04-12 12:34:30.905148: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 15 Chunks of size 1024 totalling 15.0KiB
  931. 2018-04-12 12:34:30.908981: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 1280 totalling 1.3KiB
  932. 2018-04-12 12:34:30.912202: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 22 Chunks of size 2048 totalling 44.0KiB
  933. 2018-04-12 12:34:30.916331: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 2304 totalling 2.3KiB
  934. 2018-04-12 12:34:30.920083: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 8 Chunks of size 4096 totalling 32.0KiB
  935. 2018-04-12 12:34:30.923380: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 9 Chunks of size 8192 totalling 72.0KiB
  936. 2018-04-12 12:34:30.926962: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 7 Chunks of size 19200 totalling 131.3KiB
  937. 2018-04-12 12:34:30.930147: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 20736 totalling 20.3KiB
  938. 2018-04-12 12:34:30.934256: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 8 Chunks of size 38400 totalling 300.0KiB
  939. 2018-04-12 12:34:30.937838: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 49152 totalling 48.0KiB
  940. 2018-04-12 12:34:30.941185: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 50432 totalling 49.3KiB
  941. 2018-04-12 12:34:30.945047: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 10 Chunks of size 65536 totalling 640.0KiB
  942. 2018-04-12 12:34:30.948213: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 77568 totalling 75.8KiB
  943. 2018-04-12 12:34:30.951393: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 2 Chunks of size 131072 totalling 256.0KiB
  944. 2018-04-12 12:34:30.955500: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 195328 totalling 190.8KiB
  945. 2018-04-12 12:34:30.959135: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 2 Chunks of size 262144 totalling 512.0KiB
  946. 2018-04-12 12:34:30.963059: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 292096 totalling 285.3KiB
  947. 2018-04-12 12:34:30.966957: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 2 Chunks of size 524288 totalling 1.00MiB
  948. 2018-04-12 12:34:30.970957: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 597760 totalling 583.8KiB
  949. 2018-04-12 12:34:30.974120: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 7 Chunks of size 1179648 totalling 7.88MiB
  950. 2018-04-12 12:34:30.977449: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 2097152 totalling 2.00MiB
  951. 2018-04-12 12:34:30.982325: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 8 Chunks of size 3276800 totalling 25.00MiB
  952. 2018-04-12 12:34:30.985664: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 8 Chunks of size 4718592 totalling 36.00MiB
  953. 2018-04-12 12:34:30.989797: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 8388608 totalling 8.00MiB
  954. 2018-04-12 12:34:30.993195: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 8 Chunks of size 13107200 totalling 100.00MiB
  955. 2018-04-12 12:34:30.997009: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 8 Chunks of size 18874368 totalling 144.00MiB
  956. 2018-04-12 12:34:31.001188: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 11 Chunks of size 33554432 totalling 352.00MiB
  957. 2018-04-12 12:34:31.004396: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 8 Chunks of size 67108864 totalling 512.00MiB
  958. 2018-04-12 12:34:31.008228: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 10 Chunks of size 75497472 totalling 720.00MiB
  959. 2018-04-12 12:34:31.011420: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 111927808 totalling 106.74MiB
  960. 2018-04-12 12:34:31.014757: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:678] Sum Total of in-use chunks: 1.97GiB
  961. 2018-04-12 12:34:31.018444: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:680] Stats:
  962. Limit: 2143564595
  963. InUse: 2115835392
  964. MaxInUse: 2128993024
  965. NumAllocs: 854
  966. MaxAllocSize: 573792256
  967.  
  968. 2018-04-12 12:34:31.026747: W T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:279] **************************************************************************************************xx
  969. 2018-04-12 12:34:31.031040: W T:\src\github\tensorflow\tensorflow\core\framework\op_kernel.cc:1273] OP_REQUIRES failed at cwise_ops_common.cc:70 : Resource exhausted: OOM when allocating tensor with shape[32768,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
  970. Exception in thread Thread-1:
  971. Traceback (most recent call last):
  972. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\tensorflow\python\client\session.py", line 1327, in _do_call
  973. return fn(*args)
  974. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\tensorflow\python\client\session.py", line 1312, in _run_fn
  975. options, feed_dict, fetch_list, target_list, run_metadata)
  976. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\tensorflow\python\client\session.py", line 1420, in _call_tf_sessionrun
  977. status, run_metadata)
  978. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 516, in __exit__
  979. c_api.TF_GetCode(self.status.status))
  980. tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[2048,1024,3,3] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
  981. [[Node: training_1/Adam/gradients/model_1/conv2d_4/convolution_grad/Conv2DBackpropFilter = Conv2DBackpropFilter[T=DT_FLOAT, _class=["loc:@model_1/conv2d_4/convolution"], data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](training_1/Adam/gradients/model_1/conv2d_4/convolution_grad/Conv2DBackpropFilter-0-TransposeNHWCToNCHW-LayoutOptimizer, training_1/Adam/gradients/model_1/conv2d_4/convolution_grad/Const, training_1/Adam/gradients/AddN_3)]]
  982. Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
  983.  
  984.  
  985. During handling of the above exception, another exception occurred:
  986.  
  987. Traceback (most recent call last):
  988. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\threading.py", line 916, in _bootstrap_inner
  989. self.run()
  990. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\threading.py", line 864, in run
  991. self._target(*self._args, **self._kwargs)
  992. File "C:\OpenFaceSwap\faceswap\scripts\train.py", line 181, in processThread
  993. raise e
  994. File "C:\OpenFaceSwap\faceswap\scripts\train.py", line 161, in processThread
  995. trainer.train_one_step(epoch, self.show if (save_iteration or self.save_now) else None)
  996. File "C:\OpenFaceSwap\faceswap\plugins\Model_LowMem\Trainer.py", line 27, in train_one_step
  997. loss_B = self.model.autoencoder_B.train_on_batch(warped_B, target_B)
  998. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\keras\engine\training.py", line 1839, in train_on_batch
  999. outputs = self.train_function(ins)
  1000. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\keras\backend\tensorflow_backend.py", line 2357, in __call__
  1001. **self.session_kwargs)
  1002. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\tensorflow\python\client\session.py", line 905, in run
  1003. run_metadata_ptr)
  1004. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\tensorflow\python\client\session.py", line 1140, in _run
  1005. feed_dict_tensor, options, run_metadata)
  1006. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\tensorflow\python\client\session.py", line 1321, in _do_run
  1007. run_metadata)
  1008. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\tensorflow\python\client\session.py", line 1340, in _do_call
  1009. raise type(e)(node_def, op, message)
  1010. tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[2048,1024,3,3] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
  1011. [[Node: training_1/Adam/gradients/model_1/conv2d_4/convolution_grad/Conv2DBackpropFilter = Conv2DBackpropFilter[T=DT_FLOAT, _class=["loc:@model_1/conv2d_4/convolution"], data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](training_1/Adam/gradients/model_1/conv2d_4/convolution_grad/Conv2DBackpropFilter-0-TransposeNHWCToNCHW-LayoutOptimizer, training_1/Adam/gradients/model_1/conv2d_4/convolution_grad/Const, training_1/Adam/gradients/AddN_3)]]
  1012. Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
  1013.  
  1014.  
  1015. Caused by op 'training_1/Adam/gradients/model_1/conv2d_4/convolution_grad/Conv2DBackpropFilter', defined at:
  1016. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\threading.py", line 884, in _bootstrap
  1017. self._bootstrap_inner()
  1018. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\threading.py", line 916, in _bootstrap_inner
  1019. self.run()
  1020. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\threading.py", line 864, in run
  1021. self._target(*self._args, **self._kwargs)
  1022. File "C:\OpenFaceSwap\faceswap\scripts\train.py", line 161, in processThread
  1023. trainer.train_one_step(epoch, self.show if (save_iteration or self.save_now) else None)
  1024. File "C:\OpenFaceSwap\faceswap\plugins\Model_LowMem\Trainer.py", line 27, in train_one_step
  1025. loss_B = self.model.autoencoder_B.train_on_batch(warped_B, target_B)
  1026. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\keras\engine\training.py", line 1838, in train_on_batch
  1027. self._make_train_function()
  1028. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\keras\engine\training.py", line 990, in _make_train_function
  1029. loss=self.total_loss)
  1030. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\keras\legacy\interfaces.py", line 87, in wrapper
  1031. return func(*args, **kwargs)
  1032. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\keras\optimizers.py", line 415, in get_updates
  1033. grads = self.get_gradients(loss, params)
  1034. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\keras\optimizers.py", line 73, in get_gradients
  1035. grads = K.gradients(loss, params)
  1036. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\keras\backend\tensorflow_backend.py", line 2394, in gradients
  1037. return tf.gradients(loss, variables, colocate_gradients_with_ops=True)
  1038. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\tensorflow\python\ops\gradients_impl.py", line 488, in gradients
  1039. gate_gradients, aggregation_method, stop_gradients)
  1040. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\tensorflow\python\ops\gradients_impl.py", line 625, in _GradientsHelper
  1041. lambda: grad_fn(op, *out_grads))
  1042. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\tensorflow\python\ops\gradients_impl.py", line 379, in _MaybeCompile
  1043. return grad_fn() # Exit early
  1044. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\tensorflow\python\ops\gradients_impl.py", line 625, in <lambda>
  1045. lambda: grad_fn(op, *out_grads))
  1046. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\tensorflow\python\ops\nn_grad.py", line 523, in _Conv2DGrad
  1047. data_format=data_format)
  1048. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\tensorflow\python\ops\gen_nn_ops.py", line 1187, in conv2d_backprop_filter
  1049. dilations=dilations, name=name)
  1050. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
  1051. op_def=op_def)
  1052. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\tensorflow\python\framework\ops.py", line 3290, in create_op
  1053. op_def=op_def)
  1054. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\tensorflow\python\framework\ops.py", line 1654, in __init__
  1055. self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
  1056.  
  1057. ...which was originally created as op 'model_1/conv2d_4/convolution', defined at:
  1058. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\threading.py", line 884, in _bootstrap
  1059. self._bootstrap_inner()
  1060. [elided 1 identical lines from previous traceback]
  1061. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\threading.py", line 864, in run
  1062. self._target(*self._args, **self._kwargs)
  1063. File "C:\OpenFaceSwap\faceswap\scripts\train.py", line 147, in processThread
  1064. model = PluginLoader.get_model(trainer)(get_folder(self.arguments.model_dir), self.arguments.gpus)
  1065. File "C:\OpenFaceSwap\faceswap\plugins\Model_LowMem\AutoEncoder.py", line 16, in __init__
  1066. self.initModel()
  1067. File "C:\OpenFaceSwap\faceswap\plugins\Model_LowMem\Model.py", line 22, in initModel
  1068. self.autoencoder_A = KerasModel(x, self.decoder_A(self.encoder(x)))
  1069. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\keras\engine\topology.py", line 603, in __call__
  1070. output = self.call(inputs, **kwargs)
  1071. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\keras\engine\topology.py", line 2061, in call
  1072. output_tensors, _, _ = self.run_internal_graph(inputs, masks)
  1073. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\keras\engine\topology.py", line 2212, in run_internal_graph
  1074. output_tensors = _to_list(layer.call(computed_tensor, **kwargs))
  1075. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\keras\layers\convolutional.py", line 164, in call
  1076. dilation_rate=self.dilation_rate)
  1077. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\keras\backend\tensorflow_backend.py", line 3195, in conv2d
  1078. data_format=tf_data_format)
  1079. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 782, in convolution
  1080. return op(input, filter)
  1081. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 870, in __call__
  1082. return self.conv_op(inp, filter)
  1083. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 522, in __call__
  1084. return self.call(inp, filter)
  1085. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 206, in __call__
  1086. name=self.name)
  1087. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\tensorflow\python\ops\gen_nn_ops.py", line 1039, in conv2d
  1088. data_format=data_format, dilations=dilations, name=name)
  1089. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
  1090. op_def=op_def)
  1091. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\tensorflow\python\framework\ops.py", line 3290, in create_op
  1092. op_def=op_def)
  1093. File "C:\OpenFaceSwap\python\python-3.6.3.amd64\lib\site-packages\tensorflow\python\framework\ops.py", line 1654, in __init__
  1094. self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
  1095.  
  1096. ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[2048,1024,3,3] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
  1097. [[Node: training_1/Adam/gradients/model_1/conv2d_4/convolution_grad/Conv2DBackpropFilter = Conv2DBackpropFilter[T=DT_FLOAT, _class=["loc:@model_1/conv2d_4/convolution"], data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](training_1/Adam/gradients/model_1/conv2d_4/convolution_grad/Conv2DBackpropFilter-0-TransposeNHWCToNCHW-LayoutOptimizer, training_1/Adam/gradients/model_1/conv2d_4/convolution_grad/Const, training_1/Adam/gradients/AddN_3)]]
  1098. Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!