ning training
2019-04-23 02:25:41.950021: W tensorflow/core/common_runtime/bfc_allocator.cc:215] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.28GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-04-23 02:25:41.950077: W tensorflow/core/common_runtime/bfc_allocator.cc:215] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.28GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-04-23 02:25:42.608347: W tensorflow/core/common_runtime/bfc_allocator.cc:215] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.09GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-04-23 02:25:42.608375: W tensorflow/core/common_runtime/bfc_allocator.cc:215] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.09GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-04-23 02:25:42.699015: W tensorflow/core/common_runtime/bfc_allocator.cc:215] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.38GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-04-23 02:25:42.699066: W tensorflow/core/common_runtime/bfc_allocator.cc:215] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.38GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-04-23 02:25:43.010736: W tensorflow/core/common_runtime/bfc_allocator.cc:215] Allocator (GPU_0_bfc) ran out of memory trying to allocate 340.05MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-04-23 02:25:43.010774: W tensorflow/core/common_runtime/bfc_allocator.cc:215] Allocator (GPU_0_bfc) ran out of memory trying to allocate 340.05MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-04-23 02:25:43.010791: W tensorflow/core/common_runtime/bfc_allocator.cc:215] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.17GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-04-23 02:25:43.010803: W tensorflow/core/common_runtime/bfc_allocator.cc:215] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.17GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-04-23 02:25:53.190433: W tensorflow/core/common_runtime/bfc_allocator.cc:271] Allocator (GPU_0_bfc) ran out of memory trying to allocate 346.88MiB.  Current allocation summary follows.
2019-04-23 02:25:53.190545: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (256): 	Total Chunks: 166, Chunks in use: 165. 41.5KiB allocated for chunks. 41.2KiB in use in bin. 3.5KiB client-requested in use in bin.
2019-04-23 02:25:53.190588: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (512): 	Total Chunks: 16, Chunks in use: 16. 8.0KiB allocated for chunks. 8.0KiB in use in bin. 8.0KiB client-requested in use in bin.
2019-04-23 02:25:53.190628: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (1024): 	Total Chunks: 17, Chunks in use: 17. 17.2KiB allocated for chunks. 17.2KiB in use in bin. 17.0KiB client-requested in use in bin.
2019-04-23 02:25:53.190668: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (2048): 	Total Chunks: 30, Chunks in use: 30. 60.5KiB allocated for chunks. 60.5KiB in use in bin. 60.0KiB client-requested in use in bin.
2019-04-23 02:25:53.190702: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (4096): 	Total Chunks: 10, Chunks in use: 9. 53.2KiB allocated for chunks. 47.0KiB in use in bin. 47.0KiB client-requested in use in bin.
2019-04-23 02:25:53.190744: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (8192): 	Total Chunks: 6, Chunks in use: 6. 59.2KiB allocated for chunks. 59.2KiB in use in bin. 56.8KiB client-requested in use in bin.
2019-04-23 02:25:53.190778: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (16384): 	Total Chunks: 1, Chunks in use: 0. 30.5KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-04-23 02:25:53.190803: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (32768): 	Total Chunks: 1, Chunks in use: 0. 62.0KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-04-23 02:25:53.190828: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (65536): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-04-23 02:25:53.190863: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (131072): 	Total Chunks: 5, Chunks in use: 5. 720.0KiB allocated for chunks. 720.0KiB in use in bin. 720.0KiB client-requested in use in bin.
2019-04-23 02:25:53.190890: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (262144): 	Total Chunks: 6, Chunks in use: 5. 1.77MiB allocated for chunks. 1.48MiB in use in bin. 1.41MiB client-requested in use in bin.
2019-04-23 02:25:53.190924: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (524288): 	Total Chunks: 11, Chunks in use: 10. 5.88MiB allocated for chunks. 5.31MiB in use in bin. 5.31MiB client-requested in use in bin.
2019-04-23 02:25:53.190957: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (1048576): 	Total Chunks: 6, Chunks in use: 5. 6.75MiB allocated for chunks. 5.62MiB in use in bin. 5.62MiB client-requested in use in bin.
2019-04-23 02:25:53.190993: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (2097152): 	Total Chunks: 10, Chunks in use: 10. 22.50MiB allocated for chunks. 22.50MiB in use in bin. 22.50MiB client-requested in use in bin.
2019-04-23 02:25:53.191030: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (4194304): 	Total Chunks: 6, Chunks in use: 5. 27.00MiB allocated for chunks. 22.50MiB in use in bin. 22.50MiB client-requested in use in bin.
2019-04-23 02:25:53.191067: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (8388608): 	Total Chunks: 22, Chunks in use: 21. 198.00MiB allocated for chunks. 189.00MiB in use in bin. 189.00MiB client-requested in use in bin.
2019-04-23 02:25:53.191099: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (16777216): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-04-23 02:25:53.191129: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (33554432): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-04-23 02:25:53.191168: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (67108864): 	Total Chunks: 7, Chunks in use: 6. 497.92MiB allocated for chunks. 425.92MiB in use in bin. 425.92MiB client-requested in use in bin.
2019-04-23 02:25:53.191199: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (134217728): 	Total Chunks: 2, Chunks in use: 1. 325.18MiB allocated for chunks. 173.44MiB in use in bin. 173.44MiB client-requested in use in bin.
2019-04-23 02:25:53.191226: I tensorflow/core/common_runtime/bfc_allocator.cc:610] Bin (268435456): 	Total Chunks: 7, Chunks in use: 7. 5.15GiB allocated for chunks. 5.15GiB in use in bin. 5.14GiB client-requested in use in bin.
2019-04-23 02:25:53.191261: I tensorflow/core/common_runtime/bfc_allocator.cc:626] Bin for 346.88MiB was 256.00MiB, Chunk State: