API tools faq
paste
Advertisement
Guest User

CUDA Error in Tensorflow for Jetson Nano

a guest
Aug 8th, 2021
47
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 23.47 KB | None | 0 0
raw download clone embed print report
  1.  
  2. 2021-08-09 00:22:20.834807: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.10.2
  3. 2021-08-09 00:22:30.900789: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1
  4. 2021-08-09 00:22:30.940465: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
  5. 2021-08-09 00:22:30.940650: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1734] Found device 0 with properties:
  6. pciBusID: 0000:00:00.0 name: NVIDIA Tegra X1 computeCapability: 5.3
  7. coreClock: 0.9216GHz coreCount: 1 deviceMemorySize: 3.87GiB deviceMemoryBandwidth: 194.55MiB/s
  8. 2021-08-09 00:22:30.940764: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.10.2
  9. 2021-08-09 00:22:31.108329: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.10
  10. 2021-08-09 00:22:31.108637: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.10
  11. 2021-08-09 00:22:31.183954: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcufft.so.10
  12. 2021-08-09 00:22:31.294431: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcurand.so.10
  13. 2021-08-09 00:22:31.418350: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusolver.so.10
  14. 2021-08-09 00:22:31.502921: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusparse.so.10
  15. 2021-08-09 00:22:31.506656: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
  16. 2021-08-09 00:22:31.507015: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
  17. 2021-08-09 00:22:31.507369: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
  18. 2021-08-09 00:22:31.507514: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1872] Adding visible gpu devices: 0
  19. 2021-08-09 00:22:31.737166: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
  20. 2021-08-09 00:22:31.737323: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1734] Found device 0 with properties:
  21. pciBusID: 0000:00:00.0 name: NVIDIA Tegra X1 computeCapability: 5.3
  22. coreClock: 0.9216GHz coreCount: 1 deviceMemorySize: 3.87GiB deviceMemoryBandwidth: 194.55MiB/s
  23. 2021-08-09 00:22:31.737640: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
  24. 2021-08-09 00:22:31.737892: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
  25. 2021-08-09 00:22:31.738041: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1872] Adding visible gpu devices: 0
  26. 2021-08-09 00:22:31.738199: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.10.2
  27. 2021-08-09 00:22:37.375891: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
  28. 2021-08-09 00:22:37.376070: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264] 0
  29. 2021-08-09 00:22:37.376114: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0: N
  30. 2021-08-09 00:22:37.376550: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
  31. 2021-08-09 00:22:37.376892: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
  32. 2021-08-09 00:22:37.377164: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
  33. 2021-08-09 00:22:37.377302: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1024 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
  34. 2021-08-09 00:22:50.220662: W tensorflow/core/common_runtime/bfc_allocator.cc:456] Allocator (GPU_0_bfc) ran out of memory trying to allocate 392.00MiB (rounded to 411041792)requested by op Add
  35. If the cause is memory fragmentation maybe the environment variable 'TF_GPU_ALLOCATOR=cuda_malloc_async' will improve the situation.
  36. Current allocation summary follows.
  37. Current allocation summary follows.
  38. 2021-08-09 00:22:50.220856: I tensorflow/core/common_runtime/bfc_allocator.cc:991] BFCAllocator dump for GPU_0_bfc
  39. 2021-08-09 00:22:50.220892: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (256): Total Chunks: 22, Chunks in use: 22. 5.5KiB allocated for chunks. 5.5KiB in use in bin. 596B client-requested in use in bin.
  40. 2021-08-09 00:22:50.220934: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (512): Total Chunks: 2, Chunks in use: 2. 1.0KiB allocated for chunks. 1.0KiB in use in bin. 1.0KiB client-requested in use in bin.
  41. 2021-08-09 00:22:50.220971: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (1024): Total Chunks: 4, Chunks in use: 4. 4.8KiB allocated for chunks. 4.8KiB in use in bin. 4.0KiB client-requested in use in bin.
  42. 2021-08-09 00:22:50.221004: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (2048): Total Chunks: 6, Chunks in use: 6. 13.0KiB allocated for chunks. 13.0KiB in use in bin. 12.0KiB client-requested in use in bin.
  43. 2021-08-09 00:22:50.221043: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (4096): Total Chunks: 1, Chunks in use: 1. 6.8KiB allocated for chunks. 6.8KiB in use in bin. 6.8KiB client-requested in use in bin.
  44. 2021-08-09 00:22:50.221078: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
  45. 2021-08-09 00:22:50.221108: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
  46. 2021-08-09 00:22:50.221145: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (32768): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
  47. 2021-08-09 00:22:50.221183: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (65536): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
  48. 2021-08-09 00:22:50.221219: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (131072): Total Chunks: 1, Chunks in use: 1. 144.0KiB allocated for chunks. 144.0KiB in use in bin. 144.0KiB client-requested in use in bin.
  49. 2021-08-09 00:22:50.221263: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (262144): Total Chunks: 3, Chunks in use: 1. 855.5KiB allocated for chunks. 288.0KiB in use in bin. 288.0KiB client-requested in use in bin.
  50. 2021-08-09 00:22:50.221312: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (524288): Total Chunks: 1, Chunks in use: 1. 576.0KiB allocated for chunks. 576.0KiB in use in bin. 576.0KiB client-requested in use in bin.
  51. 2021-08-09 00:22:50.221355: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (1048576): Total Chunks: 3, Chunks in use: 1. 3.38MiB allocated for chunks. 1.12MiB in use in bin. 1.12MiB client-requested in use in bin.
  52. 2021-08-09 00:22:50.221391: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (2097152): Total Chunks: 2, Chunks in use: 2. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
  53. 2021-08-09 00:22:50.221433: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (4194304): Total Chunks: 3, Chunks in use: 1. 13.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
  54. 2021-08-09 00:22:50.221484: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (8388608): Total Chunks: 5, Chunks in use: 5. 45.00MiB allocated for chunks. 45.00MiB in use in bin. 45.00MiB client-requested in use in bin.
  55. 2021-08-09 00:22:50.221527: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (16777216): Total Chunks: 1, Chunks in use: 0. 18.00MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
  56. 2021-08-09 00:22:50.221566: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
  57. 2021-08-09 00:22:50.221604: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
  58. 2021-08-09 00:22:50.221638: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (134217728): Total Chunks: 1, Chunks in use: 0. 154.06MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
  59. 2021-08-09 00:22:50.221684: I tensorflow/core/common_runtime/bfc_allocator.cc:998] Bin (268435456): Total Chunks: 2, Chunks in use: 2. 784.00MiB allocated for chunks. 784.00MiB in use in bin. 784.00MiB client-requested in use in bin.
  60. 2021-08-09 00:22:50.221766: I tensorflow/core/common_runtime/bfc_allocator.cc:1014] Bin for 392.00MiB was 256.00MiB, Chunk State:
  61. 2021-08-09 00:22:50.221803: I tensorflow/core/common_runtime/bfc_allocator.cc:1027] Next region of size 1073741824
  62. 2021-08-09 00:22:50.221853: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f00970000 of size 1280 by op ScratchBuffer action_count 3921231748848353800 step 0 next 1
  63. 2021-08-09 00:22:50.221893: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f00970500 of size 256 by op AssignVariableOp action_count 3921231748848353801 step 0 next 2
  64. 2021-08-09 00:22:50.221985: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f00970600 of size 256 by op Fill action_count 3921231748848353811 step 0 next 6
  65. 2021-08-09 00:22:50.222025: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f00970700 of size 256 by op Fill action_count 3921231748848353812 step 0 next 9
  66. 2021-08-09 00:22:50.222059: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f00970800 of size 256 by op Sub action_count 3921231748848353814 step 0 next 11
  67. 2021-08-09 00:22:50.222093: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f00970900 of size 256 by op Sub action_count 3921231748848353815 step 0 next 12
  68. 2021-08-09 00:22:50.222128: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f00970a00 of size 256 by op Fill action_count 3921231748848353822 step 0 next 10
  69. 2021-08-09 00:22:50.222161: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f00970b00 of size 256 by op Sub action_count 3921231748848353824 step 0 next 15
  70. 2021-08-09 00:22:50.222194: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f00970c00 of size 256 by op Sub action_count 3921231748848353825 step 0 next 16
  71. 2021-08-09 00:22:50.222228: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f00970d00 of size 512 by op Fill action_count 3921231748848353832 step 0 next 19
  72. 2021-08-09 00:22:50.222262: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f00970f00 of size 256 by op Sub action_count 3921231748848353834 step 0 next 21
  73. 2021-08-09 00:22:50.222295: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f00971000 of size 256 by op Sub action_count 3921231748848353835 step 0 next 22
  74. 2021-08-09 00:22:50.222329: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f00971100 of size 512 by op Fill action_count 3921231748848353842 step 0 next 20
  75. 2021-08-09 00:22:50.222363: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f00971300 of size 256 by op Sub action_count 3921231748848353844 step 0 next 25
  76. 2021-08-09 00:22:50.222396: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f00971400 of size 256 by op Sub action_count 3921231748848353845 step 0 next 26
  77. 2021-08-09 00:22:50.222429: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f00971500 of size 1024 by op Fill action_count 3921231748848353852 step 0 next 29
  78. 2021-08-09 00:22:50.222462: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f00971900 of size 256 by op Sub action_count 3921231748848353854 step 0 next 31
  79. 2021-08-09 00:22:50.222495: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f00971a00 of size 256 by op Sub action_count 3921231748848353855 step 0 next 32
  80. 2021-08-09 00:22:50.222529: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f00971b00 of size 1536 by op Fill action_count 3921231748848353862 step 0 next 3
  81. 2021-08-09 00:22:50.222564: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f00972100 of size 256 by op Sub action_count 3921231748848353803 step 0 next 4
  82. 2021-08-09 00:22:50.222599: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f00972200 of size 256 by op Sub action_count 3921231748848353804 step 0 next 5
  83. 2021-08-09 00:22:50.222632: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f00972300 of size 1024 by op Fill action_count 3921231748848353870 step 0 next 30
  84. 2021-08-09 00:22:50.222664: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f00972700 of size 256 by op Sub action_count 3921231748848353872 step 0 next 36
  85. 2021-08-09 00:22:50.222697: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f00972800 of size 256 by op Sub action_count 3921231748848353873 step 0 next 37
  86. 2021-08-09 00:22:50.222730: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f00972900 of size 2048 by op Fill action_count 3921231748848353880 step 0 next 40
  87. 2021-08-09 00:22:50.222764: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f00973100 of size 256 by op Sub action_count 3921231748848353882 step 0 next 42
  88. 2021-08-09 00:22:50.222797: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f00973200 of size 256 by op Sub action_count 3921231748848353883 step 0 next 43
  89. 2021-08-09 00:22:50.222830: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f00973300 of size 3072 by op Fill action_count 3921231748848353890 step 0 next 7
  90. 2021-08-09 00:22:50.222866: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f00973f00 of size 6912 by op Add action_count 3921231748848353808 step 0 next 8
  91. 2021-08-09 00:22:50.222901: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f00975a00 of size 2048 by op Fill action_count 3921231748848353898 step 0 next 41
  92. 2021-08-09 00:22:50.222935: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f00976200 of size 2048 by op Fill action_count 3921231748848353906 step 0 next 47
  93. 2021-08-09 00:22:50.222969: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f00976a00 of size 2048 by op Fill action_count 3921231748848353914 step 0 next 49
  94. 2021-08-09 00:22:50.223002: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f00977200 of size 2048 by op Fill action_count 3921231748848353922 step 0 next 51
  95. 2021-08-09 00:22:50.223036: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f00977a00 of size 256 by op Sub action_count 3921231748848353924 step 0 next 54
  96. 2021-08-09 00:22:50.223064: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f00977b00 of size 256 by op Sub action_count 3921231748848353925 step 0 next 55
  97. 2021-08-09 00:22:50.223098: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Free at f00977c00 of size 286208 by op UNUSED action_count 3921231748848353928 step 0 next 14
  98. 2021-08-09 00:22:50.223132: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f009bda00 of size 147456 by op Add action_count 3921231748848353819 step 0 next 13
  99. 2021-08-09 00:22:50.223162: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Free at f009e1a00 of size 294912 by op UNUSED action_count 3921231748848353830 step 0 next 18
  100. 2021-08-09 00:22:50.223200: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f00a29a00 of size 294912 by op Add action_count 3921231748848353829 step 0 next 17
  101. 2021-08-09 00:22:50.223235: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Free at f00a71a00 of size 1179648 by op UNUSED action_count 3921231748848353851 step 0 next 24
  102. 2021-08-09 00:22:50.223265: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f00b91a00 of size 589824 by op Add action_count 3921231748848353839 step 0 next 23
  103. 2021-08-09 00:22:50.223297: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Free at f00c21a00 of size 1179648 by op UNUSED action_count 3921231748848353850 step 0 next 28
  104. 2021-08-09 00:22:50.223332: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f00d41a00 of size 1179648 by op Add action_count 3921231748848353849 step 0 next 27
  105. 2021-08-09 00:22:50.223365: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Free at f00e61a00 of size 4718592 by op UNUSED action_count 3921231748848353879 step 0 next 34
  106. 2021-08-09 00:22:50.223398: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f012e1a00 of size 2359296 by op Add action_count 3921231748848353859 step 0 next 33
  107. 2021-08-09 00:22:50.223433: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f01521a00 of size 2359296 by op Add action_count 3921231748848353867 step 0 next 35
  108. 2021-08-09 00:22:50.223468: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Free at f01761a00 of size 4718592 by op UNUSED action_count 3921231748848353878 step 0 next 39
  109. 2021-08-09 00:22:50.223502: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f01be1a00 of size 4718592 by op Add action_count 3921231748848353877 step 0 next 38
  110. 2021-08-09 00:22:50.223535: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Free at f02061a00 of size 18874368 by op UNUSED action_count 3921231748848353921 step 0 next 45
  111. 2021-08-09 00:22:50.223568: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f03261a00 of size 9437184 by op Add action_count 3921231748848353887 step 0 next 44
  112. 2021-08-09 00:22:50.223602: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f03b61a00 of size 9437184 by op Add action_count 3921231748848353895 step 0 next 46
  113. 2021-08-09 00:22:50.223635: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f04461a00 of size 9437184 by op Add action_count 3921231748848353903 step 0 next 48
  114. 2021-08-09 00:22:50.223669: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f04d61a00 of size 9437184 by op Add action_count 3921231748848353911 step 0 next 50
  115. 2021-08-09 00:22:50.223704: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f05661a00 of size 9437184 by op Add action_count 3921231748848353919 step 0 next 52
  116. 2021-08-09 00:22:50.223741: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f05f61a00 of size 411041792 by op RandomUniform action_count 3921231748848353923 step 0 next 53
  117. 2021-08-09 00:22:50.223775: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at f1e761a00 of size 411041792 by op Mul action_count 3921231748848353927 step 0 next 57
  118. 2021-08-09 00:22:50.223809: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Free at f36f61a00 of size 161539584 by op UNUSED action_count 0 step 0 next 18446744073709551615
  119. 2021-08-09 00:22:50.223839: I tensorflow/core/common_runtime/bfc_allocator.cc:1051] Summary of in-use Chunks by size:
  120. 2021-08-09 00:22:50.223877: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 22 Chunks of size 256 totalling 5.5KiB
  121. 2021-08-09 00:22:50.223914: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 2 Chunks of size 512 totalling 1.0KiB
  122. 2021-08-09 00:22:50.223949: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 2 Chunks of size 1024 totalling 2.0KiB
  123. 2021-08-09 00:22:50.223983: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 1280 totalling 1.2KiB
  124. 2021-08-09 00:22:50.224019: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 1536 totalling 1.5KiB
  125. 2021-08-09 00:22:50.224054: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 5 Chunks of size 2048 totalling 10.0KiB
  126. 2021-08-09 00:22:50.224089: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 3072 totalling 3.0KiB
  127. 2021-08-09 00:22:50.224129: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 6912 totalling 6.8KiB
  128. 2021-08-09 00:22:50.224166: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 147456 totalling 144.0KiB
  129. 2021-08-09 00:22:50.224197: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 294912 totalling 288.0KiB
  130. 2021-08-09 00:22:50.224234: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 589824 totalling 576.0KiB
  131. 2021-08-09 00:22:50.224269: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 1179648 totalling 1.12MiB
  132. 2021-08-09 00:22:50.224303: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 2 Chunks of size 2359296 totalling 4.50MiB
  133. 2021-08-09 00:22:50.224338: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 4718592 totalling 4.50MiB
  134. 2021-08-09 00:22:50.224374: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 5 Chunks of size 9437184 totalling 45.00MiB
  135. 2021-08-09 00:22:50.224411: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 2 Chunks of size 411041792 totalling 784.00MiB
  136. 2021-08-09 00:22:50.224448: I tensorflow/core/common_runtime/bfc_allocator.cc:1058] Sum Total of in-use chunks: 840.14MiB
  137. 2021-08-09 00:22:50.224480: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] total_region_allocated_bytes_: 1073741824 memory_limit_: 1073741824 available bytes: 0 curr_region_allocation_bytes_: 2147483648
  138. 2021-08-09 00:22:50.224526: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Stats:
  139. Limit: 1073741824
  140. InUse: 880950272
  141. MaxInUse: 880950528
  142. NumAllocs: 89
  143. MaxAllocSize: 411041792
  144. Reserved: 0
  145. PeakReserved: 0
  146. LargestFreeBlock: 0
  147.  
  148. 2021-08-09 00:22:50.224593: W tensorflow/core/common_runtime/bfc_allocator.cc:467] *************************************************************************************_______________
  149. 2021-08-09 00:22:50.346121: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:818] failed to record completion event; therefore, failed to create inter-stream dependency
  150. 2021-08-09 00:22:50.346237: E tensorflow/stream_executor/cuda/cuda_driver.cc:1175] failed to enqueue async memcpy from host to device: CUDA_ERROR_LAUNCH_FAILED: unspecified launch failure; GPU dst: 0xf0097bc00; host src: 0x20175600; size: 4=0x4
  151. 2021-08-09 00:22:50.361214: E tensorflow/stream_executor/stream.cc:356] Error recording event in stream: Error recording CUDA event: CUDA_ERROR_LAUNCH_FAILED: unspecified launch failure; not marking stream as bad, as the Event object may be at fault. Monitor for further errors.
  152. 2021-08-09 00:22:50.361288: E tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_LAUNCH_FAILED: unspecified launch failure
  153. 2021-08-09 00:22:50.361381: F tensorflow/core/common_runtime/device/device_event_mgr.cc:221] Unexpected Event status: 1
  154. Aborted (core dumped)
  155.  
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!