Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- 2020-06-30 23:15:59.776929: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
- 2020-06-30 23:15:59.796188: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
- 2020-06-30 23:15:59.796397: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
- name: Quadro T2000 major: 7 minor: 5 memoryClockRate(GHz): 1.62
- pciBusID: 0000:01:00.0
- 2020-06-30 23:15:59.796547: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
- 2020-06-30 23:15:59.797490: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
- 2020-06-30 23:15:59.798348: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
- 2020-06-30 23:15:59.798516: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
- 2020-06-30 23:15:59.799329: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
- 2020-06-30 23:15:59.799763: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
- 2020-06-30 23:15:59.801529: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
- 2020-06-30 23:15:59.801607: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
- 2020-06-30 23:15:59.801909: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
- 2020-06-30 23:15:59.802083: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
- 2020-06-30 23:15:59.802301: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
- 2020-06-30 23:15:59.818502: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2299965000 Hz
- 2020-06-30 23:15:59.819047: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55a6ae7b7070 executing computations on platform Host. Devices:
- 2020-06-30 23:15:59.819064: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): <undefined>, <undefined>
- 2020-06-30 23:15:59.819184: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
- 2020-06-30 23:15:59.819399: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
- name: Quadro T2000 major: 7 minor: 5 memoryClockRate(GHz): 1.62
- pciBusID: 0000:01:00.0
- 2020-06-30 23:15:59.819429: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
- 2020-06-30 23:15:59.819454: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
- 2020-06-30 23:15:59.819485: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
- 2020-06-30 23:15:59.819496: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
- 2020-06-30 23:15:59.819507: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
- 2020-06-30 23:15:59.819517: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
- 2020-06-30 23:15:59.819528: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
- 2020-06-30 23:15:59.819569: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
- 2020-06-30 23:15:59.819741: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
- 2020-06-30 23:15:59.819883: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
- 2020-06-30 23:15:59.819904: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
- 2020-06-30 23:15:59.924123: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
- 2020-06-30 23:15:59.924146: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0
- 2020-06-30 23:15:59.924153: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N
- 2020-06-30 23:15:59.924275: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
- 2020-06-30 23:15:59.924478: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
- 2020-06-30 23:15:59.924657: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
- 2020-06-30 23:15:59.924817: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1795 MB memory) -> physical GPU (device: 0, name: Quadro T2000, pci bus id: 0000:01:00.0, compute capability: 7.5)
- 2020-06-30 23:15:59.926087: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55a6b2233800 executing computations on platform CUDA. Devices:
- 2020-06-30 23:15:59.926098: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): Quadro T2000, Compute Capability 7.5
- 2020-06-30 23:16:00.032734: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set. If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU. To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
- 2020-06-30 23:16:00.500687: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
- 2020-06-30 23:16:00.500849: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
- name: Quadro T2000 major: 7 minor: 5 memoryClockRate(GHz): 1.62
- pciBusID: 0000:01:00.0
- 2020-06-30 23:16:00.500887: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
- 2020-06-30 23:16:00.500902: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
- 2020-06-30 23:16:00.500916: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
- 2020-06-30 23:16:00.500929: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
- 2020-06-30 23:16:00.500942: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
- 2020-06-30 23:16:00.500955: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
- 2020-06-30 23:16:00.500968: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
- 2020-06-30 23:16:00.501013: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
- 2020-06-30 23:16:00.501183: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
- 2020-06-30 23:16:00.501314: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
- 2020-06-30 23:16:00.501333: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
- 2020-06-30 23:16:00.501339: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0
- 2020-06-30 23:16:00.501344: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N
- 2020-06-30 23:16:00.501430: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
- 2020-06-30 23:16:00.501606: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
- 2020-06-30 23:16:00.501759: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1795 MB memory) -> physical GPU (device: 0, name: Quadro T2000, pci bus id: 0000:01:00.0, compute capability: 7.5)
- 2020-06-30 23:16:10.673804: W tensorflow/core/common_runtime/bfc_allocator.cc:314] Allocator (GPU_0_bfc) ran out of memory trying to allocate 512.00MiB (rounded to 536870912). Current allocation summary follows.
- 2020-06-30 23:16:10.673858: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (256): Total Chunks: 9, Chunks in use: 9. 2.2KiB allocated for chunks. 2.2KiB in use in bin. 44B client-requested in use in bin.
- 2020-06-30 23:16:10.673880: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (512): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
- 2020-06-30 23:16:10.673885: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (1024): Total Chunks: 1, Chunks in use: 1. 1.2KiB allocated for chunks. 1.2KiB in use in bin. 1.0KiB client-requested in use in bin.
- 2020-06-30 23:16:10.673890: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (2048): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
- 2020-06-30 23:16:10.673907: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (4096): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
- 2020-06-30 23:16:10.673911: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
- 2020-06-30 23:16:10.673937: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
- 2020-06-30 23:16:10.673957: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (32768): Total Chunks: 1, Chunks in use: 1. 62.5KiB allocated for chunks. 62.5KiB in use in bin. 62.5KiB client-requested in use in bin.
- 2020-06-30 23:16:10.673962: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (65536): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
- 2020-06-30 23:16:10.673989: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (131072): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
- 2020-06-30 23:16:10.673994: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (262144): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
- 2020-06-30 23:16:10.673999: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (524288): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
- 2020-06-30 23:16:10.674003: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (1048576): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
- 2020-06-30 23:16:10.674008: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (2097152): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
- 2020-06-30 23:16:10.674012: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (4194304): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
- 2020-06-30 23:16:10.674017: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
- 2020-06-30 23:16:10.674037: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
- 2020-06-30 23:16:10.674046: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (33554432): Total Chunks: 3, Chunks in use: 1. 133.56MiB allocated for chunks. 44.86MiB in use in bin. 44.86MiB client-requested in use in bin.
- 2020-06-30 23:16:10.674053: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
- 2020-06-30 23:16:10.674059: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
- 2020-06-30 23:16:10.674067: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (268435456): Total Chunks: 5, Chunks in use: 5. 1.62GiB allocated for chunks. 1.62GiB in use in bin. 1.50GiB client-requested in use in bin.
- 2020-06-30 23:16:10.674075: I tensorflow/core/common_runtime/bfc_allocator.cc:780] Bin for 512.00MiB was 256.00MiB, Chunk State:
- 2020-06-30 23:16:10.674081: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 1882783744
- 2020-06-30 23:16:10.674089: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2c62000000 next 10 of size 256
- 2020-06-30 23:16:10.674116: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2c62000100 next 11 of size 256
- 2020-06-30 23:16:10.674123: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2c62000200 next 9 of size 256
- 2020-06-30 23:16:10.674129: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2c62000300 next 8 of size 256
- 2020-06-30 23:16:10.674136: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2c62000400 next 7 of size 256
- 2020-06-30 23:16:10.674185: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2c62000500 next 6 of size 64000
- 2020-06-30 23:16:10.674207: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2c6200ff00 next 5 of size 256
- 2020-06-30 23:16:10.674211: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2c62010000 next 4 of size 256
- 2020-06-30 23:16:10.674215: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2c62010100 next 3 of size 256
- 2020-06-30 23:16:10.674218: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2c62010200 next 2 of size 256
- 2020-06-30 23:16:10.674223: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2c62010300 next 1 of size 1280
- 2020-06-30 23:16:10.674248: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free at 0x7f2c62010800 next 17 of size 45958656
- 2020-06-30 23:16:10.674252: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2c64be4e00 next 14 of size 47043072
- 2020-06-30 23:16:10.674272: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free at 0x7f2c678c2000 next 15 of size 47043072
- 2020-06-30 23:16:10.674276: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2c6a59f200 next 13 of size 268435456
- 2020-06-30 23:16:10.674279: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2c7a59f200 next 18 of size 268435456
- 2020-06-30 23:16:10.674283: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2c8a59f200 next 21 of size 268435456
- 2020-06-30 23:16:10.674287: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2c9a59f200 next 20 of size 268435456
- 2020-06-30 23:16:10.674290: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7f2caa59f200 next 18446744073709551615 of size 668929536
- 2020-06-30 23:16:10.674294: I tensorflow/core/common_runtime/bfc_allocator.cc:809] Summary of in-use Chunks by size:
- 2020-06-30 23:16:10.674300: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 9 Chunks of size 256 totalling 2.2KiB
- 2020-06-30 23:16:10.674304: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 1280 totalling 1.2KiB
- 2020-06-30 23:16:10.674308: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 64000 totalling 62.5KiB
- 2020-06-30 23:16:10.674311: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 47043072 totalling 44.86MiB
- 2020-06-30 23:16:10.674315: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 4 Chunks of size 268435456 totalling 1.00GiB
- 2020-06-30 23:16:10.674319: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 668929536 totalling 637.94MiB
- 2020-06-30 23:16:10.674335: I tensorflow/core/common_runtime/bfc_allocator.cc:816] Sum Total of in-use chunks: 1.67GiB
- 2020-06-30 23:16:10.674339: I tensorflow/core/common_runtime/bfc_allocator.cc:818] total_region_allocated_bytes_: 1882783744 memory_limit_: 1882783744 available bytes: 0 curr_region_allocation_bytes_: 3765567488
- 2020-06-30 23:16:10.674363: I tensorflow/core/common_runtime/bfc_allocator.cc:824] Stats:
- Limit: 1882783744
- InUse: 1789782016
- MaxInUse: 1789782016
- NumAllocs: 56
- MaxAllocSize: 668929536
- 2020-06-30 23:16:10.674370: W tensorflow/core/common_runtime/bfc_allocator.cc:319] *_***__**************************************************************************************xxxxxxx
- 2020-06-30 23:16:10.674391: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at fft_ops.cc:96 : Resource exhausted: OOM when allocating tensor with shape[16,2048,2048] and type complex64 on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
- [W 23:16:37.140 NotebookApp] 404 GET /api/kernels/16cb1fde-d900-4f24-9e1d-2f672c698619/channels?session_id=3cf932ea4fbc43238dc573e8f15f6a2e (::1): Kernel does not exist: 16cb1fde-d900-4f24-9e1d-2f672c698619
- [W 23:16:37.141 NotebookApp] 404 GET /api/kernels/16cb1fde-d900-4f24-9e1d-2f672c698619/channels?session_id=3cf932ea4fbc43238dc573e8f15f6a2e (::1) 1.70ms referer=None
Add Comment
Please, Sign In to add comment