Advertisement
Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- mona@pascal:~/computer_vision/VPilot$ python train.py
- Using TensorFlow backend.
- I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so.8.0 locally
- I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcudnn.so.5.0 locally
- I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so.8.0 locally
- I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally
- I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so.8.0 locally
- /usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py:1938: UserWarning: Expected no kwargs, you passed 1
- kwargs passed to function are ignored with Tensorflow backend
- warnings.warn('\n'.join(msg))
- Epoch 1/1000
- I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 0 with properties:
- name: Tesla K40c
- major: 3 minor: 5 memoryClockRate (GHz) 0.8755
- pciBusID 0000:03:00.0
- Total memory: 11.92GiB
- Free memory: 11.85GiB
- W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x4750d80
- I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 1 with properties:
- name: Tesla K40c
- major: 3 minor: 5 memoryClockRate (GHz) 0.8755
- pciBusID 0000:83:00.0
- Total memory: 11.92GiB
- Free memory: 11.85GiB
- I tensorflow/core/common_runtime/gpu/gpu_device.cc:855] cannot enable peer access from device ordinal 0 to device ordinal 1
- I tensorflow/core/common_runtime/gpu/gpu_device.cc:855] cannot enable peer access from device ordinal 1 to device ordinal 0
- I tensorflow/core/common_runtime/gpu/gpu_device.cc:972] DMA: 0 1
- I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] 0: Y N
- I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] 1: N Y
- I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Tesla K40c, pci bus id: 0000:03:00.0)
- I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Creating TensorFlow device (/gpu:1) -> (device: 1, name: Tesla K40c, pci bus id: 0000:83:00.0)
- I tensorflow/core/common_runtime/gpu/pool_allocator.cc:245] PoolAllocator: After 4777 get requests, put_count=3270 evicted_count=1000 eviction_rate=0.30581 and unsatisfied allocation rate=0.54574
- I tensorflow/core/common_runtime/gpu/pool_allocator.cc:257] Raising pool_size_limit_ from 100 to 110
- 4/70629 [..............................] - ETA: 364851s - loss: 0.5890I tensorflow/core/common_runtime/gpu/pool_allocator.cc:245] PoolAllocator: After 755 get requests, put_count=1771 evicted_count=1000 eviction_rate=0.564653 and unsatisfied allocation rate=0
- 8/70629 [..............................] - ETA: 194931s - loss: 0.5553I tensorflow/core/common_runtime/gpu/pool_allocator.cc:245] PoolAllocator: After 247 get requests, put_count=1270 evicted_count=1000 eviction_rate=0.787402 and unsatisfied allocation rate=0
- 13/70629 [..............................] - ETA: 129454s - loss: 0.5582I tensorflow/core/common_runtime/gpu/pool_allocator.cc:245] PoolAllocator: After 5071 get requests, put_count=4961 evicted_count=2000 eviction_rate=0.403145 and unsatisfied allocation rate=0.423979
- I tensorflow/core/common_runtime/gpu/pool_allocator.cc:257] Raising pool_size_limit_ from 449 to 493
- 18/70629 [..............................] - ETA: 100341s - loss: 0.5194I tensorflow/core/common_runtime/gpu/pool_allocator.cc:245] PoolAllocator: After 5145 get requests, put_count=5327 evicted_count=2000 eviction_rate=0.375446 and unsatisfied allocation rate=0.365986
- I tensorflow/core/common_runtime/gpu/pool_allocator.cc:257] Raising pool_size_limit_ from 720 to 792
- 25/70629 [..............................] - ETA: 79355s - loss: 0.5875I tensorflow/core/common_runtime/gpu/pool_allocator.cc:245] PoolAllocator: After 5137 get requests, put_count=5388 evicted_count=1000 eviction_rate=0.185598 and unsatisfied allocation rate=0.175784
- I tensorflow/core/common_runtime/gpu/pool_allocator.cc:257] Raising pool_size_limit_ from 1694 to 1863
- 70629/70629 [==============================] - 25358s - loss: nan - val_loss: nan
- Epoch 2/1000
- 70629/70629 [==============================] - 24899s - loss: nan - val_loss: nan
- Epoch 3/1000
- 70629/70629 [==============================] - 24967s - loss: nan - val_loss: nan
- Epoch 4/1000
- 70629/70629 [==============================] - 24987s - loss: nan - val_loss: nan
- Epoch 5/1000
- 70629/70629 [==============================] - 24855s - loss: nan - val_loss: nan
- Epoch 6/1000
- 70629/70629 [==============================] - 24977s - loss: nan - val_loss: nan
- Epoch 7/1000
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement