Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- mona@pascal:~/computer_vision/VPilot$ python train.py
- Using TensorFlow backend.
- I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so.8.0 locally
- I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcudnn.so.5.0 locally
- I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so.8.0 locally
- I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally
- I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so.8.0 locally
- /usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py:1938: UserWarning: Expected no kwargs, you passed 1
- kwargs passed to function are ignored with Tensorflow backend
- warnings.warn('\n'.join(msg))
- Epoch 1/1000
- I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 0 with properties:
- name: Tesla K40c
- major: 3 minor: 5 memoryClockRate (GHz) 0.8755
- pciBusID 0000:03:00.0
- Total memory: 11.92GiB
- Free memory: 11.85GiB
- W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x47913e0
- I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 1 with properties:
- name: Tesla K40c
- major: 3 minor: 5 memoryClockRate (GHz) 0.8755
- pciBusID 0000:83:00.0
- Total memory: 11.92GiB
- Free memory: 11.85GiB
- I tensorflow/core/common_runtime/gpu/gpu_device.cc:855] cannot enable peer access from device ordinal 0 to device ordinal 1
- I tensorflow/core/common_runtime/gpu/gpu_device.cc:855] cannot enable peer access from device ordinal 1 to device ordinal 0
- I tensorflow/core/common_runtime/gpu/gpu_device.cc:972] DMA: 0 1
- I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] 0: Y N
- I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] 1: N Y
- I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Tesla K40c, pci bus id: 0000:03:00.0)
- I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Creating TensorFlow device (/gpu:1) -> (device: 1, name: Tesla K40c, pci bus id: 0000:83:00.0)
- I tensorflow/core/common_runtime/gpu/pool_allocator.cc:245] PoolAllocator: After 4777 get requests, put_count=3272 evicted_count=1000 eviction_rate=0.305623 and unsatisfied allocation rate=0.545321
- I tensorflow/core/common_runtime/gpu/pool_allocator.cc:257] Raising pool_size_limit_ from 100 to 110
- 4/17035 [..............................] - ETA: 34257s - loss: 0.3268I tensorflow/core/common_runtime/gpu/pool_allocator.cc:245] PoolAllocator: After 1090 get requests, put_count=1951 evicted_count=1000 eviction_rate=0.512558 and unsatisfied allocation rate=0.142202
- I tensorflow/core/common_runtime/gpu/pool_allocator.cc:257] Raising pool_size_limit_ from 176 to 193
- 8/17035 [..............................] - ETA: 20115s - loss: 0.4547I tensorflow/core/common_runtime/gpu/pool_allocator.cc:245] PoolAllocator: After 247 get requests, put_count=1272 evicted_count=1000 eviction_rate=0.786164 and unsatisfied allocation rate=0
- 13/17035 [..............................] - ETA: 14711s - loss: 0.5380I tensorflow/core/common_runtime/gpu/pool_allocator.cc:245] PoolAllocator: After 5059 get requests, put_count=4904 evicted_count=2000 eviction_rate=0.40783 and unsatisfied allocation rate=0.433287
- I tensorflow/core/common_runtime/gpu/pool_allocator.cc:257] Raising pool_size_limit_ from 409 to 449
- 18/17035 [..............................] - ETA: 12283s - loss: 0.4809I tensorflow/core/common_runtime/gpu/pool_allocator.cc:245] PoolAllocator: After 4903 get requests, put_count=4087 evicted_count=1000 eviction_rate=0.244678 and unsatisfied allocation rate=0.383643
- I tensorflow/core/common_runtime/gpu/pool_allocator.cc:257] Raising pool_size_limit_ from 720 to 792
- 24/17035 [..............................] - ETA: 10673s - loss: 0.4934I tensorflow/core/common_runtime/gpu/pool_allocator.cc:245] PoolAllocator: After 241 get requests, put_count=1380 evicted_count=1000 eviction_rate=0.724638 and unsatisfied allocation rate=0.00414938
- I tensorflow/core/common_runtime/gpu/pool_allocator.cc:257] Raising pool_size_limit_ from 1540 to 1694
- 17035/17035 [==============================] - 6235s - loss: 0.1774 - val_loss: 0.1062
- Epoch 2/1000
- 17035/17035 [==============================] - 6092s - loss: nan - val_loss: nan
- Epoch 3/1000
- 11030/17035 [==================>...........] - ETA: 2007s - loss: nan^Z
- [4]+ Stopped python train.py
- mona@pascal:~/computer_vision/VPilot$ ls
- drive.py frames1 frames2 model.h5 model.h5_bk model.py model.pyc __pycache__ README.md train.py
- mona@pascal:~/computer_vision/VPilot$ ls -ltr
- total 436392
- drwxrwxr-x 2 mona mona 4096 Jan 18 18:46 __pycache__
- -rw-rw-r-- 1 mona mona 223406216 Jan 19 14:26 model.h5_bk
- -rw-rw-r-- 1 mona mona 1697 Jan 20 16:56 README.md
- -rw-rw-r-- 1 mona mona 2292 Jan 24 16:07 drive.py
- drwxrwxr-x 2 mona mona 4096 Jan 26 15:39 frames1
- drwxrwxr-x 2 mona mona 4096 Jan 26 15:39 frames2
- -rw-rw-r-- 1 mona mona 1778 Jan 26 17:20 train.py
- -rw-rw-r-- 1 mona mona 5567 Jan 26 17:21 model.py
- -rw-rw-r-- 1 mona mona 6038 Jan 26 17:21 model.pyc
- -rw-rw-r-- 1 mona mona 223405664 Jan 26 19:19 model.h5
Advertisement
Add Comment
Please, Sign In to add comment