aitchisonl@c04u01:~$ run_lsf -g 1 nvidia-smi &
[1] 45654
aitchisonl@c04u01:~$ Scheduling job on 5 CPUs, 1 GPUs, 25600 MB in /groups/turaga/home/aitchisonl
Job <29600257> is submitted to queue <slowpoke>.
<<Waiting for dispatch ...>>
run_lsf -g 1 nvidia-smi &
[2] 45742
aitchisonl@c04u01:~$ Scheduling job on 5 CPUs, 1 GPUs, 25600 MB in /groups/turaga/home/aitchisonl
Job <29600258> is submitted to queue <slowpoke>.
<<Waiting for dispatch ...>>
run_lsf -g 1 nvidia-smi &
[3] 45816
aitchisonl@c04u01:~$ Scheduling job on 5 CPUs, 1 GPUs, 25600 MB in /groups/turaga/home/aitchisonl
Job <29600259> is submitted to queue <slowpoke>.
<<Waiting for dispatch ...>>
run_lsf -g 1 nvidia-smi &
[4] 45900
aitchisonl@c04u01:~$ Scheduling job on 5 CPUs, 1 GPUs, 25600 MB in /groups/turaga/home/aitchisonl
run_lsf -g 1 nvidia-smi &Job <29600260> is submitted to queue <slowpoke>.
<<Waiting for dispatch ...>>

[5] 45977
aitchisonl@c04u01:~$ run_lsf -g 1 nvidia-smi &Scheduling job on 5 CPUs, 1 GPUs, 25600 MB in /groups/turaga/home/aitchisonl
Job <29600261> is submitted to queue <slowpoke>.
<<Waiting for dispatch ...>>

[6] 46049
aitchisonl@c04u01:~$ Scheduling job on 5 CPUs, 1 GPUs, 25600 MB in /groups/turaga/home/aitchisonl
Job <29600262> is submitted to queue <slowpoke>.
<<Waiting for dispatch ...>>
run_lsf -g 1 nvidia-smi &<<Starting on c04u17>>
Sat Feb 24 06:59:42 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 387.26                 Driver Version: 387.26                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX TIT...  Off  | 00000000:04:00.0 Off |                  N/A |
| 22%   21C    P8    15W / 250W |     11MiB / 12207MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

[7] 46121
[2]   Done                    run_lsf -g 1 nvidia-smi
aitchisonl@c04u01:~$ Scheduling job on 5 CPUs, 1 GPUs, 25600 MB in /groups/turaga/home/aitchisonl
Job <29600263> is submitted to queue <slowpoke>.
<<Waiting for dispatch ...>>
run_lsf -g 1 nvidia-smi &<<Starting on c04u17>>
<<Starting on c04u07>>
Sat Feb 24 06:59:42 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 387.26                 Driver Version: 387.26                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|

[8] 46194
aitchisonl@c04u01:~$ |   0  GeForce GTX TIT...  Off  | 00000000:09:00.0 Off |                  N/A |
| 22%   25C    P8    15W / 250W |     11MiB / 12207MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
Scheduling job on 5 CPUs, 1 GPUs, 25600 MB in /groups/turaga/home/aitchisonl
run_lsf -g 1 nvidia-smi &Job <29600264> is submitted to queue <slowpoke>.
<<Waiting for dispatch ...>>

[9] 46267
[3]   Done                    run_lsf -g 1 nvidia-smi
aitchisonl@c04u01:~$ <<Starting on c04u17>>
Scheduling job on 5 CPUs, 1 GPUs, 25600 MB in /groups/turaga/home/aitchisonl
Sat Feb 24 06:59:42 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 387.26                 Driver Version: 387.26                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX TIT...  Off  | 00000000:86:00.0 Off |                  N/A |
| 22%   26C    P8    15W / 250W |     11MiB / 12207MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
Job <29600265> is submitted to queue <slowpoke>.
<<Waiting for dispatch ...>>
Sat Feb 24 06:59:42 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 387.26                 Driver Version: 387.26                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX TIT...  Off  | 00000000:89:00.0 Off |                  N/A |
| 22%   21C    P8    15W / 250W |     11MiB / 12207MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
<<Starting on c04u17>>
Sat Feb 24 06:59:43 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 387.26                 Driver Version: 387.26                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX TIT...  Off  | 00000000:89:00.0 Off |                  N/A |
| 22%   19C    P8    14W / 250W |     11MiB / 12207MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
<<Starting on c04u17>>
Sat Feb 24 06:59:43 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 387.26                 Driver Version: 387.26                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX TIT...  Off  | 00000000:8A:00.0 Off |                  N/A |
| 22%   22C    P8    15W / 250W |     11MiB / 12207MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
<<Starting on c04u31>>
Sat Feb 24 06:59:43 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 387.26                 Driver Version: 387.26                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  TITAN Xp            Off  | 00000000:05:00.0 Off |                  N/A |
| 23%   30C    P5    20W / 250W |  11731MiB / 12189MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     79019      C   python                                     11713MiB |
+-----------------------------------------------------------------------------+
<<Starting on c04u31>>
Sat Feb 24 06:59:44 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 387.26                 Driver Version: 387.26                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  TITAN Xp            Off  | 00000000:08:00.0 Off |                  N/A |
| 23%   20C    P8    15W / 250W |  11591MiB / 12189MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     79019      C   python                                     11573MiB |
+-----------------------------------------------------------------------------+
<<Starting on c04u31>>
Sat Feb 24 06:59:44 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 387.26                 Driver Version: 387.26                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  TITAN Xp            Off  | 00000000:09:00.0 Off |                  N/A |
| 23%   21C    P8    15W / 250W |  11591MiB / 12189MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     79019      C   python                                     11573MiB |
+-----------------------------------------------------------------------------+
^C

[1]   Done                    run_lsf -g 1 nvidia-smi
[4]   Done                    run_lsf -g 1 nvidia-smi
[5]   Done                    run_lsf -g 1 nvidia-smi
[6]   Done                    run_lsf -g 1 nvidia-smi
[7]   Done                    run_lsf -g 1 nvidia-smi
[8]-  Done                    run_lsf -g 1 nvidia-smi
[9]+  Done                    run_lsf -g 1 nvidia-smi