API tools faq
paste
Advertisement
Guest User

big_pipe_log

a guest
Jul 28th, 2017
110
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 422.84 KB | None | 0 0
raw download clone embed print report
  1.  
  2. Jul-28 00:34:18.780 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  3. Jul-28 00:34:23.776 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4. job: 4727741: RUNNING
  5. job: 4727744: RUNNING
  6. job: 4728288: RUNNING
  7.  
  8. Jul-28 00:34:23.777 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  9. Jul-28 00:34:23.778 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  10. job: 4727741: RUNNING
  11. job: 4727744: RUNNING
  12. job: 4728288: RUNNING
  13.  
  14. Jul-28 00:34:23.778 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  15. Jul-28 00:34:28.777 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q short
  16. Jul-28 00:34:28.912 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  17. 4727741,RUN,Jul 27 20:22
  18. 4727744,RUN,Jul 27 20:22
  19. 4728288,RUN,Jul 27 23:44
  20.  
  21.  
  22. Jul-28 00:34:28.912 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  23. job: 4727741: RUNNING
  24. job: 4727744: RUNNING
  25. job: 4728288: RUNNING
  26.  
  27. Jul-28 00:34:28.912 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  28. Jul-28 00:34:28.914 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  29. job: 4727741: RUNNING
  30. job: 4727744: RUNNING
  31. job: 4728288: RUNNING
  32.  
  33. Jul-28 00:34:28.914 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  34. Jul-28 00:34:33.777 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  35. job: 4727741: RUNNING
  36. job: 4727744: RUNNING
  37. job: 4728288: RUNNING
  38.  
  39. Jul-28 00:34:33.778 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  40. Jul-28 00:34:33.779 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  41. job: 4727741: RUNNING
  42. job: 4727744: RUNNING
  43. job: 4728288: RUNNING
  44.  
  45. Jul-28 00:34:33.779 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  46. Jul-28 00:34:38.777 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  47. job: 4727741: RUNNING
  48. job: 4727744: RUNNING
  49. job: 4728288: RUNNING
  50.  
  51. Jul-28 00:34:38.777 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  52. Jul-28 00:34:38.779 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  53. job: 4727741: RUNNING
  54. job: 4727744: RUNNING
  55. job: 4728288: RUNNING
  56.  
  57. Jul-28 00:34:38.779 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  58. Jul-28 00:34:43.776 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  59. job: 4727741: RUNNING
  60. job: 4727744: RUNNING
  61. job: 4728288: RUNNING
  62.  
  63. Jul-28 00:34:43.777 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  64. Jul-28 00:34:43.778 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  65. job: 4727741: RUNNING
  66. job: 4727744: RUNNING
  67. job: 4728288: RUNNING
  68.  
  69. Jul-28 00:34:43.778 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  70. Jul-28 00:34:48.776 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  71. job: 4727741: RUNNING
  72. job: 4727744: RUNNING
  73. job: 4728288: RUNNING
  74.  
  75. Jul-28 00:34:48.776 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  76. Jul-28 00:34:48.778 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  77. job: 4727741: RUNNING
  78. job: 4727744: RUNNING
  79. job: 4728288: RUNNING
  80.  
  81. Jul-28 00:34:48.778 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  82. Jul-28 00:34:53.776 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  83. job: 4727741: RUNNING
  84. job: 4727744: RUNNING
  85. job: 4728288: RUNNING
  86.  
  87. Jul-28 00:34:53.776 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  88. Jul-28 00:34:53.777 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  89. job: 4727741: RUNNING
  90. job: 4727744: RUNNING
  91. job: 4728288: RUNNING
  92.  
  93. Jul-28 00:34:53.777 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  94. Jul-28 00:34:58.779 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  95. job: 4727741: RUNNING
  96. job: 4727744: RUNNING
  97. job: 4728288: RUNNING
  98.  
  99. Jul-28 00:34:58.780 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  100. Jul-28 00:34:58.781 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  101. job: 4727741: RUNNING
  102. job: 4727744: RUNNING
  103. job: 4728288: RUNNING
  104.  
  105. Jul-28 00:34:58.781 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  106. Jul-28 00:35:03.779 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  107. job: 4727741: RUNNING
  108. job: 4727744: RUNNING
  109. job: 4728288: RUNNING
  110.  
  111. Jul-28 00:35:03.780 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  112. Jul-28 00:35:03.781 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  113. job: 4727741: RUNNING
  114. job: 4727744: RUNNING
  115. job: 4728288: RUNNING
  116.  
  117. Jul-28 00:35:03.781 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  118. Jul-28 00:35:08.778 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor lsf > tasks to be completed: 2 -- pending tasks are shown below
  119. ~> TaskHandler[jobId: 4727741; id: 146; name: merge_pairsam_into_runs (library:HeLa1 run:lane1); status: RUNNING; exit: -; error: -; workDir: /farline/umw_job_dekker/HPCC/sv49w/distiller-nf/work/bd/4c7a06e2f9df6f36089f01ca73027b started: 1501202287725; exited: -; ]
  120. ~> TaskHandler[jobId: 4727744; id: 148; name: merge_pairsam_into_runs (library:HeLa1 run:lane3); status: RUNNING; exit: -; error: -; workDir: /farline/umw_job_dekker/HPCC/sv49w/distiller-nf/work/c0/e9870618be7003929109a336301bca started: 1501203352792; exited: -; ]
  121. Jul-28 00:35:08.778 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  122. job: 4727741: RUNNING
  123. job: 4727744: RUNNING
  124. job: 4728288: RUNNING
  125.  
  126. Jul-28 00:35:08.778 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  127. Jul-28 00:35:08.778 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  128. job: 4727741: RUNNING
  129. job: 4727744: RUNNING
  130. job: 4728288: RUNNING
  131.  
  132. Jul-28 00:35:08.778 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  133. Jul-28 00:35:13.782 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  134. job: 4727741: RUNNING
  135. job: 4727744: RUNNING
  136. job: 4728288: RUNNING
  137.  
  138. Jul-28 00:35:13.782 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  139. Jul-28 00:35:13.783 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  140. job: 4727741: RUNNING
  141. job: 4727744: RUNNING
  142. job: 4728288: RUNNING
  143.  
  144. Jul-28 00:35:13.783 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  145. Jul-28 00:35:18.782 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  146. job: 4727741: RUNNING
  147. job: 4727744: RUNNING
  148. job: 4728288: RUNNING
  149.  
  150. Jul-28 00:35:18.782 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  151. Jul-28 00:35:18.782 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  152. job: 4727741: RUNNING
  153. job: 4727744: RUNNING
  154. job: 4728288: RUNNING
  155.  
  156. Jul-28 00:35:18.782 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  157. Jul-28 00:35:23.782 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  158. job: 4727741: RUNNING
  159. job: 4727744: RUNNING
  160. job: 4728288: RUNNING
  161.  
  162. Jul-28 00:35:23.783 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  163. Jul-28 00:35:23.784 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  164. job: 4727741: RUNNING
  165. job: 4727744: RUNNING
  166. job: 4728288: RUNNING
  167.  
  168. Jul-28 00:35:23.784 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  169. Jul-28 00:35:28.783 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q short
  170. Jul-28 00:35:28.917 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  171. 4727741,RUN,Jul 27 20:22
  172. 4727744,RUN,Jul 27 20:22
  173.  
  174.  
  175. Jul-28 00:35:28.917 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  176. job: 4727741: RUNNING
  177. job: 4727744: RUNNING
  178.  
  179. Jul-28 00:35:28.917 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  180. Jul-28 00:35:28.919 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  181. job: 4727741: RUNNING
  182. job: 4727744: RUNNING
  183.  
  184. Jul-28 00:35:28.919 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  185. Jul-28 00:35:33.782 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  186. job: 4727741: RUNNING
  187. job: 4727744: RUNNING
  188.  
  189. Jul-28 00:35:33.782 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  190. Jul-28 00:35:33.783 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  191. job: 4727741: RUNNING
  192. job: 4727744: RUNNING
  193.  
  194. Jul-28 00:35:33.783 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  195. Jul-28 00:35:38.783 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  196. job: 4727741: RUNNING
  197. job: 4727744: RUNNING
  198.  
  199. Jul-28 00:35:38.783 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  200. Jul-28 00:35:38.784 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  201. job: 4727741: RUNNING
  202. job: 4727744: RUNNING
  203.  
  204. Jul-28 00:35:38.784 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  205. Jul-28 00:35:43.783 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  206. job: 4727741: RUNNING
  207. job: 4727744: RUNNING
  208.  
  209. Jul-28 00:35:43.783 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  210. Jul-28 00:35:43.784 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  211. job: 4727741: RUNNING
  212. job: 4727744: RUNNING
  213.  
  214. Jul-28 00:35:43.784 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  215. Jul-28 00:35:48.783 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  216. job: 4727741: RUNNING
  217. job: 4727744: RUNNING
  218.  
  219. Jul-28 00:35:48.783 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  220. Jul-28 00:35:48.784 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  221. job: 4727741: RUNNING
  222. job: 4727744: RUNNING
  223.  
  224. Jul-28 00:35:48.784 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  225. Jul-28 00:35:53.783 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  226. job: 4727741: RUNNING
  227. job: 4727744: RUNNING
  228.  
  229. Jul-28 00:35:53.783 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  230. Jul-28 00:35:53.784 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  231. job: 4727741: RUNNING
  232. job: 4727744: RUNNING
  233.  
  234. Jul-28 00:35:53.784 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  235. Jul-28 00:35:58.783 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  236. job: 4727741: RUNNING
  237. job: 4727744: RUNNING
  238.  
  239. Jul-28 00:35:58.783 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  240. Jul-28 00:35:58.783 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  241. job: 4727741: RUNNING
  242. job: 4727744: RUNNING
  243.  
  244. Jul-28 00:35:58.783 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  245. Jul-28 00:36:03.783 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  246. job: 4727741: RUNNING
  247. job: 4727744: RUNNING
  248.  
  249. Jul-28 00:36:03.783 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  250. Jul-28 00:36:03.784 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  251. job: 4727741: RUNNING
  252. job: 4727744: RUNNING
  253.  
  254. Jul-28 00:36:03.784 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  255. Jul-28 00:36:08.786 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  256. job: 4727741: RUNNING
  257. job: 4727744: RUNNING
  258.  
  259. Jul-28 00:36:08.786 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  260. Jul-28 00:36:08.788 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  261. job: 4727741: RUNNING
  262. job: 4727744: RUNNING
  263.  
  264. Jul-28 00:36:08.788 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  265. Jul-28 00:36:13.786 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  266. job: 4727741: RUNNING
  267. job: 4727744: RUNNING
  268.  
  269. Jul-28 00:36:13.787 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  270. Jul-28 00:36:13.788 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  271. job: 4727741: RUNNING
  272. job: 4727744: RUNNING
  273.  
  274. Jul-28 00:36:13.788 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  275. Jul-28 00:36:18.787 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  276. job: 4727741: RUNNING
  277. job: 4727744: RUNNING
  278.  
  279. Jul-28 00:36:18.787 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  280. Jul-28 00:36:18.789 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  281. job: 4727741: RUNNING
  282. job: 4727744: RUNNING
  283.  
  284. Jul-28 00:36:18.789 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  285. Jul-28 00:36:23.784 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  286. job: 4727741: RUNNING
  287. job: 4727744: RUNNING
  288.  
  289. Jul-28 00:36:23.785 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  290. Jul-28 00:36:23.786 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  291. job: 4727741: RUNNING
  292. job: 4727744: RUNNING
  293.  
  294. Jul-28 00:36:23.786 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  295. Jul-28 00:36:28.784 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q short
  296. Jul-28 00:36:28.919 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  297. 4727741,RUN,Jul 27 20:22
  298. 4727744,RUN,Jul 27 20:22
  299.  
  300.  
  301. Jul-28 00:36:28.919 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  302. job: 4727741: RUNNING
  303. job: 4727744: RUNNING
  304.  
  305. Jul-28 00:36:28.919 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  306. Jul-28 00:36:28.920 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  307. job: 4727741: RUNNING
  308. job: 4727744: RUNNING
  309.  
  310. Jul-28 00:36:28.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  311. Jul-28 00:36:33.784 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  312. job: 4727741: RUNNING
  313. job: 4727744: RUNNING
  314.  
  315. Jul-28 00:36:33.784 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  316. Jul-28 00:36:33.785 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  317. job: 4727741: RUNNING
  318. job: 4727744: RUNNING
  319.  
  320. Jul-28 00:36:33.785 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  321. Jul-28 00:36:38.784 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  322. job: 4727741: RUNNING
  323. job: 4727744: RUNNING
  324.  
  325. Jul-28 00:36:38.785 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  326. Jul-28 00:36:38.786 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  327. job: 4727741: RUNNING
  328. job: 4727744: RUNNING
  329.  
  330. Jul-28 00:36:38.786 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  331. Jul-28 00:36:43.783 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  332. job: 4727741: RUNNING
  333. job: 4727744: RUNNING
  334.  
  335. Jul-28 00:36:43.783 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  336. Jul-28 00:36:43.784 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  337. job: 4727741: RUNNING
  338. job: 4727744: RUNNING
  339.  
  340. Jul-28 00:36:43.784 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  341. Jul-28 00:36:48.788 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  342. job: 4727741: RUNNING
  343. job: 4727744: RUNNING
  344.  
  345. Jul-28 00:36:48.788 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  346. Jul-28 00:36:48.790 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  347. job: 4727741: RUNNING
  348. job: 4727744: RUNNING
  349.  
  350. Jul-28 00:36:48.790 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  351. Jul-28 00:36:53.787 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  352. job: 4727741: RUNNING
  353. job: 4727744: RUNNING
  354.  
  355. Jul-28 00:36:53.787 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  356. Jul-28 00:36:53.788 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  357. job: 4727741: RUNNING
  358. job: 4727744: RUNNING
  359.  
  360. Jul-28 00:36:53.788 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  361. Jul-28 00:36:58.785 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  362. job: 4727741: RUNNING
  363. job: 4727744: RUNNING
  364.  
  365. Jul-28 00:36:58.785 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  366. Jul-28 00:36:58.786 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  367. job: 4727741: RUNNING
  368. job: 4727744: RUNNING
  369.  
  370. Jul-28 00:36:58.786 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  371. Jul-28 00:37:03.786 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  372. job: 4727741: RUNNING
  373. job: 4727744: RUNNING
  374.  
  375. Jul-28 00:37:03.786 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  376. Jul-28 00:37:03.787 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  377. job: 4727741: RUNNING
  378. job: 4727744: RUNNING
  379.  
  380. Jul-28 00:37:03.787 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  381. Jul-28 00:37:08.785 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  382. job: 4727741: RUNNING
  383. job: 4727744: RUNNING
  384.  
  385. Jul-28 00:37:08.785 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  386. Jul-28 00:37:08.786 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  387. job: 4727741: RUNNING
  388. job: 4727744: RUNNING
  389.  
  390. Jul-28 00:37:08.786 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  391. Jul-28 00:37:13.788 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  392. job: 4727741: RUNNING
  393. job: 4727744: RUNNING
  394.  
  395. Jul-28 00:37:13.789 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  396. Jul-28 00:37:13.790 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  397. job: 4727741: RUNNING
  398. job: 4727744: RUNNING
  399.  
  400. Jul-28 00:37:13.790 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  401. Jul-28 00:37:18.785 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  402. job: 4727741: RUNNING
  403. job: 4727744: RUNNING
  404.  
  405. Jul-28 00:37:18.785 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  406. Jul-28 00:37:18.786 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  407. job: 4727741: RUNNING
  408. job: 4727744: RUNNING
  409.  
  410. Jul-28 00:37:18.786 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  411. Jul-28 00:37:23.785 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  412. job: 4727741: RUNNING
  413. job: 4727744: RUNNING
  414.  
  415. Jul-28 00:37:23.785 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  416. Jul-28 00:37:23.786 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  417. job: 4727741: RUNNING
  418. job: 4727744: RUNNING
  419.  
  420. Jul-28 00:37:23.786 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  421. Jul-28 00:37:28.788 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q short
  422. Jul-28 00:37:28.923 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  423. 4727741,RUN,Jul 27 20:22
  424. 4727744,RUN,Jul 27 20:22
  425.  
  426.  
  427. Jul-28 00:37:28.923 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  428. job: 4727741: RUNNING
  429. job: 4727744: RUNNING
  430.  
  431. Jul-28 00:37:28.923 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  432. Jul-28 00:37:28.924 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  433. job: 4727741: RUNNING
  434. job: 4727744: RUNNING
  435.  
  436. Jul-28 00:37:28.924 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  437. Jul-28 00:37:33.788 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  438. job: 4727741: RUNNING
  439. job: 4727744: RUNNING
  440.  
  441. Jul-28 00:37:33.789 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  442. Jul-28 00:37:33.791 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  443. job: 4727741: RUNNING
  444. job: 4727744: RUNNING
  445.  
  446. Jul-28 00:37:33.791 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  447. Jul-28 00:37:38.785 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  448. job: 4727741: RUNNING
  449. job: 4727744: RUNNING
  450.  
  451. Jul-28 00:37:38.785 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  452. Jul-28 00:37:38.786 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  453. job: 4727741: RUNNING
  454. job: 4727744: RUNNING
  455.  
  456. Jul-28 00:37:38.786 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  457. Jul-28 00:37:43.786 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  458. job: 4727741: RUNNING
  459. job: 4727744: RUNNING
  460.  
  461. Jul-28 00:37:43.786 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  462. Jul-28 00:37:43.787 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  463. job: 4727741: RUNNING
  464. job: 4727744: RUNNING
  465.  
  466. Jul-28 00:37:43.787 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  467. Jul-28 00:37:48.787 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  468. job: 4727741: RUNNING
  469. job: 4727744: RUNNING
  470.  
  471. Jul-28 00:37:48.787 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  472. Jul-28 00:37:48.789 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  473. job: 4727741: RUNNING
  474. job: 4727744: RUNNING
  475.  
  476. Jul-28 00:37:48.789 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  477. Jul-28 00:37:53.787 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  478. job: 4727741: RUNNING
  479. job: 4727744: RUNNING
  480.  
  481. Jul-28 00:37:53.787 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  482. Jul-28 00:37:53.788 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  483. job: 4727741: RUNNING
  484. job: 4727744: RUNNING
  485.  
  486. Jul-28 00:37:53.788 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  487. Jul-28 00:37:58.787 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  488. job: 4727741: RUNNING
  489. job: 4727744: RUNNING
  490.  
  491. Jul-28 00:37:58.788 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  492. Jul-28 00:37:58.789 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  493. job: 4727741: RUNNING
  494. job: 4727744: RUNNING
  495.  
  496. Jul-28 00:37:58.789 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  497. Jul-28 00:38:03.791 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  498. job: 4727741: RUNNING
  499. job: 4727744: RUNNING
  500.  
  501. Jul-28 00:38:03.792 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  502. Jul-28 00:38:03.793 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  503. job: 4727741: RUNNING
  504. job: 4727744: RUNNING
  505.  
  506. Jul-28 00:38:03.793 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  507. Jul-28 00:38:08.786 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  508. job: 4727741: RUNNING
  509. job: 4727744: RUNNING
  510.  
  511. Jul-28 00:38:08.786 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  512. Jul-28 00:38:08.787 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  513. job: 4727741: RUNNING
  514. job: 4727744: RUNNING
  515.  
  516. Jul-28 00:38:08.787 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  517. Jul-28 00:38:13.786 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  518. job: 4727741: RUNNING
  519. job: 4727744: RUNNING
  520.  
  521. Jul-28 00:38:13.787 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  522. Jul-28 00:38:13.788 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  523. job: 4727741: RUNNING
  524. job: 4727744: RUNNING
  525.  
  526. Jul-28 00:38:13.788 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  527. Jul-28 00:38:18.786 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  528. job: 4727741: RUNNING
  529. job: 4727744: RUNNING
  530.  
  531. Jul-28 00:38:18.787 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  532. Jul-28 00:38:18.788 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  533. job: 4727741: RUNNING
  534. job: 4727744: RUNNING
  535.  
  536. Jul-28 00:38:18.788 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  537. Jul-28 00:38:23.786 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  538. job: 4727741: RUNNING
  539. job: 4727744: RUNNING
  540.  
  541. Jul-28 00:38:23.786 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  542. Jul-28 00:38:23.787 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  543. job: 4727741: RUNNING
  544. job: 4727744: RUNNING
  545.  
  546. Jul-28 00:38:23.787 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  547. Jul-28 00:38:28.786 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  548. job: 4727741: RUNNING
  549. job: 4727744: RUNNING
  550.  
  551. Jul-28 00:38:28.787 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  552. Jul-28 00:38:28.787 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  553. job: 4727741: RUNNING
  554. job: 4727744: RUNNING
  555.  
  556. Jul-28 00:38:28.787 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  557. Jul-28 00:38:33.791 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q short
  558. Jul-28 00:38:34.247 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  559. 4727741,RUN,Jul 27 20:22
  560. 4727744,RUN,Jul 27 20:22
  561.  
  562.  
  563. Jul-28 00:38:34.247 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  564. job: 4727741: RUNNING
  565. job: 4727744: RUNNING
  566.  
  567. Jul-28 00:38:34.248 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  568. Jul-28 00:38:34.249 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  569. job: 4727741: RUNNING
  570. job: 4727744: RUNNING
  571.  
  572. Jul-28 00:38:34.249 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  573. Jul-28 00:38:38.787 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  574. job: 4727741: RUNNING
  575. job: 4727744: RUNNING
  576.  
  577. Jul-28 00:38:38.787 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  578. Jul-28 00:38:38.788 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  579. job: 4727741: RUNNING
  580. job: 4727744: RUNNING
  581.  
  582. Jul-28 00:38:38.788 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  583. Jul-28 00:38:43.787 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  584. job: 4727741: RUNNING
  585. job: 4727744: RUNNING
  586.  
  587. Jul-28 00:38:43.788 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  588. Jul-28 00:38:43.789 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  589. job: 4727741: RUNNING
  590. job: 4727744: RUNNING
  591.  
  592. Jul-28 00:38:43.789 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  593. Jul-28 00:38:48.787 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  594. job: 4727741: RUNNING
  595. job: 4727744: RUNNING
  596.  
  597. Jul-28 00:38:48.787 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  598. Jul-28 00:38:48.788 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  599. job: 4727741: RUNNING
  600. job: 4727744: RUNNING
  601.  
  602. Jul-28 00:38:48.788 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  603. Jul-28 00:38:53.787 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  604. job: 4727741: RUNNING
  605. job: 4727744: RUNNING
  606.  
  607. Jul-28 00:38:53.788 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  608. Jul-28 00:38:53.789 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  609. job: 4727741: RUNNING
  610. job: 4727744: RUNNING
  611.  
  612. Jul-28 00:38:53.789 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  613. Jul-28 00:38:58.787 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  614. job: 4727741: RUNNING
  615. job: 4727744: RUNNING
  616.  
  617. Jul-28 00:38:58.787 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  618. Jul-28 00:38:58.788 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  619. job: 4727741: RUNNING
  620. job: 4727744: RUNNING
  621.  
  622. Jul-28 00:38:58.788 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  623. Jul-28 00:39:03.787 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  624. job: 4727741: RUNNING
  625. job: 4727744: RUNNING
  626.  
  627. Jul-28 00:39:03.788 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  628. Jul-28 00:39:03.788 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  629. job: 4727741: RUNNING
  630. job: 4727744: RUNNING
  631.  
  632. Jul-28 00:39:03.788 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  633. Jul-28 00:39:08.788 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  634. job: 4727741: RUNNING
  635. job: 4727744: RUNNING
  636.  
  637. Jul-28 00:39:08.788 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  638. Jul-28 00:39:08.789 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  639. job: 4727741: RUNNING
  640. job: 4727744: RUNNING
  641.  
  642. Jul-28 00:39:08.789 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  643. Jul-28 00:39:13.788 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  644. job: 4727741: RUNNING
  645. job: 4727744: RUNNING
  646.  
  647. Jul-28 00:39:13.789 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  648. Jul-28 00:39:13.789 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  649. job: 4727741: RUNNING
  650. job: 4727744: RUNNING
  651.  
  652. Jul-28 00:39:13.789 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  653. Jul-28 00:39:18.788 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  654. job: 4727741: RUNNING
  655. job: 4727744: RUNNING
  656.  
  657. Jul-28 00:39:18.788 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  658. Jul-28 00:39:18.789 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  659. job: 4727741: RUNNING
  660. job: 4727744: RUNNING
  661.  
  662. Jul-28 00:39:18.789 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  663. Jul-28 00:39:23.788 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  664. job: 4727741: RUNNING
  665. job: 4727744: RUNNING
  666.  
  667. Jul-28 00:39:23.788 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  668. Jul-28 00:39:23.789 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  669. job: 4727741: RUNNING
  670. job: 4727744: RUNNING
  671.  
  672. Jul-28 00:39:23.789 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  673. Jul-28 00:39:28.788 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  674. job: 4727741: RUNNING
  675. job: 4727744: RUNNING
  676.  
  677. Jul-28 00:39:28.788 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  678. Jul-28 00:39:28.789 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  679. job: 4727741: RUNNING
  680. job: 4727744: RUNNING
  681.  
  682. Jul-28 00:39:28.789 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  683. Jul-28 00:39:33.788 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  684. job: 4727741: RUNNING
  685. job: 4727744: RUNNING
  686.  
  687. Jul-28 00:39:33.789 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  688. Jul-28 00:39:33.789 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  689. job: 4727741: RUNNING
  690. job: 4727744: RUNNING
  691.  
  692. Jul-28 00:39:33.789 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  693. Jul-28 00:39:38.790 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q short
  694. Jul-28 00:39:38.891 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  695. 4727741,RUN,Jul 27 20:22
  696. 4727744,RUN,Jul 27 20:22
  697.  
  698.  
  699. Jul-28 00:39:38.891 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  700. job: 4727741: RUNNING
  701. job: 4727744: RUNNING
  702.  
  703. Jul-28 00:39:38.891 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  704. Jul-28 00:39:38.892 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  705. job: 4727741: RUNNING
  706. job: 4727744: RUNNING
  707.  
  708. Jul-28 00:39:38.892 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  709. Jul-28 00:39:43.790 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  710. job: 4727741: RUNNING
  711. job: 4727744: RUNNING
  712.  
  713. Jul-28 00:39:43.791 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  714. Jul-28 00:39:43.791 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  715. job: 4727741: RUNNING
  716. job: 4727744: RUNNING
  717.  
  718. Jul-28 00:39:43.791 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  719. Jul-28 00:39:48.790 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  720. job: 4727741: RUNNING
  721. job: 4727744: RUNNING
  722.  
  723. Jul-28 00:39:48.791 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  724. Jul-28 00:39:48.792 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  725. job: 4727741: RUNNING
  726. job: 4727744: RUNNING
  727.  
  728. Jul-28 00:39:48.792 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  729. Jul-28 00:39:53.790 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  730. job: 4727741: RUNNING
  731. job: 4727744: RUNNING
  732.  
  733. Jul-28 00:39:53.791 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  734. Jul-28 00:39:53.791 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  735. job: 4727741: RUNNING
  736. job: 4727744: RUNNING
  737.  
  738. Jul-28 00:39:53.791 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  739. Jul-28 00:39:58.790 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  740. job: 4727741: RUNNING
  741. job: 4727744: RUNNING
  742.  
  743. Jul-28 00:39:58.790 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  744. Jul-28 00:39:58.791 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  745. job: 4727741: RUNNING
  746. job: 4727744: RUNNING
  747.  
  748. Jul-28 00:39:58.791 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  749. Jul-28 00:40:03.790 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  750. job: 4727741: RUNNING
  751. job: 4727744: RUNNING
  752.  
  753. Jul-28 00:40:03.791 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  754. Jul-28 00:40:03.792 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  755. job: 4727741: RUNNING
  756. job: 4727744: RUNNING
  757.  
  758. Jul-28 00:40:03.792 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  759. Jul-28 00:40:08.792 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor lsf > tasks to be completed: 2 -- pending tasks are shown below
  760. ~> TaskHandler[jobId: 4727741; id: 146; name: merge_pairsam_into_runs (library:HeLa1 run:lane1); status: RUNNING; exit: -; error: -; workDir: /farline/umw_job_dekker/HPCC/sv49w/distiller-nf/work/bd/4c7a06e2f9df6f36089f01ca73027b started: 1501202287725; exited: -; ]
  761. ~> TaskHandler[jobId: 4727744; id: 148; name: merge_pairsam_into_runs (library:HeLa1 run:lane3); status: RUNNING; exit: -; error: -; workDir: /farline/umw_job_dekker/HPCC/sv49w/distiller-nf/work/c0/e9870618be7003929109a336301bca started: 1501203352792; exited: -; ]
  762. Jul-28 00:40:08.793 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  763. job: 4727741: RUNNING
  764. job: 4727744: RUNNING
  765.  
  766. Jul-28 00:40:08.793 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  767. Jul-28 00:40:08.793 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  768. job: 4727741: RUNNING
  769. job: 4727744: RUNNING
  770.  
  771. Jul-28 00:40:08.793 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  772. Jul-28 00:40:13.796 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  773. job: 4727741: RUNNING
  774. job: 4727744: RUNNING
  775.  
  776. Jul-28 00:40:13.796 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  777. Jul-28 00:40:13.797 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  778. job: 4727741: RUNNING
  779. job: 4727744: RUNNING
  780.  
  781. Jul-28 00:40:13.797 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  782. Jul-28 00:40:18.796 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  783. job: 4727741: RUNNING
  784. job: 4727744: RUNNING
  785.  
  786. Jul-28 00:40:18.796 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  787. Jul-28 00:40:18.797 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  788. job: 4727741: RUNNING
  789. job: 4727744: RUNNING
  790.  
  791. Jul-28 00:40:18.797 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  792. Jul-28 00:40:23.795 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  793. job: 4727741: RUNNING
  794. job: 4727744: RUNNING
  795.  
  796. Jul-28 00:40:23.796 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  797. Jul-28 00:40:23.797 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  798. job: 4727741: RUNNING
  799. job: 4727744: RUNNING
  800.  
  801. Jul-28 00:40:23.797 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  802. Jul-28 00:40:28.795 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  803. job: 4727741: RUNNING
  804. job: 4727744: RUNNING
  805.  
  806. Jul-28 00:40:28.795 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  807. Jul-28 00:40:28.796 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  808. job: 4727741: RUNNING
  809. job: 4727744: RUNNING
  810.  
  811. Jul-28 00:40:28.796 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  812. Jul-28 00:40:33.795 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  813. job: 4727741: RUNNING
  814. job: 4727744: RUNNING
  815.  
  816. Jul-28 00:40:33.796 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  817. Jul-28 00:40:33.797 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  818. job: 4727741: RUNNING
  819. job: 4727744: RUNNING
  820.  
  821. Jul-28 00:40:33.797 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  822. Jul-28 00:40:38.795 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q short
  823. Jul-28 00:40:38.903 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  824. 4727741,RUN,Jul 27 20:22
  825. 4727744,RUN,Jul 27 20:22
  826.  
  827.  
  828. Jul-28 00:40:38.903 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  829. job: 4727741: RUNNING
  830. job: 4727744: RUNNING
  831.  
  832. Jul-28 00:40:38.904 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  833. Jul-28 00:40:38.905 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  834. job: 4727741: RUNNING
  835. job: 4727744: RUNNING
  836.  
  837. Jul-28 00:40:38.905 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  838. Jul-28 00:40:43.795 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  839. job: 4727741: RUNNING
  840. job: 4727744: RUNNING
  841.  
  842. Jul-28 00:40:43.795 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  843. Jul-28 00:40:43.796 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  844. job: 4727741: RUNNING
  845. job: 4727744: RUNNING
  846.  
  847. Jul-28 00:40:43.796 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  848. Jul-28 00:40:48.795 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  849. job: 4727741: RUNNING
  850. job: 4727744: RUNNING
  851.  
  852. Jul-28 00:40:48.795 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  853. Jul-28 00:40:48.796 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  854. job: 4727741: RUNNING
  855. job: 4727744: RUNNING
  856.  
  857. Jul-28 00:40:48.796 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  858. Jul-28 00:40:53.796 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  859. job: 4727741: RUNNING
  860. job: 4727744: RUNNING
  861.  
  862. Jul-28 00:40:53.797 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727741` active status: true
  863. Jul-28 00:40:53.797 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  864. job: 4727741: RUNNING
  865. job: 4727744: RUNNING
  866.  
  867. Jul-28 00:40:53.797 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  868. Jul-28 00:40:58.818 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[jobId: 4727741; id: 146; name: merge_pairsam_into_runs (library:HeLa1 run:lane1); status: COMPLETED; exit: 140; error: -; workDir: /farline/umw_job_dekker/HPCC/sv49w/distiller-nf/work/bd/4c7a06e2f9df6f36089f01ca73027b started: 1501202287725; exited: 2017-07-28T04:40:54.417519Z; ]
  869. Jul-28 00:40:58.837 [Task monitor] WARN nextflow.processor.TaskProcessor - Process `merge_pairsam_into_runs (library:HeLa1 run:lane1)` terminated with an error exit status (140) -- Execution is retried (1)
  870. Jul-28 00:40:58.859 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  871. job: 4727741: RUNNING
  872. job: 4727744: RUNNING
  873.  
  874. Jul-28 00:40:58.859 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  875. Jul-28 00:40:59.119 [Task submitter] DEBUG nextflow.executor.GridTaskHandler - Submitted process merge_pairsam_into_runs (library:HeLa1 run:lane1) > lsf jobId: 4728323; workDir: /farline/umw_job_dekker/HPCC/sv49w/distiller-nf/work/81/e67006b79d110316327d1d4be71a71
  876. Jul-28 00:40:59.119 [Task submitter] INFO nextflow.Session - [81/e67006] Re-submitted process > merge_pairsam_into_runs (library:HeLa1 run:lane1)
  877. Jul-28 00:41:03.797 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  878. job: 4727741: RUNNING
  879. job: 4727744: RUNNING
  880.  
  881. Jul-28 00:41:03.797 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  882. Jul-28 00:41:08.797 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  883. job: 4727741: RUNNING
  884. job: 4727744: RUNNING
  885.  
  886. Jul-28 00:41:08.797 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  887. Jul-28 00:41:13.797 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  888. job: 4727741: RUNNING
  889. job: 4727744: RUNNING
  890.  
  891. Jul-28 00:41:13.797 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  892. Jul-28 00:41:18.798 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  893. job: 4727741: RUNNING
  894. job: 4727744: RUNNING
  895.  
  896. Jul-28 00:41:18.799 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  897. Jul-28 00:41:18.803 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  898. job: 4727741: RUNNING
  899. job: 4727744: RUNNING
  900.  
  901. Jul-28 00:41:18.803 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  902. Jul-28 00:41:23.797 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  903. job: 4727741: RUNNING
  904. job: 4727744: RUNNING
  905.  
  906. Jul-28 00:41:23.797 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  907. Jul-28 00:41:23.798 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  908. job: 4727741: RUNNING
  909. job: 4727744: RUNNING
  910.  
  911. Jul-28 00:41:23.798 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  912. Jul-28 00:41:28.797 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  913. job: 4727741: RUNNING
  914. job: 4727744: RUNNING
  915.  
  916. Jul-28 00:41:28.797 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  917. Jul-28 00:41:28.798 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  918. job: 4727741: RUNNING
  919. job: 4727744: RUNNING
  920.  
  921. Jul-28 00:41:28.798 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  922. Jul-28 00:41:33.797 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  923. job: 4727741: RUNNING
  924. job: 4727744: RUNNING
  925.  
  926. Jul-28 00:41:33.797 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  927. Jul-28 00:41:33.798 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  928. job: 4727741: RUNNING
  929. job: 4727744: RUNNING
  930.  
  931. Jul-28 00:41:33.798 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  932. Jul-28 00:41:38.797 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q short
  933. Jul-28 00:41:38.911 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  934. 4727744,RUN,Jul 27 20:22
  935.  
  936.  
  937. Jul-28 00:41:38.911 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  938. job: 4727744: RUNNING
  939.  
  940. Jul-28 00:41:38.911 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  941. Jul-28 00:41:38.912 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  942. job: 4727744: RUNNING
  943.  
  944. Jul-28 00:41:38.912 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  945. Jul-28 00:41:43.796 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  946. job: 4727744: RUNNING
  947.  
  948. Jul-28 00:41:43.796 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  949. Jul-28 00:41:43.797 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  950. job: 4727744: RUNNING
  951.  
  952. Jul-28 00:41:43.797 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  953. Jul-28 00:41:48.796 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  954. job: 4727744: RUNNING
  955.  
  956. Jul-28 00:41:48.797 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  957. Jul-28 00:41:48.797 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  958. job: 4727744: RUNNING
  959.  
  960. Jul-28 00:41:48.797 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  961. Jul-28 00:41:53.797 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  962. job: 4727744: RUNNING
  963.  
  964. Jul-28 00:41:53.798 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  965. Jul-28 00:41:53.799 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  966. job: 4727744: RUNNING
  967.  
  968. Jul-28 00:41:53.799 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  969. Jul-28 00:41:58.797 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  970. job: 4727744: RUNNING
  971.  
  972. Jul-28 00:41:58.797 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  973. Jul-28 00:41:58.798 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  974. job: 4727744: RUNNING
  975.  
  976. Jul-28 00:41:58.798 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  977. Jul-28 00:42:03.797 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  978. job: 4727744: RUNNING
  979.  
  980. Jul-28 00:42:03.798 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  981. Jul-28 00:42:03.799 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  982. job: 4727744: RUNNING
  983.  
  984. Jul-28 00:42:03.799 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  985. Jul-28 00:42:08.797 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  986. job: 4727744: RUNNING
  987.  
  988. Jul-28 00:42:08.798 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  989. Jul-28 00:42:08.798 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  990. job: 4727744: RUNNING
  991.  
  992. Jul-28 00:42:08.798 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  993. Jul-28 00:42:13.798 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  994. job: 4727744: RUNNING
  995.  
  996. Jul-28 00:42:13.798 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  997. Jul-28 00:42:13.799 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  998. job: 4727744: RUNNING
  999.  
  1000. Jul-28 00:42:13.799 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1001. Jul-28 00:42:18.798 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1002. job: 4727744: RUNNING
  1003.  
  1004. Jul-28 00:42:18.799 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1005. Jul-28 00:42:18.800 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1006. job: 4727744: RUNNING
  1007.  
  1008. Jul-28 00:42:18.800 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1009. Jul-28 00:42:23.799 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1010. job: 4727744: RUNNING
  1011.  
  1012. Jul-28 00:42:23.800 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1013. Jul-28 00:42:23.801 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1014. job: 4727744: RUNNING
  1015.  
  1016. Jul-28 00:42:23.801 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1017. Jul-28 00:42:28.799 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1018. job: 4727744: RUNNING
  1019.  
  1020. Jul-28 00:42:28.799 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1021. Jul-28 00:42:28.800 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1022. job: 4727744: RUNNING
  1023.  
  1024. Jul-28 00:42:28.800 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1025. Jul-28 00:42:33.799 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1026. job: 4727744: RUNNING
  1027.  
  1028. Jul-28 00:42:33.799 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1029. Jul-28 00:42:33.800 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1030. job: 4727744: RUNNING
  1031.  
  1032. Jul-28 00:42:33.800 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1033. Jul-28 00:42:38.799 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q short
  1034. Jul-28 00:42:38.999 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  1035. 4727744,RUN,Jul 27 20:22
  1036.  
  1037.  
  1038. Jul-28 00:42:39.000 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1039. job: 4727744: RUNNING
  1040.  
  1041. Jul-28 00:42:39.000 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1042. Jul-28 00:42:39.001 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1043. job: 4727744: RUNNING
  1044.  
  1045. Jul-28 00:42:39.001 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1046. Jul-28 00:42:43.799 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1047. job: 4727744: RUNNING
  1048.  
  1049. Jul-28 00:42:43.799 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1050. Jul-28 00:42:43.800 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1051. job: 4727744: RUNNING
  1052.  
  1053. Jul-28 00:42:43.800 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1054. Jul-28 00:42:48.798 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1055. job: 4727744: RUNNING
  1056.  
  1057. Jul-28 00:42:48.799 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1058. Jul-28 00:42:48.800 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1059. job: 4727744: RUNNING
  1060.  
  1061. Jul-28 00:42:48.800 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1062. Jul-28 00:42:53.799 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1063. job: 4727744: RUNNING
  1064.  
  1065. Jul-28 00:42:53.799 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1066. Jul-28 00:42:53.800 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1067. job: 4727744: RUNNING
  1068.  
  1069. Jul-28 00:42:53.800 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1070. Jul-28 00:42:58.798 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1071. job: 4727744: RUNNING
  1072.  
  1073. Jul-28 00:42:58.799 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1074. Jul-28 00:42:58.799 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1075. job: 4727744: RUNNING
  1076.  
  1077. Jul-28 00:42:58.799 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1078. Jul-28 00:43:03.799 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1079. job: 4727744: RUNNING
  1080.  
  1081. Jul-28 00:43:03.799 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1082. Jul-28 00:43:03.800 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1083. job: 4727744: RUNNING
  1084.  
  1085. Jul-28 00:43:03.800 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1086. Jul-28 00:43:08.798 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1087. job: 4727744: RUNNING
  1088.  
  1089. Jul-28 00:43:08.799 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1090. Jul-28 00:43:08.799 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1091. job: 4727744: RUNNING
  1092.  
  1093. Jul-28 00:43:08.799 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1094. Jul-28 00:43:13.799 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1095. job: 4727744: RUNNING
  1096.  
  1097. Jul-28 00:43:13.800 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1098. Jul-28 00:43:13.801 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1099. job: 4727744: RUNNING
  1100.  
  1101. Jul-28 00:43:13.801 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1102. Jul-28 00:43:18.799 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1103. job: 4727744: RUNNING
  1104.  
  1105. Jul-28 00:43:18.799 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1106. Jul-28 00:43:18.800 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1107. job: 4727744: RUNNING
  1108.  
  1109. Jul-28 00:43:18.800 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1110. Jul-28 00:43:23.799 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1111. job: 4727744: RUNNING
  1112.  
  1113. Jul-28 00:43:23.800 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1114. Jul-28 00:43:23.800 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1115. job: 4727744: RUNNING
  1116.  
  1117. Jul-28 00:43:23.800 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1118. Jul-28 00:43:28.799 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1119. job: 4727744: RUNNING
  1120.  
  1121. Jul-28 00:43:28.799 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1122. Jul-28 00:43:28.800 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1123. job: 4727744: RUNNING
  1124.  
  1125. Jul-28 00:43:28.800 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1126. Jul-28 00:43:33.800 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1127. job: 4727744: RUNNING
  1128.  
  1129. Jul-28 00:43:33.800 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1130. Jul-28 00:43:33.801 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1131. job: 4727744: RUNNING
  1132.  
  1133. Jul-28 00:43:33.801 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1134. Jul-28 00:43:38.799 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1135. job: 4727744: RUNNING
  1136.  
  1137. Jul-28 00:43:38.800 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1138. Jul-28 00:43:38.800 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  1139. Jul-28 00:43:38.934 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  1140. 4713446,RUN,Jul 27 11:44
  1141. 4728323,RUN,Jul 28 00:40
  1142. 4728321,RUN,Jul 28 00:34
  1143. 4728286,RUN,Jul 27 23:44
  1144.  
  1145.  
  1146. Jul-28 00:43:38.934 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1147. job: 4713446: RUNNING
  1148. job: 4728323: RUNNING
  1149. job: 4728321: RUNNING
  1150. job: 4728286: RUNNING
  1151.  
  1152. Jul-28 00:43:38.934 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728323` active status: true
  1153. Jul-28 00:43:43.800 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1154. job: 4713446: RUNNING
  1155. job: 4728323: RUNNING
  1156. job: 4728321: RUNNING
  1157. job: 4728286: RUNNING
  1158.  
  1159. Jul-28 00:43:43.800 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4727744`
  1160. Jul-28 00:43:43.801 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1161. job: 4713446: RUNNING
  1162. job: 4728323: RUNNING
  1163. job: 4728321: RUNNING
  1164. job: 4728286: RUNNING
  1165.  
  1166. Jul-28 00:43:43.801 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728323` active status: true
  1167. Jul-28 00:43:48.796 [Task monitor] DEBUG nextflow.file.FileHelper - NFS path (true): /farline/umw_job_dekker/HPCC/sv49w/distiller-nf/work
  1168. Jul-28 00:43:48.853 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1169. job: 4713446: RUNNING
  1170. job: 4728323: RUNNING
  1171. job: 4728321: RUNNING
  1172. job: 4728286: RUNNING
  1173.  
  1174. Jul-28 00:43:48.853 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4727744`
  1175. Jul-28 00:43:48.854 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1176. job: 4713446: RUNNING
  1177. job: 4728323: RUNNING
  1178. job: 4728321: RUNNING
  1179. job: 4728286: RUNNING
  1180.  
  1181. Jul-28 00:43:48.854 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728323` active status: true
  1182. Jul-28 00:43:53.826 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1183. job: 4713446: RUNNING
  1184. job: 4728323: RUNNING
  1185. job: 4728321: RUNNING
  1186. job: 4728286: RUNNING
  1187.  
  1188. Jul-28 00:43:53.826 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4727744`
  1189. Jul-28 00:43:53.826 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1190. job: 4713446: RUNNING
  1191. job: 4728323: RUNNING
  1192. job: 4728321: RUNNING
  1193. job: 4728286: RUNNING
  1194.  
  1195. Jul-28 00:43:53.826 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728323` active status: true
  1196. Jul-28 00:43:58.825 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1197. job: 4713446: RUNNING
  1198. job: 4728323: RUNNING
  1199. job: 4728321: RUNNING
  1200. job: 4728286: RUNNING
  1201.  
  1202. Jul-28 00:43:58.825 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4727744`
  1203. Jul-28 00:43:58.826 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1204. job: 4713446: RUNNING
  1205. job: 4728323: RUNNING
  1206. job: 4728321: RUNNING
  1207. job: 4728286: RUNNING
  1208.  
  1209. Jul-28 00:43:58.826 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728323` active status: true
  1210. Jul-28 00:44:03.831 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1211. job: 4713446: RUNNING
  1212. job: 4728323: RUNNING
  1213. job: 4728321: RUNNING
  1214. job: 4728286: RUNNING
  1215.  
  1216. Jul-28 00:44:03.831 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4727744`
  1217. Jul-28 00:44:03.832 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1218. job: 4713446: RUNNING
  1219. job: 4728323: RUNNING
  1220. job: 4728321: RUNNING
  1221. job: 4728286: RUNNING
  1222.  
  1223. Jul-28 00:44:03.832 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728323` active status: true
  1224. Jul-28 00:44:08.823 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1225. job: 4713446: RUNNING
  1226. job: 4728323: RUNNING
  1227. job: 4728321: RUNNING
  1228. job: 4728286: RUNNING
  1229.  
  1230. Jul-28 00:44:08.823 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4727744`
  1231. Jul-28 00:44:08.824 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1232. job: 4713446: RUNNING
  1233. job: 4728323: RUNNING
  1234. job: 4728321: RUNNING
  1235. job: 4728286: RUNNING
  1236.  
  1237. Jul-28 00:44:08.824 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728323` active status: true
  1238. Jul-28 00:44:13.825 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1239. job: 4713446: RUNNING
  1240. job: 4728323: RUNNING
  1241. job: 4728321: RUNNING
  1242. job: 4728286: RUNNING
  1243.  
  1244. Jul-28 00:44:13.825 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4727744`
  1245. Jul-28 00:44:13.826 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1246. job: 4713446: RUNNING
  1247. job: 4728323: RUNNING
  1248. job: 4728321: RUNNING
  1249. job: 4728286: RUNNING
  1250.  
  1251. Jul-28 00:44:13.826 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728323` active status: true
  1252. Jul-28 00:44:18.817 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1253. job: 4713446: RUNNING
  1254. job: 4728323: RUNNING
  1255. job: 4728321: RUNNING
  1256. job: 4728286: RUNNING
  1257.  
  1258. Jul-28 00:44:18.817 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4727744`
  1259. Jul-28 00:44:18.817 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1260. job: 4713446: RUNNING
  1261. job: 4728323: RUNNING
  1262. job: 4728321: RUNNING
  1263. job: 4728286: RUNNING
  1264.  
  1265. Jul-28 00:44:18.817 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728323` active status: true
  1266. Jul-28 00:44:24.132 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1267. job: 4713446: RUNNING
  1268. job: 4728323: RUNNING
  1269. job: 4728321: RUNNING
  1270. job: 4728286: RUNNING
  1271.  
  1272. Jul-28 00:44:24.132 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4727744`
  1273. Jul-28 00:44:24.133 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1274. job: 4713446: RUNNING
  1275. job: 4728323: RUNNING
  1276. job: 4728321: RUNNING
  1277. job: 4728286: RUNNING
  1278.  
  1279. Jul-28 00:44:24.133 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728323` active status: true
  1280. Jul-28 00:44:28.830 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1281. job: 4713446: RUNNING
  1282. job: 4728323: RUNNING
  1283. job: 4728321: RUNNING
  1284. job: 4728286: RUNNING
  1285.  
  1286. Jul-28 00:44:28.830 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4727744`
  1287. Jul-28 00:44:28.831 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1288. job: 4713446: RUNNING
  1289. job: 4728323: RUNNING
  1290. job: 4728321: RUNNING
  1291. job: 4728286: RUNNING
  1292.  
  1293. Jul-28 00:44:28.831 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728323` active status: true
  1294. Jul-28 00:44:34.023 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1295. job: 4713446: RUNNING
  1296. job: 4728323: RUNNING
  1297. job: 4728321: RUNNING
  1298. job: 4728286: RUNNING
  1299.  
  1300. Jul-28 00:44:34.023 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4727744`
  1301. Jul-28 00:44:34.024 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1302. job: 4713446: RUNNING
  1303. job: 4728323: RUNNING
  1304. job: 4728321: RUNNING
  1305. job: 4728286: RUNNING
  1306.  
  1307. Jul-28 00:44:34.024 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728323` active status: true
  1308. Jul-28 00:44:38.822 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q short
  1309. Jul-28 00:44:38.887 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  1310. 4727744,RUN,Jul 27 20:22
  1311.  
  1312.  
  1313. Jul-28 00:44:38.887 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1314. job: 4727744: RUNNING
  1315.  
  1316. Jul-28 00:44:38.887 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1317. Jul-28 00:44:38.889 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1318. job: 4727744: RUNNING
  1319.  
  1320. Jul-28 00:44:38.889 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1321. Jul-28 00:44:43.826 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1322. job: 4727744: RUNNING
  1323.  
  1324. Jul-28 00:44:43.826 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1325. Jul-28 00:44:43.842 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1326. job: 4727744: RUNNING
  1327.  
  1328. Jul-28 00:44:43.842 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1329. Jul-28 00:44:48.823 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1330. job: 4727744: RUNNING
  1331.  
  1332. Jul-28 00:44:48.823 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1333. Jul-28 00:44:48.839 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1334. job: 4727744: RUNNING
  1335.  
  1336. Jul-28 00:44:48.839 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1337. Jul-28 00:44:53.826 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1338. job: 4727744: RUNNING
  1339.  
  1340. Jul-28 00:44:53.826 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1341. Jul-28 00:44:53.846 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1342. job: 4727744: RUNNING
  1343.  
  1344. Jul-28 00:44:53.846 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1345. Jul-28 00:44:58.826 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1346. job: 4727744: RUNNING
  1347.  
  1348. Jul-28 00:44:58.826 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1349. Jul-28 00:44:58.843 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1350. job: 4727744: RUNNING
  1351.  
  1352. Jul-28 00:44:58.843 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1353. Jul-28 00:45:03.825 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1354. job: 4727744: RUNNING
  1355.  
  1356. Jul-28 00:45:03.825 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1357. Jul-28 00:45:03.841 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1358. job: 4727744: RUNNING
  1359.  
  1360. Jul-28 00:45:03.841 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1361. Jul-28 00:45:08.805 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor lsf > tasks to be completed: 2 -- pending tasks are shown below
  1362. ~> TaskHandler[jobId: 4727744; id: 148; name: merge_pairsam_into_runs (library:HeLa1 run:lane3); status: RUNNING; exit: -; error: -; workDir: /farline/umw_job_dekker/HPCC/sv49w/distiller-nf/work/c0/e9870618be7003929109a336301bca started: 1501203352792; exited: -; ]
  1363. ~> TaskHandler[jobId: 4728323; id: 151; name: merge_pairsam_into_runs (library:HeLa1 run:lane1); status: RUNNING; exit: -; error: -; workDir: /farline/umw_job_dekker/HPCC/sv49w/distiller-nf/work/81/e67006b79d110316327d1d4be71a71 started: 1501216878802; exited: -; ]
  1364. Jul-28 00:45:08.831 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1365. job: 4727744: RUNNING
  1366.  
  1367. Jul-28 00:45:08.832 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1368. Jul-28 00:45:08.849 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1369. job: 4727744: RUNNING
  1370.  
  1371. Jul-28 00:45:08.849 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1372. Jul-28 00:45:13.845 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1373. job: 4727744: RUNNING
  1374.  
  1375. Jul-28 00:45:13.845 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1376. Jul-28 00:45:13.866 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1377. job: 4727744: RUNNING
  1378.  
  1379. Jul-28 00:45:13.867 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1380. Jul-28 00:45:18.844 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1381. job: 4727744: RUNNING
  1382.  
  1383. Jul-28 00:45:18.844 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1384. Jul-28 00:45:18.867 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1385. job: 4727744: RUNNING
  1386.  
  1387. Jul-28 00:45:18.867 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1388. Jul-28 00:45:23.829 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1389. job: 4727744: RUNNING
  1390.  
  1391. Jul-28 00:45:23.829 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1392. Jul-28 00:45:23.846 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1393. job: 4727744: RUNNING
  1394.  
  1395. Jul-28 00:45:23.846 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1396. Jul-28 00:45:28.825 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1397. job: 4727744: RUNNING
  1398.  
  1399. Jul-28 00:45:28.826 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1400. Jul-28 00:45:28.842 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1401. job: 4727744: RUNNING
  1402.  
  1403. Jul-28 00:45:28.842 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1404. Jul-28 00:45:33.828 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1405. job: 4727744: RUNNING
  1406.  
  1407. Jul-28 00:45:33.828 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1408. Jul-28 00:45:33.843 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1409. job: 4727744: RUNNING
  1410.  
  1411. Jul-28 00:45:33.843 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1412. Jul-28 00:45:39.138 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q short
  1413. Jul-28 00:45:39.633 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  1414. 4727744,RUN,Jul 27 20:22
  1415.  
  1416.  
  1417. Jul-28 00:45:39.633 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1418. job: 4727744: RUNNING
  1419.  
  1420. Jul-28 00:45:39.633 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1421. Jul-28 00:45:39.650 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1422. job: 4727744: RUNNING
  1423.  
  1424. Jul-28 00:45:39.650 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1425. Jul-28 00:45:43.836 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1426. job: 4727744: RUNNING
  1427.  
  1428. Jul-28 00:45:43.836 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1429. Jul-28 00:45:43.854 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1430. job: 4727744: RUNNING
  1431.  
  1432. Jul-28 00:45:43.854 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1433. Jul-28 00:45:48.832 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1434. job: 4727744: RUNNING
  1435.  
  1436. Jul-28 00:45:48.832 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1437. Jul-28 00:45:48.848 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1438. job: 4727744: RUNNING
  1439.  
  1440. Jul-28 00:45:48.848 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1441. Jul-28 00:45:53.831 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1442. job: 4727744: RUNNING
  1443.  
  1444. Jul-28 00:45:53.831 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1445. Jul-28 00:45:53.849 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1446. job: 4727744: RUNNING
  1447.  
  1448. Jul-28 00:45:53.849 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1449. Jul-28 00:45:58.832 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1450. job: 4727744: RUNNING
  1451.  
  1452. Jul-28 00:45:58.832 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1453. Jul-28 00:45:58.847 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1454. job: 4727744: RUNNING
  1455.  
  1456. Jul-28 00:45:58.847 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1457. Jul-28 00:46:03.829 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1458. job: 4727744: RUNNING
  1459.  
  1460. Jul-28 00:46:03.829 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1461. Jul-28 00:46:03.844 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1462. job: 4727744: RUNNING
  1463.  
  1464. Jul-28 00:46:03.845 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1465. Jul-28 00:46:08.831 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1466. job: 4727744: RUNNING
  1467.  
  1468. Jul-28 00:46:08.831 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1469. Jul-28 00:46:08.847 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1470. job: 4727744: RUNNING
  1471.  
  1472. Jul-28 00:46:08.847 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1473. Jul-28 00:46:13.831 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1474. job: 4727744: RUNNING
  1475.  
  1476. Jul-28 00:46:13.831 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1477. Jul-28 00:46:13.846 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1478. job: 4727744: RUNNING
  1479.  
  1480. Jul-28 00:46:13.846 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1481. Jul-28 00:46:18.845 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1482. job: 4727744: RUNNING
  1483.  
  1484. Jul-28 00:46:18.845 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1485. Jul-28 00:46:18.865 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1486. job: 4727744: RUNNING
  1487.  
  1488. Jul-28 00:46:18.865 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1489. Jul-28 00:46:23.842 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1490. job: 4727744: RUNNING
  1491.  
  1492. Jul-28 00:46:23.842 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1493. Jul-28 00:46:23.860 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1494. job: 4727744: RUNNING
  1495.  
  1496. Jul-28 00:46:23.860 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1497. Jul-28 00:46:28.841 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1498. job: 4727744: RUNNING
  1499.  
  1500. Jul-28 00:46:28.841 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1501. Jul-28 00:46:28.860 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1502. job: 4727744: RUNNING
  1503.  
  1504. Jul-28 00:46:28.860 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1505. Jul-28 00:46:33.847 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1506. job: 4727744: RUNNING
  1507.  
  1508. Jul-28 00:46:33.847 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1509. Jul-28 00:46:33.866 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1510. job: 4727744: RUNNING
  1511.  
  1512. Jul-28 00:46:33.866 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1513. Jul-28 00:46:38.826 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1514. job: 4727744: RUNNING
  1515.  
  1516. Jul-28 00:46:38.826 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1517. Jul-28 00:46:38.837 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1518. job: 4727744: RUNNING
  1519.  
  1520. Jul-28 00:46:38.837 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1521. Jul-28 00:46:43.835 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q short
  1522. Jul-28 00:46:43.967 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  1523. 4727744,RUN,Jul 27 20:22
  1524.  
  1525.  
  1526. Jul-28 00:46:43.967 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1527. job: 4727744: RUNNING
  1528.  
  1529. Jul-28 00:46:43.967 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1530. Jul-28 00:46:43.985 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1531. job: 4727744: RUNNING
  1532.  
  1533. Jul-28 00:46:43.985 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1534. Jul-28 00:46:48.827 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1535. job: 4727744: RUNNING
  1536.  
  1537. Jul-28 00:46:48.827 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1538. Jul-28 00:46:48.838 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1539. job: 4727744: RUNNING
  1540.  
  1541. Jul-28 00:46:48.838 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1542. Jul-28 00:46:53.827 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1543. job: 4727744: RUNNING
  1544.  
  1545. Jul-28 00:46:53.827 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1546. Jul-28 00:46:53.841 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1547. job: 4727744: RUNNING
  1548.  
  1549. Jul-28 00:46:53.841 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1550. Jul-28 00:46:58.834 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1551. job: 4727744: RUNNING
  1552.  
  1553. Jul-28 00:46:58.834 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1554. Jul-28 00:46:58.850 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1555. job: 4727744: RUNNING
  1556.  
  1557. Jul-28 00:46:58.850 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1558. Jul-28 00:47:03.835 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1559. job: 4727744: RUNNING
  1560.  
  1561. Jul-28 00:47:03.835 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1562. Jul-28 00:47:03.854 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1563. job: 4727744: RUNNING
  1564.  
  1565. Jul-28 00:47:03.854 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1566. Jul-28 00:47:08.827 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1567. job: 4727744: RUNNING
  1568.  
  1569. Jul-28 00:47:08.827 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1570. Jul-28 00:47:08.838 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1571. job: 4727744: RUNNING
  1572.  
  1573. Jul-28 00:47:08.838 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1574. Jul-28 00:47:13.832 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1575. job: 4727744: RUNNING
  1576.  
  1577. Jul-28 00:47:13.833 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1578. Jul-28 00:47:13.849 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1579. job: 4727744: RUNNING
  1580.  
  1581. Jul-28 00:47:13.849 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1582. Jul-28 00:47:18.832 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1583. job: 4727744: RUNNING
  1584.  
  1585. Jul-28 00:47:18.832 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1586. Jul-28 00:47:18.849 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1587. job: 4727744: RUNNING
  1588.  
  1589. Jul-28 00:47:18.849 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1590. Jul-28 00:47:23.836 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1591. job: 4727744: RUNNING
  1592.  
  1593. Jul-28 00:47:23.836 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1594. Jul-28 00:47:23.852 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1595. job: 4727744: RUNNING
  1596.  
  1597. Jul-28 00:47:23.852 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1598. Jul-28 00:47:28.841 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1599. job: 4727744: RUNNING
  1600.  
  1601. Jul-28 00:47:28.841 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1602. Jul-28 00:47:28.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1603. job: 4727744: RUNNING
  1604.  
  1605. Jul-28 00:47:28.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1606. Jul-28 00:47:33.841 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1607. job: 4727744: RUNNING
  1608.  
  1609. Jul-28 00:47:33.841 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1610. Jul-28 00:47:33.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1611. job: 4727744: RUNNING
  1612.  
  1613. Jul-28 00:47:33.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1614. Jul-28 00:47:38.848 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1615. job: 4727744: RUNNING
  1616.  
  1617. Jul-28 00:47:38.848 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1618. Jul-28 00:47:38.867 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1619. job: 4727744: RUNNING
  1620.  
  1621. Jul-28 00:47:38.868 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1622. Jul-28 00:47:43.844 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q short
  1623. Jul-28 00:47:43.981 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  1624. 4727744,RUN,Jul 27 20:22
  1625.  
  1626.  
  1627. Jul-28 00:47:43.982 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1628. job: 4727744: RUNNING
  1629.  
  1630. Jul-28 00:47:43.982 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1631. Jul-28 00:47:44.008 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1632. job: 4727744: RUNNING
  1633.  
  1634. Jul-28 00:47:44.008 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1635. Jul-28 00:47:48.836 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1636. job: 4727744: RUNNING
  1637.  
  1638. Jul-28 00:47:48.837 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1639. Jul-28 00:47:48.853 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1640. job: 4727744: RUNNING
  1641.  
  1642. Jul-28 00:47:48.853 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1643. Jul-28 00:47:53.833 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1644. job: 4727744: RUNNING
  1645.  
  1646. Jul-28 00:47:53.833 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1647. Jul-28 00:47:53.849 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1648. job: 4727744: RUNNING
  1649.  
  1650. Jul-28 00:47:53.849 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1651. Jul-28 00:47:58.833 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1652. job: 4727744: RUNNING
  1653.  
  1654. Jul-28 00:47:58.833 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1655. Jul-28 00:47:58.849 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1656. job: 4727744: RUNNING
  1657.  
  1658. Jul-28 00:47:58.849 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1659. Jul-28 00:48:04.138 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1660. job: 4727744: RUNNING
  1661.  
  1662. Jul-28 00:48:04.138 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1663. Jul-28 00:48:04.167 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1664. job: 4727744: RUNNING
  1665.  
  1666. Jul-28 00:48:04.167 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1667. Jul-28 00:48:08.829 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1668. job: 4727744: RUNNING
  1669.  
  1670. Jul-28 00:48:08.829 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1671. Jul-28 00:48:08.839 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1672. job: 4727744: RUNNING
  1673.  
  1674. Jul-28 00:48:08.839 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1675. Jul-28 00:48:13.834 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1676. job: 4727744: RUNNING
  1677.  
  1678. Jul-28 00:48:13.834 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1679. Jul-28 00:48:13.845 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1680. job: 4727744: RUNNING
  1681.  
  1682. Jul-28 00:48:13.845 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1683. Jul-28 00:48:18.844 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1684. job: 4727744: RUNNING
  1685.  
  1686. Jul-28 00:48:18.845 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1687. Jul-28 00:48:18.868 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1688. job: 4727744: RUNNING
  1689.  
  1690. Jul-28 00:48:18.868 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1691. Jul-28 00:48:23.833 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1692. job: 4727744: RUNNING
  1693.  
  1694. Jul-28 00:48:23.834 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1695. Jul-28 00:48:24.048 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1696. job: 4727744: RUNNING
  1697.  
  1698. Jul-28 00:48:24.048 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1699. Jul-28 00:48:28.836 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1700. job: 4727744: RUNNING
  1701.  
  1702. Jul-28 00:48:28.836 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1703. Jul-28 00:48:28.851 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1704. job: 4727744: RUNNING
  1705.  
  1706. Jul-28 00:48:28.851 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1707. Jul-28 00:48:33.840 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1708. job: 4727744: RUNNING
  1709.  
  1710. Jul-28 00:48:33.840 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1711. Jul-28 00:48:33.873 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1712. job: 4727744: RUNNING
  1713.  
  1714. Jul-28 00:48:33.873 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1715. Jul-28 00:48:38.835 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1716. job: 4727744: RUNNING
  1717.  
  1718. Jul-28 00:48:38.835 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1719. Jul-28 00:48:38.853 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1720. job: 4727744: RUNNING
  1721.  
  1722. Jul-28 00:48:38.853 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1723. Jul-28 00:48:43.846 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q short
  1724. Jul-28 00:48:43.985 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  1725. 4727744,RUN,Jul 27 20:22
  1726.  
  1727.  
  1728. Jul-28 00:48:43.985 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1729. job: 4727744: RUNNING
  1730.  
  1731. Jul-28 00:48:43.985 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1732. Jul-28 00:48:44.010 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1733. job: 4727744: RUNNING
  1734.  
  1735. Jul-28 00:48:44.010 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1736. Jul-28 00:48:48.833 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1737. job: 4727744: RUNNING
  1738.  
  1739. Jul-28 00:48:48.833 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1740. Jul-28 00:48:48.850 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1741. job: 4727744: RUNNING
  1742.  
  1743. Jul-28 00:48:48.850 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1744. Jul-28 00:48:53.834 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1745. job: 4727744: RUNNING
  1746.  
  1747. Jul-28 00:48:53.834 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1748. Jul-28 00:48:53.851 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1749. job: 4727744: RUNNING
  1750.  
  1751. Jul-28 00:48:53.851 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1752. Jul-28 00:48:58.837 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1753. job: 4727744: RUNNING
  1754.  
  1755. Jul-28 00:48:58.837 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1756. Jul-28 00:48:58.853 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1757. job: 4727744: RUNNING
  1758.  
  1759. Jul-28 00:48:58.853 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1760. Jul-28 00:49:03.845 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1761. job: 4727744: RUNNING
  1762.  
  1763. Jul-28 00:49:03.845 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1764. Jul-28 00:49:03.866 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1765. job: 4727744: RUNNING
  1766.  
  1767. Jul-28 00:49:03.866 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1768. Jul-28 00:49:08.837 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1769. job: 4727744: RUNNING
  1770.  
  1771. Jul-28 00:49:08.837 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1772. Jul-28 00:49:08.853 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1773. job: 4727744: RUNNING
  1774.  
  1775. Jul-28 00:49:08.853 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1776. Jul-28 00:49:13.838 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1777. job: 4727744: RUNNING
  1778.  
  1779. Jul-28 00:49:13.838 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1780. Jul-28 00:49:13.854 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1781. job: 4727744: RUNNING
  1782.  
  1783. Jul-28 00:49:13.854 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4728323`
  1784. Jul-28 00:49:13.856 [Task monitor] DEBUG nextflow.executor.GridTaskHandler - Failed to get exist status for process TaskHandler[jobId: 4728323; id: 151; name: merge_pairsam_into_runs (library:HeLa1 run:lane1); status: RUNNING; exit: -; error: -; workDir: /farline/umw_job_dekker/HPCC/sv49w/distiller-nf/work/81/e67006b79d110316327d1d4be71a71 started: 1501216878802; exited: -; ] -- exitStatusReadTimeoutMillis: 270000; delta: 274965
  1785. Current queue status:
  1786. > job: 4727744: RUNNING
  1787.  
  1788. Content of workDir: /farline/umw_job_dekker/HPCC/sv49w/distiller-nf/work/81/e67006b79d110316327d1d4be71a71
  1789. > total 133
  1790. > drwxrwxr-x 2 sv49w umw_job_dekker 183 Jul 28 00:41 .
  1791. > drwxrwxr-x 3 sv49w umw_job_dekker 48 Jul 28 00:40 ..
  1792. > -rw-rw-r-- 1 sv49w umw_job_dekker 0 Jul 28 00:41 .command.begin
  1793. > -rw-rw-r-- 1 sv49w umw_job_dekker 21 Jul 28 00:40 .command.env
  1794. > -rw-rw-r-- 1 sv49w umw_job_dekker 43 Jul 28 00:41 .command.log
  1795. > -rw-rw-r-- 1 sv49w umw_job_dekker 3486 Jul 28 00:40 .command.run
  1796. > -rw-rw-r-- 1 sv49w umw_job_dekker 2672 Jul 28 00:40 .command.run.1
  1797. > -rw-rw-r-- 1 sv49w umw_job_dekker 232 Jul 28 00:40 .command.sh
  1798.  
  1799. Jul-28 00:49:13.856 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[jobId: 4728323; id: 151; name: merge_pairsam_into_runs (library:HeLa1 run:lane1); status: COMPLETED; exit: -; error: -; workDir: /farline/umw_job_dekker/HPCC/sv49w/distiller-nf/work/81/e67006b79d110316327d1d4be71a71 started: 1501216878802; exited: -; ]
  1800. Jul-28 00:49:13.858 [Task monitor] WARN nextflow.processor.TaskProcessor - Process `merge_pairsam_into_runs (library:HeLa1 run:lane1)` terminated for an unknown reason -- Likely it has been terminated by the external system -- Execution is retried (2)
  1801. Jul-28 00:49:14.060 [Task submitter] DEBUG nextflow.executor.GridTaskHandler - Submitted process merge_pairsam_into_runs (library:HeLa1 run:lane1) > lsf jobId: 4728325; workDir: /farline/umw_job_dekker/HPCC/sv49w/distiller-nf/work/3b/37ed2fa2ba160f54196e762e24c83b
  1802. Jul-28 00:49:14.060 [Task submitter] INFO nextflow.Session - [3b/37ed2f] Re-submitted process > merge_pairsam_into_runs (library:HeLa1 run:lane1)
  1803. Jul-28 00:49:18.839 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1804. job: 4727744: RUNNING
  1805.  
  1806. Jul-28 00:49:18.839 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1807. Jul-28 00:49:23.847 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1808. job: 4727744: RUNNING
  1809.  
  1810. Jul-28 00:49:23.847 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1811. Jul-28 00:49:28.829 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1812. job: 4727744: RUNNING
  1813.  
  1814. Jul-28 00:49:28.829 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1815. Jul-28 00:49:33.837 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1816. job: 4727744: RUNNING
  1817.  
  1818. Jul-28 00:49:33.837 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1819. Jul-28 00:49:38.836 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1820. job: 4727744: RUNNING
  1821.  
  1822. Jul-28 00:49:38.836 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1823. Jul-28 00:49:43.839 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1824. job: 4727744: RUNNING
  1825.  
  1826. Jul-28 00:49:43.839 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1827. Jul-28 00:49:48.848 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q short
  1828. Jul-28 00:49:48.986 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  1829. 4727744,RUN,Jul 27 20:22
  1830.  
  1831.  
  1832. Jul-28 00:49:48.987 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1833. job: 4727744: RUNNING
  1834.  
  1835. Jul-28 00:49:48.987 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1836. Jul-28 00:49:53.847 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1837. job: 4727744: RUNNING
  1838.  
  1839. Jul-28 00:49:53.847 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1840. Jul-28 00:49:58.840 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1841. job: 4727744: RUNNING
  1842.  
  1843. Jul-28 00:49:58.840 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1844. Jul-28 00:50:03.843 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1845. job: 4727744: RUNNING
  1846.  
  1847. Jul-28 00:50:03.843 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1848. Jul-28 00:50:08.821 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor lsf > tasks to be completed: 2 -- pending tasks are shown below
  1849. ~> TaskHandler[jobId: 4727744; id: 148; name: merge_pairsam_into_runs (library:HeLa1 run:lane3); status: RUNNING; exit: -; error: -; workDir: /farline/umw_job_dekker/HPCC/sv49w/distiller-nf/work/c0/e9870618be7003929109a336301bca started: 1501203352792; exited: -; ]
  1850. ~> TaskHandler[jobId: 4728325; id: 152; name: merge_pairsam_into_runs (library:HeLa1 run:lane1); status: SUBMITTED; exit: -; error: -; workDir: /farline/umw_job_dekker/HPCC/sv49w/distiller-nf/work/3b/37ed2fa2ba160f54196e762e24c83b started: -; exited: -; ]
  1851. Jul-28 00:50:08.844 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1852. job: 4727744: RUNNING
  1853.  
  1854. Jul-28 00:50:08.844 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1855. Jul-28 00:50:13.846 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1856. job: 4727744: RUNNING
  1857.  
  1858. Jul-28 00:50:13.846 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1859. Jul-28 00:50:18.846 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1860. job: 4727744: RUNNING
  1861.  
  1862. Jul-28 00:50:18.846 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1863. Jul-28 00:50:23.846 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1864. job: 4727744: RUNNING
  1865.  
  1866. Jul-28 00:50:23.847 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1867. Jul-28 00:50:28.850 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1868. job: 4727744: RUNNING
  1869.  
  1870. Jul-28 00:50:28.850 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1871. Jul-28 00:50:33.859 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1872. job: 4727744: RUNNING
  1873.  
  1874. Jul-28 00:50:33.860 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1875. Jul-28 00:50:38.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1876. job: 4727744: RUNNING
  1877.  
  1878. Jul-28 00:50:38.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1879. Jul-28 00:50:43.854 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1880. job: 4727744: RUNNING
  1881.  
  1882. Jul-28 00:50:43.855 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1883. Jul-28 00:50:48.851 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q short
  1884. Jul-28 00:50:48.979 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  1885. 4727744,RUN,Jul 27 20:22
  1886.  
  1887.  
  1888. Jul-28 00:50:48.979 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1889. job: 4727744: RUNNING
  1890.  
  1891. Jul-28 00:50:48.979 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1892. Jul-28 00:50:53.847 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1893. job: 4727744: RUNNING
  1894.  
  1895. Jul-28 00:50:53.847 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1896. Jul-28 00:50:58.849 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1897. job: 4727744: RUNNING
  1898.  
  1899. Jul-28 00:50:58.849 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1900. Jul-28 00:51:03.847 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1901. job: 4727744: RUNNING
  1902.  
  1903. Jul-28 00:51:03.847 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1904. Jul-28 00:51:08.848 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1905. job: 4727744: RUNNING
  1906.  
  1907. Jul-28 00:51:08.848 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1908. Jul-28 00:51:13.840 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1909. job: 4727744: RUNNING
  1910.  
  1911. Jul-28 00:51:13.840 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1912. Jul-28 00:51:18.845 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1913. job: 4727744: RUNNING
  1914.  
  1915. Jul-28 00:51:18.845 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1916. Jul-28 00:51:23.847 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1917. job: 4727744: RUNNING
  1918.  
  1919. Jul-28 00:51:23.847 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1920. Jul-28 00:51:28.849 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1921. job: 4727744: RUNNING
  1922.  
  1923. Jul-28 00:51:28.849 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1924. Jul-28 00:51:33.848 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1925. job: 4727744: RUNNING
  1926.  
  1927. Jul-28 00:51:33.848 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1928. Jul-28 00:51:38.855 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1929. job: 4727744: RUNNING
  1930.  
  1931. Jul-28 00:51:38.855 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1932. Jul-28 00:51:43.862 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1933. job: 4727744: RUNNING
  1934.  
  1935. Jul-28 00:51:43.862 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1936. Jul-28 00:51:48.850 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1937. job: 4727744: RUNNING
  1938.  
  1939. Jul-28 00:51:48.850 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4727744` active status: true
  1940. Jul-28 00:51:48.855 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  1941. Jul-28 00:51:48.987 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  1942. 4713446,RUN,Jul 27 11:44
  1943. 4728323,RUN,Jul 28 00:40
  1944. 4728325,RUN,Jul 28 00:49
  1945. 4728321,RUN,Jul 28 00:34
  1946. 4728286,RUN,Jul 27 23:44
  1947.  
  1948.  
  1949. Jul-28 00:51:48.987 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1950. job: 4713446: RUNNING
  1951. job: 4728323: RUNNING
  1952. job: 4728325: RUNNING
  1953. job: 4728321: RUNNING
  1954. job: 4728286: RUNNING
  1955.  
  1956. Jul-28 00:51:48.987 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  1957. Jul-28 00:51:53.842 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1958. job: 4713446: RUNNING
  1959. job: 4728323: RUNNING
  1960. job: 4728325: RUNNING
  1961. job: 4728321: RUNNING
  1962. job: 4728286: RUNNING
  1963.  
  1964. Jul-28 00:51:53.842 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status map does not contain jobId: `4727744`
  1965. Jul-28 00:51:53.843 [Task monitor] DEBUG nextflow.executor.GridTaskHandler - Failed to get exist status for process TaskHandler[jobId: 4727744; id: 148; name: merge_pairsam_into_runs (library:HeLa1 run:lane3); status: RUNNING; exit: -; error: -; workDir: /farline/umw_job_dekker/HPCC/sv49w/distiller-nf/work/c0/e9870618be7003929109a336301bca started: 1501203352792; exited: -; ] -- exitStatusReadTimeoutMillis: 270000; delta: 490042
  1966. Current queue status:
  1967. > job: 4713446: RUNNING
  1968. > job: 4728323: RUNNING
  1969. > job: 4728325: RUNNING
  1970. > job: 4728321: RUNNING
  1971. > job: 4728286: RUNNING
  1972.  
  1973. Content of workDir: /farline/umw_job_dekker/HPCC/sv49w/distiller-nf/work/c0/e9870618be7003929109a336301bca
  1974. > total 133
  1975. > drwxrwxr-x 2 sv49w umw_job_dekker 183 Jul 27 20:55 .
  1976. > drwxrwxr-x 4 sv49w umw_job_dekker 96 Jul 27 20:22 ..
  1977. > -rw-rw-r-- 1 sv49w umw_job_dekker 0 Jul 27 20:55 .command.begin
  1978. > -rw-rw-r-- 1 sv49w umw_job_dekker 21 Jul 27 20:22 .command.env
  1979. > -rw-rw-r-- 1 sv49w umw_job_dekker 43 Jul 27 20:55 .command.log
  1980. > -rw-rw-r-- 1 sv49w umw_job_dekker 3487 Jul 27 20:22 .command.run
  1981. > -rw-rw-r-- 1 sv49w umw_job_dekker 2672 Jul 27 20:22 .command.run.1
  1982. > -rw-rw-r-- 1 sv49w umw_job_dekker 232 Jul 27 20:22 .command.sh
  1983.  
  1984. Jul-28 00:51:53.844 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[jobId: 4727744; id: 148; name: merge_pairsam_into_runs (library:HeLa1 run:lane3); status: COMPLETED; exit: -; error: -; workDir: /farline/umw_job_dekker/HPCC/sv49w/distiller-nf/work/c0/e9870618be7003929109a336301bca started: 1501203352792; exited: -; ]
  1985. Jul-28 00:51:53.844 [Task monitor] WARN nextflow.processor.TaskProcessor - Process `merge_pairsam_into_runs (library:HeLa1 run:lane3)` terminated for an unknown reason -- Likely it has been terminated by the external system -- Execution is retried (1)
  1986. Jul-28 00:51:53.846 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1987. job: 4713446: RUNNING
  1988. job: 4728323: RUNNING
  1989. job: 4728325: RUNNING
  1990. job: 4728321: RUNNING
  1991. job: 4728286: RUNNING
  1992.  
  1993. Jul-28 00:51:53.846 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  1994. Jul-28 00:51:54.045 [Task submitter] DEBUG nextflow.executor.GridTaskHandler - Submitted process merge_pairsam_into_runs (library:HeLa1 run:lane3) > lsf jobId: 4728326; workDir: /farline/umw_job_dekker/HPCC/sv49w/distiller-nf/work/00/2633edc9da9ad5649d3d6dcffe5a44
  1995. Jul-28 00:51:54.045 [Task submitter] INFO nextflow.Session - [00/2633ed] Re-submitted process > merge_pairsam_into_runs (library:HeLa1 run:lane3)
  1996. Jul-28 00:51:58.829 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  1997. job: 4713446: RUNNING
  1998. job: 4728323: RUNNING
  1999. job: 4728325: RUNNING
  2000. job: 4728321: RUNNING
  2001. job: 4728286: RUNNING
  2002.  
  2003. Jul-28 00:51:58.830 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2004. Jul-28 00:52:03.828 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2005. job: 4713446: RUNNING
  2006. job: 4728323: RUNNING
  2007. job: 4728325: RUNNING
  2008. job: 4728321: RUNNING
  2009. job: 4728286: RUNNING
  2010.  
  2011. Jul-28 00:52:03.828 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2012. Jul-28 00:52:08.828 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2013. job: 4713446: RUNNING
  2014. job: 4728323: RUNNING
  2015. job: 4728325: RUNNING
  2016. job: 4728321: RUNNING
  2017. job: 4728286: RUNNING
  2018.  
  2019. Jul-28 00:52:08.828 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2020. Jul-28 00:52:13.828 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2021. job: 4713446: RUNNING
  2022. job: 4728323: RUNNING
  2023. job: 4728325: RUNNING
  2024. job: 4728321: RUNNING
  2025. job: 4728286: RUNNING
  2026.  
  2027. Jul-28 00:52:13.829 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2028. Jul-28 00:52:18.831 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2029. job: 4713446: RUNNING
  2030. job: 4728323: RUNNING
  2031. job: 4728325: RUNNING
  2032. job: 4728321: RUNNING
  2033. job: 4728286: RUNNING
  2034.  
  2035. Jul-28 00:52:18.831 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2036. Jul-28 00:52:23.830 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2037. job: 4713446: RUNNING
  2038. job: 4728323: RUNNING
  2039. job: 4728325: RUNNING
  2040. job: 4728321: RUNNING
  2041. job: 4728286: RUNNING
  2042.  
  2043. Jul-28 00:52:23.831 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2044. Jul-28 00:52:28.830 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2045. job: 4713446: RUNNING
  2046. job: 4728323: RUNNING
  2047. job: 4728325: RUNNING
  2048. job: 4728321: RUNNING
  2049. job: 4728286: RUNNING
  2050.  
  2051. Jul-28 00:52:28.830 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2052. Jul-28 00:52:33.828 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2053. job: 4713446: RUNNING
  2054. job: 4728323: RUNNING
  2055. job: 4728325: RUNNING
  2056. job: 4728321: RUNNING
  2057. job: 4728286: RUNNING
  2058.  
  2059. Jul-28 00:52:33.829 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2060. Jul-28 00:52:38.832 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2061. job: 4713446: RUNNING
  2062. job: 4728323: RUNNING
  2063. job: 4728325: RUNNING
  2064. job: 4728321: RUNNING
  2065. job: 4728286: RUNNING
  2066.  
  2067. Jul-28 00:52:38.833 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2068. Jul-28 00:52:43.829 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2069. job: 4713446: RUNNING
  2070. job: 4728323: RUNNING
  2071. job: 4728325: RUNNING
  2072. job: 4728321: RUNNING
  2073. job: 4728286: RUNNING
  2074.  
  2075. Jul-28 00:52:43.830 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2076. Jul-28 00:52:48.832 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2077. job: 4713446: RUNNING
  2078. job: 4728323: RUNNING
  2079. job: 4728325: RUNNING
  2080. job: 4728321: RUNNING
  2081. job: 4728286: RUNNING
  2082.  
  2083. Jul-28 00:52:48.832 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2084. Jul-28 00:52:53.832 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  2085. Jul-28 00:52:53.978 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  2086. 4713446,RUN,Jul 27 11:44
  2087. 4728323,RUN,Jul 28 00:40
  2088. 4728325,RUN,Jul 28 00:49
  2089. 4728321,RUN,Jul 28 00:34
  2090. 4728286,RUN,Jul 27 23:44
  2091. 4728326,PEND,Jul 28 00:51
  2092.  
  2093.  
  2094. Jul-28 00:52:53.979 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2095. job: 4713446: RUNNING
  2096. job: 4728323: RUNNING
  2097. job: 4728325: RUNNING
  2098. job: 4728321: RUNNING
  2099. job: 4728286: RUNNING
  2100. job: 4728326: PENDING
  2101.  
  2102. Jul-28 00:52:53.979 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2103. Jul-28 00:52:58.831 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2104. job: 4713446: RUNNING
  2105. job: 4728323: RUNNING
  2106. job: 4728325: RUNNING
  2107. job: 4728321: RUNNING
  2108. job: 4728286: RUNNING
  2109. job: 4728326: PENDING
  2110.  
  2111. Jul-28 00:52:58.832 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2112. Jul-28 00:53:03.832 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2113. job: 4713446: RUNNING
  2114. job: 4728323: RUNNING
  2115. job: 4728325: RUNNING
  2116. job: 4728321: RUNNING
  2117. job: 4728286: RUNNING
  2118. job: 4728326: PENDING
  2119.  
  2120. Jul-28 00:53:03.833 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2121. Jul-28 00:53:08.830 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2122. job: 4713446: RUNNING
  2123. job: 4728323: RUNNING
  2124. job: 4728325: RUNNING
  2125. job: 4728321: RUNNING
  2126. job: 4728286: RUNNING
  2127. job: 4728326: PENDING
  2128.  
  2129. Jul-28 00:53:08.831 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2130. Jul-28 00:53:13.831 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2131. job: 4713446: RUNNING
  2132. job: 4728323: RUNNING
  2133. job: 4728325: RUNNING
  2134. job: 4728321: RUNNING
  2135. job: 4728286: RUNNING
  2136. job: 4728326: PENDING
  2137.  
  2138. Jul-28 00:53:13.832 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2139. Jul-28 00:53:18.831 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2140. job: 4713446: RUNNING
  2141. job: 4728323: RUNNING
  2142. job: 4728325: RUNNING
  2143. job: 4728321: RUNNING
  2144. job: 4728286: RUNNING
  2145. job: 4728326: PENDING
  2146.  
  2147. Jul-28 00:53:18.832 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2148. Jul-28 00:53:23.832 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2149. job: 4713446: RUNNING
  2150. job: 4728323: RUNNING
  2151. job: 4728325: RUNNING
  2152. job: 4728321: RUNNING
  2153. job: 4728286: RUNNING
  2154. job: 4728326: PENDING
  2155.  
  2156. Jul-28 00:53:23.832 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2157. Jul-28 00:53:28.835 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2158. job: 4713446: RUNNING
  2159. job: 4728323: RUNNING
  2160. job: 4728325: RUNNING
  2161. job: 4728321: RUNNING
  2162. job: 4728286: RUNNING
  2163. job: 4728326: PENDING
  2164.  
  2165. Jul-28 00:53:28.836 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2166. Jul-28 00:53:33.831 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2167. job: 4713446: RUNNING
  2168. job: 4728323: RUNNING
  2169. job: 4728325: RUNNING
  2170. job: 4728321: RUNNING
  2171. job: 4728286: RUNNING
  2172. job: 4728326: PENDING
  2173.  
  2174. Jul-28 00:53:33.831 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2175. Jul-28 00:53:38.831 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2176. job: 4713446: RUNNING
  2177. job: 4728323: RUNNING
  2178. job: 4728325: RUNNING
  2179. job: 4728321: RUNNING
  2180. job: 4728286: RUNNING
  2181. job: 4728326: PENDING
  2182.  
  2183. Jul-28 00:53:38.831 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2184. Jul-28 00:53:43.830 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2185. job: 4713446: RUNNING
  2186. job: 4728323: RUNNING
  2187. job: 4728325: RUNNING
  2188. job: 4728321: RUNNING
  2189. job: 4728286: RUNNING
  2190. job: 4728326: PENDING
  2191.  
  2192. Jul-28 00:53:43.831 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2193. Jul-28 00:53:48.831 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2194. job: 4713446: RUNNING
  2195. job: 4728323: RUNNING
  2196. job: 4728325: RUNNING
  2197. job: 4728321: RUNNING
  2198. job: 4728286: RUNNING
  2199. job: 4728326: PENDING
  2200.  
  2201. Jul-28 00:53:48.832 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2202. Jul-28 00:53:53.831 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2203. job: 4713446: RUNNING
  2204. job: 4728323: RUNNING
  2205. job: 4728325: RUNNING
  2206. job: 4728321: RUNNING
  2207. job: 4728286: RUNNING
  2208. job: 4728326: PENDING
  2209.  
  2210. Jul-28 00:53:53.831 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2211. Jul-28 00:53:58.831 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  2212. Jul-28 00:53:58.958 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  2213. 4713446,RUN,Jul 27 11:44
  2214. 4728323,RUN,Jul 28 00:40
  2215. 4728325,RUN,Jul 28 00:49
  2216. 4728321,RUN,Jul 28 00:34
  2217. 4728286,RUN,Jul 27 23:44
  2218. 4728326,PEND,Jul 28 00:51
  2219.  
  2220.  
  2221. Jul-28 00:53:58.958 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2222. job: 4713446: RUNNING
  2223. job: 4728323: RUNNING
  2224. job: 4728325: RUNNING
  2225. job: 4728321: RUNNING
  2226. job: 4728286: RUNNING
  2227. job: 4728326: PENDING
  2228.  
  2229. Jul-28 00:53:58.958 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2230. Jul-28 00:54:03.832 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2231. job: 4713446: RUNNING
  2232. job: 4728323: RUNNING
  2233. job: 4728325: RUNNING
  2234. job: 4728321: RUNNING
  2235. job: 4728286: RUNNING
  2236. job: 4728326: PENDING
  2237.  
  2238. Jul-28 00:54:03.832 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2239. Jul-28 00:54:08.834 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2240. job: 4713446: RUNNING
  2241. job: 4728323: RUNNING
  2242. job: 4728325: RUNNING
  2243. job: 4728321: RUNNING
  2244. job: 4728286: RUNNING
  2245. job: 4728326: PENDING
  2246.  
  2247. Jul-28 00:54:08.835 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2248. Jul-28 00:54:13.835 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2249. job: 4713446: RUNNING
  2250. job: 4728323: RUNNING
  2251. job: 4728325: RUNNING
  2252. job: 4728321: RUNNING
  2253. job: 4728286: RUNNING
  2254. job: 4728326: PENDING
  2255.  
  2256. Jul-28 00:54:13.835 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2257. Jul-28 00:54:18.836 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2258. job: 4713446: RUNNING
  2259. job: 4728323: RUNNING
  2260. job: 4728325: RUNNING
  2261. job: 4728321: RUNNING
  2262. job: 4728286: RUNNING
  2263. job: 4728326: PENDING
  2264.  
  2265. Jul-28 00:54:18.837 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2266. Jul-28 00:54:23.832 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2267. job: 4713446: RUNNING
  2268. job: 4728323: RUNNING
  2269. job: 4728325: RUNNING
  2270. job: 4728321: RUNNING
  2271. job: 4728286: RUNNING
  2272. job: 4728326: PENDING
  2273.  
  2274. Jul-28 00:54:23.832 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2275. Jul-28 00:54:28.832 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2276. job: 4713446: RUNNING
  2277. job: 4728323: RUNNING
  2278. job: 4728325: RUNNING
  2279. job: 4728321: RUNNING
  2280. job: 4728286: RUNNING
  2281. job: 4728326: PENDING
  2282.  
  2283. Jul-28 00:54:28.833 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2284. Jul-28 00:54:33.833 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2285. job: 4713446: RUNNING
  2286. job: 4728323: RUNNING
  2287. job: 4728325: RUNNING
  2288. job: 4728321: RUNNING
  2289. job: 4728286: RUNNING
  2290. job: 4728326: PENDING
  2291.  
  2292. Jul-28 00:54:33.833 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2293. Jul-28 00:54:38.834 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2294. job: 4713446: RUNNING
  2295. job: 4728323: RUNNING
  2296. job: 4728325: RUNNING
  2297. job: 4728321: RUNNING
  2298. job: 4728286: RUNNING
  2299. job: 4728326: PENDING
  2300.  
  2301. Jul-28 00:54:38.835 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2302. Jul-28 00:54:43.836 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2303. job: 4713446: RUNNING
  2304. job: 4728323: RUNNING
  2305. job: 4728325: RUNNING
  2306. job: 4728321: RUNNING
  2307. job: 4728286: RUNNING
  2308. job: 4728326: PENDING
  2309.  
  2310. Jul-28 00:54:43.836 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2311. Jul-28 00:54:48.835 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2312. job: 4713446: RUNNING
  2313. job: 4728323: RUNNING
  2314. job: 4728325: RUNNING
  2315. job: 4728321: RUNNING
  2316. job: 4728286: RUNNING
  2317. job: 4728326: PENDING
  2318.  
  2319. Jul-28 00:54:48.836 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2320. Jul-28 00:54:53.834 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2321. job: 4713446: RUNNING
  2322. job: 4728323: RUNNING
  2323. job: 4728325: RUNNING
  2324. job: 4728321: RUNNING
  2325. job: 4728286: RUNNING
  2326. job: 4728326: PENDING
  2327.  
  2328. Jul-28 00:54:53.834 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2329. Jul-28 00:54:58.833 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  2330. Jul-28 00:54:58.959 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  2331. 4713446,RUN,Jul 27 11:44
  2332. 4728323,RUN,Jul 28 00:40
  2333. 4728325,RUN,Jul 28 00:49
  2334. 4728321,RUN,Jul 28 00:34
  2335. 4728286,RUN,Jul 27 23:44
  2336. 4728326,PEND,Jul 28 00:51
  2337.  
  2338.  
  2339. Jul-28 00:54:58.959 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2340. job: 4713446: RUNNING
  2341. job: 4728323: RUNNING
  2342. job: 4728325: RUNNING
  2343. job: 4728321: RUNNING
  2344. job: 4728286: RUNNING
  2345. job: 4728326: PENDING
  2346.  
  2347. Jul-28 00:54:58.959 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2348. Jul-28 00:55:03.837 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2349. job: 4713446: RUNNING
  2350. job: 4728323: RUNNING
  2351. job: 4728325: RUNNING
  2352. job: 4728321: RUNNING
  2353. job: 4728286: RUNNING
  2354. job: 4728326: PENDING
  2355.  
  2356. Jul-28 00:55:03.837 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2357. Jul-28 00:55:08.835 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor lsf > tasks to be completed: 2 -- pending tasks are shown below
  2358. ~> TaskHandler[jobId: 4728325; id: 152; name: merge_pairsam_into_runs (library:HeLa1 run:lane1); status: RUNNING; exit: -; error: -; workDir: /farline/umw_job_dekker/HPCC/sv49w/distiller-nf/work/3b/37ed2fa2ba160f54196e762e24c83b started: 1501217508853; exited: -; ]
  2359. ~> TaskHandler[jobId: 4728326; id: 153; name: merge_pairsam_into_runs (library:HeLa1 run:lane3); status: SUBMITTED; exit: -; error: -; workDir: /farline/umw_job_dekker/HPCC/sv49w/distiller-nf/work/00/2633edc9da9ad5649d3d6dcffe5a44 started: -; exited: -; ]
  2360. Jul-28 00:55:08.836 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2361. job: 4713446: RUNNING
  2362. job: 4728323: RUNNING
  2363. job: 4728325: RUNNING
  2364. job: 4728321: RUNNING
  2365. job: 4728286: RUNNING
  2366. job: 4728326: PENDING
  2367.  
  2368. Jul-28 00:55:08.836 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2369. Jul-28 00:55:13.837 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2370. job: 4713446: RUNNING
  2371. job: 4728323: RUNNING
  2372. job: 4728325: RUNNING
  2373. job: 4728321: RUNNING
  2374. job: 4728286: RUNNING
  2375. job: 4728326: PENDING
  2376.  
  2377. Jul-28 00:55:13.838 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2378. Jul-28 00:55:18.838 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2379. job: 4713446: RUNNING
  2380. job: 4728323: RUNNING
  2381. job: 4728325: RUNNING
  2382. job: 4728321: RUNNING
  2383. job: 4728286: RUNNING
  2384. job: 4728326: PENDING
  2385.  
  2386. Jul-28 00:55:18.838 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2387. Jul-28 00:55:23.840 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2388. job: 4713446: RUNNING
  2389. job: 4728323: RUNNING
  2390. job: 4728325: RUNNING
  2391. job: 4728321: RUNNING
  2392. job: 4728286: RUNNING
  2393. job: 4728326: PENDING
  2394.  
  2395. Jul-28 00:55:23.841 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2396. Jul-28 00:55:28.839 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2397. job: 4713446: RUNNING
  2398. job: 4728323: RUNNING
  2399. job: 4728325: RUNNING
  2400. job: 4728321: RUNNING
  2401. job: 4728286: RUNNING
  2402. job: 4728326: PENDING
  2403.  
  2404. Jul-28 00:55:28.840 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2405. Jul-28 00:55:33.837 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2406. job: 4713446: RUNNING
  2407. job: 4728323: RUNNING
  2408. job: 4728325: RUNNING
  2409. job: 4728321: RUNNING
  2410. job: 4728286: RUNNING
  2411. job: 4728326: PENDING
  2412.  
  2413. Jul-28 00:55:33.837 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2414. Jul-28 00:55:38.840 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2415. job: 4713446: RUNNING
  2416. job: 4728323: RUNNING
  2417. job: 4728325: RUNNING
  2418. job: 4728321: RUNNING
  2419. job: 4728286: RUNNING
  2420. job: 4728326: PENDING
  2421.  
  2422. Jul-28 00:55:38.841 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2423. Jul-28 00:55:43.840 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2424. job: 4713446: RUNNING
  2425. job: 4728323: RUNNING
  2426. job: 4728325: RUNNING
  2427. job: 4728321: RUNNING
  2428. job: 4728286: RUNNING
  2429. job: 4728326: PENDING
  2430.  
  2431. Jul-28 00:55:43.840 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2432. Jul-28 00:55:48.838 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2433. job: 4713446: RUNNING
  2434. job: 4728323: RUNNING
  2435. job: 4728325: RUNNING
  2436. job: 4728321: RUNNING
  2437. job: 4728286: RUNNING
  2438. job: 4728326: PENDING
  2439.  
  2440. Jul-28 00:55:48.838 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2441. Jul-28 00:55:53.838 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2442. job: 4713446: RUNNING
  2443. job: 4728323: RUNNING
  2444. job: 4728325: RUNNING
  2445. job: 4728321: RUNNING
  2446. job: 4728286: RUNNING
  2447. job: 4728326: PENDING
  2448.  
  2449. Jul-28 00:55:53.838 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2450. Jul-28 00:55:58.838 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  2451. Jul-28 00:55:58.973 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  2452. 4713446,RUN,Jul 27 11:44
  2453. 4728323,RUN,Jul 28 00:40
  2454. 4728325,RUN,Jul 28 00:49
  2455. 4728321,RUN,Jul 28 00:34
  2456. 4728286,RUN,Jul 27 23:44
  2457. 4728326,PEND,Jul 28 00:51
  2458.  
  2459.  
  2460. Jul-28 00:55:58.973 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2461. job: 4713446: RUNNING
  2462. job: 4728323: RUNNING
  2463. job: 4728325: RUNNING
  2464. job: 4728321: RUNNING
  2465. job: 4728286: RUNNING
  2466. job: 4728326: PENDING
  2467.  
  2468. Jul-28 00:55:58.973 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2469. Jul-28 00:56:03.841 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2470. job: 4713446: RUNNING
  2471. job: 4728323: RUNNING
  2472. job: 4728325: RUNNING
  2473. job: 4728321: RUNNING
  2474. job: 4728286: RUNNING
  2475. job: 4728326: PENDING
  2476.  
  2477. Jul-28 00:56:03.841 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2478. Jul-28 00:56:08.841 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2479. job: 4713446: RUNNING
  2480. job: 4728323: RUNNING
  2481. job: 4728325: RUNNING
  2482. job: 4728321: RUNNING
  2483. job: 4728286: RUNNING
  2484. job: 4728326: PENDING
  2485.  
  2486. Jul-28 00:56:08.841 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2487. Jul-28 00:56:13.840 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2488. job: 4713446: RUNNING
  2489. job: 4728323: RUNNING
  2490. job: 4728325: RUNNING
  2491. job: 4728321: RUNNING
  2492. job: 4728286: RUNNING
  2493. job: 4728326: PENDING
  2494.  
  2495. Jul-28 00:56:13.841 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2496. Jul-28 00:56:18.840 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2497. job: 4713446: RUNNING
  2498. job: 4728323: RUNNING
  2499. job: 4728325: RUNNING
  2500. job: 4728321: RUNNING
  2501. job: 4728286: RUNNING
  2502. job: 4728326: PENDING
  2503.  
  2504. Jul-28 00:56:18.841 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2505. Jul-28 00:56:23.839 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2506. job: 4713446: RUNNING
  2507. job: 4728323: RUNNING
  2508. job: 4728325: RUNNING
  2509. job: 4728321: RUNNING
  2510. job: 4728286: RUNNING
  2511. job: 4728326: PENDING
  2512.  
  2513. Jul-28 00:56:23.839 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2514. Jul-28 00:56:28.839 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2515. job: 4713446: RUNNING
  2516. job: 4728323: RUNNING
  2517. job: 4728325: RUNNING
  2518. job: 4728321: RUNNING
  2519. job: 4728286: RUNNING
  2520. job: 4728326: PENDING
  2521.  
  2522. Jul-28 00:56:28.840 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2523. Jul-28 00:56:33.839 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2524. job: 4713446: RUNNING
  2525. job: 4728323: RUNNING
  2526. job: 4728325: RUNNING
  2527. job: 4728321: RUNNING
  2528. job: 4728286: RUNNING
  2529. job: 4728326: PENDING
  2530.  
  2531. Jul-28 00:56:33.840 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2532. Jul-28 00:56:38.839 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2533. job: 4713446: RUNNING
  2534. job: 4728323: RUNNING
  2535. job: 4728325: RUNNING
  2536. job: 4728321: RUNNING
  2537. job: 4728286: RUNNING
  2538. job: 4728326: PENDING
  2539.  
  2540. Jul-28 00:56:38.840 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2541. Jul-28 00:56:43.839 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2542. job: 4713446: RUNNING
  2543. job: 4728323: RUNNING
  2544. job: 4728325: RUNNING
  2545. job: 4728321: RUNNING
  2546. job: 4728286: RUNNING
  2547. job: 4728326: PENDING
  2548.  
  2549. Jul-28 00:56:43.839 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2550. Jul-28 00:56:48.839 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2551. job: 4713446: RUNNING
  2552. job: 4728323: RUNNING
  2553. job: 4728325: RUNNING
  2554. job: 4728321: RUNNING
  2555. job: 4728286: RUNNING
  2556. job: 4728326: PENDING
  2557.  
  2558. Jul-28 00:56:48.840 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2559. Jul-28 00:56:53.840 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2560. job: 4713446: RUNNING
  2561. job: 4728323: RUNNING
  2562. job: 4728325: RUNNING
  2563. job: 4728321: RUNNING
  2564. job: 4728286: RUNNING
  2565. job: 4728326: PENDING
  2566.  
  2567. Jul-28 00:56:53.840 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2568. Jul-28 00:56:58.843 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  2569. Jul-28 00:56:58.986 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  2570. 4713446,RUN,Jul 27 11:44
  2571. 4728323,RUN,Jul 28 00:40
  2572. 4728325,RUN,Jul 28 00:49
  2573. 4728321,RUN,Jul 28 00:34
  2574. 4728286,RUN,Jul 27 23:44
  2575. 4728326,PEND,Jul 28 00:51
  2576.  
  2577.  
  2578. Jul-28 00:56:58.986 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2579. job: 4713446: RUNNING
  2580. job: 4728323: RUNNING
  2581. job: 4728325: RUNNING
  2582. job: 4728321: RUNNING
  2583. job: 4728286: RUNNING
  2584. job: 4728326: PENDING
  2585.  
  2586. Jul-28 00:56:58.986 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2587. Jul-28 00:57:03.840 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2588. job: 4713446: RUNNING
  2589. job: 4728323: RUNNING
  2590. job: 4728325: RUNNING
  2591. job: 4728321: RUNNING
  2592. job: 4728286: RUNNING
  2593. job: 4728326: PENDING
  2594.  
  2595. Jul-28 00:57:03.840 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2596. Jul-28 00:57:08.841 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2597. job: 4713446: RUNNING
  2598. job: 4728323: RUNNING
  2599. job: 4728325: RUNNING
  2600. job: 4728321: RUNNING
  2601. job: 4728286: RUNNING
  2602. job: 4728326: PENDING
  2603.  
  2604. Jul-28 00:57:08.841 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2605. Jul-28 00:57:13.841 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2606. job: 4713446: RUNNING
  2607. job: 4728323: RUNNING
  2608. job: 4728325: RUNNING
  2609. job: 4728321: RUNNING
  2610. job: 4728286: RUNNING
  2611. job: 4728326: PENDING
  2612.  
  2613. Jul-28 00:57:13.841 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2614. Jul-28 00:57:18.842 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2615. job: 4713446: RUNNING
  2616. job: 4728323: RUNNING
  2617. job: 4728325: RUNNING
  2618. job: 4728321: RUNNING
  2619. job: 4728286: RUNNING
  2620. job: 4728326: PENDING
  2621.  
  2622. Jul-28 00:57:18.843 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2623. Jul-28 00:57:23.840 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2624. job: 4713446: RUNNING
  2625. job: 4728323: RUNNING
  2626. job: 4728325: RUNNING
  2627. job: 4728321: RUNNING
  2628. job: 4728286: RUNNING
  2629. job: 4728326: PENDING
  2630.  
  2631. Jul-28 00:57:23.840 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2632. Jul-28 00:57:28.844 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2633. job: 4713446: RUNNING
  2634. job: 4728323: RUNNING
  2635. job: 4728325: RUNNING
  2636. job: 4728321: RUNNING
  2637. job: 4728286: RUNNING
  2638. job: 4728326: PENDING
  2639.  
  2640. Jul-28 00:57:28.845 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2641. Jul-28 00:57:33.840 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2642. job: 4713446: RUNNING
  2643. job: 4728323: RUNNING
  2644. job: 4728325: RUNNING
  2645. job: 4728321: RUNNING
  2646. job: 4728286: RUNNING
  2647. job: 4728326: PENDING
  2648.  
  2649. Jul-28 00:57:33.840 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2650. Jul-28 00:57:38.839 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2651. job: 4713446: RUNNING
  2652. job: 4728323: RUNNING
  2653. job: 4728325: RUNNING
  2654. job: 4728321: RUNNING
  2655. job: 4728286: RUNNING
  2656. job: 4728326: PENDING
  2657.  
  2658. Jul-28 00:57:38.840 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2659. Jul-28 00:57:43.840 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2660. job: 4713446: RUNNING
  2661. job: 4728323: RUNNING
  2662. job: 4728325: RUNNING
  2663. job: 4728321: RUNNING
  2664. job: 4728286: RUNNING
  2665. job: 4728326: PENDING
  2666.  
  2667. Jul-28 00:57:43.840 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2668. Jul-28 00:57:48.841 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2669. job: 4713446: RUNNING
  2670. job: 4728323: RUNNING
  2671. job: 4728325: RUNNING
  2672. job: 4728321: RUNNING
  2673. job: 4728286: RUNNING
  2674. job: 4728326: PENDING
  2675.  
  2676. Jul-28 00:57:48.842 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2677. Jul-28 00:57:53.840 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2678. job: 4713446: RUNNING
  2679. job: 4728323: RUNNING
  2680. job: 4728325: RUNNING
  2681. job: 4728321: RUNNING
  2682. job: 4728286: RUNNING
  2683. job: 4728326: PENDING
  2684.  
  2685. Jul-28 00:57:53.840 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2686. Jul-28 00:57:58.840 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2687. job: 4713446: RUNNING
  2688. job: 4728323: RUNNING
  2689. job: 4728325: RUNNING
  2690. job: 4728321: RUNNING
  2691. job: 4728286: RUNNING
  2692. job: 4728326: PENDING
  2693.  
  2694. Jul-28 00:57:58.841 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2695. Jul-28 00:58:03.839 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  2696. Jul-28 00:58:03.973 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  2697. 4713446,RUN,Jul 27 11:44
  2698. 4728323,RUN,Jul 28 00:40
  2699. 4728325,RUN,Jul 28 00:49
  2700. 4728321,RUN,Jul 28 00:34
  2701. 4728286,RUN,Jul 27 23:44
  2702. 4728326,PEND,Jul 28 00:51
  2703.  
  2704.  
  2705. Jul-28 00:58:03.973 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2706. job: 4713446: RUNNING
  2707. job: 4728323: RUNNING
  2708. job: 4728325: RUNNING
  2709. job: 4728321: RUNNING
  2710. job: 4728286: RUNNING
  2711. job: 4728326: PENDING
  2712.  
  2713. Jul-28 00:58:03.973 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2714. Jul-28 00:58:08.840 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2715. job: 4713446: RUNNING
  2716. job: 4728323: RUNNING
  2717. job: 4728325: RUNNING
  2718. job: 4728321: RUNNING
  2719. job: 4728286: RUNNING
  2720. job: 4728326: PENDING
  2721.  
  2722. Jul-28 00:58:08.840 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2723. Jul-28 00:58:13.843 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2724. job: 4713446: RUNNING
  2725. job: 4728323: RUNNING
  2726. job: 4728325: RUNNING
  2727. job: 4728321: RUNNING
  2728. job: 4728286: RUNNING
  2729. job: 4728326: PENDING
  2730.  
  2731. Jul-28 00:58:13.844 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2732. Jul-28 00:58:18.840 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2733. job: 4713446: RUNNING
  2734. job: 4728323: RUNNING
  2735. job: 4728325: RUNNING
  2736. job: 4728321: RUNNING
  2737. job: 4728286: RUNNING
  2738. job: 4728326: PENDING
  2739.  
  2740. Jul-28 00:58:18.841 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2741. Jul-28 00:58:23.840 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2742. job: 4713446: RUNNING
  2743. job: 4728323: RUNNING
  2744. job: 4728325: RUNNING
  2745. job: 4728321: RUNNING
  2746. job: 4728286: RUNNING
  2747. job: 4728326: PENDING
  2748.  
  2749. Jul-28 00:58:23.841 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2750. Jul-28 00:58:28.845 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2751. job: 4713446: RUNNING
  2752. job: 4728323: RUNNING
  2753. job: 4728325: RUNNING
  2754. job: 4728321: RUNNING
  2755. job: 4728286: RUNNING
  2756. job: 4728326: PENDING
  2757.  
  2758. Jul-28 00:58:28.846 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2759. Jul-28 00:58:33.842 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2760. job: 4713446: RUNNING
  2761. job: 4728323: RUNNING
  2762. job: 4728325: RUNNING
  2763. job: 4728321: RUNNING
  2764. job: 4728286: RUNNING
  2765. job: 4728326: PENDING
  2766.  
  2767. Jul-28 00:58:33.843 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2768. Jul-28 00:58:38.843 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2769. job: 4713446: RUNNING
  2770. job: 4728323: RUNNING
  2771. job: 4728325: RUNNING
  2772. job: 4728321: RUNNING
  2773. job: 4728286: RUNNING
  2774. job: 4728326: PENDING
  2775.  
  2776. Jul-28 00:58:38.843 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2777. Jul-28 00:58:43.843 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2778. job: 4713446: RUNNING
  2779. job: 4728323: RUNNING
  2780. job: 4728325: RUNNING
  2781. job: 4728321: RUNNING
  2782. job: 4728286: RUNNING
  2783. job: 4728326: PENDING
  2784.  
  2785. Jul-28 00:58:43.844 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2786. Jul-28 00:58:48.844 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2787. job: 4713446: RUNNING
  2788. job: 4728323: RUNNING
  2789. job: 4728325: RUNNING
  2790. job: 4728321: RUNNING
  2791. job: 4728286: RUNNING
  2792. job: 4728326: PENDING
  2793.  
  2794. Jul-28 00:58:48.844 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2795. Jul-28 00:58:53.845 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2796. job: 4713446: RUNNING
  2797. job: 4728323: RUNNING
  2798. job: 4728325: RUNNING
  2799. job: 4728321: RUNNING
  2800. job: 4728286: RUNNING
  2801. job: 4728326: PENDING
  2802.  
  2803. Jul-28 00:58:53.845 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2804. Jul-28 00:58:58.845 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2805. job: 4713446: RUNNING
  2806. job: 4728323: RUNNING
  2807. job: 4728325: RUNNING
  2808. job: 4728321: RUNNING
  2809. job: 4728286: RUNNING
  2810. job: 4728326: PENDING
  2811.  
  2812. Jul-28 00:58:58.845 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2813. Jul-28 00:58:58.849 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2814. job: 4713446: RUNNING
  2815. job: 4728323: RUNNING
  2816. job: 4728325: RUNNING
  2817. job: 4728321: RUNNING
  2818. job: 4728286: RUNNING
  2819. job: 4728326: PENDING
  2820.  
  2821. Jul-28 00:58:58.849 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: false
  2822. Jul-28 00:59:03.846 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  2823. Jul-28 00:59:03.982 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  2824. 4713446,RUN,Jul 27 11:44
  2825. 4728323,RUN,Jul 28 00:40
  2826. 4728326,RUN,Jul 28 00:51
  2827. 4728325,RUN,Jul 28 00:49
  2828. 4728321,RUN,Jul 28 00:34
  2829. 4728286,RUN,Jul 27 23:44
  2830.  
  2831.  
  2832. Jul-28 00:59:03.982 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2833. job: 4713446: RUNNING
  2834. job: 4728323: RUNNING
  2835. job: 4728326: RUNNING
  2836. job: 4728325: RUNNING
  2837. job: 4728321: RUNNING
  2838. job: 4728286: RUNNING
  2839.  
  2840. Jul-28 00:59:03.982 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2841. Jul-28 00:59:03.983 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2842. job: 4713446: RUNNING
  2843. job: 4728323: RUNNING
  2844. job: 4728326: RUNNING
  2845. job: 4728325: RUNNING
  2846. job: 4728321: RUNNING
  2847. job: 4728286: RUNNING
  2848.  
  2849. Jul-28 00:59:03.983 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  2850. Jul-28 00:59:08.846 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2851. job: 4713446: RUNNING
  2852. job: 4728323: RUNNING
  2853. job: 4728326: RUNNING
  2854. job: 4728325: RUNNING
  2855. job: 4728321: RUNNING
  2856. job: 4728286: RUNNING
  2857.  
  2858. Jul-28 00:59:08.847 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2859. Jul-28 00:59:08.848 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2860. job: 4713446: RUNNING
  2861. job: 4728323: RUNNING
  2862. job: 4728326: RUNNING
  2863. job: 4728325: RUNNING
  2864. job: 4728321: RUNNING
  2865. job: 4728286: RUNNING
  2866.  
  2867. Jul-28 00:59:08.848 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  2868. Jul-28 00:59:13.846 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2869. job: 4713446: RUNNING
  2870. job: 4728323: RUNNING
  2871. job: 4728326: RUNNING
  2872. job: 4728325: RUNNING
  2873. job: 4728321: RUNNING
  2874. job: 4728286: RUNNING
  2875.  
  2876. Jul-28 00:59:13.847 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2877. Jul-28 00:59:13.848 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2878. job: 4713446: RUNNING
  2879. job: 4728323: RUNNING
  2880. job: 4728326: RUNNING
  2881. job: 4728325: RUNNING
  2882. job: 4728321: RUNNING
  2883. job: 4728286: RUNNING
  2884.  
  2885. Jul-28 00:59:13.848 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  2886. Jul-28 00:59:18.846 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2887. job: 4713446: RUNNING
  2888. job: 4728323: RUNNING
  2889. job: 4728326: RUNNING
  2890. job: 4728325: RUNNING
  2891. job: 4728321: RUNNING
  2892. job: 4728286: RUNNING
  2893.  
  2894. Jul-28 00:59:18.846 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2895. Jul-28 00:59:18.847 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2896. job: 4713446: RUNNING
  2897. job: 4728323: RUNNING
  2898. job: 4728326: RUNNING
  2899. job: 4728325: RUNNING
  2900. job: 4728321: RUNNING
  2901. job: 4728286: RUNNING
  2902.  
  2903. Jul-28 00:59:18.847 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  2904. Jul-28 00:59:23.847 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2905. job: 4713446: RUNNING
  2906. job: 4728323: RUNNING
  2907. job: 4728326: RUNNING
  2908. job: 4728325: RUNNING
  2909. job: 4728321: RUNNING
  2910. job: 4728286: RUNNING
  2911.  
  2912. Jul-28 00:59:23.847 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2913. Jul-28 00:59:23.848 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2914. job: 4713446: RUNNING
  2915. job: 4728323: RUNNING
  2916. job: 4728326: RUNNING
  2917. job: 4728325: RUNNING
  2918. job: 4728321: RUNNING
  2919. job: 4728286: RUNNING
  2920.  
  2921. Jul-28 00:59:23.848 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  2922. Jul-28 00:59:28.847 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2923. job: 4713446: RUNNING
  2924. job: 4728323: RUNNING
  2925. job: 4728326: RUNNING
  2926. job: 4728325: RUNNING
  2927. job: 4728321: RUNNING
  2928. job: 4728286: RUNNING
  2929.  
  2930. Jul-28 00:59:28.847 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2931. Jul-28 00:59:28.848 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2932. job: 4713446: RUNNING
  2933. job: 4728323: RUNNING
  2934. job: 4728326: RUNNING
  2935. job: 4728325: RUNNING
  2936. job: 4728321: RUNNING
  2937. job: 4728286: RUNNING
  2938.  
  2939. Jul-28 00:59:28.848 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  2940. Jul-28 00:59:33.847 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2941. job: 4713446: RUNNING
  2942. job: 4728323: RUNNING
  2943. job: 4728326: RUNNING
  2944. job: 4728325: RUNNING
  2945. job: 4728321: RUNNING
  2946. job: 4728286: RUNNING
  2947.  
  2948. Jul-28 00:59:33.848 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2949. Jul-28 00:59:33.849 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2950. job: 4713446: RUNNING
  2951. job: 4728323: RUNNING
  2952. job: 4728326: RUNNING
  2953. job: 4728325: RUNNING
  2954. job: 4728321: RUNNING
  2955. job: 4728286: RUNNING
  2956.  
  2957. Jul-28 00:59:33.849 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  2958. Jul-28 00:59:38.847 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2959. job: 4713446: RUNNING
  2960. job: 4728323: RUNNING
  2961. job: 4728326: RUNNING
  2962. job: 4728325: RUNNING
  2963. job: 4728321: RUNNING
  2964. job: 4728286: RUNNING
  2965.  
  2966. Jul-28 00:59:38.847 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2967. Jul-28 00:59:38.848 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2968. job: 4713446: RUNNING
  2969. job: 4728323: RUNNING
  2970. job: 4728326: RUNNING
  2971. job: 4728325: RUNNING
  2972. job: 4728321: RUNNING
  2973. job: 4728286: RUNNING
  2974.  
  2975. Jul-28 00:59:38.848 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  2976. Jul-28 00:59:43.847 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2977. job: 4713446: RUNNING
  2978. job: 4728323: RUNNING
  2979. job: 4728326: RUNNING
  2980. job: 4728325: RUNNING
  2981. job: 4728321: RUNNING
  2982. job: 4728286: RUNNING
  2983.  
  2984. Jul-28 00:59:43.848 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  2985. Jul-28 00:59:43.849 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2986. job: 4713446: RUNNING
  2987. job: 4728323: RUNNING
  2988. job: 4728326: RUNNING
  2989. job: 4728325: RUNNING
  2990. job: 4728321: RUNNING
  2991. job: 4728286: RUNNING
  2992.  
  2993. Jul-28 00:59:43.849 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  2994. Jul-28 00:59:48.848 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  2995. job: 4713446: RUNNING
  2996. job: 4728323: RUNNING
  2997. job: 4728326: RUNNING
  2998. job: 4728325: RUNNING
  2999. job: 4728321: RUNNING
  3000. job: 4728286: RUNNING
  3001.  
  3002. Jul-28 00:59:48.849 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3003. Jul-28 00:59:48.849 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3004. job: 4713446: RUNNING
  3005. job: 4728323: RUNNING
  3006. job: 4728326: RUNNING
  3007. job: 4728325: RUNNING
  3008. job: 4728321: RUNNING
  3009. job: 4728286: RUNNING
  3010.  
  3011. Jul-28 00:59:48.849 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3012. Jul-28 00:59:53.847 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3013. job: 4713446: RUNNING
  3014. job: 4728323: RUNNING
  3015. job: 4728326: RUNNING
  3016. job: 4728325: RUNNING
  3017. job: 4728321: RUNNING
  3018. job: 4728286: RUNNING
  3019.  
  3020. Jul-28 00:59:53.847 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3021. Jul-28 00:59:53.848 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3022. job: 4713446: RUNNING
  3023. job: 4728323: RUNNING
  3024. job: 4728326: RUNNING
  3025. job: 4728325: RUNNING
  3026. job: 4728321: RUNNING
  3027. job: 4728286: RUNNING
  3028.  
  3029. Jul-28 00:59:53.848 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3030. Jul-28 00:59:58.848 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3031. job: 4713446: RUNNING
  3032. job: 4728323: RUNNING
  3033. job: 4728326: RUNNING
  3034. job: 4728325: RUNNING
  3035. job: 4728321: RUNNING
  3036. job: 4728286: RUNNING
  3037.  
  3038. Jul-28 00:59:58.848 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3039. Jul-28 00:59:58.849 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3040. job: 4713446: RUNNING
  3041. job: 4728323: RUNNING
  3042. job: 4728326: RUNNING
  3043. job: 4728325: RUNNING
  3044. job: 4728321: RUNNING
  3045. job: 4728286: RUNNING
  3046.  
  3047. Jul-28 00:59:58.849 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3048. Jul-28 01:00:03.847 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  3049. Jul-28 01:00:03.981 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  3050. 4713446,RUN,Jul 27 11:44
  3051. 4728323,RUN,Jul 28 00:40
  3052. 4728326,RUN,Jul 28 00:51
  3053. 4728325,RUN,Jul 28 00:49
  3054. 4728321,RUN,Jul 28 00:34
  3055. 4728286,RUN,Jul 27 23:44
  3056.  
  3057.  
  3058. Jul-28 01:00:03.981 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3059. job: 4713446: RUNNING
  3060. job: 4728323: RUNNING
  3061. job: 4728326: RUNNING
  3062. job: 4728325: RUNNING
  3063. job: 4728321: RUNNING
  3064. job: 4728286: RUNNING
  3065.  
  3066. Jul-28 01:00:03.981 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3067. Jul-28 01:00:03.982 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3068. job: 4713446: RUNNING
  3069. job: 4728323: RUNNING
  3070. job: 4728326: RUNNING
  3071. job: 4728325: RUNNING
  3072. job: 4728321: RUNNING
  3073. job: 4728286: RUNNING
  3074.  
  3075. Jul-28 01:00:03.982 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3076. Jul-28 01:00:08.851 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor lsf > tasks to be completed: 2 -- pending tasks are shown below
  3077. ~> TaskHandler[jobId: 4728325; id: 152; name: merge_pairsam_into_runs (library:HeLa1 run:lane1); status: RUNNING; exit: -; error: -; workDir: /farline/umw_job_dekker/HPCC/sv49w/distiller-nf/work/3b/37ed2fa2ba160f54196e762e24c83b started: 1501217508853; exited: -; ]
  3078. ~> TaskHandler[jobId: 4728326; id: 153; name: merge_pairsam_into_runs (library:HeLa1 run:lane3); status: RUNNING; exit: -; error: -; workDir: /farline/umw_job_dekker/HPCC/sv49w/distiller-nf/work/00/2633edc9da9ad5649d3d6dcffe5a44 started: 1501217938848; exited: -; ]
  3079. Jul-28 01:00:08.851 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3080. job: 4713446: RUNNING
  3081. job: 4728323: RUNNING
  3082. job: 4728326: RUNNING
  3083. job: 4728325: RUNNING
  3084. job: 4728321: RUNNING
  3085. job: 4728286: RUNNING
  3086.  
  3087. Jul-28 01:00:08.851 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3088. Jul-28 01:00:08.851 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3089. job: 4713446: RUNNING
  3090. job: 4728323: RUNNING
  3091. job: 4728326: RUNNING
  3092. job: 4728325: RUNNING
  3093. job: 4728321: RUNNING
  3094. job: 4728286: RUNNING
  3095.  
  3096. Jul-28 01:00:08.851 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3097. Jul-28 01:00:13.855 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3098. job: 4713446: RUNNING
  3099. job: 4728323: RUNNING
  3100. job: 4728326: RUNNING
  3101. job: 4728325: RUNNING
  3102. job: 4728321: RUNNING
  3103. job: 4728286: RUNNING
  3104.  
  3105. Jul-28 01:00:13.855 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3106. Jul-28 01:00:13.856 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3107. job: 4713446: RUNNING
  3108. job: 4728323: RUNNING
  3109. job: 4728326: RUNNING
  3110. job: 4728325: RUNNING
  3111. job: 4728321: RUNNING
  3112. job: 4728286: RUNNING
  3113.  
  3114. Jul-28 01:00:13.856 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3115. Jul-28 01:00:18.855 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3116. job: 4713446: RUNNING
  3117. job: 4728323: RUNNING
  3118. job: 4728326: RUNNING
  3119. job: 4728325: RUNNING
  3120. job: 4728321: RUNNING
  3121. job: 4728286: RUNNING
  3122.  
  3123. Jul-28 01:00:18.855 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3124. Jul-28 01:00:18.856 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3125. job: 4713446: RUNNING
  3126. job: 4728323: RUNNING
  3127. job: 4728326: RUNNING
  3128. job: 4728325: RUNNING
  3129. job: 4728321: RUNNING
  3130. job: 4728286: RUNNING
  3131.  
  3132. Jul-28 01:00:18.856 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3133. Jul-28 01:00:23.856 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3134. job: 4713446: RUNNING
  3135. job: 4728323: RUNNING
  3136. job: 4728326: RUNNING
  3137. job: 4728325: RUNNING
  3138. job: 4728321: RUNNING
  3139. job: 4728286: RUNNING
  3140.  
  3141. Jul-28 01:00:23.856 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3142. Jul-28 01:00:23.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3143. job: 4713446: RUNNING
  3144. job: 4728323: RUNNING
  3145. job: 4728326: RUNNING
  3146. job: 4728325: RUNNING
  3147. job: 4728321: RUNNING
  3148. job: 4728286: RUNNING
  3149.  
  3150. Jul-28 01:00:23.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3151. Jul-28 01:00:28.855 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3152. job: 4713446: RUNNING
  3153. job: 4728323: RUNNING
  3154. job: 4728326: RUNNING
  3155. job: 4728325: RUNNING
  3156. job: 4728321: RUNNING
  3157. job: 4728286: RUNNING
  3158.  
  3159. Jul-28 01:00:28.856 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3160. Jul-28 01:00:28.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3161. job: 4713446: RUNNING
  3162. job: 4728323: RUNNING
  3163. job: 4728326: RUNNING
  3164. job: 4728325: RUNNING
  3165. job: 4728321: RUNNING
  3166. job: 4728286: RUNNING
  3167.  
  3168. Jul-28 01:00:28.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3169. Jul-28 01:00:33.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3170. job: 4713446: RUNNING
  3171. job: 4728323: RUNNING
  3172. job: 4728326: RUNNING
  3173. job: 4728325: RUNNING
  3174. job: 4728321: RUNNING
  3175. job: 4728286: RUNNING
  3176.  
  3177. Jul-28 01:00:33.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3178. Jul-28 01:00:33.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3179. job: 4713446: RUNNING
  3180. job: 4728323: RUNNING
  3181. job: 4728326: RUNNING
  3182. job: 4728325: RUNNING
  3183. job: 4728321: RUNNING
  3184. job: 4728286: RUNNING
  3185.  
  3186. Jul-28 01:00:33.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3187. Jul-28 01:00:38.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3188. job: 4713446: RUNNING
  3189. job: 4728323: RUNNING
  3190. job: 4728326: RUNNING
  3191. job: 4728325: RUNNING
  3192. job: 4728321: RUNNING
  3193. job: 4728286: RUNNING
  3194.  
  3195. Jul-28 01:00:38.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3196. Jul-28 01:00:38.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3197. job: 4713446: RUNNING
  3198. job: 4728323: RUNNING
  3199. job: 4728326: RUNNING
  3200. job: 4728325: RUNNING
  3201. job: 4728321: RUNNING
  3202. job: 4728286: RUNNING
  3203.  
  3204. Jul-28 01:00:38.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3205. Jul-28 01:00:43.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3206. job: 4713446: RUNNING
  3207. job: 4728323: RUNNING
  3208. job: 4728326: RUNNING
  3209. job: 4728325: RUNNING
  3210. job: 4728321: RUNNING
  3211. job: 4728286: RUNNING
  3212.  
  3213. Jul-28 01:00:43.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3214. Jul-28 01:00:43.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3215. job: 4713446: RUNNING
  3216. job: 4728323: RUNNING
  3217. job: 4728326: RUNNING
  3218. job: 4728325: RUNNING
  3219. job: 4728321: RUNNING
  3220. job: 4728286: RUNNING
  3221.  
  3222. Jul-28 01:00:43.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3223. Jul-28 01:00:48.856 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3224. job: 4713446: RUNNING
  3225. job: 4728323: RUNNING
  3226. job: 4728326: RUNNING
  3227. job: 4728325: RUNNING
  3228. job: 4728321: RUNNING
  3229. job: 4728286: RUNNING
  3230.  
  3231. Jul-28 01:00:48.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3232. Jul-28 01:00:48.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3233. job: 4713446: RUNNING
  3234. job: 4728323: RUNNING
  3235. job: 4728326: RUNNING
  3236. job: 4728325: RUNNING
  3237. job: 4728321: RUNNING
  3238. job: 4728286: RUNNING
  3239.  
  3240. Jul-28 01:00:48.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3241. Jul-28 01:00:53.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3242. job: 4713446: RUNNING
  3243. job: 4728323: RUNNING
  3244. job: 4728326: RUNNING
  3245. job: 4728325: RUNNING
  3246. job: 4728321: RUNNING
  3247. job: 4728286: RUNNING
  3248.  
  3249. Jul-28 01:00:53.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3250. Jul-28 01:00:53.859 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3251. job: 4713446: RUNNING
  3252. job: 4728323: RUNNING
  3253. job: 4728326: RUNNING
  3254. job: 4728325: RUNNING
  3255. job: 4728321: RUNNING
  3256. job: 4728286: RUNNING
  3257.  
  3258. Jul-28 01:00:53.859 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3259. Jul-28 01:00:58.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3260. job: 4713446: RUNNING
  3261. job: 4728323: RUNNING
  3262. job: 4728326: RUNNING
  3263. job: 4728325: RUNNING
  3264. job: 4728321: RUNNING
  3265. job: 4728286: RUNNING
  3266.  
  3267. Jul-28 01:00:58.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3268. Jul-28 01:00:58.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3269. job: 4713446: RUNNING
  3270. job: 4728323: RUNNING
  3271. job: 4728326: RUNNING
  3272. job: 4728325: RUNNING
  3273. job: 4728321: RUNNING
  3274. job: 4728286: RUNNING
  3275.  
  3276. Jul-28 01:00:58.859 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3277. Jul-28 01:01:03.856 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  3278. Jul-28 01:01:03.982 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  3279. 4713446,RUN,Jul 27 11:44
  3280. 4728323,RUN,Jul 28 00:40
  3281. 4728326,RUN,Jul 28 00:51
  3282. 4728325,RUN,Jul 28 00:49
  3283. 4728321,RUN,Jul 28 00:34
  3284. 4728286,RUN,Jul 27 23:44
  3285.  
  3286.  
  3287. Jul-28 01:01:03.982 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3288. job: 4713446: RUNNING
  3289. job: 4728323: RUNNING
  3290. job: 4728326: RUNNING
  3291. job: 4728325: RUNNING
  3292. job: 4728321: RUNNING
  3293. job: 4728286: RUNNING
  3294.  
  3295. Jul-28 01:01:03.982 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3296. Jul-28 01:01:03.983 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3297. job: 4713446: RUNNING
  3298. job: 4728323: RUNNING
  3299. job: 4728326: RUNNING
  3300. job: 4728325: RUNNING
  3301. job: 4728321: RUNNING
  3302. job: 4728286: RUNNING
  3303.  
  3304. Jul-28 01:01:03.983 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3305. Jul-28 01:01:08.856 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3306. job: 4713446: RUNNING
  3307. job: 4728323: RUNNING
  3308. job: 4728326: RUNNING
  3309. job: 4728325: RUNNING
  3310. job: 4728321: RUNNING
  3311. job: 4728286: RUNNING
  3312.  
  3313. Jul-28 01:01:08.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3314. Jul-28 01:01:08.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3315. job: 4713446: RUNNING
  3316. job: 4728323: RUNNING
  3317. job: 4728326: RUNNING
  3318. job: 4728325: RUNNING
  3319. job: 4728321: RUNNING
  3320. job: 4728286: RUNNING
  3321.  
  3322. Jul-28 01:01:08.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3323. Jul-28 01:01:13.856 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3324. job: 4713446: RUNNING
  3325. job: 4728323: RUNNING
  3326. job: 4728326: RUNNING
  3327. job: 4728325: RUNNING
  3328. job: 4728321: RUNNING
  3329. job: 4728286: RUNNING
  3330.  
  3331. Jul-28 01:01:13.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3332. Jul-28 01:01:13.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3333. job: 4713446: RUNNING
  3334. job: 4728323: RUNNING
  3335. job: 4728326: RUNNING
  3336. job: 4728325: RUNNING
  3337. job: 4728321: RUNNING
  3338. job: 4728286: RUNNING
  3339.  
  3340. Jul-28 01:01:13.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3341. Jul-28 01:01:18.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3342. job: 4713446: RUNNING
  3343. job: 4728323: RUNNING
  3344. job: 4728326: RUNNING
  3345. job: 4728325: RUNNING
  3346. job: 4728321: RUNNING
  3347. job: 4728286: RUNNING
  3348.  
  3349. Jul-28 01:01:18.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3350. Jul-28 01:01:18.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3351. job: 4713446: RUNNING
  3352. job: 4728323: RUNNING
  3353. job: 4728326: RUNNING
  3354. job: 4728325: RUNNING
  3355. job: 4728321: RUNNING
  3356. job: 4728286: RUNNING
  3357.  
  3358. Jul-28 01:01:18.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3359. Jul-28 01:01:23.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3360. job: 4713446: RUNNING
  3361. job: 4728323: RUNNING
  3362. job: 4728326: RUNNING
  3363. job: 4728325: RUNNING
  3364. job: 4728321: RUNNING
  3365. job: 4728286: RUNNING
  3366.  
  3367. Jul-28 01:01:23.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3368. Jul-28 01:01:23.859 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3369. job: 4713446: RUNNING
  3370. job: 4728323: RUNNING
  3371. job: 4728326: RUNNING
  3372. job: 4728325: RUNNING
  3373. job: 4728321: RUNNING
  3374. job: 4728286: RUNNING
  3375.  
  3376. Jul-28 01:01:23.859 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3377. Jul-28 01:01:28.856 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3378. job: 4713446: RUNNING
  3379. job: 4728323: RUNNING
  3380. job: 4728326: RUNNING
  3381. job: 4728325: RUNNING
  3382. job: 4728321: RUNNING
  3383. job: 4728286: RUNNING
  3384.  
  3385. Jul-28 01:01:28.856 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3386. Jul-28 01:01:28.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3387. job: 4713446: RUNNING
  3388. job: 4728323: RUNNING
  3389. job: 4728326: RUNNING
  3390. job: 4728325: RUNNING
  3391. job: 4728321: RUNNING
  3392. job: 4728286: RUNNING
  3393.  
  3394. Jul-28 01:01:28.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3395. Jul-28 01:01:33.856 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3396. job: 4713446: RUNNING
  3397. job: 4728323: RUNNING
  3398. job: 4728326: RUNNING
  3399. job: 4728325: RUNNING
  3400. job: 4728321: RUNNING
  3401. job: 4728286: RUNNING
  3402.  
  3403. Jul-28 01:01:33.856 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3404. Jul-28 01:01:33.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3405. job: 4713446: RUNNING
  3406. job: 4728323: RUNNING
  3407. job: 4728326: RUNNING
  3408. job: 4728325: RUNNING
  3409. job: 4728321: RUNNING
  3410. job: 4728286: RUNNING
  3411.  
  3412. Jul-28 01:01:33.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3413. Jul-28 01:01:38.856 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3414. job: 4713446: RUNNING
  3415. job: 4728323: RUNNING
  3416. job: 4728326: RUNNING
  3417. job: 4728325: RUNNING
  3418. job: 4728321: RUNNING
  3419. job: 4728286: RUNNING
  3420.  
  3421. Jul-28 01:01:38.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3422. Jul-28 01:01:38.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3423. job: 4713446: RUNNING
  3424. job: 4728323: RUNNING
  3425. job: 4728326: RUNNING
  3426. job: 4728325: RUNNING
  3427. job: 4728321: RUNNING
  3428. job: 4728286: RUNNING
  3429.  
  3430. Jul-28 01:01:38.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3431. Jul-28 01:01:43.856 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3432. job: 4713446: RUNNING
  3433. job: 4728323: RUNNING
  3434. job: 4728326: RUNNING
  3435. job: 4728325: RUNNING
  3436. job: 4728321: RUNNING
  3437. job: 4728286: RUNNING
  3438.  
  3439. Jul-28 01:01:43.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3440. Jul-28 01:01:43.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3441. job: 4713446: RUNNING
  3442. job: 4728323: RUNNING
  3443. job: 4728326: RUNNING
  3444. job: 4728325: RUNNING
  3445. job: 4728321: RUNNING
  3446. job: 4728286: RUNNING
  3447.  
  3448. Jul-28 01:01:43.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3449. Jul-28 01:01:48.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3450. job: 4713446: RUNNING
  3451. job: 4728323: RUNNING
  3452. job: 4728326: RUNNING
  3453. job: 4728325: RUNNING
  3454. job: 4728321: RUNNING
  3455. job: 4728286: RUNNING
  3456.  
  3457. Jul-28 01:01:48.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3458. Jul-28 01:01:48.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3459. job: 4713446: RUNNING
  3460. job: 4728323: RUNNING
  3461. job: 4728326: RUNNING
  3462. job: 4728325: RUNNING
  3463. job: 4728321: RUNNING
  3464. job: 4728286: RUNNING
  3465.  
  3466. Jul-28 01:01:48.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3467. Jul-28 01:01:53.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3468. job: 4713446: RUNNING
  3469. job: 4728323: RUNNING
  3470. job: 4728326: RUNNING
  3471. job: 4728325: RUNNING
  3472. job: 4728321: RUNNING
  3473. job: 4728286: RUNNING
  3474.  
  3475. Jul-28 01:01:53.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3476. Jul-28 01:01:53.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3477. job: 4713446: RUNNING
  3478. job: 4728323: RUNNING
  3479. job: 4728326: RUNNING
  3480. job: 4728325: RUNNING
  3481. job: 4728321: RUNNING
  3482. job: 4728286: RUNNING
  3483.  
  3484. Jul-28 01:01:53.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3485. Jul-28 01:01:58.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3486. job: 4713446: RUNNING
  3487. job: 4728323: RUNNING
  3488. job: 4728326: RUNNING
  3489. job: 4728325: RUNNING
  3490. job: 4728321: RUNNING
  3491. job: 4728286: RUNNING
  3492.  
  3493. Jul-28 01:01:58.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3494. Jul-28 01:01:58.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3495. job: 4713446: RUNNING
  3496. job: 4728323: RUNNING
  3497. job: 4728326: RUNNING
  3498. job: 4728325: RUNNING
  3499. job: 4728321: RUNNING
  3500. job: 4728286: RUNNING
  3501.  
  3502. Jul-28 01:01:58.859 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3503. Jul-28 01:02:03.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  3504. Jul-28 01:02:03.992 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  3505. 4713446,RUN,Jul 27 11:44
  3506. 4728323,RUN,Jul 28 00:40
  3507. 4728326,RUN,Jul 28 00:51
  3508. 4728325,RUN,Jul 28 00:49
  3509. 4728321,RUN,Jul 28 00:34
  3510. 4728286,RUN,Jul 27 23:44
  3511.  
  3512.  
  3513. Jul-28 01:02:03.992 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3514. job: 4713446: RUNNING
  3515. job: 4728323: RUNNING
  3516. job: 4728326: RUNNING
  3517. job: 4728325: RUNNING
  3518. job: 4728321: RUNNING
  3519. job: 4728286: RUNNING
  3520.  
  3521. Jul-28 01:02:03.992 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3522. Jul-28 01:02:03.993 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3523. job: 4713446: RUNNING
  3524. job: 4728323: RUNNING
  3525. job: 4728326: RUNNING
  3526. job: 4728325: RUNNING
  3527. job: 4728321: RUNNING
  3528. job: 4728286: RUNNING
  3529.  
  3530. Jul-28 01:02:03.993 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3531. Jul-28 01:02:08.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3532. job: 4713446: RUNNING
  3533. job: 4728323: RUNNING
  3534. job: 4728326: RUNNING
  3535. job: 4728325: RUNNING
  3536. job: 4728321: RUNNING
  3537. job: 4728286: RUNNING
  3538.  
  3539. Jul-28 01:02:08.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3540. Jul-28 01:02:08.859 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3541. job: 4713446: RUNNING
  3542. job: 4728323: RUNNING
  3543. job: 4728326: RUNNING
  3544. job: 4728325: RUNNING
  3545. job: 4728321: RUNNING
  3546. job: 4728286: RUNNING
  3547.  
  3548. Jul-28 01:02:08.859 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3549. Jul-28 01:02:13.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3550. job: 4713446: RUNNING
  3551. job: 4728323: RUNNING
  3552. job: 4728326: RUNNING
  3553. job: 4728325: RUNNING
  3554. job: 4728321: RUNNING
  3555. job: 4728286: RUNNING
  3556.  
  3557. Jul-28 01:02:13.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3558. Jul-28 01:02:13.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3559. job: 4713446: RUNNING
  3560. job: 4728323: RUNNING
  3561. job: 4728326: RUNNING
  3562. job: 4728325: RUNNING
  3563. job: 4728321: RUNNING
  3564. job: 4728286: RUNNING
  3565.  
  3566. Jul-28 01:02:13.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3567. Jul-28 01:02:18.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3568. job: 4713446: RUNNING
  3569. job: 4728323: RUNNING
  3570. job: 4728326: RUNNING
  3571. job: 4728325: RUNNING
  3572. job: 4728321: RUNNING
  3573. job: 4728286: RUNNING
  3574.  
  3575. Jul-28 01:02:18.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3576. Jul-28 01:02:18.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3577. job: 4713446: RUNNING
  3578. job: 4728323: RUNNING
  3579. job: 4728326: RUNNING
  3580. job: 4728325: RUNNING
  3581. job: 4728321: RUNNING
  3582. job: 4728286: RUNNING
  3583.  
  3584. Jul-28 01:02:18.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3585. Jul-28 01:02:23.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3586. job: 4713446: RUNNING
  3587. job: 4728323: RUNNING
  3588. job: 4728326: RUNNING
  3589. job: 4728325: RUNNING
  3590. job: 4728321: RUNNING
  3591. job: 4728286: RUNNING
  3592.  
  3593. Jul-28 01:02:23.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3594. Jul-28 01:02:23.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3595. job: 4713446: RUNNING
  3596. job: 4728323: RUNNING
  3597. job: 4728326: RUNNING
  3598. job: 4728325: RUNNING
  3599. job: 4728321: RUNNING
  3600. job: 4728286: RUNNING
  3601.  
  3602. Jul-28 01:02:23.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3603. Jul-28 01:02:28.856 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3604. job: 4713446: RUNNING
  3605. job: 4728323: RUNNING
  3606. job: 4728326: RUNNING
  3607. job: 4728325: RUNNING
  3608. job: 4728321: RUNNING
  3609. job: 4728286: RUNNING
  3610.  
  3611. Jul-28 01:02:28.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3612. Jul-28 01:02:28.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3613. job: 4713446: RUNNING
  3614. job: 4728323: RUNNING
  3615. job: 4728326: RUNNING
  3616. job: 4728325: RUNNING
  3617. job: 4728321: RUNNING
  3618. job: 4728286: RUNNING
  3619.  
  3620. Jul-28 01:02:28.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3621. Jul-28 01:02:33.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3622. job: 4713446: RUNNING
  3623. job: 4728323: RUNNING
  3624. job: 4728326: RUNNING
  3625. job: 4728325: RUNNING
  3626. job: 4728321: RUNNING
  3627. job: 4728286: RUNNING
  3628.  
  3629. Jul-28 01:02:33.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3630. Jul-28 01:02:33.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3631. job: 4713446: RUNNING
  3632. job: 4728323: RUNNING
  3633. job: 4728326: RUNNING
  3634. job: 4728325: RUNNING
  3635. job: 4728321: RUNNING
  3636. job: 4728286: RUNNING
  3637.  
  3638. Jul-28 01:02:33.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3639. Jul-28 01:02:38.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3640. job: 4713446: RUNNING
  3641. job: 4728323: RUNNING
  3642. job: 4728326: RUNNING
  3643. job: 4728325: RUNNING
  3644. job: 4728321: RUNNING
  3645. job: 4728286: RUNNING
  3646.  
  3647. Jul-28 01:02:38.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3648. Jul-28 01:02:38.859 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3649. job: 4713446: RUNNING
  3650. job: 4728323: RUNNING
  3651. job: 4728326: RUNNING
  3652. job: 4728325: RUNNING
  3653. job: 4728321: RUNNING
  3654. job: 4728286: RUNNING
  3655.  
  3656. Jul-28 01:02:38.859 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3657. Jul-28 01:02:43.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3658. job: 4713446: RUNNING
  3659. job: 4728323: RUNNING
  3660. job: 4728326: RUNNING
  3661. job: 4728325: RUNNING
  3662. job: 4728321: RUNNING
  3663. job: 4728286: RUNNING
  3664.  
  3665. Jul-28 01:02:43.859 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3666. Jul-28 01:02:43.859 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3667. job: 4713446: RUNNING
  3668. job: 4728323: RUNNING
  3669. job: 4728326: RUNNING
  3670. job: 4728325: RUNNING
  3671. job: 4728321: RUNNING
  3672. job: 4728286: RUNNING
  3673.  
  3674. Jul-28 01:02:43.860 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3675. Jul-28 01:02:48.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3676. job: 4713446: RUNNING
  3677. job: 4728323: RUNNING
  3678. job: 4728326: RUNNING
  3679. job: 4728325: RUNNING
  3680. job: 4728321: RUNNING
  3681. job: 4728286: RUNNING
  3682.  
  3683. Jul-28 01:02:48.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3684. Jul-28 01:02:48.859 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3685. job: 4713446: RUNNING
  3686. job: 4728323: RUNNING
  3687. job: 4728326: RUNNING
  3688. job: 4728325: RUNNING
  3689. job: 4728321: RUNNING
  3690. job: 4728286: RUNNING
  3691.  
  3692. Jul-28 01:02:48.859 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3693. Jul-28 01:02:53.859 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3694. job: 4713446: RUNNING
  3695. job: 4728323: RUNNING
  3696. job: 4728326: RUNNING
  3697. job: 4728325: RUNNING
  3698. job: 4728321: RUNNING
  3699. job: 4728286: RUNNING
  3700.  
  3701. Jul-28 01:02:53.859 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3702. Jul-28 01:02:53.860 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3703. job: 4713446: RUNNING
  3704. job: 4728323: RUNNING
  3705. job: 4728326: RUNNING
  3706. job: 4728325: RUNNING
  3707. job: 4728321: RUNNING
  3708. job: 4728286: RUNNING
  3709.  
  3710. Jul-28 01:02:53.860 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3711. Jul-28 01:02:58.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3712. job: 4713446: RUNNING
  3713. job: 4728323: RUNNING
  3714. job: 4728326: RUNNING
  3715. job: 4728325: RUNNING
  3716. job: 4728321: RUNNING
  3717. job: 4728286: RUNNING
  3718.  
  3719. Jul-28 01:02:58.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3720. Jul-28 01:02:58.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3721. job: 4713446: RUNNING
  3722. job: 4728323: RUNNING
  3723. job: 4728326: RUNNING
  3724. job: 4728325: RUNNING
  3725. job: 4728321: RUNNING
  3726. job: 4728286: RUNNING
  3727.  
  3728. Jul-28 01:02:58.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3729. Jul-28 01:03:03.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3730. job: 4713446: RUNNING
  3731. job: 4728323: RUNNING
  3732. job: 4728326: RUNNING
  3733. job: 4728325: RUNNING
  3734. job: 4728321: RUNNING
  3735. job: 4728286: RUNNING
  3736.  
  3737. Jul-28 01:03:03.857 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3738. Jul-28 01:03:03.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  3739. Jul-28 01:03:03.986 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  3740. 4713446,RUN,Jul 27 11:44
  3741. 4728323,RUN,Jul 28 00:40
  3742. 4728326,RUN,Jul 28 00:51
  3743. 4728325,RUN,Jul 28 00:49
  3744. 4728321,RUN,Jul 28 00:34
  3745. 4728286,RUN,Jul 27 23:44
  3746.  
  3747.  
  3748. Jul-28 01:03:03.986 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3749. job: 4713446: RUNNING
  3750. job: 4728323: RUNNING
  3751. job: 4728326: RUNNING
  3752. job: 4728325: RUNNING
  3753. job: 4728321: RUNNING
  3754. job: 4728286: RUNNING
  3755.  
  3756. Jul-28 01:03:03.986 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3757. Jul-28 01:03:08.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3758. job: 4713446: RUNNING
  3759. job: 4728323: RUNNING
  3760. job: 4728326: RUNNING
  3761. job: 4728325: RUNNING
  3762. job: 4728321: RUNNING
  3763. job: 4728286: RUNNING
  3764.  
  3765. Jul-28 01:03:08.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3766. Jul-28 01:03:08.859 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3767. job: 4713446: RUNNING
  3768. job: 4728323: RUNNING
  3769. job: 4728326: RUNNING
  3770. job: 4728325: RUNNING
  3771. job: 4728321: RUNNING
  3772. job: 4728286: RUNNING
  3773.  
  3774. Jul-28 01:03:08.859 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3775. Jul-28 01:03:13.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3776. job: 4713446: RUNNING
  3777. job: 4728323: RUNNING
  3778. job: 4728326: RUNNING
  3779. job: 4728325: RUNNING
  3780. job: 4728321: RUNNING
  3781. job: 4728286: RUNNING
  3782.  
  3783. Jul-28 01:03:13.859 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3784. Jul-28 01:03:13.859 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3785. job: 4713446: RUNNING
  3786. job: 4728323: RUNNING
  3787. job: 4728326: RUNNING
  3788. job: 4728325: RUNNING
  3789. job: 4728321: RUNNING
  3790. job: 4728286: RUNNING
  3791.  
  3792. Jul-28 01:03:13.859 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3793. Jul-28 01:03:18.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3794. job: 4713446: RUNNING
  3795. job: 4728323: RUNNING
  3796. job: 4728326: RUNNING
  3797. job: 4728325: RUNNING
  3798. job: 4728321: RUNNING
  3799. job: 4728286: RUNNING
  3800.  
  3801. Jul-28 01:03:18.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3802. Jul-28 01:03:18.859 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3803. job: 4713446: RUNNING
  3804. job: 4728323: RUNNING
  3805. job: 4728326: RUNNING
  3806. job: 4728325: RUNNING
  3807. job: 4728321: RUNNING
  3808. job: 4728286: RUNNING
  3809.  
  3810. Jul-28 01:03:18.859 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3811. Jul-28 01:03:23.858 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3812. job: 4713446: RUNNING
  3813. job: 4728323: RUNNING
  3814. job: 4728326: RUNNING
  3815. job: 4728325: RUNNING
  3816. job: 4728321: RUNNING
  3817. job: 4728286: RUNNING
  3818.  
  3819. Jul-28 01:03:23.859 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3820. Jul-28 01:03:23.859 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3821. job: 4713446: RUNNING
  3822. job: 4728323: RUNNING
  3823. job: 4728326: RUNNING
  3824. job: 4728325: RUNNING
  3825. job: 4728321: RUNNING
  3826. job: 4728286: RUNNING
  3827.  
  3828. Jul-28 01:03:23.859 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3829. Jul-28 01:03:28.859 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3830. job: 4713446: RUNNING
  3831. job: 4728323: RUNNING
  3832. job: 4728326: RUNNING
  3833. job: 4728325: RUNNING
  3834. job: 4728321: RUNNING
  3835. job: 4728286: RUNNING
  3836.  
  3837. Jul-28 01:03:28.859 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3838. Jul-28 01:03:28.860 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3839. job: 4713446: RUNNING
  3840. job: 4728323: RUNNING
  3841. job: 4728326: RUNNING
  3842. job: 4728325: RUNNING
  3843. job: 4728321: RUNNING
  3844. job: 4728286: RUNNING
  3845.  
  3846. Jul-28 01:03:28.860 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3847. Jul-28 01:03:33.859 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3848. job: 4713446: RUNNING
  3849. job: 4728323: RUNNING
  3850. job: 4728326: RUNNING
  3851. job: 4728325: RUNNING
  3852. job: 4728321: RUNNING
  3853. job: 4728286: RUNNING
  3854.  
  3855. Jul-28 01:03:33.860 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3856. Jul-28 01:03:33.860 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3857. job: 4713446: RUNNING
  3858. job: 4728323: RUNNING
  3859. job: 4728326: RUNNING
  3860. job: 4728325: RUNNING
  3861. job: 4728321: RUNNING
  3862. job: 4728286: RUNNING
  3863.  
  3864. Jul-28 01:03:33.860 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3865. Jul-28 01:03:38.860 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3866. job: 4713446: RUNNING
  3867. job: 4728323: RUNNING
  3868. job: 4728326: RUNNING
  3869. job: 4728325: RUNNING
  3870. job: 4728321: RUNNING
  3871. job: 4728286: RUNNING
  3872.  
  3873. Jul-28 01:03:38.860 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3874. Jul-28 01:03:38.861 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3875. job: 4713446: RUNNING
  3876. job: 4728323: RUNNING
  3877. job: 4728326: RUNNING
  3878. job: 4728325: RUNNING
  3879. job: 4728321: RUNNING
  3880. job: 4728286: RUNNING
  3881.  
  3882. Jul-28 01:03:38.861 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3883. Jul-28 01:03:43.860 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3884. job: 4713446: RUNNING
  3885. job: 4728323: RUNNING
  3886. job: 4728326: RUNNING
  3887. job: 4728325: RUNNING
  3888. job: 4728321: RUNNING
  3889. job: 4728286: RUNNING
  3890.  
  3891. Jul-28 01:03:43.861 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3892. Jul-28 01:03:43.862 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3893. job: 4713446: RUNNING
  3894. job: 4728323: RUNNING
  3895. job: 4728326: RUNNING
  3896. job: 4728325: RUNNING
  3897. job: 4728321: RUNNING
  3898. job: 4728286: RUNNING
  3899.  
  3900. Jul-28 01:03:43.862 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3901. Jul-28 01:03:48.860 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3902. job: 4713446: RUNNING
  3903. job: 4728323: RUNNING
  3904. job: 4728326: RUNNING
  3905. job: 4728325: RUNNING
  3906. job: 4728321: RUNNING
  3907. job: 4728286: RUNNING
  3908.  
  3909. Jul-28 01:03:48.860 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3910. Jul-28 01:03:48.861 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3911. job: 4713446: RUNNING
  3912. job: 4728323: RUNNING
  3913. job: 4728326: RUNNING
  3914. job: 4728325: RUNNING
  3915. job: 4728321: RUNNING
  3916. job: 4728286: RUNNING
  3917.  
  3918. Jul-28 01:03:48.861 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3919. Jul-28 01:03:53.862 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3920. job: 4713446: RUNNING
  3921. job: 4728323: RUNNING
  3922. job: 4728326: RUNNING
  3923. job: 4728325: RUNNING
  3924. job: 4728321: RUNNING
  3925. job: 4728286: RUNNING
  3926.  
  3927. Jul-28 01:03:53.863 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3928. Jul-28 01:03:53.864 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3929. job: 4713446: RUNNING
  3930. job: 4728323: RUNNING
  3931. job: 4728326: RUNNING
  3932. job: 4728325: RUNNING
  3933. job: 4728321: RUNNING
  3934. job: 4728286: RUNNING
  3935.  
  3936. Jul-28 01:03:53.864 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3937. Jul-28 01:03:58.861 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3938. job: 4713446: RUNNING
  3939. job: 4728323: RUNNING
  3940. job: 4728326: RUNNING
  3941. job: 4728325: RUNNING
  3942. job: 4728321: RUNNING
  3943. job: 4728286: RUNNING
  3944.  
  3945. Jul-28 01:03:58.861 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3946. Jul-28 01:03:58.862 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3947. job: 4713446: RUNNING
  3948. job: 4728323: RUNNING
  3949. job: 4728326: RUNNING
  3950. job: 4728325: RUNNING
  3951. job: 4728321: RUNNING
  3952. job: 4728286: RUNNING
  3953.  
  3954. Jul-28 01:03:58.862 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3955. Jul-28 01:04:03.861 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  3956. Jul-28 01:04:03.987 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  3957. 4713446,RUN,Jul 27 11:44
  3958. 4728323,RUN,Jul 28 00:40
  3959. 4728326,RUN,Jul 28 00:51
  3960. 4728325,RUN,Jul 28 00:49
  3961. 4728321,RUN,Jul 28 00:34
  3962. 4728286,RUN,Jul 27 23:44
  3963.  
  3964.  
  3965. Jul-28 01:04:03.987 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3966. job: 4713446: RUNNING
  3967. job: 4728323: RUNNING
  3968. job: 4728326: RUNNING
  3969. job: 4728325: RUNNING
  3970. job: 4728321: RUNNING
  3971. job: 4728286: RUNNING
  3972.  
  3973. Jul-28 01:04:03.987 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3974. Jul-28 01:04:03.988 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3975. job: 4713446: RUNNING
  3976. job: 4728323: RUNNING
  3977. job: 4728326: RUNNING
  3978. job: 4728325: RUNNING
  3979. job: 4728321: RUNNING
  3980. job: 4728286: RUNNING
  3981.  
  3982. Jul-28 01:04:03.988 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  3983. Jul-28 01:04:08.861 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3984. job: 4713446: RUNNING
  3985. job: 4728323: RUNNING
  3986. job: 4728326: RUNNING
  3987. job: 4728325: RUNNING
  3988. job: 4728321: RUNNING
  3989. job: 4728286: RUNNING
  3990.  
  3991. Jul-28 01:04:08.862 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  3992. Jul-28 01:04:08.862 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  3993. job: 4713446: RUNNING
  3994. job: 4728323: RUNNING
  3995. job: 4728326: RUNNING
  3996. job: 4728325: RUNNING
  3997. job: 4728321: RUNNING
  3998. job: 4728286: RUNNING
  3999.  
  4000. Jul-28 01:04:08.863 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4001. Jul-28 01:04:13.860 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4002. job: 4713446: RUNNING
  4003. job: 4728323: RUNNING
  4004. job: 4728326: RUNNING
  4005. job: 4728325: RUNNING
  4006. job: 4728321: RUNNING
  4007. job: 4728286: RUNNING
  4008.  
  4009. Jul-28 01:04:13.861 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4010. Jul-28 01:04:13.862 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4011. job: 4713446: RUNNING
  4012. job: 4728323: RUNNING
  4013. job: 4728326: RUNNING
  4014. job: 4728325: RUNNING
  4015. job: 4728321: RUNNING
  4016. job: 4728286: RUNNING
  4017.  
  4018. Jul-28 01:04:13.862 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4019. Jul-28 01:04:18.860 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4020. job: 4713446: RUNNING
  4021. job: 4728323: RUNNING
  4022. job: 4728326: RUNNING
  4023. job: 4728325: RUNNING
  4024. job: 4728321: RUNNING
  4025. job: 4728286: RUNNING
  4026.  
  4027. Jul-28 01:04:18.861 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4028. Jul-28 01:04:18.862 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4029. job: 4713446: RUNNING
  4030. job: 4728323: RUNNING
  4031. job: 4728326: RUNNING
  4032. job: 4728325: RUNNING
  4033. job: 4728321: RUNNING
  4034. job: 4728286: RUNNING
  4035.  
  4036. Jul-28 01:04:18.862 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4037. Jul-28 01:04:23.861 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4038. job: 4713446: RUNNING
  4039. job: 4728323: RUNNING
  4040. job: 4728326: RUNNING
  4041. job: 4728325: RUNNING
  4042. job: 4728321: RUNNING
  4043. job: 4728286: RUNNING
  4044.  
  4045. Jul-28 01:04:23.861 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4046. Jul-28 01:04:23.862 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4047. job: 4713446: RUNNING
  4048. job: 4728323: RUNNING
  4049. job: 4728326: RUNNING
  4050. job: 4728325: RUNNING
  4051. job: 4728321: RUNNING
  4052. job: 4728286: RUNNING
  4053.  
  4054. Jul-28 01:04:23.862 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4055. Jul-28 01:04:28.861 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4056. job: 4713446: RUNNING
  4057. job: 4728323: RUNNING
  4058. job: 4728326: RUNNING
  4059. job: 4728325: RUNNING
  4060. job: 4728321: RUNNING
  4061. job: 4728286: RUNNING
  4062.  
  4063. Jul-28 01:04:28.862 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4064. Jul-28 01:04:28.863 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4065. job: 4713446: RUNNING
  4066. job: 4728323: RUNNING
  4067. job: 4728326: RUNNING
  4068. job: 4728325: RUNNING
  4069. job: 4728321: RUNNING
  4070. job: 4728286: RUNNING
  4071.  
  4072. Jul-28 01:04:28.863 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4073. Jul-28 01:04:33.860 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4074. job: 4713446: RUNNING
  4075. job: 4728323: RUNNING
  4076. job: 4728326: RUNNING
  4077. job: 4728325: RUNNING
  4078. job: 4728321: RUNNING
  4079. job: 4728286: RUNNING
  4080.  
  4081. Jul-28 01:04:33.860 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4082. Jul-28 01:04:33.861 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4083. job: 4713446: RUNNING
  4084. job: 4728323: RUNNING
  4085. job: 4728326: RUNNING
  4086. job: 4728325: RUNNING
  4087. job: 4728321: RUNNING
  4088. job: 4728286: RUNNING
  4089.  
  4090. Jul-28 01:04:33.861 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4091. Jul-28 01:04:38.860 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4092. job: 4713446: RUNNING
  4093. job: 4728323: RUNNING
  4094. job: 4728326: RUNNING
  4095. job: 4728325: RUNNING
  4096. job: 4728321: RUNNING
  4097. job: 4728286: RUNNING
  4098.  
  4099. Jul-28 01:04:38.860 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4100. Jul-28 01:04:38.861 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4101. job: 4713446: RUNNING
  4102. job: 4728323: RUNNING
  4103. job: 4728326: RUNNING
  4104. job: 4728325: RUNNING
  4105. job: 4728321: RUNNING
  4106. job: 4728286: RUNNING
  4107.  
  4108. Jul-28 01:04:38.861 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4109. Jul-28 01:04:43.860 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4110. job: 4713446: RUNNING
  4111. job: 4728323: RUNNING
  4112. job: 4728326: RUNNING
  4113. job: 4728325: RUNNING
  4114. job: 4728321: RUNNING
  4115. job: 4728286: RUNNING
  4116.  
  4117. Jul-28 01:04:43.861 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4118. Jul-28 01:04:43.861 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4119. job: 4713446: RUNNING
  4120. job: 4728323: RUNNING
  4121. job: 4728326: RUNNING
  4122. job: 4728325: RUNNING
  4123. job: 4728321: RUNNING
  4124. job: 4728286: RUNNING
  4125.  
  4126. Jul-28 01:04:43.861 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4127. Jul-28 01:04:48.861 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4128. job: 4713446: RUNNING
  4129. job: 4728323: RUNNING
  4130. job: 4728326: RUNNING
  4131. job: 4728325: RUNNING
  4132. job: 4728321: RUNNING
  4133. job: 4728286: RUNNING
  4134.  
  4135. Jul-28 01:04:48.861 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4136. Jul-28 01:04:48.862 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4137. job: 4713446: RUNNING
  4138. job: 4728323: RUNNING
  4139. job: 4728326: RUNNING
  4140. job: 4728325: RUNNING
  4141. job: 4728321: RUNNING
  4142. job: 4728286: RUNNING
  4143.  
  4144. Jul-28 01:04:48.862 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4145. Jul-28 01:04:53.861 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4146. job: 4713446: RUNNING
  4147. job: 4728323: RUNNING
  4148. job: 4728326: RUNNING
  4149. job: 4728325: RUNNING
  4150. job: 4728321: RUNNING
  4151. job: 4728286: RUNNING
  4152.  
  4153. Jul-28 01:04:53.862 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4154. Jul-28 01:04:53.863 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4155. job: 4713446: RUNNING
  4156. job: 4728323: RUNNING
  4157. job: 4728326: RUNNING
  4158. job: 4728325: RUNNING
  4159. job: 4728321: RUNNING
  4160. job: 4728286: RUNNING
  4161.  
  4162. Jul-28 01:04:53.863 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4163. Jul-28 01:04:58.861 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4164. job: 4713446: RUNNING
  4165. job: 4728323: RUNNING
  4166. job: 4728326: RUNNING
  4167. job: 4728325: RUNNING
  4168. job: 4728321: RUNNING
  4169. job: 4728286: RUNNING
  4170.  
  4171. Jul-28 01:04:58.861 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4172. Jul-28 01:04:58.862 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4173. job: 4713446: RUNNING
  4174. job: 4728323: RUNNING
  4175. job: 4728326: RUNNING
  4176. job: 4728325: RUNNING
  4177. job: 4728321: RUNNING
  4178. job: 4728286: RUNNING
  4179.  
  4180. Jul-28 01:04:58.862 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4181. Jul-28 01:05:03.862 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  4182. Jul-28 01:05:03.998 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  4183. 4713446,RUN,Jul 27 11:44
  4184. 4728323,RUN,Jul 28 00:40
  4185. 4728326,RUN,Jul 28 00:51
  4186. 4728325,RUN,Jul 28 00:49
  4187. 4728321,RUN,Jul 28 00:34
  4188. 4728286,RUN,Jul 27 23:44
  4189.  
  4190.  
  4191. Jul-28 01:05:03.998 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4192. job: 4713446: RUNNING
  4193. job: 4728323: RUNNING
  4194. job: 4728326: RUNNING
  4195. job: 4728325: RUNNING
  4196. job: 4728321: RUNNING
  4197. job: 4728286: RUNNING
  4198.  
  4199. Jul-28 01:05:03.998 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4200. Jul-28 01:05:03.999 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4201. job: 4713446: RUNNING
  4202. job: 4728323: RUNNING
  4203. job: 4728326: RUNNING
  4204. job: 4728325: RUNNING
  4205. job: 4728321: RUNNING
  4206. job: 4728286: RUNNING
  4207.  
  4208. Jul-28 01:05:03.999 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4209. Jul-28 01:05:08.867 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor lsf > tasks to be completed: 2 -- pending tasks are shown below
  4210. ~> TaskHandler[jobId: 4728325; id: 152; name: merge_pairsam_into_runs (library:HeLa1 run:lane1); status: RUNNING; exit: -; error: -; workDir: /farline/umw_job_dekker/HPCC/sv49w/distiller-nf/work/3b/37ed2fa2ba160f54196e762e24c83b started: 1501217508853; exited: -; ]
  4211. ~> TaskHandler[jobId: 4728326; id: 153; name: merge_pairsam_into_runs (library:HeLa1 run:lane3); status: RUNNING; exit: -; error: -; workDir: /farline/umw_job_dekker/HPCC/sv49w/distiller-nf/work/00/2633edc9da9ad5649d3d6dcffe5a44 started: 1501217938848; exited: -; ]
  4212. Jul-28 01:05:08.867 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4213. job: 4713446: RUNNING
  4214. job: 4728323: RUNNING
  4215. job: 4728326: RUNNING
  4216. job: 4728325: RUNNING
  4217. job: 4728321: RUNNING
  4218. job: 4728286: RUNNING
  4219.  
  4220. Jul-28 01:05:08.867 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4221. Jul-28 01:05:08.867 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4222. job: 4713446: RUNNING
  4223. job: 4728323: RUNNING
  4224. job: 4728326: RUNNING
  4225. job: 4728325: RUNNING
  4226. job: 4728321: RUNNING
  4227. job: 4728286: RUNNING
  4228.  
  4229. Jul-28 01:05:08.868 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4230. Jul-28 01:05:13.870 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4231. job: 4713446: RUNNING
  4232. job: 4728323: RUNNING
  4233. job: 4728326: RUNNING
  4234. job: 4728325: RUNNING
  4235. job: 4728321: RUNNING
  4236. job: 4728286: RUNNING
  4237.  
  4238. Jul-28 01:05:13.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4239. Jul-28 01:05:13.872 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4240. job: 4713446: RUNNING
  4241. job: 4728323: RUNNING
  4242. job: 4728326: RUNNING
  4243. job: 4728325: RUNNING
  4244. job: 4728321: RUNNING
  4245. job: 4728286: RUNNING
  4246.  
  4247. Jul-28 01:05:13.872 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4248. Jul-28 01:05:18.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4249. job: 4713446: RUNNING
  4250. job: 4728323: RUNNING
  4251. job: 4728326: RUNNING
  4252. job: 4728325: RUNNING
  4253. job: 4728321: RUNNING
  4254. job: 4728286: RUNNING
  4255.  
  4256. Jul-28 01:05:18.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4257. Jul-28 01:05:18.872 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4258. job: 4713446: RUNNING
  4259. job: 4728323: RUNNING
  4260. job: 4728326: RUNNING
  4261. job: 4728325: RUNNING
  4262. job: 4728321: RUNNING
  4263. job: 4728286: RUNNING
  4264.  
  4265. Jul-28 01:05:18.872 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4266. Jul-28 01:05:23.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4267. job: 4713446: RUNNING
  4268. job: 4728323: RUNNING
  4269. job: 4728326: RUNNING
  4270. job: 4728325: RUNNING
  4271. job: 4728321: RUNNING
  4272. job: 4728286: RUNNING
  4273.  
  4274. Jul-28 01:05:23.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4275. Jul-28 01:05:23.872 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4276. job: 4713446: RUNNING
  4277. job: 4728323: RUNNING
  4278. job: 4728326: RUNNING
  4279. job: 4728325: RUNNING
  4280. job: 4728321: RUNNING
  4281. job: 4728286: RUNNING
  4282.  
  4283. Jul-28 01:05:23.872 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4284. Jul-28 01:05:28.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4285. job: 4713446: RUNNING
  4286. job: 4728323: RUNNING
  4287. job: 4728326: RUNNING
  4288. job: 4728325: RUNNING
  4289. job: 4728321: RUNNING
  4290. job: 4728286: RUNNING
  4291.  
  4292. Jul-28 01:05:28.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4293. Jul-28 01:05:28.872 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4294. job: 4713446: RUNNING
  4295. job: 4728323: RUNNING
  4296. job: 4728326: RUNNING
  4297. job: 4728325: RUNNING
  4298. job: 4728321: RUNNING
  4299. job: 4728286: RUNNING
  4300.  
  4301. Jul-28 01:05:28.872 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4302. Jul-28 01:05:33.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4303. job: 4713446: RUNNING
  4304. job: 4728323: RUNNING
  4305. job: 4728326: RUNNING
  4306. job: 4728325: RUNNING
  4307. job: 4728321: RUNNING
  4308. job: 4728286: RUNNING
  4309.  
  4310. Jul-28 01:05:33.872 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4311. Jul-28 01:05:33.873 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4312. job: 4713446: RUNNING
  4313. job: 4728323: RUNNING
  4314. job: 4728326: RUNNING
  4315. job: 4728325: RUNNING
  4316. job: 4728321: RUNNING
  4317. job: 4728286: RUNNING
  4318.  
  4319. Jul-28 01:05:33.873 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4320. Jul-28 01:05:38.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4321. job: 4713446: RUNNING
  4322. job: 4728323: RUNNING
  4323. job: 4728326: RUNNING
  4324. job: 4728325: RUNNING
  4325. job: 4728321: RUNNING
  4326. job: 4728286: RUNNING
  4327.  
  4328. Jul-28 01:05:38.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4329. Jul-28 01:05:38.872 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4330. job: 4713446: RUNNING
  4331. job: 4728323: RUNNING
  4332. job: 4728326: RUNNING
  4333. job: 4728325: RUNNING
  4334. job: 4728321: RUNNING
  4335. job: 4728286: RUNNING
  4336.  
  4337. Jul-28 01:05:38.872 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4338. Jul-28 01:05:43.870 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4339. job: 4713446: RUNNING
  4340. job: 4728323: RUNNING
  4341. job: 4728326: RUNNING
  4342. job: 4728325: RUNNING
  4343. job: 4728321: RUNNING
  4344. job: 4728286: RUNNING
  4345.  
  4346. Jul-28 01:05:43.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4347. Jul-28 01:05:43.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4348. job: 4713446: RUNNING
  4349. job: 4728323: RUNNING
  4350. job: 4728326: RUNNING
  4351. job: 4728325: RUNNING
  4352. job: 4728321: RUNNING
  4353. job: 4728286: RUNNING
  4354.  
  4355. Jul-28 01:05:43.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4356. Jul-28 01:05:48.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4357. job: 4713446: RUNNING
  4358. job: 4728323: RUNNING
  4359. job: 4728326: RUNNING
  4360. job: 4728325: RUNNING
  4361. job: 4728321: RUNNING
  4362. job: 4728286: RUNNING
  4363.  
  4364. Jul-28 01:05:48.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4365. Jul-28 01:05:48.872 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4366. job: 4713446: RUNNING
  4367. job: 4728323: RUNNING
  4368. job: 4728326: RUNNING
  4369. job: 4728325: RUNNING
  4370. job: 4728321: RUNNING
  4371. job: 4728286: RUNNING
  4372.  
  4373. Jul-28 01:05:48.872 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4374. Jul-28 01:05:53.870 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4375. job: 4713446: RUNNING
  4376. job: 4728323: RUNNING
  4377. job: 4728326: RUNNING
  4378. job: 4728325: RUNNING
  4379. job: 4728321: RUNNING
  4380. job: 4728286: RUNNING
  4381.  
  4382. Jul-28 01:05:53.870 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4383. Jul-28 01:05:53.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4384. job: 4713446: RUNNING
  4385. job: 4728323: RUNNING
  4386. job: 4728326: RUNNING
  4387. job: 4728325: RUNNING
  4388. job: 4728321: RUNNING
  4389. job: 4728286: RUNNING
  4390.  
  4391. Jul-28 01:05:53.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4392. Jul-28 01:05:58.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4393. job: 4713446: RUNNING
  4394. job: 4728323: RUNNING
  4395. job: 4728326: RUNNING
  4396. job: 4728325: RUNNING
  4397. job: 4728321: RUNNING
  4398. job: 4728286: RUNNING
  4399.  
  4400. Jul-28 01:05:58.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4401. Jul-28 01:05:58.872 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4402. job: 4713446: RUNNING
  4403. job: 4728323: RUNNING
  4404. job: 4728326: RUNNING
  4405. job: 4728325: RUNNING
  4406. job: 4728321: RUNNING
  4407. job: 4728286: RUNNING
  4408.  
  4409. Jul-28 01:05:58.872 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4410. Jul-28 01:06:03.869 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  4411. Jul-28 01:06:04.001 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  4412. 4713446,RUN,Jul 27 11:44
  4413. 4728323,RUN,Jul 28 00:40
  4414. 4728326,RUN,Jul 28 00:51
  4415. 4728325,RUN,Jul 28 00:49
  4416. 4728321,RUN,Jul 28 00:34
  4417. 4728286,RUN,Jul 27 23:44
  4418.  
  4419.  
  4420. Jul-28 01:06:04.001 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4421. job: 4713446: RUNNING
  4422. job: 4728323: RUNNING
  4423. job: 4728326: RUNNING
  4424. job: 4728325: RUNNING
  4425. job: 4728321: RUNNING
  4426. job: 4728286: RUNNING
  4427.  
  4428. Jul-28 01:06:04.001 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4429. Jul-28 01:06:04.002 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4430. job: 4713446: RUNNING
  4431. job: 4728323: RUNNING
  4432. job: 4728326: RUNNING
  4433. job: 4728325: RUNNING
  4434. job: 4728321: RUNNING
  4435. job: 4728286: RUNNING
  4436.  
  4437. Jul-28 01:06:04.002 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4438. Jul-28 01:06:08.870 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4439. job: 4713446: RUNNING
  4440. job: 4728323: RUNNING
  4441. job: 4728326: RUNNING
  4442. job: 4728325: RUNNING
  4443. job: 4728321: RUNNING
  4444. job: 4728286: RUNNING
  4445.  
  4446. Jul-28 01:06:08.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4447. Jul-28 01:06:08.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4448. job: 4713446: RUNNING
  4449. job: 4728323: RUNNING
  4450. job: 4728326: RUNNING
  4451. job: 4728325: RUNNING
  4452. job: 4728321: RUNNING
  4453. job: 4728286: RUNNING
  4454.  
  4455. Jul-28 01:06:08.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4456. Jul-28 01:06:13.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4457. job: 4713446: RUNNING
  4458. job: 4728323: RUNNING
  4459. job: 4728326: RUNNING
  4460. job: 4728325: RUNNING
  4461. job: 4728321: RUNNING
  4462. job: 4728286: RUNNING
  4463.  
  4464. Jul-28 01:06:13.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4465. Jul-28 01:06:13.872 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4466. job: 4713446: RUNNING
  4467. job: 4728323: RUNNING
  4468. job: 4728326: RUNNING
  4469. job: 4728325: RUNNING
  4470. job: 4728321: RUNNING
  4471. job: 4728286: RUNNING
  4472.  
  4473. Jul-28 01:06:13.872 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4474. Jul-28 01:06:18.887 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4475. job: 4713446: RUNNING
  4476. job: 4728323: RUNNING
  4477. job: 4728326: RUNNING
  4478. job: 4728325: RUNNING
  4479. job: 4728321: RUNNING
  4480. job: 4728286: RUNNING
  4481.  
  4482. Jul-28 01:06:18.888 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4483. Jul-28 01:06:18.889 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4484. job: 4713446: RUNNING
  4485. job: 4728323: RUNNING
  4486. job: 4728326: RUNNING
  4487. job: 4728325: RUNNING
  4488. job: 4728321: RUNNING
  4489. job: 4728286: RUNNING
  4490.  
  4491. Jul-28 01:06:18.889 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4492. Jul-28 01:06:23.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4493. job: 4713446: RUNNING
  4494. job: 4728323: RUNNING
  4495. job: 4728326: RUNNING
  4496. job: 4728325: RUNNING
  4497. job: 4728321: RUNNING
  4498. job: 4728286: RUNNING
  4499.  
  4500. Jul-28 01:06:23.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4501. Jul-28 01:06:23.872 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4502. job: 4713446: RUNNING
  4503. job: 4728323: RUNNING
  4504. job: 4728326: RUNNING
  4505. job: 4728325: RUNNING
  4506. job: 4728321: RUNNING
  4507. job: 4728286: RUNNING
  4508.  
  4509. Jul-28 01:06:23.872 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4510. Jul-28 01:06:28.870 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4511. job: 4713446: RUNNING
  4512. job: 4728323: RUNNING
  4513. job: 4728326: RUNNING
  4514. job: 4728325: RUNNING
  4515. job: 4728321: RUNNING
  4516. job: 4728286: RUNNING
  4517.  
  4518. Jul-28 01:06:28.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4519. Jul-28 01:06:28.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4520. job: 4713446: RUNNING
  4521. job: 4728323: RUNNING
  4522. job: 4728326: RUNNING
  4523. job: 4728325: RUNNING
  4524. job: 4728321: RUNNING
  4525. job: 4728286: RUNNING
  4526.  
  4527. Jul-28 01:06:28.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4528. Jul-28 01:06:33.870 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4529. job: 4713446: RUNNING
  4530. job: 4728323: RUNNING
  4531. job: 4728326: RUNNING
  4532. job: 4728325: RUNNING
  4533. job: 4728321: RUNNING
  4534. job: 4728286: RUNNING
  4535.  
  4536. Jul-28 01:06:33.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4537. Jul-28 01:06:33.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4538. job: 4713446: RUNNING
  4539. job: 4728323: RUNNING
  4540. job: 4728326: RUNNING
  4541. job: 4728325: RUNNING
  4542. job: 4728321: RUNNING
  4543. job: 4728286: RUNNING
  4544.  
  4545. Jul-28 01:06:33.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4546. Jul-28 01:06:38.870 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4547. job: 4713446: RUNNING
  4548. job: 4728323: RUNNING
  4549. job: 4728326: RUNNING
  4550. job: 4728325: RUNNING
  4551. job: 4728321: RUNNING
  4552. job: 4728286: RUNNING
  4553.  
  4554. Jul-28 01:06:38.870 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4555. Jul-28 01:06:38.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4556. job: 4713446: RUNNING
  4557. job: 4728323: RUNNING
  4558. job: 4728326: RUNNING
  4559. job: 4728325: RUNNING
  4560. job: 4728321: RUNNING
  4561. job: 4728286: RUNNING
  4562.  
  4563. Jul-28 01:06:38.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4564. Jul-28 01:06:43.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4565. job: 4713446: RUNNING
  4566. job: 4728323: RUNNING
  4567. job: 4728326: RUNNING
  4568. job: 4728325: RUNNING
  4569. job: 4728321: RUNNING
  4570. job: 4728286: RUNNING
  4571.  
  4572. Jul-28 01:06:43.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4573. Jul-28 01:06:43.872 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4574. job: 4713446: RUNNING
  4575. job: 4728323: RUNNING
  4576. job: 4728326: RUNNING
  4577. job: 4728325: RUNNING
  4578. job: 4728321: RUNNING
  4579. job: 4728286: RUNNING
  4580.  
  4581. Jul-28 01:06:43.872 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4582. Jul-28 01:06:48.870 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4583. job: 4713446: RUNNING
  4584. job: 4728323: RUNNING
  4585. job: 4728326: RUNNING
  4586. job: 4728325: RUNNING
  4587. job: 4728321: RUNNING
  4588. job: 4728286: RUNNING
  4589.  
  4590. Jul-28 01:06:48.870 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4591. Jul-28 01:06:48.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4592. job: 4713446: RUNNING
  4593. job: 4728323: RUNNING
  4594. job: 4728326: RUNNING
  4595. job: 4728325: RUNNING
  4596. job: 4728321: RUNNING
  4597. job: 4728286: RUNNING
  4598.  
  4599. Jul-28 01:06:48.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4600. Jul-28 01:06:53.873 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4601. job: 4713446: RUNNING
  4602. job: 4728323: RUNNING
  4603. job: 4728326: RUNNING
  4604. job: 4728325: RUNNING
  4605. job: 4728321: RUNNING
  4606. job: 4728286: RUNNING
  4607.  
  4608. Jul-28 01:06:53.873 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4609. Jul-28 01:06:53.875 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4610. job: 4713446: RUNNING
  4611. job: 4728323: RUNNING
  4612. job: 4728326: RUNNING
  4613. job: 4728325: RUNNING
  4614. job: 4728321: RUNNING
  4615. job: 4728286: RUNNING
  4616.  
  4617. Jul-28 01:06:53.875 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4618. Jul-28 01:06:58.869 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4619. job: 4713446: RUNNING
  4620. job: 4728323: RUNNING
  4621. job: 4728326: RUNNING
  4622. job: 4728325: RUNNING
  4623. job: 4728321: RUNNING
  4624. job: 4728286: RUNNING
  4625.  
  4626. Jul-28 01:06:58.869 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4627. Jul-28 01:06:58.870 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4628. job: 4713446: RUNNING
  4629. job: 4728323: RUNNING
  4630. job: 4728326: RUNNING
  4631. job: 4728325: RUNNING
  4632. job: 4728321: RUNNING
  4633. job: 4728286: RUNNING
  4634.  
  4635. Jul-28 01:06:58.870 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4636. Jul-28 01:07:03.870 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  4637. Jul-28 01:07:04.002 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  4638. 4713446,RUN,Jul 27 11:44
  4639. 4728323,RUN,Jul 28 00:40
  4640. 4728326,RUN,Jul 28 00:51
  4641. 4728325,RUN,Jul 28 00:49
  4642. 4728321,RUN,Jul 28 00:34
  4643. 4728286,RUN,Jul 27 23:44
  4644.  
  4645.  
  4646. Jul-28 01:07:04.002 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4647. job: 4713446: RUNNING
  4648. job: 4728323: RUNNING
  4649. job: 4728326: RUNNING
  4650. job: 4728325: RUNNING
  4651. job: 4728321: RUNNING
  4652. job: 4728286: RUNNING
  4653.  
  4654. Jul-28 01:07:04.002 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4655. Jul-28 01:07:04.003 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4656. job: 4713446: RUNNING
  4657. job: 4728323: RUNNING
  4658. job: 4728326: RUNNING
  4659. job: 4728325: RUNNING
  4660. job: 4728321: RUNNING
  4661. job: 4728286: RUNNING
  4662.  
  4663. Jul-28 01:07:04.003 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4664. Jul-28 01:07:08.870 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4665. job: 4713446: RUNNING
  4666. job: 4728323: RUNNING
  4667. job: 4728326: RUNNING
  4668. job: 4728325: RUNNING
  4669. job: 4728321: RUNNING
  4670. job: 4728286: RUNNING
  4671.  
  4672. Jul-28 01:07:08.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4673. Jul-28 01:07:08.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4674. job: 4713446: RUNNING
  4675. job: 4728323: RUNNING
  4676. job: 4728326: RUNNING
  4677. job: 4728325: RUNNING
  4678. job: 4728321: RUNNING
  4679. job: 4728286: RUNNING
  4680.  
  4681. Jul-28 01:07:08.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4682. Jul-28 01:07:13.872 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4683. job: 4713446: RUNNING
  4684. job: 4728323: RUNNING
  4685. job: 4728326: RUNNING
  4686. job: 4728325: RUNNING
  4687. job: 4728321: RUNNING
  4688. job: 4728286: RUNNING
  4689.  
  4690. Jul-28 01:07:13.872 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4691. Jul-28 01:07:13.873 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4692. job: 4713446: RUNNING
  4693. job: 4728323: RUNNING
  4694. job: 4728326: RUNNING
  4695. job: 4728325: RUNNING
  4696. job: 4728321: RUNNING
  4697. job: 4728286: RUNNING
  4698.  
  4699. Jul-28 01:07:13.873 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4700. Jul-28 01:07:18.872 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4701. job: 4713446: RUNNING
  4702. job: 4728323: RUNNING
  4703. job: 4728326: RUNNING
  4704. job: 4728325: RUNNING
  4705. job: 4728321: RUNNING
  4706. job: 4728286: RUNNING
  4707.  
  4708. Jul-28 01:07:18.872 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4709. Jul-28 01:07:18.873 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4710. job: 4713446: RUNNING
  4711. job: 4728323: RUNNING
  4712. job: 4728326: RUNNING
  4713. job: 4728325: RUNNING
  4714. job: 4728321: RUNNING
  4715. job: 4728286: RUNNING
  4716.  
  4717. Jul-28 01:07:18.873 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4718. Jul-28 01:07:23.872 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4719. job: 4713446: RUNNING
  4720. job: 4728323: RUNNING
  4721. job: 4728326: RUNNING
  4722. job: 4728325: RUNNING
  4723. job: 4728321: RUNNING
  4724. job: 4728286: RUNNING
  4725.  
  4726. Jul-28 01:07:23.872 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4727. Jul-28 01:07:23.873 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4728. job: 4713446: RUNNING
  4729. job: 4728323: RUNNING
  4730. job: 4728326: RUNNING
  4731. job: 4728325: RUNNING
  4732. job: 4728321: RUNNING
  4733. job: 4728286: RUNNING
  4734.  
  4735. Jul-28 01:07:23.873 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4736. Jul-28 01:07:28.871 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4737. job: 4713446: RUNNING
  4738. job: 4728323: RUNNING
  4739. job: 4728326: RUNNING
  4740. job: 4728325: RUNNING
  4741. job: 4728321: RUNNING
  4742. job: 4728286: RUNNING
  4743.  
  4744. Jul-28 01:07:28.872 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4745. Jul-28 01:07:28.872 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4746. job: 4713446: RUNNING
  4747. job: 4728323: RUNNING
  4748. job: 4728326: RUNNING
  4749. job: 4728325: RUNNING
  4750. job: 4728321: RUNNING
  4751. job: 4728286: RUNNING
  4752.  
  4753. Jul-28 01:07:28.872 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4754. Jul-28 01:07:33.873 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4755. job: 4713446: RUNNING
  4756. job: 4728323: RUNNING
  4757. job: 4728326: RUNNING
  4758. job: 4728325: RUNNING
  4759. job: 4728321: RUNNING
  4760. job: 4728286: RUNNING
  4761.  
  4762. Jul-28 01:07:33.873 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4763. Jul-28 01:07:33.874 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4764. job: 4713446: RUNNING
  4765. job: 4728323: RUNNING
  4766. job: 4728326: RUNNING
  4767. job: 4728325: RUNNING
  4768. job: 4728321: RUNNING
  4769. job: 4728286: RUNNING
  4770.  
  4771. Jul-28 01:07:33.874 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4772. Jul-28 01:07:38.876 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4773. job: 4713446: RUNNING
  4774. job: 4728323: RUNNING
  4775. job: 4728326: RUNNING
  4776. job: 4728325: RUNNING
  4777. job: 4728321: RUNNING
  4778. job: 4728286: RUNNING
  4779.  
  4780. Jul-28 01:07:38.877 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4781. Jul-28 01:07:38.879 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4782. job: 4713446: RUNNING
  4783. job: 4728323: RUNNING
  4784. job: 4728326: RUNNING
  4785. job: 4728325: RUNNING
  4786. job: 4728321: RUNNING
  4787. job: 4728286: RUNNING
  4788.  
  4789. Jul-28 01:07:38.879 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4790. Jul-28 01:07:43.872 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4791. job: 4713446: RUNNING
  4792. job: 4728323: RUNNING
  4793. job: 4728326: RUNNING
  4794. job: 4728325: RUNNING
  4795. job: 4728321: RUNNING
  4796. job: 4728286: RUNNING
  4797.  
  4798. Jul-28 01:07:43.872 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4799. Jul-28 01:07:43.873 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4800. job: 4713446: RUNNING
  4801. job: 4728323: RUNNING
  4802. job: 4728326: RUNNING
  4803. job: 4728325: RUNNING
  4804. job: 4728321: RUNNING
  4805. job: 4728286: RUNNING
  4806.  
  4807. Jul-28 01:07:43.873 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4808. Jul-28 01:07:48.873 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4809. job: 4713446: RUNNING
  4810. job: 4728323: RUNNING
  4811. job: 4728326: RUNNING
  4812. job: 4728325: RUNNING
  4813. job: 4728321: RUNNING
  4814. job: 4728286: RUNNING
  4815.  
  4816. Jul-28 01:07:48.874 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4817. Jul-28 01:07:48.875 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4818. job: 4713446: RUNNING
  4819. job: 4728323: RUNNING
  4820. job: 4728326: RUNNING
  4821. job: 4728325: RUNNING
  4822. job: 4728321: RUNNING
  4823. job: 4728286: RUNNING
  4824.  
  4825. Jul-28 01:07:48.875 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4826. Jul-28 01:07:53.873 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4827. job: 4713446: RUNNING
  4828. job: 4728323: RUNNING
  4829. job: 4728326: RUNNING
  4830. job: 4728325: RUNNING
  4831. job: 4728321: RUNNING
  4832. job: 4728286: RUNNING
  4833.  
  4834. Jul-28 01:07:53.873 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4835. Jul-28 01:07:53.874 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4836. job: 4713446: RUNNING
  4837. job: 4728323: RUNNING
  4838. job: 4728326: RUNNING
  4839. job: 4728325: RUNNING
  4840. job: 4728321: RUNNING
  4841. job: 4728286: RUNNING
  4842.  
  4843. Jul-28 01:07:53.874 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4844. Jul-28 01:07:58.873 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4845. job: 4713446: RUNNING
  4846. job: 4728323: RUNNING
  4847. job: 4728326: RUNNING
  4848. job: 4728325: RUNNING
  4849. job: 4728321: RUNNING
  4850. job: 4728286: RUNNING
  4851.  
  4852. Jul-28 01:07:58.874 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4853. Jul-28 01:07:58.875 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4854. job: 4713446: RUNNING
  4855. job: 4728323: RUNNING
  4856. job: 4728326: RUNNING
  4857. job: 4728325: RUNNING
  4858. job: 4728321: RUNNING
  4859. job: 4728286: RUNNING
  4860.  
  4861. Jul-28 01:07:58.875 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4862. Jul-28 01:08:03.876 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  4863. Jul-28 01:08:04.018 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  4864. 4713446,RUN,Jul 27 11:44
  4865. 4728323,RUN,Jul 28 00:40
  4866. 4728326,RUN,Jul 28 00:51
  4867. 4728325,RUN,Jul 28 00:49
  4868. 4728321,RUN,Jul 28 00:34
  4869. 4728286,RUN,Jul 27 23:44
  4870.  
  4871.  
  4872. Jul-28 01:08:04.018 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4873. job: 4713446: RUNNING
  4874. job: 4728323: RUNNING
  4875. job: 4728326: RUNNING
  4876. job: 4728325: RUNNING
  4877. job: 4728321: RUNNING
  4878. job: 4728286: RUNNING
  4879.  
  4880. Jul-28 01:08:04.018 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4881. Jul-28 01:08:04.020 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4882. job: 4713446: RUNNING
  4883. job: 4728323: RUNNING
  4884. job: 4728326: RUNNING
  4885. job: 4728325: RUNNING
  4886. job: 4728321: RUNNING
  4887. job: 4728286: RUNNING
  4888.  
  4889. Jul-28 01:08:04.020 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4890. Jul-28 01:08:08.873 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4891. job: 4713446: RUNNING
  4892. job: 4728323: RUNNING
  4893. job: 4728326: RUNNING
  4894. job: 4728325: RUNNING
  4895. job: 4728321: RUNNING
  4896. job: 4728286: RUNNING
  4897.  
  4898. Jul-28 01:08:08.874 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4899. Jul-28 01:08:08.874 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4900. job: 4713446: RUNNING
  4901. job: 4728323: RUNNING
  4902. job: 4728326: RUNNING
  4903. job: 4728325: RUNNING
  4904. job: 4728321: RUNNING
  4905. job: 4728286: RUNNING
  4906.  
  4907. Jul-28 01:08:08.874 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4908. Jul-28 01:08:13.873 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4909. job: 4713446: RUNNING
  4910. job: 4728323: RUNNING
  4911. job: 4728326: RUNNING
  4912. job: 4728325: RUNNING
  4913. job: 4728321: RUNNING
  4914. job: 4728286: RUNNING
  4915.  
  4916. Jul-28 01:08:13.873 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4917. Jul-28 01:08:13.874 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4918. job: 4713446: RUNNING
  4919. job: 4728323: RUNNING
  4920. job: 4728326: RUNNING
  4921. job: 4728325: RUNNING
  4922. job: 4728321: RUNNING
  4923. job: 4728286: RUNNING
  4924.  
  4925. Jul-28 01:08:13.874 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4926. Jul-28 01:08:18.872 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4927. job: 4713446: RUNNING
  4928. job: 4728323: RUNNING
  4929. job: 4728326: RUNNING
  4930. job: 4728325: RUNNING
  4931. job: 4728321: RUNNING
  4932. job: 4728286: RUNNING
  4933.  
  4934. Jul-28 01:08:18.873 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4935. Jul-28 01:08:18.873 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4936. job: 4713446: RUNNING
  4937. job: 4728323: RUNNING
  4938. job: 4728326: RUNNING
  4939. job: 4728325: RUNNING
  4940. job: 4728321: RUNNING
  4941. job: 4728286: RUNNING
  4942.  
  4943. Jul-28 01:08:18.873 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4944. Jul-28 01:08:23.873 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4945. job: 4713446: RUNNING
  4946. job: 4728323: RUNNING
  4947. job: 4728326: RUNNING
  4948. job: 4728325: RUNNING
  4949. job: 4728321: RUNNING
  4950. job: 4728286: RUNNING
  4951.  
  4952. Jul-28 01:08:23.873 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4953. Jul-28 01:08:23.874 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4954. job: 4713446: RUNNING
  4955. job: 4728323: RUNNING
  4956. job: 4728326: RUNNING
  4957. job: 4728325: RUNNING
  4958. job: 4728321: RUNNING
  4959. job: 4728286: RUNNING
  4960.  
  4961. Jul-28 01:08:23.874 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4962. Jul-28 01:08:28.876 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4963. job: 4713446: RUNNING
  4964. job: 4728323: RUNNING
  4965. job: 4728326: RUNNING
  4966. job: 4728325: RUNNING
  4967. job: 4728321: RUNNING
  4968. job: 4728286: RUNNING
  4969.  
  4970. Jul-28 01:08:28.876 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4971. Jul-28 01:08:28.878 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4972. job: 4713446: RUNNING
  4973. job: 4728323: RUNNING
  4974. job: 4728326: RUNNING
  4975. job: 4728325: RUNNING
  4976. job: 4728321: RUNNING
  4977. job: 4728286: RUNNING
  4978.  
  4979. Jul-28 01:08:28.878 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4980. Jul-28 01:08:33.878 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4981. job: 4713446: RUNNING
  4982. job: 4728323: RUNNING
  4983. job: 4728326: RUNNING
  4984. job: 4728325: RUNNING
  4985. job: 4728321: RUNNING
  4986. job: 4728286: RUNNING
  4987.  
  4988. Jul-28 01:08:33.878 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  4989. Jul-28 01:08:33.880 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4990. job: 4713446: RUNNING
  4991. job: 4728323: RUNNING
  4992. job: 4728326: RUNNING
  4993. job: 4728325: RUNNING
  4994. job: 4728321: RUNNING
  4995. job: 4728286: RUNNING
  4996.  
  4997. Jul-28 01:08:33.880 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  4998. Jul-28 01:08:38.874 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  4999. job: 4713446: RUNNING
  5000. job: 4728323: RUNNING
  5001. job: 4728326: RUNNING
  5002. job: 4728325: RUNNING
  5003. job: 4728321: RUNNING
  5004. job: 4728286: RUNNING
  5005.  
  5006. Jul-28 01:08:38.875 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5007. Jul-28 01:08:38.875 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5008. job: 4713446: RUNNING
  5009. job: 4728323: RUNNING
  5010. job: 4728326: RUNNING
  5011. job: 4728325: RUNNING
  5012. job: 4728321: RUNNING
  5013. job: 4728286: RUNNING
  5014.  
  5015. Jul-28 01:08:38.875 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5016. Jul-28 01:08:43.876 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5017. job: 4713446: RUNNING
  5018. job: 4728323: RUNNING
  5019. job: 4728326: RUNNING
  5020. job: 4728325: RUNNING
  5021. job: 4728321: RUNNING
  5022. job: 4728286: RUNNING
  5023.  
  5024. Jul-28 01:08:43.876 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5025. Jul-28 01:08:43.877 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5026. job: 4713446: RUNNING
  5027. job: 4728323: RUNNING
  5028. job: 4728326: RUNNING
  5029. job: 4728325: RUNNING
  5030. job: 4728321: RUNNING
  5031. job: 4728286: RUNNING
  5032.  
  5033. Jul-28 01:08:43.877 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5034. Jul-28 01:08:48.874 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5035. job: 4713446: RUNNING
  5036. job: 4728323: RUNNING
  5037. job: 4728326: RUNNING
  5038. job: 4728325: RUNNING
  5039. job: 4728321: RUNNING
  5040. job: 4728286: RUNNING
  5041.  
  5042. Jul-28 01:08:48.875 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5043. Jul-28 01:08:48.875 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5044. job: 4713446: RUNNING
  5045. job: 4728323: RUNNING
  5046. job: 4728326: RUNNING
  5047. job: 4728325: RUNNING
  5048. job: 4728321: RUNNING
  5049. job: 4728286: RUNNING
  5050.  
  5051. Jul-28 01:08:48.875 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5052. Jul-28 01:08:53.876 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5053. job: 4713446: RUNNING
  5054. job: 4728323: RUNNING
  5055. job: 4728326: RUNNING
  5056. job: 4728325: RUNNING
  5057. job: 4728321: RUNNING
  5058. job: 4728286: RUNNING
  5059.  
  5060. Jul-28 01:08:53.876 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5061. Jul-28 01:08:53.877 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5062. job: 4713446: RUNNING
  5063. job: 4728323: RUNNING
  5064. job: 4728326: RUNNING
  5065. job: 4728325: RUNNING
  5066. job: 4728321: RUNNING
  5067. job: 4728286: RUNNING
  5068.  
  5069. Jul-28 01:08:53.877 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5070. Jul-28 01:08:58.875 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5071. job: 4713446: RUNNING
  5072. job: 4728323: RUNNING
  5073. job: 4728326: RUNNING
  5074. job: 4728325: RUNNING
  5075. job: 4728321: RUNNING
  5076. job: 4728286: RUNNING
  5077.  
  5078. Jul-28 01:08:58.875 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5079. Jul-28 01:08:58.876 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5080. job: 4713446: RUNNING
  5081. job: 4728323: RUNNING
  5082. job: 4728326: RUNNING
  5083. job: 4728325: RUNNING
  5084. job: 4728321: RUNNING
  5085. job: 4728286: RUNNING
  5086.  
  5087. Jul-28 01:08:58.876 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5088. Jul-28 01:09:03.875 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5089. job: 4713446: RUNNING
  5090. job: 4728323: RUNNING
  5091. job: 4728326: RUNNING
  5092. job: 4728325: RUNNING
  5093. job: 4728321: RUNNING
  5094. job: 4728286: RUNNING
  5095.  
  5096. Jul-28 01:09:03.875 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5097. Jul-28 01:09:03.876 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5098. job: 4713446: RUNNING
  5099. job: 4728323: RUNNING
  5100. job: 4728326: RUNNING
  5101. job: 4728325: RUNNING
  5102. job: 4728321: RUNNING
  5103. job: 4728286: RUNNING
  5104.  
  5105. Jul-28 01:09:03.876 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5106. Jul-28 01:09:08.878 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  5107. Jul-28 01:09:09.020 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  5108. 4713446,RUN,Jul 27 11:44
  5109. 4728323,RUN,Jul 28 00:40
  5110. 4728326,RUN,Jul 28 00:51
  5111. 4728325,RUN,Jul 28 00:49
  5112. 4728321,RUN,Jul 28 00:34
  5113. 4728286,RUN,Jul 27 23:44
  5114.  
  5115.  
  5116. Jul-28 01:09:09.020 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5117. job: 4713446: RUNNING
  5118. job: 4728323: RUNNING
  5119. job: 4728326: RUNNING
  5120. job: 4728325: RUNNING
  5121. job: 4728321: RUNNING
  5122. job: 4728286: RUNNING
  5123.  
  5124. Jul-28 01:09:09.020 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5125. Jul-28 01:09:09.021 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5126. job: 4713446: RUNNING
  5127. job: 4728323: RUNNING
  5128. job: 4728326: RUNNING
  5129. job: 4728325: RUNNING
  5130. job: 4728321: RUNNING
  5131. job: 4728286: RUNNING
  5132.  
  5133. Jul-28 01:09:09.021 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5134. Jul-28 01:09:13.875 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5135. job: 4713446: RUNNING
  5136. job: 4728323: RUNNING
  5137. job: 4728326: RUNNING
  5138. job: 4728325: RUNNING
  5139. job: 4728321: RUNNING
  5140. job: 4728286: RUNNING
  5141.  
  5142. Jul-28 01:09:13.875 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5143. Jul-28 01:09:13.876 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5144. job: 4713446: RUNNING
  5145. job: 4728323: RUNNING
  5146. job: 4728326: RUNNING
  5147. job: 4728325: RUNNING
  5148. job: 4728321: RUNNING
  5149. job: 4728286: RUNNING
  5150.  
  5151. Jul-28 01:09:13.876 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5152. Jul-28 01:09:18.875 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5153. job: 4713446: RUNNING
  5154. job: 4728323: RUNNING
  5155. job: 4728326: RUNNING
  5156. job: 4728325: RUNNING
  5157. job: 4728321: RUNNING
  5158. job: 4728286: RUNNING
  5159.  
  5160. Jul-28 01:09:18.875 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5161. Jul-28 01:09:18.876 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5162. job: 4713446: RUNNING
  5163. job: 4728323: RUNNING
  5164. job: 4728326: RUNNING
  5165. job: 4728325: RUNNING
  5166. job: 4728321: RUNNING
  5167. job: 4728286: RUNNING
  5168.  
  5169. Jul-28 01:09:18.876 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5170. Jul-28 01:09:23.875 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5171. job: 4713446: RUNNING
  5172. job: 4728323: RUNNING
  5173. job: 4728326: RUNNING
  5174. job: 4728325: RUNNING
  5175. job: 4728321: RUNNING
  5176. job: 4728286: RUNNING
  5177.  
  5178. Jul-28 01:09:23.876 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5179. Jul-28 01:09:23.876 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5180. job: 4713446: RUNNING
  5181. job: 4728323: RUNNING
  5182. job: 4728326: RUNNING
  5183. job: 4728325: RUNNING
  5184. job: 4728321: RUNNING
  5185. job: 4728286: RUNNING
  5186.  
  5187. Jul-28 01:09:23.876 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5188. Jul-28 01:09:28.878 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5189. job: 4713446: RUNNING
  5190. job: 4728323: RUNNING
  5191. job: 4728326: RUNNING
  5192. job: 4728325: RUNNING
  5193. job: 4728321: RUNNING
  5194. job: 4728286: RUNNING
  5195.  
  5196. Jul-28 01:09:28.879 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5197. Jul-28 01:09:28.880 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5198. job: 4713446: RUNNING
  5199. job: 4728323: RUNNING
  5200. job: 4728326: RUNNING
  5201. job: 4728325: RUNNING
  5202. job: 4728321: RUNNING
  5203. job: 4728286: RUNNING
  5204.  
  5205. Jul-28 01:09:28.880 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5206. Jul-28 01:09:33.880 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5207. job: 4713446: RUNNING
  5208. job: 4728323: RUNNING
  5209. job: 4728326: RUNNING
  5210. job: 4728325: RUNNING
  5211. job: 4728321: RUNNING
  5212. job: 4728286: RUNNING
  5213.  
  5214. Jul-28 01:09:33.881 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5215. Jul-28 01:09:33.882 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5216. job: 4713446: RUNNING
  5217. job: 4728323: RUNNING
  5218. job: 4728326: RUNNING
  5219. job: 4728325: RUNNING
  5220. job: 4728321: RUNNING
  5221. job: 4728286: RUNNING
  5222.  
  5223. Jul-28 01:09:33.882 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5224. Jul-28 01:09:38.880 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5225. job: 4713446: RUNNING
  5226. job: 4728323: RUNNING
  5227. job: 4728326: RUNNING
  5228. job: 4728325: RUNNING
  5229. job: 4728321: RUNNING
  5230. job: 4728286: RUNNING
  5231.  
  5232. Jul-28 01:09:38.881 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5233. Jul-28 01:09:38.882 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5234. job: 4713446: RUNNING
  5235. job: 4728323: RUNNING
  5236. job: 4728326: RUNNING
  5237. job: 4728325: RUNNING
  5238. job: 4728321: RUNNING
  5239. job: 4728286: RUNNING
  5240.  
  5241. Jul-28 01:09:38.882 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5242. Jul-28 01:09:43.880 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5243. job: 4713446: RUNNING
  5244. job: 4728323: RUNNING
  5245. job: 4728326: RUNNING
  5246. job: 4728325: RUNNING
  5247. job: 4728321: RUNNING
  5248. job: 4728286: RUNNING
  5249.  
  5250. Jul-28 01:09:43.881 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5251. Jul-28 01:09:43.882 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5252. job: 4713446: RUNNING
  5253. job: 4728323: RUNNING
  5254. job: 4728326: RUNNING
  5255. job: 4728325: RUNNING
  5256. job: 4728321: RUNNING
  5257. job: 4728286: RUNNING
  5258.  
  5259. Jul-28 01:09:43.882 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5260. Jul-28 01:09:48.882 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5261. job: 4713446: RUNNING
  5262. job: 4728323: RUNNING
  5263. job: 4728326: RUNNING
  5264. job: 4728325: RUNNING
  5265. job: 4728321: RUNNING
  5266. job: 4728286: RUNNING
  5267.  
  5268. Jul-28 01:09:48.882 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5269. Jul-28 01:09:48.884 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5270. job: 4713446: RUNNING
  5271. job: 4728323: RUNNING
  5272. job: 4728326: RUNNING
  5273. job: 4728325: RUNNING
  5274. job: 4728321: RUNNING
  5275. job: 4728286: RUNNING
  5276.  
  5277. Jul-28 01:09:48.884 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5278. Jul-28 01:09:53.879 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5279. job: 4713446: RUNNING
  5280. job: 4728323: RUNNING
  5281. job: 4728326: RUNNING
  5282. job: 4728325: RUNNING
  5283. job: 4728321: RUNNING
  5284. job: 4728286: RUNNING
  5285.  
  5286. Jul-28 01:09:53.879 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5287. Jul-28 01:09:53.880 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5288. job: 4713446: RUNNING
  5289. job: 4728323: RUNNING
  5290. job: 4728326: RUNNING
  5291. job: 4728325: RUNNING
  5292. job: 4728321: RUNNING
  5293. job: 4728286: RUNNING
  5294.  
  5295. Jul-28 01:09:53.880 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5296. Jul-28 01:09:58.879 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5297. job: 4713446: RUNNING
  5298. job: 4728323: RUNNING
  5299. job: 4728326: RUNNING
  5300. job: 4728325: RUNNING
  5301. job: 4728321: RUNNING
  5302. job: 4728286: RUNNING
  5303.  
  5304. Jul-28 01:09:58.880 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5305. Jul-28 01:09:58.881 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5306. job: 4713446: RUNNING
  5307. job: 4728323: RUNNING
  5308. job: 4728326: RUNNING
  5309. job: 4728325: RUNNING
  5310. job: 4728321: RUNNING
  5311. job: 4728286: RUNNING
  5312.  
  5313. Jul-28 01:09:58.881 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5314. Jul-28 01:10:03.879 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5315. job: 4713446: RUNNING
  5316. job: 4728323: RUNNING
  5317. job: 4728326: RUNNING
  5318. job: 4728325: RUNNING
  5319. job: 4728321: RUNNING
  5320. job: 4728286: RUNNING
  5321.  
  5322. Jul-28 01:10:03.879 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5323. Jul-28 01:10:03.880 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5324. job: 4713446: RUNNING
  5325. job: 4728323: RUNNING
  5326. job: 4728326: RUNNING
  5327. job: 4728325: RUNNING
  5328. job: 4728321: RUNNING
  5329. job: 4728286: RUNNING
  5330.  
  5331. Jul-28 01:10:03.881 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5332. Jul-28 01:10:08.882 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor lsf > tasks to be completed: 2 -- pending tasks are shown below
  5333. ~> TaskHandler[jobId: 4728325; id: 152; name: merge_pairsam_into_runs (library:HeLa1 run:lane1); status: RUNNING; exit: -; error: -; workDir: /farline/umw_job_dekker/HPCC/sv49w/distiller-nf/work/3b/37ed2fa2ba160f54196e762e24c83b started: 1501217508853; exited: -; ]
  5334. ~> TaskHandler[jobId: 4728326; id: 153; name: merge_pairsam_into_runs (library:HeLa1 run:lane3); status: RUNNING; exit: -; error: -; workDir: /farline/umw_job_dekker/HPCC/sv49w/distiller-nf/work/00/2633edc9da9ad5649d3d6dcffe5a44 started: 1501217938848; exited: -; ]
  5335. Jul-28 01:10:08.882 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  5336. Jul-28 01:10:09.019 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  5337. 4713446,RUN,Jul 27 11:44
  5338. 4728323,RUN,Jul 28 00:40
  5339. 4728326,RUN,Jul 28 00:51
  5340. 4728325,RUN,Jul 28 00:49
  5341. 4728321,RUN,Jul 28 00:34
  5342. 4728286,RUN,Jul 27 23:44
  5343.  
  5344.  
  5345. Jul-28 01:10:09.019 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5346. job: 4713446: RUNNING
  5347. job: 4728323: RUNNING
  5348. job: 4728326: RUNNING
  5349. job: 4728325: RUNNING
  5350. job: 4728321: RUNNING
  5351. job: 4728286: RUNNING
  5352.  
  5353. Jul-28 01:10:09.019 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5354. Jul-28 01:10:09.020 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5355. job: 4713446: RUNNING
  5356. job: 4728323: RUNNING
  5357. job: 4728326: RUNNING
  5358. job: 4728325: RUNNING
  5359. job: 4728321: RUNNING
  5360. job: 4728286: RUNNING
  5361.  
  5362. Jul-28 01:10:09.020 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5363. Jul-28 01:10:13.885 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5364. job: 4713446: RUNNING
  5365. job: 4728323: RUNNING
  5366. job: 4728326: RUNNING
  5367. job: 4728325: RUNNING
  5368. job: 4728321: RUNNING
  5369. job: 4728286: RUNNING
  5370.  
  5371. Jul-28 01:10:13.885 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5372. Jul-28 01:10:13.887 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5373. job: 4713446: RUNNING
  5374. job: 4728323: RUNNING
  5375. job: 4728326: RUNNING
  5376. job: 4728325: RUNNING
  5377. job: 4728321: RUNNING
  5378. job: 4728286: RUNNING
  5379.  
  5380. Jul-28 01:10:13.887 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5381. Jul-28 01:10:18.885 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5382. job: 4713446: RUNNING
  5383. job: 4728323: RUNNING
  5384. job: 4728326: RUNNING
  5385. job: 4728325: RUNNING
  5386. job: 4728321: RUNNING
  5387. job: 4728286: RUNNING
  5388.  
  5389. Jul-28 01:10:18.886 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5390. Jul-28 01:10:18.886 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5391. job: 4713446: RUNNING
  5392. job: 4728323: RUNNING
  5393. job: 4728326: RUNNING
  5394. job: 4728325: RUNNING
  5395. job: 4728321: RUNNING
  5396. job: 4728286: RUNNING
  5397.  
  5398. Jul-28 01:10:18.886 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5399. Jul-28 01:10:23.885 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5400. job: 4713446: RUNNING
  5401. job: 4728323: RUNNING
  5402. job: 4728326: RUNNING
  5403. job: 4728325: RUNNING
  5404. job: 4728321: RUNNING
  5405. job: 4728286: RUNNING
  5406.  
  5407. Jul-28 01:10:23.886 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5408. Jul-28 01:10:23.886 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5409. job: 4713446: RUNNING
  5410. job: 4728323: RUNNING
  5411. job: 4728326: RUNNING
  5412. job: 4728325: RUNNING
  5413. job: 4728321: RUNNING
  5414. job: 4728286: RUNNING
  5415.  
  5416. Jul-28 01:10:23.886 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5417. Jul-28 01:10:28.886 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5418. job: 4713446: RUNNING
  5419. job: 4728323: RUNNING
  5420. job: 4728326: RUNNING
  5421. job: 4728325: RUNNING
  5422. job: 4728321: RUNNING
  5423. job: 4728286: RUNNING
  5424.  
  5425. Jul-28 01:10:28.886 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5426. Jul-28 01:10:28.887 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5427. job: 4713446: RUNNING
  5428. job: 4728323: RUNNING
  5429. job: 4728326: RUNNING
  5430. job: 4728325: RUNNING
  5431. job: 4728321: RUNNING
  5432. job: 4728286: RUNNING
  5433.  
  5434. Jul-28 01:10:28.887 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5435. Jul-28 01:10:33.888 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5436. job: 4713446: RUNNING
  5437. job: 4728323: RUNNING
  5438. job: 4728326: RUNNING
  5439. job: 4728325: RUNNING
  5440. job: 4728321: RUNNING
  5441. job: 4728286: RUNNING
  5442.  
  5443. Jul-28 01:10:33.889 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5444. Jul-28 01:10:33.891 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5445. job: 4713446: RUNNING
  5446. job: 4728323: RUNNING
  5447. job: 4728326: RUNNING
  5448. job: 4728325: RUNNING
  5449. job: 4728321: RUNNING
  5450. job: 4728286: RUNNING
  5451.  
  5452. Jul-28 01:10:33.891 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5453. Jul-28 01:10:38.886 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5454. job: 4713446: RUNNING
  5455. job: 4728323: RUNNING
  5456. job: 4728326: RUNNING
  5457. job: 4728325: RUNNING
  5458. job: 4728321: RUNNING
  5459. job: 4728286: RUNNING
  5460.  
  5461. Jul-28 01:10:38.887 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5462. Jul-28 01:10:38.887 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5463. job: 4713446: RUNNING
  5464. job: 4728323: RUNNING
  5465. job: 4728326: RUNNING
  5466. job: 4728325: RUNNING
  5467. job: 4728321: RUNNING
  5468. job: 4728286: RUNNING
  5469.  
  5470. Jul-28 01:10:38.887 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5471. Jul-28 01:10:43.887 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5472. job: 4713446: RUNNING
  5473. job: 4728323: RUNNING
  5474. job: 4728326: RUNNING
  5475. job: 4728325: RUNNING
  5476. job: 4728321: RUNNING
  5477. job: 4728286: RUNNING
  5478.  
  5479. Jul-28 01:10:43.888 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5480. Jul-28 01:10:43.888 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5481. job: 4713446: RUNNING
  5482. job: 4728323: RUNNING
  5483. job: 4728326: RUNNING
  5484. job: 4728325: RUNNING
  5485. job: 4728321: RUNNING
  5486. job: 4728286: RUNNING
  5487.  
  5488. Jul-28 01:10:43.889 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5489. Jul-28 01:10:48.889 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5490. job: 4713446: RUNNING
  5491. job: 4728323: RUNNING
  5492. job: 4728326: RUNNING
  5493. job: 4728325: RUNNING
  5494. job: 4728321: RUNNING
  5495. job: 4728286: RUNNING
  5496.  
  5497. Jul-28 01:10:48.889 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5498. Jul-28 01:10:48.891 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5499. job: 4713446: RUNNING
  5500. job: 4728323: RUNNING
  5501. job: 4728326: RUNNING
  5502. job: 4728325: RUNNING
  5503. job: 4728321: RUNNING
  5504. job: 4728286: RUNNING
  5505.  
  5506. Jul-28 01:10:48.891 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5507. Jul-28 01:10:53.885 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5508. job: 4713446: RUNNING
  5509. job: 4728323: RUNNING
  5510. job: 4728326: RUNNING
  5511. job: 4728325: RUNNING
  5512. job: 4728321: RUNNING
  5513. job: 4728286: RUNNING
  5514.  
  5515. Jul-28 01:10:53.885 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5516. Jul-28 01:10:53.886 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5517. job: 4713446: RUNNING
  5518. job: 4728323: RUNNING
  5519. job: 4728326: RUNNING
  5520. job: 4728325: RUNNING
  5521. job: 4728321: RUNNING
  5522. job: 4728286: RUNNING
  5523.  
  5524. Jul-28 01:10:53.886 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5525. Jul-28 01:10:58.886 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5526. job: 4713446: RUNNING
  5527. job: 4728323: RUNNING
  5528. job: 4728326: RUNNING
  5529. job: 4728325: RUNNING
  5530. job: 4728321: RUNNING
  5531. job: 4728286: RUNNING
  5532.  
  5533. Jul-28 01:10:58.886 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5534. Jul-28 01:10:58.887 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5535. job: 4713446: RUNNING
  5536. job: 4728323: RUNNING
  5537. job: 4728326: RUNNING
  5538. job: 4728325: RUNNING
  5539. job: 4728321: RUNNING
  5540. job: 4728286: RUNNING
  5541.  
  5542. Jul-28 01:10:58.887 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5543. Jul-28 01:11:03.885 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5544. job: 4713446: RUNNING
  5545. job: 4728323: RUNNING
  5546. job: 4728326: RUNNING
  5547. job: 4728325: RUNNING
  5548. job: 4728321: RUNNING
  5549. job: 4728286: RUNNING
  5550.  
  5551. Jul-28 01:11:03.886 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5552. Jul-28 01:11:03.886 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5553. job: 4713446: RUNNING
  5554. job: 4728323: RUNNING
  5555. job: 4728326: RUNNING
  5556. job: 4728325: RUNNING
  5557. job: 4728321: RUNNING
  5558. job: 4728286: RUNNING
  5559.  
  5560. Jul-28 01:11:03.886 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5561. Jul-28 01:11:08.886 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  5562. Jul-28 01:11:09.017 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  5563. 4713446,RUN,Jul 27 11:44
  5564. 4728323,RUN,Jul 28 00:40
  5565. 4728326,RUN,Jul 28 00:51
  5566. 4728325,RUN,Jul 28 00:49
  5567. 4728321,RUN,Jul 28 00:34
  5568. 4728286,RUN,Jul 27 23:44
  5569.  
  5570.  
  5571. Jul-28 01:11:09.017 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5572. job: 4713446: RUNNING
  5573. job: 4728323: RUNNING
  5574. job: 4728326: RUNNING
  5575. job: 4728325: RUNNING
  5576. job: 4728321: RUNNING
  5577. job: 4728286: RUNNING
  5578.  
  5579. Jul-28 01:11:09.017 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5580. Jul-28 01:11:09.018 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5581. job: 4713446: RUNNING
  5582. job: 4728323: RUNNING
  5583. job: 4728326: RUNNING
  5584. job: 4728325: RUNNING
  5585. job: 4728321: RUNNING
  5586. job: 4728286: RUNNING
  5587.  
  5588. Jul-28 01:11:09.018 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5589. Jul-28 01:11:13.890 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5590. job: 4713446: RUNNING
  5591. job: 4728323: RUNNING
  5592. job: 4728326: RUNNING
  5593. job: 4728325: RUNNING
  5594. job: 4728321: RUNNING
  5595. job: 4728286: RUNNING
  5596.  
  5597. Jul-28 01:11:13.891 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5598. Jul-28 01:11:13.892 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5599. job: 4713446: RUNNING
  5600. job: 4728323: RUNNING
  5601. job: 4728326: RUNNING
  5602. job: 4728325: RUNNING
  5603. job: 4728321: RUNNING
  5604. job: 4728286: RUNNING
  5605.  
  5606. Jul-28 01:11:13.892 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5607. Jul-28 01:11:18.891 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5608. job: 4713446: RUNNING
  5609. job: 4728323: RUNNING
  5610. job: 4728326: RUNNING
  5611. job: 4728325: RUNNING
  5612. job: 4728321: RUNNING
  5613. job: 4728286: RUNNING
  5614.  
  5615. Jul-28 01:11:18.892 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5616. Jul-28 01:11:18.894 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5617. job: 4713446: RUNNING
  5618. job: 4728323: RUNNING
  5619. job: 4728326: RUNNING
  5620. job: 4728325: RUNNING
  5621. job: 4728321: RUNNING
  5622. job: 4728286: RUNNING
  5623.  
  5624. Jul-28 01:11:18.894 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5625. Jul-28 01:11:23.886 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5626. job: 4713446: RUNNING
  5627. job: 4728323: RUNNING
  5628. job: 4728326: RUNNING
  5629. job: 4728325: RUNNING
  5630. job: 4728321: RUNNING
  5631. job: 4728286: RUNNING
  5632.  
  5633. Jul-28 01:11:23.887 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5634. Jul-28 01:11:23.887 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5635. job: 4713446: RUNNING
  5636. job: 4728323: RUNNING
  5637. job: 4728326: RUNNING
  5638. job: 4728325: RUNNING
  5639. job: 4728321: RUNNING
  5640. job: 4728286: RUNNING
  5641.  
  5642. Jul-28 01:11:23.887 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5643. Jul-28 01:11:28.887 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5644. job: 4713446: RUNNING
  5645. job: 4728323: RUNNING
  5646. job: 4728326: RUNNING
  5647. job: 4728325: RUNNING
  5648. job: 4728321: RUNNING
  5649. job: 4728286: RUNNING
  5650.  
  5651. Jul-28 01:11:28.888 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5652. Jul-28 01:11:28.889 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5653. job: 4713446: RUNNING
  5654. job: 4728323: RUNNING
  5655. job: 4728326: RUNNING
  5656. job: 4728325: RUNNING
  5657. job: 4728321: RUNNING
  5658. job: 4728286: RUNNING
  5659.  
  5660. Jul-28 01:11:28.889 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5661. Jul-28 01:11:33.887 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5662. job: 4713446: RUNNING
  5663. job: 4728323: RUNNING
  5664. job: 4728326: RUNNING
  5665. job: 4728325: RUNNING
  5666. job: 4728321: RUNNING
  5667. job: 4728286: RUNNING
  5668.  
  5669. Jul-28 01:11:33.887 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5670. Jul-28 01:11:33.888 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5671. job: 4713446: RUNNING
  5672. job: 4728323: RUNNING
  5673. job: 4728326: RUNNING
  5674. job: 4728325: RUNNING
  5675. job: 4728321: RUNNING
  5676. job: 4728286: RUNNING
  5677.  
  5678. Jul-28 01:11:33.888 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5679. Jul-28 01:11:38.887 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5680. job: 4713446: RUNNING
  5681. job: 4728323: RUNNING
  5682. job: 4728326: RUNNING
  5683. job: 4728325: RUNNING
  5684. job: 4728321: RUNNING
  5685. job: 4728286: RUNNING
  5686.  
  5687. Jul-28 01:11:38.887 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5688. Jul-28 01:11:38.888 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5689. job: 4713446: RUNNING
  5690. job: 4728323: RUNNING
  5691. job: 4728326: RUNNING
  5692. job: 4728325: RUNNING
  5693. job: 4728321: RUNNING
  5694. job: 4728286: RUNNING
  5695.  
  5696. Jul-28 01:11:38.888 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5697. Jul-28 01:11:43.887 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5698. job: 4713446: RUNNING
  5699. job: 4728323: RUNNING
  5700. job: 4728326: RUNNING
  5701. job: 4728325: RUNNING
  5702. job: 4728321: RUNNING
  5703. job: 4728286: RUNNING
  5704.  
  5705. Jul-28 01:11:43.887 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5706. Jul-28 01:11:43.888 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5707. job: 4713446: RUNNING
  5708. job: 4728323: RUNNING
  5709. job: 4728326: RUNNING
  5710. job: 4728325: RUNNING
  5711. job: 4728321: RUNNING
  5712. job: 4728286: RUNNING
  5713.  
  5714. Jul-28 01:11:43.888 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5715. Jul-28 01:11:48.890 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5716. job: 4713446: RUNNING
  5717. job: 4728323: RUNNING
  5718. job: 4728326: RUNNING
  5719. job: 4728325: RUNNING
  5720. job: 4728321: RUNNING
  5721. job: 4728286: RUNNING
  5722.  
  5723. Jul-28 01:11:48.891 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5724. Jul-28 01:11:48.892 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5725. job: 4713446: RUNNING
  5726. job: 4728323: RUNNING
  5727. job: 4728326: RUNNING
  5728. job: 4728325: RUNNING
  5729. job: 4728321: RUNNING
  5730. job: 4728286: RUNNING
  5731.  
  5732. Jul-28 01:11:48.892 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5733. Jul-28 01:11:53.890 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5734. job: 4713446: RUNNING
  5735. job: 4728323: RUNNING
  5736. job: 4728326: RUNNING
  5737. job: 4728325: RUNNING
  5738. job: 4728321: RUNNING
  5739. job: 4728286: RUNNING
  5740.  
  5741. Jul-28 01:11:53.890 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5742. Jul-28 01:11:53.892 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5743. job: 4713446: RUNNING
  5744. job: 4728323: RUNNING
  5745. job: 4728326: RUNNING
  5746. job: 4728325: RUNNING
  5747. job: 4728321: RUNNING
  5748. job: 4728286: RUNNING
  5749.  
  5750. Jul-28 01:11:53.892 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5751. Jul-28 01:11:58.891 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5752. job: 4713446: RUNNING
  5753. job: 4728323: RUNNING
  5754. job: 4728326: RUNNING
  5755. job: 4728325: RUNNING
  5756. job: 4728321: RUNNING
  5757. job: 4728286: RUNNING
  5758.  
  5759. Jul-28 01:11:58.892 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5760. Jul-28 01:11:58.893 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5761. job: 4713446: RUNNING
  5762. job: 4728323: RUNNING
  5763. job: 4728326: RUNNING
  5764. job: 4728325: RUNNING
  5765. job: 4728321: RUNNING
  5766. job: 4728286: RUNNING
  5767.  
  5768. Jul-28 01:11:58.893 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5769. Jul-28 01:12:03.888 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5770. job: 4713446: RUNNING
  5771. job: 4728323: RUNNING
  5772. job: 4728326: RUNNING
  5773. job: 4728325: RUNNING
  5774. job: 4728321: RUNNING
  5775. job: 4728286: RUNNING
  5776.  
  5777. Jul-28 01:12:03.888 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5778. Jul-28 01:12:03.889 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5779. job: 4713446: RUNNING
  5780. job: 4728323: RUNNING
  5781. job: 4728326: RUNNING
  5782. job: 4728325: RUNNING
  5783. job: 4728321: RUNNING
  5784. job: 4728286: RUNNING
  5785.  
  5786. Jul-28 01:12:03.889 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5787. Jul-28 01:12:08.889 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  5788. Jul-28 01:12:09.020 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  5789. 4713446,RUN,Jul 27 11:44
  5790. 4728323,RUN,Jul 28 00:40
  5791. 4728326,RUN,Jul 28 00:51
  5792. 4728325,RUN,Jul 28 00:49
  5793. 4728321,RUN,Jul 28 00:34
  5794. 4728286,RUN,Jul 27 23:44
  5795.  
  5796.  
  5797. Jul-28 01:12:09.021 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5798. job: 4713446: RUNNING
  5799. job: 4728323: RUNNING
  5800. job: 4728326: RUNNING
  5801. job: 4728325: RUNNING
  5802. job: 4728321: RUNNING
  5803. job: 4728286: RUNNING
  5804.  
  5805. Jul-28 01:12:09.021 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5806. Jul-28 01:12:09.022 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5807. job: 4713446: RUNNING
  5808. job: 4728323: RUNNING
  5809. job: 4728326: RUNNING
  5810. job: 4728325: RUNNING
  5811. job: 4728321: RUNNING
  5812. job: 4728286: RUNNING
  5813.  
  5814. Jul-28 01:12:09.022 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5815. Jul-28 01:12:13.889 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5816. job: 4713446: RUNNING
  5817. job: 4728323: RUNNING
  5818. job: 4728326: RUNNING
  5819. job: 4728325: RUNNING
  5820. job: 4728321: RUNNING
  5821. job: 4728286: RUNNING
  5822.  
  5823. Jul-28 01:12:13.890 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5824. Jul-28 01:12:13.891 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5825. job: 4713446: RUNNING
  5826. job: 4728323: RUNNING
  5827. job: 4728326: RUNNING
  5828. job: 4728325: RUNNING
  5829. job: 4728321: RUNNING
  5830. job: 4728286: RUNNING
  5831.  
  5832. Jul-28 01:12:13.891 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5833. Jul-28 01:12:18.889 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5834. job: 4713446: RUNNING
  5835. job: 4728323: RUNNING
  5836. job: 4728326: RUNNING
  5837. job: 4728325: RUNNING
  5838. job: 4728321: RUNNING
  5839. job: 4728286: RUNNING
  5840.  
  5841. Jul-28 01:12:18.889 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5842. Jul-28 01:12:18.890 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5843. job: 4713446: RUNNING
  5844. job: 4728323: RUNNING
  5845. job: 4728326: RUNNING
  5846. job: 4728325: RUNNING
  5847. job: 4728321: RUNNING
  5848. job: 4728286: RUNNING
  5849.  
  5850. Jul-28 01:12:18.890 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5851. Jul-28 01:12:23.890 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5852. job: 4713446: RUNNING
  5853. job: 4728323: RUNNING
  5854. job: 4728326: RUNNING
  5855. job: 4728325: RUNNING
  5856. job: 4728321: RUNNING
  5857. job: 4728286: RUNNING
  5858.  
  5859. Jul-28 01:12:23.890 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5860. Jul-28 01:12:23.891 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5861. job: 4713446: RUNNING
  5862. job: 4728323: RUNNING
  5863. job: 4728326: RUNNING
  5864. job: 4728325: RUNNING
  5865. job: 4728321: RUNNING
  5866. job: 4728286: RUNNING
  5867.  
  5868. Jul-28 01:12:23.891 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5869. Jul-28 01:12:28.891 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5870. job: 4713446: RUNNING
  5871. job: 4728323: RUNNING
  5872. job: 4728326: RUNNING
  5873. job: 4728325: RUNNING
  5874. job: 4728321: RUNNING
  5875. job: 4728286: RUNNING
  5876.  
  5877. Jul-28 01:12:28.891 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5878. Jul-28 01:12:28.892 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5879. job: 4713446: RUNNING
  5880. job: 4728323: RUNNING
  5881. job: 4728326: RUNNING
  5882. job: 4728325: RUNNING
  5883. job: 4728321: RUNNING
  5884. job: 4728286: RUNNING
  5885.  
  5886. Jul-28 01:12:28.892 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5887. Jul-28 01:12:33.890 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5888. job: 4713446: RUNNING
  5889. job: 4728323: RUNNING
  5890. job: 4728326: RUNNING
  5891. job: 4728325: RUNNING
  5892. job: 4728321: RUNNING
  5893. job: 4728286: RUNNING
  5894.  
  5895. Jul-28 01:12:33.891 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5896. Jul-28 01:12:33.892 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5897. job: 4713446: RUNNING
  5898. job: 4728323: RUNNING
  5899. job: 4728326: RUNNING
  5900. job: 4728325: RUNNING
  5901. job: 4728321: RUNNING
  5902. job: 4728286: RUNNING
  5903.  
  5904. Jul-28 01:12:33.892 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5905. Jul-28 01:12:38.890 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5906. job: 4713446: RUNNING
  5907. job: 4728323: RUNNING
  5908. job: 4728326: RUNNING
  5909. job: 4728325: RUNNING
  5910. job: 4728321: RUNNING
  5911. job: 4728286: RUNNING
  5912.  
  5913. Jul-28 01:12:38.890 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5914. Jul-28 01:12:38.891 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5915. job: 4713446: RUNNING
  5916. job: 4728323: RUNNING
  5917. job: 4728326: RUNNING
  5918. job: 4728325: RUNNING
  5919. job: 4728321: RUNNING
  5920. job: 4728286: RUNNING
  5921.  
  5922. Jul-28 01:12:38.891 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5923. Jul-28 01:12:43.890 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5924. job: 4713446: RUNNING
  5925. job: 4728323: RUNNING
  5926. job: 4728326: RUNNING
  5927. job: 4728325: RUNNING
  5928. job: 4728321: RUNNING
  5929. job: 4728286: RUNNING
  5930.  
  5931. Jul-28 01:12:43.891 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5932. Jul-28 01:12:43.892 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5933. job: 4713446: RUNNING
  5934. job: 4728323: RUNNING
  5935. job: 4728326: RUNNING
  5936. job: 4728325: RUNNING
  5937. job: 4728321: RUNNING
  5938. job: 4728286: RUNNING
  5939.  
  5940. Jul-28 01:12:43.892 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5941. Jul-28 01:12:48.890 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5942. job: 4713446: RUNNING
  5943. job: 4728323: RUNNING
  5944. job: 4728326: RUNNING
  5945. job: 4728325: RUNNING
  5946. job: 4728321: RUNNING
  5947. job: 4728286: RUNNING
  5948.  
  5949. Jul-28 01:12:48.891 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5950. Jul-28 01:12:48.892 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5951. job: 4713446: RUNNING
  5952. job: 4728323: RUNNING
  5953. job: 4728326: RUNNING
  5954. job: 4728325: RUNNING
  5955. job: 4728321: RUNNING
  5956. job: 4728286: RUNNING
  5957.  
  5958. Jul-28 01:12:48.892 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5959. Jul-28 01:12:53.890 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5960. job: 4713446: RUNNING
  5961. job: 4728323: RUNNING
  5962. job: 4728326: RUNNING
  5963. job: 4728325: RUNNING
  5964. job: 4728321: RUNNING
  5965. job: 4728286: RUNNING
  5966.  
  5967. Jul-28 01:12:53.890 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5968. Jul-28 01:12:53.892 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5969. job: 4713446: RUNNING
  5970. job: 4728323: RUNNING
  5971. job: 4728326: RUNNING
  5972. job: 4728325: RUNNING
  5973. job: 4728321: RUNNING
  5974. job: 4728286: RUNNING
  5975.  
  5976. Jul-28 01:12:53.892 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5977. Jul-28 01:12:58.890 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5978. job: 4713446: RUNNING
  5979. job: 4728323: RUNNING
  5980. job: 4728326: RUNNING
  5981. job: 4728325: RUNNING
  5982. job: 4728321: RUNNING
  5983. job: 4728286: RUNNING
  5984.  
  5985. Jul-28 01:12:58.890 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  5986. Jul-28 01:12:58.891 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5987. job: 4713446: RUNNING
  5988. job: 4728323: RUNNING
  5989. job: 4728326: RUNNING
  5990. job: 4728325: RUNNING
  5991. job: 4728321: RUNNING
  5992. job: 4728286: RUNNING
  5993.  
  5994. Jul-28 01:12:58.891 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  5995. Jul-28 01:13:03.891 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  5996. job: 4713446: RUNNING
  5997. job: 4728323: RUNNING
  5998. job: 4728326: RUNNING
  5999. job: 4728325: RUNNING
  6000. job: 4728321: RUNNING
  6001. job: 4728286: RUNNING
  6002.  
  6003. Jul-28 01:13:03.892 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6004. Jul-28 01:13:03.893 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6005. job: 4713446: RUNNING
  6006. job: 4728323: RUNNING
  6007. job: 4728326: RUNNING
  6008. job: 4728325: RUNNING
  6009. job: 4728321: RUNNING
  6010. job: 4728286: RUNNING
  6011.  
  6012. Jul-28 01:13:03.893 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6013. Jul-28 01:13:08.889 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6014. job: 4713446: RUNNING
  6015. job: 4728323: RUNNING
  6016. job: 4728326: RUNNING
  6017. job: 4728325: RUNNING
  6018. job: 4728321: RUNNING
  6019. job: 4728286: RUNNING
  6020.  
  6021. Jul-28 01:13:08.889 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6022. Jul-28 01:13:08.890 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  6023. Jul-28 01:13:09.021 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  6024. 4713446,RUN,Jul 27 11:44
  6025. 4728323,RUN,Jul 28 00:40
  6026. 4728326,RUN,Jul 28 00:51
  6027. 4728325,RUN,Jul 28 00:49
  6028. 4728321,RUN,Jul 28 00:34
  6029. 4728286,RUN,Jul 27 23:44
  6030.  
  6031.  
  6032. Jul-28 01:13:09.021 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6033. job: 4713446: RUNNING
  6034. job: 4728323: RUNNING
  6035. job: 4728326: RUNNING
  6036. job: 4728325: RUNNING
  6037. job: 4728321: RUNNING
  6038. job: 4728286: RUNNING
  6039.  
  6040. Jul-28 01:13:09.021 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6041. Jul-28 01:13:13.889 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6042. job: 4713446: RUNNING
  6043. job: 4728323: RUNNING
  6044. job: 4728326: RUNNING
  6045. job: 4728325: RUNNING
  6046. job: 4728321: RUNNING
  6047. job: 4728286: RUNNING
  6048.  
  6049. Jul-28 01:13:13.889 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6050. Jul-28 01:13:13.890 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6051. job: 4713446: RUNNING
  6052. job: 4728323: RUNNING
  6053. job: 4728326: RUNNING
  6054. job: 4728325: RUNNING
  6055. job: 4728321: RUNNING
  6056. job: 4728286: RUNNING
  6057.  
  6058. Jul-28 01:13:13.890 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6059. Jul-28 01:13:18.890 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6060. job: 4713446: RUNNING
  6061. job: 4728323: RUNNING
  6062. job: 4728326: RUNNING
  6063. job: 4728325: RUNNING
  6064. job: 4728321: RUNNING
  6065. job: 4728286: RUNNING
  6066.  
  6067. Jul-28 01:13:18.890 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6068. Jul-28 01:13:18.891 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6069. job: 4713446: RUNNING
  6070. job: 4728323: RUNNING
  6071. job: 4728326: RUNNING
  6072. job: 4728325: RUNNING
  6073. job: 4728321: RUNNING
  6074. job: 4728286: RUNNING
  6075.  
  6076. Jul-28 01:13:18.891 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6077. Jul-28 01:13:23.889 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6078. job: 4713446: RUNNING
  6079. job: 4728323: RUNNING
  6080. job: 4728326: RUNNING
  6081. job: 4728325: RUNNING
  6082. job: 4728321: RUNNING
  6083. job: 4728286: RUNNING
  6084.  
  6085. Jul-28 01:13:23.890 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6086. Jul-28 01:13:23.890 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6087. job: 4713446: RUNNING
  6088. job: 4728323: RUNNING
  6089. job: 4728326: RUNNING
  6090. job: 4728325: RUNNING
  6091. job: 4728321: RUNNING
  6092. job: 4728286: RUNNING
  6093.  
  6094. Jul-28 01:13:23.890 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6095. Jul-28 01:13:28.890 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6096. job: 4713446: RUNNING
  6097. job: 4728323: RUNNING
  6098. job: 4728326: RUNNING
  6099. job: 4728325: RUNNING
  6100. job: 4728321: RUNNING
  6101. job: 4728286: RUNNING
  6102.  
  6103. Jul-28 01:13:28.890 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6104. Jul-28 01:13:28.891 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6105. job: 4713446: RUNNING
  6106. job: 4728323: RUNNING
  6107. job: 4728326: RUNNING
  6108. job: 4728325: RUNNING
  6109. job: 4728321: RUNNING
  6110. job: 4728286: RUNNING
  6111.  
  6112. Jul-28 01:13:28.891 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6113. Jul-28 01:13:33.890 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6114. job: 4713446: RUNNING
  6115. job: 4728323: RUNNING
  6116. job: 4728326: RUNNING
  6117. job: 4728325: RUNNING
  6118. job: 4728321: RUNNING
  6119. job: 4728286: RUNNING
  6120.  
  6121. Jul-28 01:13:33.891 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6122. Jul-28 01:13:33.892 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6123. job: 4713446: RUNNING
  6124. job: 4728323: RUNNING
  6125. job: 4728326: RUNNING
  6126. job: 4728325: RUNNING
  6127. job: 4728321: RUNNING
  6128. job: 4728286: RUNNING
  6129.  
  6130. Jul-28 01:13:33.892 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6131. Jul-28 01:13:38.890 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6132. job: 4713446: RUNNING
  6133. job: 4728323: RUNNING
  6134. job: 4728326: RUNNING
  6135. job: 4728325: RUNNING
  6136. job: 4728321: RUNNING
  6137. job: 4728286: RUNNING
  6138.  
  6139. Jul-28 01:13:38.890 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6140. Jul-28 01:13:38.891 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6141. job: 4713446: RUNNING
  6142. job: 4728323: RUNNING
  6143. job: 4728326: RUNNING
  6144. job: 4728325: RUNNING
  6145. job: 4728321: RUNNING
  6146. job: 4728286: RUNNING
  6147.  
  6148. Jul-28 01:13:38.891 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6149. Jul-28 01:13:43.890 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6150. job: 4713446: RUNNING
  6151. job: 4728323: RUNNING
  6152. job: 4728326: RUNNING
  6153. job: 4728325: RUNNING
  6154. job: 4728321: RUNNING
  6155. job: 4728286: RUNNING
  6156.  
  6157. Jul-28 01:13:43.890 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6158. Jul-28 01:13:43.891 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6159. job: 4713446: RUNNING
  6160. job: 4728323: RUNNING
  6161. job: 4728326: RUNNING
  6162. job: 4728325: RUNNING
  6163. job: 4728321: RUNNING
  6164. job: 4728286: RUNNING
  6165.  
  6166. Jul-28 01:13:43.891 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6167. Jul-28 01:13:48.890 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6168. job: 4713446: RUNNING
  6169. job: 4728323: RUNNING
  6170. job: 4728326: RUNNING
  6171. job: 4728325: RUNNING
  6172. job: 4728321: RUNNING
  6173. job: 4728286: RUNNING
  6174.  
  6175. Jul-28 01:13:48.891 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6176. Jul-28 01:13:48.891 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6177. job: 4713446: RUNNING
  6178. job: 4728323: RUNNING
  6179. job: 4728326: RUNNING
  6180. job: 4728325: RUNNING
  6181. job: 4728321: RUNNING
  6182. job: 4728286: RUNNING
  6183.  
  6184. Jul-28 01:13:48.891 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6185. Jul-28 01:13:53.890 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6186. job: 4713446: RUNNING
  6187. job: 4728323: RUNNING
  6188. job: 4728326: RUNNING
  6189. job: 4728325: RUNNING
  6190. job: 4728321: RUNNING
  6191. job: 4728286: RUNNING
  6192.  
  6193. Jul-28 01:13:53.890 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6194. Jul-28 01:13:53.891 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6195. job: 4713446: RUNNING
  6196. job: 4728323: RUNNING
  6197. job: 4728326: RUNNING
  6198. job: 4728325: RUNNING
  6199. job: 4728321: RUNNING
  6200. job: 4728286: RUNNING
  6201.  
  6202. Jul-28 01:13:53.891 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6203. Jul-28 01:13:58.896 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6204. job: 4713446: RUNNING
  6205. job: 4728323: RUNNING
  6206. job: 4728326: RUNNING
  6207. job: 4728325: RUNNING
  6208. job: 4728321: RUNNING
  6209. job: 4728286: RUNNING
  6210.  
  6211. Jul-28 01:13:58.896 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6212. Jul-28 01:13:58.898 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6213. job: 4713446: RUNNING
  6214. job: 4728323: RUNNING
  6215. job: 4728326: RUNNING
  6216. job: 4728325: RUNNING
  6217. job: 4728321: RUNNING
  6218. job: 4728286: RUNNING
  6219.  
  6220. Jul-28 01:13:58.899 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6221. Jul-28 01:14:03.894 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6222. job: 4713446: RUNNING
  6223. job: 4728323: RUNNING
  6224. job: 4728326: RUNNING
  6225. job: 4728325: RUNNING
  6226. job: 4728321: RUNNING
  6227. job: 4728286: RUNNING
  6228.  
  6229. Jul-28 01:14:03.895 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6230. Jul-28 01:14:03.895 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6231. job: 4713446: RUNNING
  6232. job: 4728323: RUNNING
  6233. job: 4728326: RUNNING
  6234. job: 4728325: RUNNING
  6235. job: 4728321: RUNNING
  6236. job: 4728286: RUNNING
  6237.  
  6238. Jul-28 01:14:03.895 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6239. Jul-28 01:14:08.894 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  6240. Jul-28 01:14:09.030 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  6241. 4713446,RUN,Jul 27 11:44
  6242. 4728323,RUN,Jul 28 00:40
  6243. 4728326,RUN,Jul 28 00:51
  6244. 4728325,RUN,Jul 28 00:49
  6245. 4728321,RUN,Jul 28 00:34
  6246. 4728286,RUN,Jul 27 23:44
  6247.  
  6248.  
  6249. Jul-28 01:14:09.030 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6250. job: 4713446: RUNNING
  6251. job: 4728323: RUNNING
  6252. job: 4728326: RUNNING
  6253. job: 4728325: RUNNING
  6254. job: 4728321: RUNNING
  6255. job: 4728286: RUNNING
  6256.  
  6257. Jul-28 01:14:09.030 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6258. Jul-28 01:14:09.031 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6259. job: 4713446: RUNNING
  6260. job: 4728323: RUNNING
  6261. job: 4728326: RUNNING
  6262. job: 4728325: RUNNING
  6263. job: 4728321: RUNNING
  6264. job: 4728286: RUNNING
  6265.  
  6266. Jul-28 01:14:09.031 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6267. Jul-28 01:14:13.895 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6268. job: 4713446: RUNNING
  6269. job: 4728323: RUNNING
  6270. job: 4728326: RUNNING
  6271. job: 4728325: RUNNING
  6272. job: 4728321: RUNNING
  6273. job: 4728286: RUNNING
  6274.  
  6275. Jul-28 01:14:13.895 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6276. Jul-28 01:14:13.896 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6277. job: 4713446: RUNNING
  6278. job: 4728323: RUNNING
  6279. job: 4728326: RUNNING
  6280. job: 4728325: RUNNING
  6281. job: 4728321: RUNNING
  6282. job: 4728286: RUNNING
  6283.  
  6284. Jul-28 01:14:13.896 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6285. Jul-28 01:14:18.894 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6286. job: 4713446: RUNNING
  6287. job: 4728323: RUNNING
  6288. job: 4728326: RUNNING
  6289. job: 4728325: RUNNING
  6290. job: 4728321: RUNNING
  6291. job: 4728286: RUNNING
  6292.  
  6293. Jul-28 01:14:18.895 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6294. Jul-28 01:14:18.896 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6295. job: 4713446: RUNNING
  6296. job: 4728323: RUNNING
  6297. job: 4728326: RUNNING
  6298. job: 4728325: RUNNING
  6299. job: 4728321: RUNNING
  6300. job: 4728286: RUNNING
  6301.  
  6302. Jul-28 01:14:18.896 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6303. Jul-28 01:14:23.895 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6304. job: 4713446: RUNNING
  6305. job: 4728323: RUNNING
  6306. job: 4728326: RUNNING
  6307. job: 4728325: RUNNING
  6308. job: 4728321: RUNNING
  6309. job: 4728286: RUNNING
  6310.  
  6311. Jul-28 01:14:23.896 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6312. Jul-28 01:14:23.897 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6313. job: 4713446: RUNNING
  6314. job: 4728323: RUNNING
  6315. job: 4728326: RUNNING
  6316. job: 4728325: RUNNING
  6317. job: 4728321: RUNNING
  6318. job: 4728286: RUNNING
  6319.  
  6320. Jul-28 01:14:23.897 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6321. Jul-28 01:14:28.894 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6322. job: 4713446: RUNNING
  6323. job: 4728323: RUNNING
  6324. job: 4728326: RUNNING
  6325. job: 4728325: RUNNING
  6326. job: 4728321: RUNNING
  6327. job: 4728286: RUNNING
  6328.  
  6329. Jul-28 01:14:28.894 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6330. Jul-28 01:14:28.896 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6331. job: 4713446: RUNNING
  6332. job: 4728323: RUNNING
  6333. job: 4728326: RUNNING
  6334. job: 4728325: RUNNING
  6335. job: 4728321: RUNNING
  6336. job: 4728286: RUNNING
  6337.  
  6338. Jul-28 01:14:28.896 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6339. Jul-28 01:14:33.894 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6340. job: 4713446: RUNNING
  6341. job: 4728323: RUNNING
  6342. job: 4728326: RUNNING
  6343. job: 4728325: RUNNING
  6344. job: 4728321: RUNNING
  6345. job: 4728286: RUNNING
  6346.  
  6347. Jul-28 01:14:33.894 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6348. Jul-28 01:14:33.895 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6349. job: 4713446: RUNNING
  6350. job: 4728323: RUNNING
  6351. job: 4728326: RUNNING
  6352. job: 4728325: RUNNING
  6353. job: 4728321: RUNNING
  6354. job: 4728286: RUNNING
  6355.  
  6356. Jul-28 01:14:33.895 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6357. Jul-28 01:14:38.893 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6358. job: 4713446: RUNNING
  6359. job: 4728323: RUNNING
  6360. job: 4728326: RUNNING
  6361. job: 4728325: RUNNING
  6362. job: 4728321: RUNNING
  6363. job: 4728286: RUNNING
  6364.  
  6365. Jul-28 01:14:38.893 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6366. Jul-28 01:14:38.894 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6367. job: 4713446: RUNNING
  6368. job: 4728323: RUNNING
  6369. job: 4728326: RUNNING
  6370. job: 4728325: RUNNING
  6371. job: 4728321: RUNNING
  6372. job: 4728286: RUNNING
  6373.  
  6374. Jul-28 01:14:38.894 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6375. Jul-28 01:14:43.894 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6376. job: 4713446: RUNNING
  6377. job: 4728323: RUNNING
  6378. job: 4728326: RUNNING
  6379. job: 4728325: RUNNING
  6380. job: 4728321: RUNNING
  6381. job: 4728286: RUNNING
  6382.  
  6383. Jul-28 01:14:43.894 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6384. Jul-28 01:14:43.895 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6385. job: 4713446: RUNNING
  6386. job: 4728323: RUNNING
  6387. job: 4728326: RUNNING
  6388. job: 4728325: RUNNING
  6389. job: 4728321: RUNNING
  6390. job: 4728286: RUNNING
  6391.  
  6392. Jul-28 01:14:43.895 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6393. Jul-28 01:14:48.894 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6394. job: 4713446: RUNNING
  6395. job: 4728323: RUNNING
  6396. job: 4728326: RUNNING
  6397. job: 4728325: RUNNING
  6398. job: 4728321: RUNNING
  6399. job: 4728286: RUNNING
  6400.  
  6401. Jul-28 01:14:48.894 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6402. Jul-28 01:14:48.895 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6403. job: 4713446: RUNNING
  6404. job: 4728323: RUNNING
  6405. job: 4728326: RUNNING
  6406. job: 4728325: RUNNING
  6407. job: 4728321: RUNNING
  6408. job: 4728286: RUNNING
  6409.  
  6410. Jul-28 01:14:48.895 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6411. Jul-28 01:14:53.894 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6412. job: 4713446: RUNNING
  6413. job: 4728323: RUNNING
  6414. job: 4728326: RUNNING
  6415. job: 4728325: RUNNING
  6416. job: 4728321: RUNNING
  6417. job: 4728286: RUNNING
  6418.  
  6419. Jul-28 01:14:53.894 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6420. Jul-28 01:14:53.895 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6421. job: 4713446: RUNNING
  6422. job: 4728323: RUNNING
  6423. job: 4728326: RUNNING
  6424. job: 4728325: RUNNING
  6425. job: 4728321: RUNNING
  6426. job: 4728286: RUNNING
  6427.  
  6428. Jul-28 01:14:53.895 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6429. Jul-28 01:14:58.894 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6430. job: 4713446: RUNNING
  6431. job: 4728323: RUNNING
  6432. job: 4728326: RUNNING
  6433. job: 4728325: RUNNING
  6434. job: 4728321: RUNNING
  6435. job: 4728286: RUNNING
  6436.  
  6437. Jul-28 01:14:58.894 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6438. Jul-28 01:14:58.895 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6439. job: 4713446: RUNNING
  6440. job: 4728323: RUNNING
  6441. job: 4728326: RUNNING
  6442. job: 4728325: RUNNING
  6443. job: 4728321: RUNNING
  6444. job: 4728286: RUNNING
  6445.  
  6446. Jul-28 01:14:58.895 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6447. Jul-28 01:15:03.893 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6448. job: 4713446: RUNNING
  6449. job: 4728323: RUNNING
  6450. job: 4728326: RUNNING
  6451. job: 4728325: RUNNING
  6452. job: 4728321: RUNNING
  6453. job: 4728286: RUNNING
  6454.  
  6455. Jul-28 01:15:03.894 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6456. Jul-28 01:15:03.894 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6457. job: 4713446: RUNNING
  6458. job: 4728323: RUNNING
  6459. job: 4728326: RUNNING
  6460. job: 4728325: RUNNING
  6461. job: 4728321: RUNNING
  6462. job: 4728286: RUNNING
  6463.  
  6464. Jul-28 01:15:03.894 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6465. Jul-28 01:15:08.895 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor lsf > tasks to be completed: 2 -- pending tasks are shown below
  6466. ~> TaskHandler[jobId: 4728325; id: 152; name: merge_pairsam_into_runs (library:HeLa1 run:lane1); status: RUNNING; exit: -; error: -; workDir: /farline/umw_job_dekker/HPCC/sv49w/distiller-nf/work/3b/37ed2fa2ba160f54196e762e24c83b started: 1501217508853; exited: -; ]
  6467. ~> TaskHandler[jobId: 4728326; id: 153; name: merge_pairsam_into_runs (library:HeLa1 run:lane3); status: RUNNING; exit: -; error: -; workDir: /farline/umw_job_dekker/HPCC/sv49w/distiller-nf/work/00/2633edc9da9ad5649d3d6dcffe5a44 started: 1501217938848; exited: -; ]
  6468. Jul-28 01:15:08.896 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  6469. Jul-28 01:15:09.028 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  6470. 4713446,RUN,Jul 27 11:44
  6471. 4728323,RUN,Jul 28 00:40
  6472. 4728326,RUN,Jul 28 00:51
  6473. 4728325,RUN,Jul 28 00:49
  6474. 4728321,RUN,Jul 28 00:34
  6475. 4728286,RUN,Jul 27 23:44
  6476.  
  6477.  
  6478. Jul-28 01:15:09.029 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6479. job: 4713446: RUNNING
  6480. job: 4728323: RUNNING
  6481. job: 4728326: RUNNING
  6482. job: 4728325: RUNNING
  6483. job: 4728321: RUNNING
  6484. job: 4728286: RUNNING
  6485.  
  6486. Jul-28 01:15:09.029 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6487. Jul-28 01:15:09.029 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6488. job: 4713446: RUNNING
  6489. job: 4728323: RUNNING
  6490. job: 4728326: RUNNING
  6491. job: 4728325: RUNNING
  6492. job: 4728321: RUNNING
  6493. job: 4728286: RUNNING
  6494.  
  6495. Jul-28 01:15:09.029 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6496. Jul-28 01:15:13.899 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6497. job: 4713446: RUNNING
  6498. job: 4728323: RUNNING
  6499. job: 4728326: RUNNING
  6500. job: 4728325: RUNNING
  6501. job: 4728321: RUNNING
  6502. job: 4728286: RUNNING
  6503.  
  6504. Jul-28 01:15:13.899 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6505. Jul-28 01:15:13.900 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6506. job: 4713446: RUNNING
  6507. job: 4728323: RUNNING
  6508. job: 4728326: RUNNING
  6509. job: 4728325: RUNNING
  6510. job: 4728321: RUNNING
  6511. job: 4728286: RUNNING
  6512.  
  6513. Jul-28 01:15:13.900 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6514. Jul-28 01:15:18.899 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6515. job: 4713446: RUNNING
  6516. job: 4728323: RUNNING
  6517. job: 4728326: RUNNING
  6518. job: 4728325: RUNNING
  6519. job: 4728321: RUNNING
  6520. job: 4728286: RUNNING
  6521.  
  6522. Jul-28 01:15:18.899 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6523. Jul-28 01:15:18.900 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6524. job: 4713446: RUNNING
  6525. job: 4728323: RUNNING
  6526. job: 4728326: RUNNING
  6527. job: 4728325: RUNNING
  6528. job: 4728321: RUNNING
  6529. job: 4728286: RUNNING
  6530.  
  6531. Jul-28 01:15:18.900 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6532. Jul-28 01:15:23.899 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6533. job: 4713446: RUNNING
  6534. job: 4728323: RUNNING
  6535. job: 4728326: RUNNING
  6536. job: 4728325: RUNNING
  6537. job: 4728321: RUNNING
  6538. job: 4728286: RUNNING
  6539.  
  6540. Jul-28 01:15:23.899 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6541. Jul-28 01:15:23.900 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6542. job: 4713446: RUNNING
  6543. job: 4728323: RUNNING
  6544. job: 4728326: RUNNING
  6545. job: 4728325: RUNNING
  6546. job: 4728321: RUNNING
  6547. job: 4728286: RUNNING
  6548.  
  6549. Jul-28 01:15:23.900 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6550. Jul-28 01:15:28.899 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6551. job: 4713446: RUNNING
  6552. job: 4728323: RUNNING
  6553. job: 4728326: RUNNING
  6554. job: 4728325: RUNNING
  6555. job: 4728321: RUNNING
  6556. job: 4728286: RUNNING
  6557.  
  6558. Jul-28 01:15:28.899 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6559. Jul-28 01:15:28.899 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6560. job: 4713446: RUNNING
  6561. job: 4728323: RUNNING
  6562. job: 4728326: RUNNING
  6563. job: 4728325: RUNNING
  6564. job: 4728321: RUNNING
  6565. job: 4728286: RUNNING
  6566.  
  6567. Jul-28 01:15:28.899 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6568. Jul-28 01:15:33.899 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6569. job: 4713446: RUNNING
  6570. job: 4728323: RUNNING
  6571. job: 4728326: RUNNING
  6572. job: 4728325: RUNNING
  6573. job: 4728321: RUNNING
  6574. job: 4728286: RUNNING
  6575.  
  6576. Jul-28 01:15:33.900 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6577. Jul-28 01:15:33.900 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6578. job: 4713446: RUNNING
  6579. job: 4728323: RUNNING
  6580. job: 4728326: RUNNING
  6581. job: 4728325: RUNNING
  6582. job: 4728321: RUNNING
  6583. job: 4728286: RUNNING
  6584.  
  6585. Jul-28 01:15:33.900 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6586. Jul-28 01:15:38.901 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6587. job: 4713446: RUNNING
  6588. job: 4728323: RUNNING
  6589. job: 4728326: RUNNING
  6590. job: 4728325: RUNNING
  6591. job: 4728321: RUNNING
  6592. job: 4728286: RUNNING
  6593.  
  6594. Jul-28 01:15:38.901 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6595. Jul-28 01:15:38.902 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6596. job: 4713446: RUNNING
  6597. job: 4728323: RUNNING
  6598. job: 4728326: RUNNING
  6599. job: 4728325: RUNNING
  6600. job: 4728321: RUNNING
  6601. job: 4728286: RUNNING
  6602.  
  6603. Jul-28 01:15:38.902 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6604. Jul-28 01:15:43.900 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6605. job: 4713446: RUNNING
  6606. job: 4728323: RUNNING
  6607. job: 4728326: RUNNING
  6608. job: 4728325: RUNNING
  6609. job: 4728321: RUNNING
  6610. job: 4728286: RUNNING
  6611.  
  6612. Jul-28 01:15:43.901 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6613. Jul-28 01:15:43.901 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6614. job: 4713446: RUNNING
  6615. job: 4728323: RUNNING
  6616. job: 4728326: RUNNING
  6617. job: 4728325: RUNNING
  6618. job: 4728321: RUNNING
  6619. job: 4728286: RUNNING
  6620.  
  6621. Jul-28 01:15:43.901 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6622. Jul-28 01:15:48.903 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6623. job: 4713446: RUNNING
  6624. job: 4728323: RUNNING
  6625. job: 4728326: RUNNING
  6626. job: 4728325: RUNNING
  6627. job: 4728321: RUNNING
  6628. job: 4728286: RUNNING
  6629.  
  6630. Jul-28 01:15:48.904 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6631. Jul-28 01:15:48.905 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6632. job: 4713446: RUNNING
  6633. job: 4728323: RUNNING
  6634. job: 4728326: RUNNING
  6635. job: 4728325: RUNNING
  6636. job: 4728321: RUNNING
  6637. job: 4728286: RUNNING
  6638.  
  6639. Jul-28 01:15:48.905 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6640. Jul-28 01:15:53.904 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6641. job: 4713446: RUNNING
  6642. job: 4728323: RUNNING
  6643. job: 4728326: RUNNING
  6644. job: 4728325: RUNNING
  6645. job: 4728321: RUNNING
  6646. job: 4728286: RUNNING
  6647.  
  6648. Jul-28 01:15:53.905 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6649. Jul-28 01:15:53.906 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6650. job: 4713446: RUNNING
  6651. job: 4728323: RUNNING
  6652. job: 4728326: RUNNING
  6653. job: 4728325: RUNNING
  6654. job: 4728321: RUNNING
  6655. job: 4728286: RUNNING
  6656.  
  6657. Jul-28 01:15:53.906 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6658. Jul-28 01:15:58.901 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6659. job: 4713446: RUNNING
  6660. job: 4728323: RUNNING
  6661. job: 4728326: RUNNING
  6662. job: 4728325: RUNNING
  6663. job: 4728321: RUNNING
  6664. job: 4728286: RUNNING
  6665.  
  6666. Jul-28 01:15:58.901 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6667. Jul-28 01:15:58.902 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6668. job: 4713446: RUNNING
  6669. job: 4728323: RUNNING
  6670. job: 4728326: RUNNING
  6671. job: 4728325: RUNNING
  6672. job: 4728321: RUNNING
  6673. job: 4728286: RUNNING
  6674.  
  6675. Jul-28 01:15:58.902 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6676. Jul-28 01:16:03.901 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6677. job: 4713446: RUNNING
  6678. job: 4728323: RUNNING
  6679. job: 4728326: RUNNING
  6680. job: 4728325: RUNNING
  6681. job: 4728321: RUNNING
  6682. job: 4728286: RUNNING
  6683.  
  6684. Jul-28 01:16:03.901 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6685. Jul-28 01:16:03.902 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6686. job: 4713446: RUNNING
  6687. job: 4728323: RUNNING
  6688. job: 4728326: RUNNING
  6689. job: 4728325: RUNNING
  6690. job: 4728321: RUNNING
  6691. job: 4728286: RUNNING
  6692.  
  6693. Jul-28 01:16:03.902 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6694. Jul-28 01:16:08.900 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  6695. Jul-28 01:16:09.028 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  6696. 4713446,RUN,Jul 27 11:44
  6697. 4728323,RUN,Jul 28 00:40
  6698. 4728326,RUN,Jul 28 00:51
  6699. 4728325,RUN,Jul 28 00:49
  6700. 4728321,RUN,Jul 28 00:34
  6701. 4728286,RUN,Jul 27 23:44
  6702.  
  6703.  
  6704. Jul-28 01:16:09.028 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6705. job: 4713446: RUNNING
  6706. job: 4728323: RUNNING
  6707. job: 4728326: RUNNING
  6708. job: 4728325: RUNNING
  6709. job: 4728321: RUNNING
  6710. job: 4728286: RUNNING
  6711.  
  6712. Jul-28 01:16:09.028 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6713. Jul-28 01:16:09.029 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6714. job: 4713446: RUNNING
  6715. job: 4728323: RUNNING
  6716. job: 4728326: RUNNING
  6717. job: 4728325: RUNNING
  6718. job: 4728321: RUNNING
  6719. job: 4728286: RUNNING
  6720.  
  6721. Jul-28 01:16:09.029 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6722. Jul-28 01:16:13.901 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6723. job: 4713446: RUNNING
  6724. job: 4728323: RUNNING
  6725. job: 4728326: RUNNING
  6726. job: 4728325: RUNNING
  6727. job: 4728321: RUNNING
  6728. job: 4728286: RUNNING
  6729.  
  6730. Jul-28 01:16:13.901 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6731. Jul-28 01:16:13.902 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6732. job: 4713446: RUNNING
  6733. job: 4728323: RUNNING
  6734. job: 4728326: RUNNING
  6735. job: 4728325: RUNNING
  6736. job: 4728321: RUNNING
  6737. job: 4728286: RUNNING
  6738.  
  6739. Jul-28 01:16:13.902 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6740. Jul-28 01:16:18.901 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6741. job: 4713446: RUNNING
  6742. job: 4728323: RUNNING
  6743. job: 4728326: RUNNING
  6744. job: 4728325: RUNNING
  6745. job: 4728321: RUNNING
  6746. job: 4728286: RUNNING
  6747.  
  6748. Jul-28 01:16:18.901 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6749. Jul-28 01:16:18.902 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6750. job: 4713446: RUNNING
  6751. job: 4728323: RUNNING
  6752. job: 4728326: RUNNING
  6753. job: 4728325: RUNNING
  6754. job: 4728321: RUNNING
  6755. job: 4728286: RUNNING
  6756.  
  6757. Jul-28 01:16:18.902 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6758. Jul-28 01:16:23.903 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6759. job: 4713446: RUNNING
  6760. job: 4728323: RUNNING
  6761. job: 4728326: RUNNING
  6762. job: 4728325: RUNNING
  6763. job: 4728321: RUNNING
  6764. job: 4728286: RUNNING
  6765.  
  6766. Jul-28 01:16:23.903 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6767. Jul-28 01:16:23.904 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6768. job: 4713446: RUNNING
  6769. job: 4728323: RUNNING
  6770. job: 4728326: RUNNING
  6771. job: 4728325: RUNNING
  6772. job: 4728321: RUNNING
  6773. job: 4728286: RUNNING
  6774.  
  6775. Jul-28 01:16:23.904 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6776. Jul-28 01:16:28.905 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6777. job: 4713446: RUNNING
  6778. job: 4728323: RUNNING
  6779. job: 4728326: RUNNING
  6780. job: 4728325: RUNNING
  6781. job: 4728321: RUNNING
  6782. job: 4728286: RUNNING
  6783.  
  6784. Jul-28 01:16:28.906 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6785. Jul-28 01:16:28.907 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6786. job: 4713446: RUNNING
  6787. job: 4728323: RUNNING
  6788. job: 4728326: RUNNING
  6789. job: 4728325: RUNNING
  6790. job: 4728321: RUNNING
  6791. job: 4728286: RUNNING
  6792.  
  6793. Jul-28 01:16:28.907 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6794. Jul-28 01:16:33.904 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6795. job: 4713446: RUNNING
  6796. job: 4728323: RUNNING
  6797. job: 4728326: RUNNING
  6798. job: 4728325: RUNNING
  6799. job: 4728321: RUNNING
  6800. job: 4728286: RUNNING
  6801.  
  6802. Jul-28 01:16:33.905 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6803. Jul-28 01:16:33.906 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6804. job: 4713446: RUNNING
  6805. job: 4728323: RUNNING
  6806. job: 4728326: RUNNING
  6807. job: 4728325: RUNNING
  6808. job: 4728321: RUNNING
  6809. job: 4728286: RUNNING
  6810.  
  6811. Jul-28 01:16:33.906 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6812. Jul-28 01:16:38.902 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6813. job: 4713446: RUNNING
  6814. job: 4728323: RUNNING
  6815. job: 4728326: RUNNING
  6816. job: 4728325: RUNNING
  6817. job: 4728321: RUNNING
  6818. job: 4728286: RUNNING
  6819.  
  6820. Jul-28 01:16:38.903 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6821. Jul-28 01:16:38.903 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6822. job: 4713446: RUNNING
  6823. job: 4728323: RUNNING
  6824. job: 4728326: RUNNING
  6825. job: 4728325: RUNNING
  6826. job: 4728321: RUNNING
  6827. job: 4728286: RUNNING
  6828.  
  6829. Jul-28 01:16:38.903 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6830. Jul-28 01:16:43.903 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6831. job: 4713446: RUNNING
  6832. job: 4728323: RUNNING
  6833. job: 4728326: RUNNING
  6834. job: 4728325: RUNNING
  6835. job: 4728321: RUNNING
  6836. job: 4728286: RUNNING
  6837.  
  6838. Jul-28 01:16:43.903 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6839. Jul-28 01:16:43.904 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6840. job: 4713446: RUNNING
  6841. job: 4728323: RUNNING
  6842. job: 4728326: RUNNING
  6843. job: 4728325: RUNNING
  6844. job: 4728321: RUNNING
  6845. job: 4728286: RUNNING
  6846.  
  6847. Jul-28 01:16:43.904 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6848. Jul-28 01:16:48.904 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6849. job: 4713446: RUNNING
  6850. job: 4728323: RUNNING
  6851. job: 4728326: RUNNING
  6852. job: 4728325: RUNNING
  6853. job: 4728321: RUNNING
  6854. job: 4728286: RUNNING
  6855.  
  6856. Jul-28 01:16:48.904 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6857. Jul-28 01:16:48.905 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6858. job: 4713446: RUNNING
  6859. job: 4728323: RUNNING
  6860. job: 4728326: RUNNING
  6861. job: 4728325: RUNNING
  6862. job: 4728321: RUNNING
  6863. job: 4728286: RUNNING
  6864.  
  6865. Jul-28 01:16:48.905 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6866. Jul-28 01:16:53.905 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6867. job: 4713446: RUNNING
  6868. job: 4728323: RUNNING
  6869. job: 4728326: RUNNING
  6870. job: 4728325: RUNNING
  6871. job: 4728321: RUNNING
  6872. job: 4728286: RUNNING
  6873.  
  6874. Jul-28 01:16:53.905 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6875. Jul-28 01:16:53.907 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6876. job: 4713446: RUNNING
  6877. job: 4728323: RUNNING
  6878. job: 4728326: RUNNING
  6879. job: 4728325: RUNNING
  6880. job: 4728321: RUNNING
  6881. job: 4728286: RUNNING
  6882.  
  6883. Jul-28 01:16:53.907 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6884. Jul-28 01:16:58.902 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6885. job: 4713446: RUNNING
  6886. job: 4728323: RUNNING
  6887. job: 4728326: RUNNING
  6888. job: 4728325: RUNNING
  6889. job: 4728321: RUNNING
  6890. job: 4728286: RUNNING
  6891.  
  6892. Jul-28 01:16:58.903 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6893. Jul-28 01:16:58.904 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6894. job: 4713446: RUNNING
  6895. job: 4728323: RUNNING
  6896. job: 4728326: RUNNING
  6897. job: 4728325: RUNNING
  6898. job: 4728321: RUNNING
  6899. job: 4728286: RUNNING
  6900.  
  6901. Jul-28 01:16:58.904 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6902. Jul-28 01:17:03.904 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6903. job: 4713446: RUNNING
  6904. job: 4728323: RUNNING
  6905. job: 4728326: RUNNING
  6906. job: 4728325: RUNNING
  6907. job: 4728321: RUNNING
  6908. job: 4728286: RUNNING
  6909.  
  6910. Jul-28 01:17:03.904 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6911. Jul-28 01:17:03.905 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6912. job: 4713446: RUNNING
  6913. job: 4728323: RUNNING
  6914. job: 4728326: RUNNING
  6915. job: 4728325: RUNNING
  6916. job: 4728321: RUNNING
  6917. job: 4728286: RUNNING
  6918.  
  6919. Jul-28 01:17:03.905 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6920. Jul-28 01:17:08.904 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  6921. Jul-28 01:17:09.039 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  6922. 4713446,RUN,Jul 27 11:44
  6923. 4728323,RUN,Jul 28 00:40
  6924. 4728326,RUN,Jul 28 00:51
  6925. 4728325,RUN,Jul 28 00:49
  6926. 4728321,RUN,Jul 28 00:34
  6927. 4728286,RUN,Jul 27 23:44
  6928.  
  6929.  
  6930. Jul-28 01:17:09.039 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6931. job: 4713446: RUNNING
  6932. job: 4728323: RUNNING
  6933. job: 4728326: RUNNING
  6934. job: 4728325: RUNNING
  6935. job: 4728321: RUNNING
  6936. job: 4728286: RUNNING
  6937.  
  6938. Jul-28 01:17:09.039 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6939. Jul-28 01:17:09.040 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6940. job: 4713446: RUNNING
  6941. job: 4728323: RUNNING
  6942. job: 4728326: RUNNING
  6943. job: 4728325: RUNNING
  6944. job: 4728321: RUNNING
  6945. job: 4728286: RUNNING
  6946.  
  6947. Jul-28 01:17:09.040 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6948. Jul-28 01:17:13.903 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6949. job: 4713446: RUNNING
  6950. job: 4728323: RUNNING
  6951. job: 4728326: RUNNING
  6952. job: 4728325: RUNNING
  6953. job: 4728321: RUNNING
  6954. job: 4728286: RUNNING
  6955.  
  6956. Jul-28 01:17:13.903 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6957. Jul-28 01:17:13.904 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6958. job: 4713446: RUNNING
  6959. job: 4728323: RUNNING
  6960. job: 4728326: RUNNING
  6961. job: 4728325: RUNNING
  6962. job: 4728321: RUNNING
  6963. job: 4728286: RUNNING
  6964.  
  6965. Jul-28 01:17:13.904 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6966. Jul-28 01:17:18.904 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6967. job: 4713446: RUNNING
  6968. job: 4728323: RUNNING
  6969. job: 4728326: RUNNING
  6970. job: 4728325: RUNNING
  6971. job: 4728321: RUNNING
  6972. job: 4728286: RUNNING
  6973.  
  6974. Jul-28 01:17:18.904 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6975. Jul-28 01:17:18.905 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6976. job: 4713446: RUNNING
  6977. job: 4728323: RUNNING
  6978. job: 4728326: RUNNING
  6979. job: 4728325: RUNNING
  6980. job: 4728321: RUNNING
  6981. job: 4728286: RUNNING
  6982.  
  6983. Jul-28 01:17:18.905 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  6984. Jul-28 01:17:23.903 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6985. job: 4713446: RUNNING
  6986. job: 4728323: RUNNING
  6987. job: 4728326: RUNNING
  6988. job: 4728325: RUNNING
  6989. job: 4728321: RUNNING
  6990. job: 4728286: RUNNING
  6991.  
  6992. Jul-28 01:17:23.904 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  6993. Jul-28 01:17:23.904 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  6994. job: 4713446: RUNNING
  6995. job: 4728323: RUNNING
  6996. job: 4728326: RUNNING
  6997. job: 4728325: RUNNING
  6998. job: 4728321: RUNNING
  6999. job: 4728286: RUNNING
  7000.  
  7001. Jul-28 01:17:23.904 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7002. Jul-28 01:17:28.906 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7003. job: 4713446: RUNNING
  7004. job: 4728323: RUNNING
  7005. job: 4728326: RUNNING
  7006. job: 4728325: RUNNING
  7007. job: 4728321: RUNNING
  7008. job: 4728286: RUNNING
  7009.  
  7010. Jul-28 01:17:28.907 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7011. Jul-28 01:17:28.908 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7012. job: 4713446: RUNNING
  7013. job: 4728323: RUNNING
  7014. job: 4728326: RUNNING
  7015. job: 4728325: RUNNING
  7016. job: 4728321: RUNNING
  7017. job: 4728286: RUNNING
  7018.  
  7019. Jul-28 01:17:28.908 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7020. Jul-28 01:17:33.908 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7021. job: 4713446: RUNNING
  7022. job: 4728323: RUNNING
  7023. job: 4728326: RUNNING
  7024. job: 4728325: RUNNING
  7025. job: 4728321: RUNNING
  7026. job: 4728286: RUNNING
  7027.  
  7028. Jul-28 01:17:33.909 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7029. Jul-28 01:17:33.910 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7030. job: 4713446: RUNNING
  7031. job: 4728323: RUNNING
  7032. job: 4728326: RUNNING
  7033. job: 4728325: RUNNING
  7034. job: 4728321: RUNNING
  7035. job: 4728286: RUNNING
  7036.  
  7037. Jul-28 01:17:33.910 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7038. Jul-28 01:17:38.908 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7039. job: 4713446: RUNNING
  7040. job: 4728323: RUNNING
  7041. job: 4728326: RUNNING
  7042. job: 4728325: RUNNING
  7043. job: 4728321: RUNNING
  7044. job: 4728286: RUNNING
  7045.  
  7046. Jul-28 01:17:38.909 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7047. Jul-28 01:17:38.911 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7048. job: 4713446: RUNNING
  7049. job: 4728323: RUNNING
  7050. job: 4728326: RUNNING
  7051. job: 4728325: RUNNING
  7052. job: 4728321: RUNNING
  7053. job: 4728286: RUNNING
  7054.  
  7055. Jul-28 01:17:38.911 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7056. Jul-28 01:17:43.905 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7057. job: 4713446: RUNNING
  7058. job: 4728323: RUNNING
  7059. job: 4728326: RUNNING
  7060. job: 4728325: RUNNING
  7061. job: 4728321: RUNNING
  7062. job: 4728286: RUNNING
  7063.  
  7064. Jul-28 01:17:43.906 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7065. Jul-28 01:17:43.907 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7066. job: 4713446: RUNNING
  7067. job: 4728323: RUNNING
  7068. job: 4728326: RUNNING
  7069. job: 4728325: RUNNING
  7070. job: 4728321: RUNNING
  7071. job: 4728286: RUNNING
  7072.  
  7073. Jul-28 01:17:43.907 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7074. Jul-28 01:17:48.905 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7075. job: 4713446: RUNNING
  7076. job: 4728323: RUNNING
  7077. job: 4728326: RUNNING
  7078. job: 4728325: RUNNING
  7079. job: 4728321: RUNNING
  7080. job: 4728286: RUNNING
  7081.  
  7082. Jul-28 01:17:48.906 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7083. Jul-28 01:17:48.906 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7084. job: 4713446: RUNNING
  7085. job: 4728323: RUNNING
  7086. job: 4728326: RUNNING
  7087. job: 4728325: RUNNING
  7088. job: 4728321: RUNNING
  7089. job: 4728286: RUNNING
  7090.  
  7091. Jul-28 01:17:48.906 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7092. Jul-28 01:17:53.905 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7093. job: 4713446: RUNNING
  7094. job: 4728323: RUNNING
  7095. job: 4728326: RUNNING
  7096. job: 4728325: RUNNING
  7097. job: 4728321: RUNNING
  7098. job: 4728286: RUNNING
  7099.  
  7100. Jul-28 01:17:53.905 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7101. Jul-28 01:17:53.906 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7102. job: 4713446: RUNNING
  7103. job: 4728323: RUNNING
  7104. job: 4728326: RUNNING
  7105. job: 4728325: RUNNING
  7106. job: 4728321: RUNNING
  7107. job: 4728286: RUNNING
  7108.  
  7109. Jul-28 01:17:53.906 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7110. Jul-28 01:17:58.905 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7111. job: 4713446: RUNNING
  7112. job: 4728323: RUNNING
  7113. job: 4728326: RUNNING
  7114. job: 4728325: RUNNING
  7115. job: 4728321: RUNNING
  7116. job: 4728286: RUNNING
  7117.  
  7118. Jul-28 01:17:58.906 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7119. Jul-28 01:17:58.906 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7120. job: 4713446: RUNNING
  7121. job: 4728323: RUNNING
  7122. job: 4728326: RUNNING
  7123. job: 4728325: RUNNING
  7124. job: 4728321: RUNNING
  7125. job: 4728286: RUNNING
  7126.  
  7127. Jul-28 01:17:58.906 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7128. Jul-28 01:18:03.906 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7129. job: 4713446: RUNNING
  7130. job: 4728323: RUNNING
  7131. job: 4728326: RUNNING
  7132. job: 4728325: RUNNING
  7133. job: 4728321: RUNNING
  7134. job: 4728286: RUNNING
  7135.  
  7136. Jul-28 01:18:03.907 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7137. Jul-28 01:18:03.908 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7138. job: 4713446: RUNNING
  7139. job: 4728323: RUNNING
  7140. job: 4728326: RUNNING
  7141. job: 4728325: RUNNING
  7142. job: 4728321: RUNNING
  7143. job: 4728286: RUNNING
  7144.  
  7145. Jul-28 01:18:03.908 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7146. Jul-28 01:18:08.906 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  7147. Jul-28 01:18:09.040 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  7148. 4713446,RUN,Jul 27 11:44
  7149. 4728323,RUN,Jul 28 00:40
  7150. 4728326,RUN,Jul 28 00:51
  7151. 4728325,RUN,Jul 28 00:49
  7152. 4728321,RUN,Jul 28 00:34
  7153. 4728286,RUN,Jul 27 23:44
  7154.  
  7155.  
  7156. Jul-28 01:18:09.040 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7157. job: 4713446: RUNNING
  7158. job: 4728323: RUNNING
  7159. job: 4728326: RUNNING
  7160. job: 4728325: RUNNING
  7161. job: 4728321: RUNNING
  7162. job: 4728286: RUNNING
  7163.  
  7164. Jul-28 01:18:09.040 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7165. Jul-28 01:18:09.042 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7166. job: 4713446: RUNNING
  7167. job: 4728323: RUNNING
  7168. job: 4728326: RUNNING
  7169. job: 4728325: RUNNING
  7170. job: 4728321: RUNNING
  7171. job: 4728286: RUNNING
  7172.  
  7173. Jul-28 01:18:09.042 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7174. Jul-28 01:18:13.909 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7175. job: 4713446: RUNNING
  7176. job: 4728323: RUNNING
  7177. job: 4728326: RUNNING
  7178. job: 4728325: RUNNING
  7179. job: 4728321: RUNNING
  7180. job: 4728286: RUNNING
  7181.  
  7182. Jul-28 01:18:13.910 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7183. Jul-28 01:18:13.911 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7184. job: 4713446: RUNNING
  7185. job: 4728323: RUNNING
  7186. job: 4728326: RUNNING
  7187. job: 4728325: RUNNING
  7188. job: 4728321: RUNNING
  7189. job: 4728286: RUNNING
  7190.  
  7191. Jul-28 01:18:13.911 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7192. Jul-28 01:18:18.911 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7193. job: 4713446: RUNNING
  7194. job: 4728323: RUNNING
  7195. job: 4728326: RUNNING
  7196. job: 4728325: RUNNING
  7197. job: 4728321: RUNNING
  7198. job: 4728286: RUNNING
  7199.  
  7200. Jul-28 01:18:18.912 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7201. Jul-28 01:18:18.914 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7202. job: 4713446: RUNNING
  7203. job: 4728323: RUNNING
  7204. job: 4728326: RUNNING
  7205. job: 4728325: RUNNING
  7206. job: 4728321: RUNNING
  7207. job: 4728286: RUNNING
  7208.  
  7209. Jul-28 01:18:18.914 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7210. Jul-28 01:18:23.908 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7211. job: 4713446: RUNNING
  7212. job: 4728323: RUNNING
  7213. job: 4728326: RUNNING
  7214. job: 4728325: RUNNING
  7215. job: 4728321: RUNNING
  7216. job: 4728286: RUNNING
  7217.  
  7218. Jul-28 01:18:23.908 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7219. Jul-28 01:18:23.909 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7220. job: 4713446: RUNNING
  7221. job: 4728323: RUNNING
  7222. job: 4728326: RUNNING
  7223. job: 4728325: RUNNING
  7224. job: 4728321: RUNNING
  7225. job: 4728286: RUNNING
  7226.  
  7227. Jul-28 01:18:23.909 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7228. Jul-28 01:18:28.909 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7229. job: 4713446: RUNNING
  7230. job: 4728323: RUNNING
  7231. job: 4728326: RUNNING
  7232. job: 4728325: RUNNING
  7233. job: 4728321: RUNNING
  7234. job: 4728286: RUNNING
  7235.  
  7236. Jul-28 01:18:28.909 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7237. Jul-28 01:18:28.910 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7238. job: 4713446: RUNNING
  7239. job: 4728323: RUNNING
  7240. job: 4728326: RUNNING
  7241. job: 4728325: RUNNING
  7242. job: 4728321: RUNNING
  7243. job: 4728286: RUNNING
  7244.  
  7245. Jul-28 01:18:28.910 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7246. Jul-28 01:18:33.911 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7247. job: 4713446: RUNNING
  7248. job: 4728323: RUNNING
  7249. job: 4728326: RUNNING
  7250. job: 4728325: RUNNING
  7251. job: 4728321: RUNNING
  7252. job: 4728286: RUNNING
  7253.  
  7254. Jul-28 01:18:33.911 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7255. Jul-28 01:18:33.912 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7256. job: 4713446: RUNNING
  7257. job: 4728323: RUNNING
  7258. job: 4728326: RUNNING
  7259. job: 4728325: RUNNING
  7260. job: 4728321: RUNNING
  7261. job: 4728286: RUNNING
  7262.  
  7263. Jul-28 01:18:33.912 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7264. Jul-28 01:18:38.912 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7265. job: 4713446: RUNNING
  7266. job: 4728323: RUNNING
  7267. job: 4728326: RUNNING
  7268. job: 4728325: RUNNING
  7269. job: 4728321: RUNNING
  7270. job: 4728286: RUNNING
  7271.  
  7272. Jul-28 01:18:38.912 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7273. Jul-28 01:18:38.914 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7274. job: 4713446: RUNNING
  7275. job: 4728323: RUNNING
  7276. job: 4728326: RUNNING
  7277. job: 4728325: RUNNING
  7278. job: 4728321: RUNNING
  7279. job: 4728286: RUNNING
  7280.  
  7281. Jul-28 01:18:38.914 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7282. Jul-28 01:18:43.912 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7283. job: 4713446: RUNNING
  7284. job: 4728323: RUNNING
  7285. job: 4728326: RUNNING
  7286. job: 4728325: RUNNING
  7287. job: 4728321: RUNNING
  7288. job: 4728286: RUNNING
  7289.  
  7290. Jul-28 01:18:43.913 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7291. Jul-28 01:18:43.915 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7292. job: 4713446: RUNNING
  7293. job: 4728323: RUNNING
  7294. job: 4728326: RUNNING
  7295. job: 4728325: RUNNING
  7296. job: 4728321: RUNNING
  7297. job: 4728286: RUNNING
  7298.  
  7299. Jul-28 01:18:43.915 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7300. Jul-28 01:18:48.909 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7301. job: 4713446: RUNNING
  7302. job: 4728323: RUNNING
  7303. job: 4728326: RUNNING
  7304. job: 4728325: RUNNING
  7305. job: 4728321: RUNNING
  7306. job: 4728286: RUNNING
  7307.  
  7308. Jul-28 01:18:48.909 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7309. Jul-28 01:18:48.910 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7310. job: 4713446: RUNNING
  7311. job: 4728323: RUNNING
  7312. job: 4728326: RUNNING
  7313. job: 4728325: RUNNING
  7314. job: 4728321: RUNNING
  7315. job: 4728286: RUNNING
  7316.  
  7317. Jul-28 01:18:48.910 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7318. Jul-28 01:18:53.909 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7319. job: 4713446: RUNNING
  7320. job: 4728323: RUNNING
  7321. job: 4728326: RUNNING
  7322. job: 4728325: RUNNING
  7323. job: 4728321: RUNNING
  7324. job: 4728286: RUNNING
  7325.  
  7326. Jul-28 01:18:53.910 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7327. Jul-28 01:18:53.911 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7328. job: 4713446: RUNNING
  7329. job: 4728323: RUNNING
  7330. job: 4728326: RUNNING
  7331. job: 4728325: RUNNING
  7332. job: 4728321: RUNNING
  7333. job: 4728286: RUNNING
  7334.  
  7335. Jul-28 01:18:53.911 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7336. Jul-28 01:18:58.908 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7337. job: 4713446: RUNNING
  7338. job: 4728323: RUNNING
  7339. job: 4728326: RUNNING
  7340. job: 4728325: RUNNING
  7341. job: 4728321: RUNNING
  7342. job: 4728286: RUNNING
  7343.  
  7344. Jul-28 01:18:58.908 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7345. Jul-28 01:18:58.909 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7346. job: 4713446: RUNNING
  7347. job: 4728323: RUNNING
  7348. job: 4728326: RUNNING
  7349. job: 4728325: RUNNING
  7350. job: 4728321: RUNNING
  7351. job: 4728286: RUNNING
  7352.  
  7353. Jul-28 01:18:58.909 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7354. Jul-28 01:19:03.909 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7355. job: 4713446: RUNNING
  7356. job: 4728323: RUNNING
  7357. job: 4728326: RUNNING
  7358. job: 4728325: RUNNING
  7359. job: 4728321: RUNNING
  7360. job: 4728286: RUNNING
  7361.  
  7362. Jul-28 01:19:03.909 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7363. Jul-28 01:19:03.910 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7364. job: 4713446: RUNNING
  7365. job: 4728323: RUNNING
  7366. job: 4728326: RUNNING
  7367. job: 4728325: RUNNING
  7368. job: 4728321: RUNNING
  7369. job: 4728286: RUNNING
  7370.  
  7371. Jul-28 01:19:03.910 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7372. Jul-28 01:19:08.909 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  7373. Jul-28 01:19:09.042 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  7374. 4713446,RUN,Jul 27 11:44
  7375. 4728323,RUN,Jul 28 00:40
  7376. 4728326,RUN,Jul 28 00:51
  7377. 4728325,RUN,Jul 28 00:49
  7378. 4728321,RUN,Jul 28 00:34
  7379. 4728286,RUN,Jul 27 23:44
  7380.  
  7381.  
  7382. Jul-28 01:19:09.042 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7383. job: 4713446: RUNNING
  7384. job: 4728323: RUNNING
  7385. job: 4728326: RUNNING
  7386. job: 4728325: RUNNING
  7387. job: 4728321: RUNNING
  7388. job: 4728286: RUNNING
  7389.  
  7390. Jul-28 01:19:09.042 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7391. Jul-28 01:19:09.043 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7392. job: 4713446: RUNNING
  7393. job: 4728323: RUNNING
  7394. job: 4728326: RUNNING
  7395. job: 4728325: RUNNING
  7396. job: 4728321: RUNNING
  7397. job: 4728286: RUNNING
  7398.  
  7399. Jul-28 01:19:09.043 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7400. Jul-28 01:19:13.909 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7401. job: 4713446: RUNNING
  7402. job: 4728323: RUNNING
  7403. job: 4728326: RUNNING
  7404. job: 4728325: RUNNING
  7405. job: 4728321: RUNNING
  7406. job: 4728286: RUNNING
  7407.  
  7408. Jul-28 01:19:13.909 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7409. Jul-28 01:19:13.910 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7410. job: 4713446: RUNNING
  7411. job: 4728323: RUNNING
  7412. job: 4728326: RUNNING
  7413. job: 4728325: RUNNING
  7414. job: 4728321: RUNNING
  7415. job: 4728286: RUNNING
  7416.  
  7417. Jul-28 01:19:13.910 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7418. Jul-28 01:19:18.909 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7419. job: 4713446: RUNNING
  7420. job: 4728323: RUNNING
  7421. job: 4728326: RUNNING
  7422. job: 4728325: RUNNING
  7423. job: 4728321: RUNNING
  7424. job: 4728286: RUNNING
  7425.  
  7426. Jul-28 01:19:18.909 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7427. Jul-28 01:19:18.910 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7428. job: 4713446: RUNNING
  7429. job: 4728323: RUNNING
  7430. job: 4728326: RUNNING
  7431. job: 4728325: RUNNING
  7432. job: 4728321: RUNNING
  7433. job: 4728286: RUNNING
  7434.  
  7435. Jul-28 01:19:18.910 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7436. Jul-28 01:19:23.919 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7437. job: 4713446: RUNNING
  7438. job: 4728323: RUNNING
  7439. job: 4728326: RUNNING
  7440. job: 4728325: RUNNING
  7441. job: 4728321: RUNNING
  7442. job: 4728286: RUNNING
  7443.  
  7444. Jul-28 01:19:23.920 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7445. Jul-28 01:19:23.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7446. job: 4713446: RUNNING
  7447. job: 4728323: RUNNING
  7448. job: 4728326: RUNNING
  7449. job: 4728325: RUNNING
  7450. job: 4728321: RUNNING
  7451. job: 4728286: RUNNING
  7452.  
  7453. Jul-28 01:19:23.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7454. Jul-28 01:19:28.917 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7455. job: 4713446: RUNNING
  7456. job: 4728323: RUNNING
  7457. job: 4728326: RUNNING
  7458. job: 4728325: RUNNING
  7459. job: 4728321: RUNNING
  7460. job: 4728286: RUNNING
  7461.  
  7462. Jul-28 01:19:28.918 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7463. Jul-28 01:19:28.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7464. job: 4713446: RUNNING
  7465. job: 4728323: RUNNING
  7466. job: 4728326: RUNNING
  7467. job: 4728325: RUNNING
  7468. job: 4728321: RUNNING
  7469. job: 4728286: RUNNING
  7470.  
  7471. Jul-28 01:19:28.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7472. Jul-28 01:19:33.911 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7473. job: 4713446: RUNNING
  7474. job: 4728323: RUNNING
  7475. job: 4728326: RUNNING
  7476. job: 4728325: RUNNING
  7477. job: 4728321: RUNNING
  7478. job: 4728286: RUNNING
  7479.  
  7480. Jul-28 01:19:33.912 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7481. Jul-28 01:19:33.913 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7482. job: 4713446: RUNNING
  7483. job: 4728323: RUNNING
  7484. job: 4728326: RUNNING
  7485. job: 4728325: RUNNING
  7486. job: 4728321: RUNNING
  7487. job: 4728286: RUNNING
  7488.  
  7489. Jul-28 01:19:33.913 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7490. Jul-28 01:19:38.911 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7491. job: 4713446: RUNNING
  7492. job: 4728323: RUNNING
  7493. job: 4728326: RUNNING
  7494. job: 4728325: RUNNING
  7495. job: 4728321: RUNNING
  7496. job: 4728286: RUNNING
  7497.  
  7498. Jul-28 01:19:38.912 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7499. Jul-28 01:19:38.913 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7500. job: 4713446: RUNNING
  7501. job: 4728323: RUNNING
  7502. job: 4728326: RUNNING
  7503. job: 4728325: RUNNING
  7504. job: 4728321: RUNNING
  7505. job: 4728286: RUNNING
  7506.  
  7507. Jul-28 01:19:38.913 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7508. Jul-28 01:19:43.911 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7509. job: 4713446: RUNNING
  7510. job: 4728323: RUNNING
  7511. job: 4728326: RUNNING
  7512. job: 4728325: RUNNING
  7513. job: 4728321: RUNNING
  7514. job: 4728286: RUNNING
  7515.  
  7516. Jul-28 01:19:43.911 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7517. Jul-28 01:19:43.912 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7518. job: 4713446: RUNNING
  7519. job: 4728323: RUNNING
  7520. job: 4728326: RUNNING
  7521. job: 4728325: RUNNING
  7522. job: 4728321: RUNNING
  7523. job: 4728286: RUNNING
  7524.  
  7525. Jul-28 01:19:43.913 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7526. Jul-28 01:19:48.912 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7527. job: 4713446: RUNNING
  7528. job: 4728323: RUNNING
  7529. job: 4728326: RUNNING
  7530. job: 4728325: RUNNING
  7531. job: 4728321: RUNNING
  7532. job: 4728286: RUNNING
  7533.  
  7534. Jul-28 01:19:48.912 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7535. Jul-28 01:19:48.913 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7536. job: 4713446: RUNNING
  7537. job: 4728323: RUNNING
  7538. job: 4728326: RUNNING
  7539. job: 4728325: RUNNING
  7540. job: 4728321: RUNNING
  7541. job: 4728286: RUNNING
  7542.  
  7543. Jul-28 01:19:48.913 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7544. Jul-28 01:19:53.912 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7545. job: 4713446: RUNNING
  7546. job: 4728323: RUNNING
  7547. job: 4728326: RUNNING
  7548. job: 4728325: RUNNING
  7549. job: 4728321: RUNNING
  7550. job: 4728286: RUNNING
  7551.  
  7552. Jul-28 01:19:53.912 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7553. Jul-28 01:19:53.914 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7554. job: 4713446: RUNNING
  7555. job: 4728323: RUNNING
  7556. job: 4728326: RUNNING
  7557. job: 4728325: RUNNING
  7558. job: 4728321: RUNNING
  7559. job: 4728286: RUNNING
  7560.  
  7561. Jul-28 01:19:53.914 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7562. Jul-28 01:19:58.910 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7563. job: 4713446: RUNNING
  7564. job: 4728323: RUNNING
  7565. job: 4728326: RUNNING
  7566. job: 4728325: RUNNING
  7567. job: 4728321: RUNNING
  7568. job: 4728286: RUNNING
  7569.  
  7570. Jul-28 01:19:58.910 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7571. Jul-28 01:19:58.911 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7572. job: 4713446: RUNNING
  7573. job: 4728323: RUNNING
  7574. job: 4728326: RUNNING
  7575. job: 4728325: RUNNING
  7576. job: 4728321: RUNNING
  7577. job: 4728286: RUNNING
  7578.  
  7579. Jul-28 01:19:58.911 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7580. Jul-28 01:20:03.909 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7581. job: 4713446: RUNNING
  7582. job: 4728323: RUNNING
  7583. job: 4728326: RUNNING
  7584. job: 4728325: RUNNING
  7585. job: 4728321: RUNNING
  7586. job: 4728286: RUNNING
  7587.  
  7588. Jul-28 01:20:03.909 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7589. Jul-28 01:20:03.910 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7590. job: 4713446: RUNNING
  7591. job: 4728323: RUNNING
  7592. job: 4728326: RUNNING
  7593. job: 4728325: RUNNING
  7594. job: 4728321: RUNNING
  7595. job: 4728286: RUNNING
  7596.  
  7597. Jul-28 01:20:03.910 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7598. Jul-28 01:20:08.910 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor lsf > tasks to be completed: 2 -- pending tasks are shown below
  7599. ~> TaskHandler[jobId: 4728325; id: 152; name: merge_pairsam_into_runs (library:HeLa1 run:lane1); status: RUNNING; exit: -; error: -; workDir: /farline/umw_job_dekker/HPCC/sv49w/distiller-nf/work/3b/37ed2fa2ba160f54196e762e24c83b started: 1501217508853; exited: -; ]
  7600. ~> TaskHandler[jobId: 4728326; id: 153; name: merge_pairsam_into_runs (library:HeLa1 run:lane3); status: RUNNING; exit: -; error: -; workDir: /farline/umw_job_dekker/HPCC/sv49w/distiller-nf/work/00/2633edc9da9ad5649d3d6dcffe5a44 started: 1501217938848; exited: -; ]
  7601. Jul-28 01:20:08.910 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  7602. Jul-28 01:20:09.039 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  7603. 4713446,RUN,Jul 27 11:44
  7604. 4728323,RUN,Jul 28 00:40
  7605. 4728326,RUN,Jul 28 00:51
  7606. 4728325,RUN,Jul 28 00:49
  7607. 4728321,RUN,Jul 28 00:34
  7608. 4728286,RUN,Jul 27 23:44
  7609.  
  7610.  
  7611. Jul-28 01:20:09.039 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7612. job: 4713446: RUNNING
  7613. job: 4728323: RUNNING
  7614. job: 4728326: RUNNING
  7615. job: 4728325: RUNNING
  7616. job: 4728321: RUNNING
  7617. job: 4728286: RUNNING
  7618.  
  7619. Jul-28 01:20:09.039 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7620. Jul-28 01:20:09.040 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7621. job: 4713446: RUNNING
  7622. job: 4728323: RUNNING
  7623. job: 4728326: RUNNING
  7624. job: 4728325: RUNNING
  7625. job: 4728321: RUNNING
  7626. job: 4728286: RUNNING
  7627.  
  7628. Jul-28 01:20:09.040 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7629. Jul-28 01:20:13.912 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7630. job: 4713446: RUNNING
  7631. job: 4728323: RUNNING
  7632. job: 4728326: RUNNING
  7633. job: 4728325: RUNNING
  7634. job: 4728321: RUNNING
  7635. job: 4728286: RUNNING
  7636.  
  7637. Jul-28 01:20:13.912 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7638. Jul-28 01:20:13.913 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7639. job: 4713446: RUNNING
  7640. job: 4728323: RUNNING
  7641. job: 4728326: RUNNING
  7642. job: 4728325: RUNNING
  7643. job: 4728321: RUNNING
  7644. job: 4728286: RUNNING
  7645.  
  7646. Jul-28 01:20:13.913 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7647. Jul-28 01:20:18.913 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7648. job: 4713446: RUNNING
  7649. job: 4728323: RUNNING
  7650. job: 4728326: RUNNING
  7651. job: 4728325: RUNNING
  7652. job: 4728321: RUNNING
  7653. job: 4728286: RUNNING
  7654.  
  7655. Jul-28 01:20:18.913 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7656. Jul-28 01:20:18.914 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7657. job: 4713446: RUNNING
  7658. job: 4728323: RUNNING
  7659. job: 4728326: RUNNING
  7660. job: 4728325: RUNNING
  7661. job: 4728321: RUNNING
  7662. job: 4728286: RUNNING
  7663.  
  7664. Jul-28 01:20:18.914 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7665. Jul-28 01:20:23.912 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7666. job: 4713446: RUNNING
  7667. job: 4728323: RUNNING
  7668. job: 4728326: RUNNING
  7669. job: 4728325: RUNNING
  7670. job: 4728321: RUNNING
  7671. job: 4728286: RUNNING
  7672.  
  7673. Jul-28 01:20:23.913 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7674. Jul-28 01:20:23.913 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7675. job: 4713446: RUNNING
  7676. job: 4728323: RUNNING
  7677. job: 4728326: RUNNING
  7678. job: 4728325: RUNNING
  7679. job: 4728321: RUNNING
  7680. job: 4728286: RUNNING
  7681.  
  7682. Jul-28 01:20:23.913 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7683. Jul-28 01:20:28.914 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7684. job: 4713446: RUNNING
  7685. job: 4728323: RUNNING
  7686. job: 4728326: RUNNING
  7687. job: 4728325: RUNNING
  7688. job: 4728321: RUNNING
  7689. job: 4728286: RUNNING
  7690.  
  7691. Jul-28 01:20:28.914 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7692. Jul-28 01:20:28.915 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7693. job: 4713446: RUNNING
  7694. job: 4728323: RUNNING
  7695. job: 4728326: RUNNING
  7696. job: 4728325: RUNNING
  7697. job: 4728321: RUNNING
  7698. job: 4728286: RUNNING
  7699.  
  7700. Jul-28 01:20:28.915 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7701. Jul-28 01:20:33.918 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7702. job: 4713446: RUNNING
  7703. job: 4728323: RUNNING
  7704. job: 4728326: RUNNING
  7705. job: 4728325: RUNNING
  7706. job: 4728321: RUNNING
  7707. job: 4728286: RUNNING
  7708.  
  7709. Jul-28 01:20:33.919 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7710. Jul-28 01:20:33.920 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7711. job: 4713446: RUNNING
  7712. job: 4728323: RUNNING
  7713. job: 4728326: RUNNING
  7714. job: 4728325: RUNNING
  7715. job: 4728321: RUNNING
  7716. job: 4728286: RUNNING
  7717.  
  7718. Jul-28 01:20:33.920 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7719. Jul-28 01:20:38.917 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7720. job: 4713446: RUNNING
  7721. job: 4728323: RUNNING
  7722. job: 4728326: RUNNING
  7723. job: 4728325: RUNNING
  7724. job: 4728321: RUNNING
  7725. job: 4728286: RUNNING
  7726.  
  7727. Jul-28 01:20:38.918 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7728. Jul-28 01:20:38.920 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7729. job: 4713446: RUNNING
  7730. job: 4728323: RUNNING
  7731. job: 4728326: RUNNING
  7732. job: 4728325: RUNNING
  7733. job: 4728321: RUNNING
  7734. job: 4728286: RUNNING
  7735.  
  7736. Jul-28 01:20:38.920 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7737. Jul-28 01:20:43.916 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7738. job: 4713446: RUNNING
  7739. job: 4728323: RUNNING
  7740. job: 4728326: RUNNING
  7741. job: 4728325: RUNNING
  7742. job: 4728321: RUNNING
  7743. job: 4728286: RUNNING
  7744.  
  7745. Jul-28 01:20:43.917 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7746. Jul-28 01:20:43.918 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7747. job: 4713446: RUNNING
  7748. job: 4728323: RUNNING
  7749. job: 4728326: RUNNING
  7750. job: 4728325: RUNNING
  7751. job: 4728321: RUNNING
  7752. job: 4728286: RUNNING
  7753.  
  7754. Jul-28 01:20:43.918 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7755. Jul-28 01:20:48.915 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7756. job: 4713446: RUNNING
  7757. job: 4728323: RUNNING
  7758. job: 4728326: RUNNING
  7759. job: 4728325: RUNNING
  7760. job: 4728321: RUNNING
  7761. job: 4728286: RUNNING
  7762.  
  7763. Jul-28 01:20:48.915 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7764. Jul-28 01:20:48.917 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7765. job: 4713446: RUNNING
  7766. job: 4728323: RUNNING
  7767. job: 4728326: RUNNING
  7768. job: 4728325: RUNNING
  7769. job: 4728321: RUNNING
  7770. job: 4728286: RUNNING
  7771.  
  7772. Jul-28 01:20:48.917 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7773. Jul-28 01:20:53.914 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7774. job: 4713446: RUNNING
  7775. job: 4728323: RUNNING
  7776. job: 4728326: RUNNING
  7777. job: 4728325: RUNNING
  7778. job: 4728321: RUNNING
  7779. job: 4728286: RUNNING
  7780.  
  7781. Jul-28 01:20:53.914 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7782. Jul-28 01:20:53.915 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7783. job: 4713446: RUNNING
  7784. job: 4728323: RUNNING
  7785. job: 4728326: RUNNING
  7786. job: 4728325: RUNNING
  7787. job: 4728321: RUNNING
  7788. job: 4728286: RUNNING
  7789.  
  7790. Jul-28 01:20:53.915 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7791. Jul-28 01:20:58.915 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7792. job: 4713446: RUNNING
  7793. job: 4728323: RUNNING
  7794. job: 4728326: RUNNING
  7795. job: 4728325: RUNNING
  7796. job: 4728321: RUNNING
  7797. job: 4728286: RUNNING
  7798.  
  7799. Jul-28 01:20:58.915 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7800. Jul-28 01:20:58.916 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7801. job: 4713446: RUNNING
  7802. job: 4728323: RUNNING
  7803. job: 4728326: RUNNING
  7804. job: 4728325: RUNNING
  7805. job: 4728321: RUNNING
  7806. job: 4728286: RUNNING
  7807.  
  7808. Jul-28 01:20:58.916 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7809. Jul-28 01:21:03.914 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7810. job: 4713446: RUNNING
  7811. job: 4728323: RUNNING
  7812. job: 4728326: RUNNING
  7813. job: 4728325: RUNNING
  7814. job: 4728321: RUNNING
  7815. job: 4728286: RUNNING
  7816.  
  7817. Jul-28 01:21:03.914 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7818. Jul-28 01:21:03.915 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7819. job: 4713446: RUNNING
  7820. job: 4728323: RUNNING
  7821. job: 4728326: RUNNING
  7822. job: 4728325: RUNNING
  7823. job: 4728321: RUNNING
  7824. job: 4728286: RUNNING
  7825.  
  7826. Jul-28 01:21:03.915 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7827. Jul-28 01:21:08.914 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  7828. Jul-28 01:21:09.048 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  7829. 4713446,RUN,Jul 27 11:44
  7830. 4728323,RUN,Jul 28 00:40
  7831. 4728326,RUN,Jul 28 00:51
  7832. 4728325,RUN,Jul 28 00:49
  7833. 4728321,RUN,Jul 28 00:34
  7834. 4728286,RUN,Jul 27 23:44
  7835.  
  7836.  
  7837. Jul-28 01:21:09.048 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7838. job: 4713446: RUNNING
  7839. job: 4728323: RUNNING
  7840. job: 4728326: RUNNING
  7841. job: 4728325: RUNNING
  7842. job: 4728321: RUNNING
  7843. job: 4728286: RUNNING
  7844.  
  7845. Jul-28 01:21:09.048 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7846. Jul-28 01:21:09.049 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7847. job: 4713446: RUNNING
  7848. job: 4728323: RUNNING
  7849. job: 4728326: RUNNING
  7850. job: 4728325: RUNNING
  7851. job: 4728321: RUNNING
  7852. job: 4728286: RUNNING
  7853.  
  7854. Jul-28 01:21:09.049 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7855. Jul-28 01:21:13.914 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7856. job: 4713446: RUNNING
  7857. job: 4728323: RUNNING
  7858. job: 4728326: RUNNING
  7859. job: 4728325: RUNNING
  7860. job: 4728321: RUNNING
  7861. job: 4728286: RUNNING
  7862.  
  7863. Jul-28 01:21:13.915 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7864. Jul-28 01:21:13.916 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7865. job: 4713446: RUNNING
  7866. job: 4728323: RUNNING
  7867. job: 4728326: RUNNING
  7868. job: 4728325: RUNNING
  7869. job: 4728321: RUNNING
  7870. job: 4728286: RUNNING
  7871.  
  7872. Jul-28 01:21:13.916 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7873. Jul-28 01:21:18.913 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7874. job: 4713446: RUNNING
  7875. job: 4728323: RUNNING
  7876. job: 4728326: RUNNING
  7877. job: 4728325: RUNNING
  7878. job: 4728321: RUNNING
  7879. job: 4728286: RUNNING
  7880.  
  7881. Jul-28 01:21:18.913 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7882. Jul-28 01:21:18.914 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7883. job: 4713446: RUNNING
  7884. job: 4728323: RUNNING
  7885. job: 4728326: RUNNING
  7886. job: 4728325: RUNNING
  7887. job: 4728321: RUNNING
  7888. job: 4728286: RUNNING
  7889.  
  7890. Jul-28 01:21:18.914 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7891. Jul-28 01:21:23.914 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7892. job: 4713446: RUNNING
  7893. job: 4728323: RUNNING
  7894. job: 4728326: RUNNING
  7895. job: 4728325: RUNNING
  7896. job: 4728321: RUNNING
  7897. job: 4728286: RUNNING
  7898.  
  7899. Jul-28 01:21:23.914 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7900. Jul-28 01:21:23.915 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7901. job: 4713446: RUNNING
  7902. job: 4728323: RUNNING
  7903. job: 4728326: RUNNING
  7904. job: 4728325: RUNNING
  7905. job: 4728321: RUNNING
  7906. job: 4728286: RUNNING
  7907.  
  7908. Jul-28 01:21:23.915 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7909. Jul-28 01:21:28.915 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7910. job: 4713446: RUNNING
  7911. job: 4728323: RUNNING
  7912. job: 4728326: RUNNING
  7913. job: 4728325: RUNNING
  7914. job: 4728321: RUNNING
  7915. job: 4728286: RUNNING
  7916.  
  7917. Jul-28 01:21:28.915 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7918. Jul-28 01:21:28.916 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7919. job: 4713446: RUNNING
  7920. job: 4728323: RUNNING
  7921. job: 4728326: RUNNING
  7922. job: 4728325: RUNNING
  7923. job: 4728321: RUNNING
  7924. job: 4728286: RUNNING
  7925.  
  7926. Jul-28 01:21:28.916 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7927. Jul-28 01:21:33.915 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7928. job: 4713446: RUNNING
  7929. job: 4728323: RUNNING
  7930. job: 4728326: RUNNING
  7931. job: 4728325: RUNNING
  7932. job: 4728321: RUNNING
  7933. job: 4728286: RUNNING
  7934.  
  7935. Jul-28 01:21:33.915 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7936. Jul-28 01:21:33.916 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7937. job: 4713446: RUNNING
  7938. job: 4728323: RUNNING
  7939. job: 4728326: RUNNING
  7940. job: 4728325: RUNNING
  7941. job: 4728321: RUNNING
  7942. job: 4728286: RUNNING
  7943.  
  7944. Jul-28 01:21:33.916 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7945. Jul-28 01:21:38.915 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7946. job: 4713446: RUNNING
  7947. job: 4728323: RUNNING
  7948. job: 4728326: RUNNING
  7949. job: 4728325: RUNNING
  7950. job: 4728321: RUNNING
  7951. job: 4728286: RUNNING
  7952.  
  7953. Jul-28 01:21:38.915 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7954. Jul-28 01:21:38.915 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7955. job: 4713446: RUNNING
  7956. job: 4728323: RUNNING
  7957. job: 4728326: RUNNING
  7958. job: 4728325: RUNNING
  7959. job: 4728321: RUNNING
  7960. job: 4728286: RUNNING
  7961.  
  7962. Jul-28 01:21:38.915 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7963. Jul-28 01:21:43.913 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7964. job: 4713446: RUNNING
  7965. job: 4728323: RUNNING
  7966. job: 4728326: RUNNING
  7967. job: 4728325: RUNNING
  7968. job: 4728321: RUNNING
  7969. job: 4728286: RUNNING
  7970.  
  7971. Jul-28 01:21:43.913 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7972. Jul-28 01:21:43.914 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7973. job: 4713446: RUNNING
  7974. job: 4728323: RUNNING
  7975. job: 4728326: RUNNING
  7976. job: 4728325: RUNNING
  7977. job: 4728321: RUNNING
  7978. job: 4728286: RUNNING
  7979.  
  7980. Jul-28 01:21:43.914 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7981. Jul-28 01:21:48.918 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7982. job: 4713446: RUNNING
  7983. job: 4728323: RUNNING
  7984. job: 4728326: RUNNING
  7985. job: 4728325: RUNNING
  7986. job: 4728321: RUNNING
  7987. job: 4728286: RUNNING
  7988.  
  7989. Jul-28 01:21:48.918 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  7990. Jul-28 01:21:48.920 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  7991. job: 4713446: RUNNING
  7992. job: 4728323: RUNNING
  7993. job: 4728326: RUNNING
  7994. job: 4728325: RUNNING
  7995. job: 4728321: RUNNING
  7996. job: 4728286: RUNNING
  7997.  
  7998. Jul-28 01:21:48.920 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  7999. Jul-28 01:21:53.917 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8000. job: 4713446: RUNNING
  8001. job: 4728323: RUNNING
  8002. job: 4728326: RUNNING
  8003. job: 4728325: RUNNING
  8004. job: 4728321: RUNNING
  8005. job: 4728286: RUNNING
  8006.  
  8007. Jul-28 01:21:53.918 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8008. Jul-28 01:21:53.919 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8009. job: 4713446: RUNNING
  8010. job: 4728323: RUNNING
  8011. job: 4728326: RUNNING
  8012. job: 4728325: RUNNING
  8013. job: 4728321: RUNNING
  8014. job: 4728286: RUNNING
  8015.  
  8016. Jul-28 01:21:53.919 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8017. Jul-28 01:21:58.920 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8018. job: 4713446: RUNNING
  8019. job: 4728323: RUNNING
  8020. job: 4728326: RUNNING
  8021. job: 4728325: RUNNING
  8022. job: 4728321: RUNNING
  8023. job: 4728286: RUNNING
  8024.  
  8025. Jul-28 01:21:58.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8026. Jul-28 01:21:58.923 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8027. job: 4713446: RUNNING
  8028. job: 4728323: RUNNING
  8029. job: 4728326: RUNNING
  8030. job: 4728325: RUNNING
  8031. job: 4728321: RUNNING
  8032. job: 4728286: RUNNING
  8033.  
  8034. Jul-28 01:21:58.923 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8035. Jul-28 01:22:03.916 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8036. job: 4713446: RUNNING
  8037. job: 4728323: RUNNING
  8038. job: 4728326: RUNNING
  8039. job: 4728325: RUNNING
  8040. job: 4728321: RUNNING
  8041. job: 4728286: RUNNING
  8042.  
  8043. Jul-28 01:22:03.917 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8044. Jul-28 01:22:03.918 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8045. job: 4713446: RUNNING
  8046. job: 4728323: RUNNING
  8047. job: 4728326: RUNNING
  8048. job: 4728325: RUNNING
  8049. job: 4728321: RUNNING
  8050. job: 4728286: RUNNING
  8051.  
  8052. Jul-28 01:22:03.918 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8053. Jul-28 01:22:08.916 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  8054. Jul-28 01:22:09.053 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  8055. 4713446,RUN,Jul 27 11:44
  8056. 4728323,RUN,Jul 28 00:40
  8057. 4728326,RUN,Jul 28 00:51
  8058. 4728325,RUN,Jul 28 00:49
  8059. 4728321,RUN,Jul 28 00:34
  8060. 4728286,RUN,Jul 27 23:44
  8061.  
  8062.  
  8063. Jul-28 01:22:09.053 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8064. job: 4713446: RUNNING
  8065. job: 4728323: RUNNING
  8066. job: 4728326: RUNNING
  8067. job: 4728325: RUNNING
  8068. job: 4728321: RUNNING
  8069. job: 4728286: RUNNING
  8070.  
  8071. Jul-28 01:22:09.053 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8072. Jul-28 01:22:09.055 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8073. job: 4713446: RUNNING
  8074. job: 4728323: RUNNING
  8075. job: 4728326: RUNNING
  8076. job: 4728325: RUNNING
  8077. job: 4728321: RUNNING
  8078. job: 4728286: RUNNING
  8079.  
  8080. Jul-28 01:22:09.055 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8081. Jul-28 01:22:13.916 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8082. job: 4713446: RUNNING
  8083. job: 4728323: RUNNING
  8084. job: 4728326: RUNNING
  8085. job: 4728325: RUNNING
  8086. job: 4728321: RUNNING
  8087. job: 4728286: RUNNING
  8088.  
  8089. Jul-28 01:22:13.916 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8090. Jul-28 01:22:13.917 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8091. job: 4713446: RUNNING
  8092. job: 4728323: RUNNING
  8093. job: 4728326: RUNNING
  8094. job: 4728325: RUNNING
  8095. job: 4728321: RUNNING
  8096. job: 4728286: RUNNING
  8097.  
  8098. Jul-28 01:22:13.917 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8099. Jul-28 01:22:18.916 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8100. job: 4713446: RUNNING
  8101. job: 4728323: RUNNING
  8102. job: 4728326: RUNNING
  8103. job: 4728325: RUNNING
  8104. job: 4728321: RUNNING
  8105. job: 4728286: RUNNING
  8106.  
  8107. Jul-28 01:22:18.917 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8108. Jul-28 01:22:18.917 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8109. job: 4713446: RUNNING
  8110. job: 4728323: RUNNING
  8111. job: 4728326: RUNNING
  8112. job: 4728325: RUNNING
  8113. job: 4728321: RUNNING
  8114. job: 4728286: RUNNING
  8115.  
  8116. Jul-28 01:22:18.917 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8117. Jul-28 01:22:23.917 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8118. job: 4713446: RUNNING
  8119. job: 4728323: RUNNING
  8120. job: 4728326: RUNNING
  8121. job: 4728325: RUNNING
  8122. job: 4728321: RUNNING
  8123. job: 4728286: RUNNING
  8124.  
  8125. Jul-28 01:22:23.918 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8126. Jul-28 01:22:23.918 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8127. job: 4713446: RUNNING
  8128. job: 4728323: RUNNING
  8129. job: 4728326: RUNNING
  8130. job: 4728325: RUNNING
  8131. job: 4728321: RUNNING
  8132. job: 4728286: RUNNING
  8133.  
  8134. Jul-28 01:22:23.918 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8135. Jul-28 01:22:28.918 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8136. job: 4713446: RUNNING
  8137. job: 4728323: RUNNING
  8138. job: 4728326: RUNNING
  8139. job: 4728325: RUNNING
  8140. job: 4728321: RUNNING
  8141. job: 4728286: RUNNING
  8142.  
  8143. Jul-28 01:22:28.919 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8144. Jul-28 01:22:28.919 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8145. job: 4713446: RUNNING
  8146. job: 4728323: RUNNING
  8147. job: 4728326: RUNNING
  8148. job: 4728325: RUNNING
  8149. job: 4728321: RUNNING
  8150. job: 4728286: RUNNING
  8151.  
  8152. Jul-28 01:22:28.919 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8153. Jul-28 01:22:33.922 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8154. job: 4713446: RUNNING
  8155. job: 4728323: RUNNING
  8156. job: 4728326: RUNNING
  8157. job: 4728325: RUNNING
  8158. job: 4728321: RUNNING
  8159. job: 4728286: RUNNING
  8160.  
  8161. Jul-28 01:22:33.923 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8162. Jul-28 01:22:33.924 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8163. job: 4713446: RUNNING
  8164. job: 4728323: RUNNING
  8165. job: 4728326: RUNNING
  8166. job: 4728325: RUNNING
  8167. job: 4728321: RUNNING
  8168. job: 4728286: RUNNING
  8169.  
  8170. Jul-28 01:22:33.924 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8171. Jul-28 01:22:38.920 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8172. job: 4713446: RUNNING
  8173. job: 4728323: RUNNING
  8174. job: 4728326: RUNNING
  8175. job: 4728325: RUNNING
  8176. job: 4728321: RUNNING
  8177. job: 4728286: RUNNING
  8178.  
  8179. Jul-28 01:22:38.920 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8180. Jul-28 01:22:38.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8181. job: 4713446: RUNNING
  8182. job: 4728323: RUNNING
  8183. job: 4728326: RUNNING
  8184. job: 4728325: RUNNING
  8185. job: 4728321: RUNNING
  8186. job: 4728286: RUNNING
  8187.  
  8188. Jul-28 01:22:38.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8189. Jul-28 01:22:43.920 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8190. job: 4713446: RUNNING
  8191. job: 4728323: RUNNING
  8192. job: 4728326: RUNNING
  8193. job: 4728325: RUNNING
  8194. job: 4728321: RUNNING
  8195. job: 4728286: RUNNING
  8196.  
  8197. Jul-28 01:22:43.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8198. Jul-28 01:22:43.922 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8199. job: 4713446: RUNNING
  8200. job: 4728323: RUNNING
  8201. job: 4728326: RUNNING
  8202. job: 4728325: RUNNING
  8203. job: 4728321: RUNNING
  8204. job: 4728286: RUNNING
  8205.  
  8206. Jul-28 01:22:43.922 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8207. Jul-28 01:22:48.920 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8208. job: 4713446: RUNNING
  8209. job: 4728323: RUNNING
  8210. job: 4728326: RUNNING
  8211. job: 4728325: RUNNING
  8212. job: 4728321: RUNNING
  8213. job: 4728286: RUNNING
  8214.  
  8215. Jul-28 01:22:48.920 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8216. Jul-28 01:22:48.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8217. job: 4713446: RUNNING
  8218. job: 4728323: RUNNING
  8219. job: 4728326: RUNNING
  8220. job: 4728325: RUNNING
  8221. job: 4728321: RUNNING
  8222. job: 4728286: RUNNING
  8223.  
  8224. Jul-28 01:22:48.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8225. Jul-28 01:22:53.920 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8226. job: 4713446: RUNNING
  8227. job: 4728323: RUNNING
  8228. job: 4728326: RUNNING
  8229. job: 4728325: RUNNING
  8230. job: 4728321: RUNNING
  8231. job: 4728286: RUNNING
  8232.  
  8233. Jul-28 01:22:53.920 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8234. Jul-28 01:22:53.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8235. job: 4713446: RUNNING
  8236. job: 4728323: RUNNING
  8237. job: 4728326: RUNNING
  8238. job: 4728325: RUNNING
  8239. job: 4728321: RUNNING
  8240. job: 4728286: RUNNING
  8241.  
  8242. Jul-28 01:22:53.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8243. Jul-28 01:22:58.923 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8244. job: 4713446: RUNNING
  8245. job: 4728323: RUNNING
  8246. job: 4728326: RUNNING
  8247. job: 4728325: RUNNING
  8248. job: 4728321: RUNNING
  8249. job: 4728286: RUNNING
  8250.  
  8251. Jul-28 01:22:58.923 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8252. Jul-28 01:22:58.925 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8253. job: 4713446: RUNNING
  8254. job: 4728323: RUNNING
  8255. job: 4728326: RUNNING
  8256. job: 4728325: RUNNING
  8257. job: 4728321: RUNNING
  8258. job: 4728286: RUNNING
  8259.  
  8260. Jul-28 01:22:58.925 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8261. Jul-28 01:23:03.923 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8262. job: 4713446: RUNNING
  8263. job: 4728323: RUNNING
  8264. job: 4728326: RUNNING
  8265. job: 4728325: RUNNING
  8266. job: 4728321: RUNNING
  8267. job: 4728286: RUNNING
  8268.  
  8269. Jul-28 01:23:03.924 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8270. Jul-28 01:23:03.925 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8271. job: 4713446: RUNNING
  8272. job: 4728323: RUNNING
  8273. job: 4728326: RUNNING
  8274. job: 4728325: RUNNING
  8275. job: 4728321: RUNNING
  8276. job: 4728286: RUNNING
  8277.  
  8278. Jul-28 01:23:03.925 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8279. Jul-28 01:23:08.920 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  8280. Jul-28 01:23:09.051 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  8281. 4713446,RUN,Jul 27 11:44
  8282. 4728323,RUN,Jul 28 00:40
  8283. 4728326,RUN,Jul 28 00:51
  8284. 4728325,RUN,Jul 28 00:49
  8285. 4728321,RUN,Jul 28 00:34
  8286. 4728286,RUN,Jul 27 23:44
  8287.  
  8288.  
  8289. Jul-28 01:23:09.052 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8290. job: 4713446: RUNNING
  8291. job: 4728323: RUNNING
  8292. job: 4728326: RUNNING
  8293. job: 4728325: RUNNING
  8294. job: 4728321: RUNNING
  8295. job: 4728286: RUNNING
  8296.  
  8297. Jul-28 01:23:09.052 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8298. Jul-28 01:23:09.053 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8299. job: 4713446: RUNNING
  8300. job: 4728323: RUNNING
  8301. job: 4728326: RUNNING
  8302. job: 4728325: RUNNING
  8303. job: 4728321: RUNNING
  8304. job: 4728286: RUNNING
  8305.  
  8306. Jul-28 01:23:09.053 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8307. Jul-28 01:23:13.920 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8308. job: 4713446: RUNNING
  8309. job: 4728323: RUNNING
  8310. job: 4728326: RUNNING
  8311. job: 4728325: RUNNING
  8312. job: 4728321: RUNNING
  8313. job: 4728286: RUNNING
  8314.  
  8315. Jul-28 01:23:13.920 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8316. Jul-28 01:23:13.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8317. job: 4713446: RUNNING
  8318. job: 4728323: RUNNING
  8319. job: 4728326: RUNNING
  8320. job: 4728325: RUNNING
  8321. job: 4728321: RUNNING
  8322. job: 4728286: RUNNING
  8323.  
  8324. Jul-28 01:23:13.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8325. Jul-28 01:23:18.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8326. job: 4713446: RUNNING
  8327. job: 4728323: RUNNING
  8328. job: 4728326: RUNNING
  8329. job: 4728325: RUNNING
  8330. job: 4728321: RUNNING
  8331. job: 4728286: RUNNING
  8332.  
  8333. Jul-28 01:23:18.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8334. Jul-28 01:23:18.922 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8335. job: 4713446: RUNNING
  8336. job: 4728323: RUNNING
  8337. job: 4728326: RUNNING
  8338. job: 4728325: RUNNING
  8339. job: 4728321: RUNNING
  8340. job: 4728286: RUNNING
  8341.  
  8342. Jul-28 01:23:18.922 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8343. Jul-28 01:23:23.920 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8344. job: 4713446: RUNNING
  8345. job: 4728323: RUNNING
  8346. job: 4728326: RUNNING
  8347. job: 4728325: RUNNING
  8348. job: 4728321: RUNNING
  8349. job: 4728286: RUNNING
  8350.  
  8351. Jul-28 01:23:23.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8352. Jul-28 01:23:23.922 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8353. job: 4713446: RUNNING
  8354. job: 4728323: RUNNING
  8355. job: 4728326: RUNNING
  8356. job: 4728325: RUNNING
  8357. job: 4728321: RUNNING
  8358. job: 4728286: RUNNING
  8359.  
  8360. Jul-28 01:23:23.922 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8361. Jul-28 01:23:28.920 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8362. job: 4713446: RUNNING
  8363. job: 4728323: RUNNING
  8364. job: 4728326: RUNNING
  8365. job: 4728325: RUNNING
  8366. job: 4728321: RUNNING
  8367. job: 4728286: RUNNING
  8368.  
  8369. Jul-28 01:23:28.920 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8370. Jul-28 01:23:28.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8371. job: 4713446: RUNNING
  8372. job: 4728323: RUNNING
  8373. job: 4728326: RUNNING
  8374. job: 4728325: RUNNING
  8375. job: 4728321: RUNNING
  8376. job: 4728286: RUNNING
  8377.  
  8378. Jul-28 01:23:28.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8379. Jul-28 01:23:33.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8380. job: 4713446: RUNNING
  8381. job: 4728323: RUNNING
  8382. job: 4728326: RUNNING
  8383. job: 4728325: RUNNING
  8384. job: 4728321: RUNNING
  8385. job: 4728286: RUNNING
  8386.  
  8387. Jul-28 01:23:33.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8388. Jul-28 01:23:33.922 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8389. job: 4713446: RUNNING
  8390. job: 4728323: RUNNING
  8391. job: 4728326: RUNNING
  8392. job: 4728325: RUNNING
  8393. job: 4728321: RUNNING
  8394. job: 4728286: RUNNING
  8395.  
  8396. Jul-28 01:23:33.922 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8397. Jul-28 01:23:38.920 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8398. job: 4713446: RUNNING
  8399. job: 4728323: RUNNING
  8400. job: 4728326: RUNNING
  8401. job: 4728325: RUNNING
  8402. job: 4728321: RUNNING
  8403. job: 4728286: RUNNING
  8404.  
  8405. Jul-28 01:23:38.920 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8406. Jul-28 01:23:38.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8407. job: 4713446: RUNNING
  8408. job: 4728323: RUNNING
  8409. job: 4728326: RUNNING
  8410. job: 4728325: RUNNING
  8411. job: 4728321: RUNNING
  8412. job: 4728286: RUNNING
  8413.  
  8414. Jul-28 01:23:38.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8415. Jul-28 01:23:43.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8416. job: 4713446: RUNNING
  8417. job: 4728323: RUNNING
  8418. job: 4728326: RUNNING
  8419. job: 4728325: RUNNING
  8420. job: 4728321: RUNNING
  8421. job: 4728286: RUNNING
  8422.  
  8423. Jul-28 01:23:43.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8424. Jul-28 01:23:43.922 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8425. job: 4713446: RUNNING
  8426. job: 4728323: RUNNING
  8427. job: 4728326: RUNNING
  8428. job: 4728325: RUNNING
  8429. job: 4728321: RUNNING
  8430. job: 4728286: RUNNING
  8431.  
  8432. Jul-28 01:23:43.922 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8433. Jul-28 01:23:48.920 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8434. job: 4713446: RUNNING
  8435. job: 4728323: RUNNING
  8436. job: 4728326: RUNNING
  8437. job: 4728325: RUNNING
  8438. job: 4728321: RUNNING
  8439. job: 4728286: RUNNING
  8440.  
  8441. Jul-28 01:23:48.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8442. Jul-28 01:23:48.922 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8443. job: 4713446: RUNNING
  8444. job: 4728323: RUNNING
  8445. job: 4728326: RUNNING
  8446. job: 4728325: RUNNING
  8447. job: 4728321: RUNNING
  8448. job: 4728286: RUNNING
  8449.  
  8450. Jul-28 01:23:48.922 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8451. Jul-28 01:23:53.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8452. job: 4713446: RUNNING
  8453. job: 4728323: RUNNING
  8454. job: 4728326: RUNNING
  8455. job: 4728325: RUNNING
  8456. job: 4728321: RUNNING
  8457. job: 4728286: RUNNING
  8458.  
  8459. Jul-28 01:23:53.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8460. Jul-28 01:23:53.922 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8461. job: 4713446: RUNNING
  8462. job: 4728323: RUNNING
  8463. job: 4728326: RUNNING
  8464. job: 4728325: RUNNING
  8465. job: 4728321: RUNNING
  8466. job: 4728286: RUNNING
  8467.  
  8468. Jul-28 01:23:53.922 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8469. Jul-28 01:23:58.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8470. job: 4713446: RUNNING
  8471. job: 4728323: RUNNING
  8472. job: 4728326: RUNNING
  8473. job: 4728325: RUNNING
  8474. job: 4728321: RUNNING
  8475. job: 4728286: RUNNING
  8476.  
  8477. Jul-28 01:23:58.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8478. Jul-28 01:23:58.922 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8479. job: 4713446: RUNNING
  8480. job: 4728323: RUNNING
  8481. job: 4728326: RUNNING
  8482. job: 4728325: RUNNING
  8483. job: 4728321: RUNNING
  8484. job: 4728286: RUNNING
  8485.  
  8486. Jul-28 01:23:58.922 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8487. Jul-28 01:24:03.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8488. job: 4713446: RUNNING
  8489. job: 4728323: RUNNING
  8490. job: 4728326: RUNNING
  8491. job: 4728325: RUNNING
  8492. job: 4728321: RUNNING
  8493. job: 4728286: RUNNING
  8494.  
  8495. Jul-28 01:24:03.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8496. Jul-28 01:24:03.922 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8497. job: 4713446: RUNNING
  8498. job: 4728323: RUNNING
  8499. job: 4728326: RUNNING
  8500. job: 4728325: RUNNING
  8501. job: 4728321: RUNNING
  8502. job: 4728286: RUNNING
  8503.  
  8504. Jul-28 01:24:03.922 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8505. Jul-28 01:24:08.920 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8506. job: 4713446: RUNNING
  8507. job: 4728323: RUNNING
  8508. job: 4728326: RUNNING
  8509. job: 4728325: RUNNING
  8510. job: 4728321: RUNNING
  8511. job: 4728286: RUNNING
  8512.  
  8513. Jul-28 01:24:08.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8514. Jul-28 01:24:08.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  8515. Jul-28 01:24:09.044 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  8516. 4713446,RUN,Jul 27 11:44
  8517. 4728323,RUN,Jul 28 00:40
  8518. 4728326,RUN,Jul 28 00:51
  8519. 4728325,RUN,Jul 28 00:49
  8520. 4728321,RUN,Jul 28 00:34
  8521. 4728286,RUN,Jul 27 23:44
  8522.  
  8523.  
  8524. Jul-28 01:24:09.044 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8525. job: 4713446: RUNNING
  8526. job: 4728323: RUNNING
  8527. job: 4728326: RUNNING
  8528. job: 4728325: RUNNING
  8529. job: 4728321: RUNNING
  8530. job: 4728286: RUNNING
  8531.  
  8532. Jul-28 01:24:09.044 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8533. Jul-28 01:24:13.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8534. job: 4713446: RUNNING
  8535. job: 4728323: RUNNING
  8536. job: 4728326: RUNNING
  8537. job: 4728325: RUNNING
  8538. job: 4728321: RUNNING
  8539. job: 4728286: RUNNING
  8540.  
  8541. Jul-28 01:24:13.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8542. Jul-28 01:24:13.922 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8543. job: 4713446: RUNNING
  8544. job: 4728323: RUNNING
  8545. job: 4728326: RUNNING
  8546. job: 4728325: RUNNING
  8547. job: 4728321: RUNNING
  8548. job: 4728286: RUNNING
  8549.  
  8550. Jul-28 01:24:13.922 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8551. Jul-28 01:24:18.920 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8552. job: 4713446: RUNNING
  8553. job: 4728323: RUNNING
  8554. job: 4728326: RUNNING
  8555. job: 4728325: RUNNING
  8556. job: 4728321: RUNNING
  8557. job: 4728286: RUNNING
  8558.  
  8559. Jul-28 01:24:18.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8560. Jul-28 01:24:18.922 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8561. job: 4713446: RUNNING
  8562. job: 4728323: RUNNING
  8563. job: 4728326: RUNNING
  8564. job: 4728325: RUNNING
  8565. job: 4728321: RUNNING
  8566. job: 4728286: RUNNING
  8567.  
  8568. Jul-28 01:24:18.922 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8569. Jul-28 01:24:23.920 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8570. job: 4713446: RUNNING
  8571. job: 4728323: RUNNING
  8572. job: 4728326: RUNNING
  8573. job: 4728325: RUNNING
  8574. job: 4728321: RUNNING
  8575. job: 4728286: RUNNING
  8576.  
  8577. Jul-28 01:24:23.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8578. Jul-28 01:24:23.922 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8579. job: 4713446: RUNNING
  8580. job: 4728323: RUNNING
  8581. job: 4728326: RUNNING
  8582. job: 4728325: RUNNING
  8583. job: 4728321: RUNNING
  8584. job: 4728286: RUNNING
  8585.  
  8586. Jul-28 01:24:23.922 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8587. Jul-28 01:24:28.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8588. job: 4713446: RUNNING
  8589. job: 4728323: RUNNING
  8590. job: 4728326: RUNNING
  8591. job: 4728325: RUNNING
  8592. job: 4728321: RUNNING
  8593. job: 4728286: RUNNING
  8594.  
  8595. Jul-28 01:24:28.922 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8596. Jul-28 01:24:28.923 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8597. job: 4713446: RUNNING
  8598. job: 4728323: RUNNING
  8599. job: 4728326: RUNNING
  8600. job: 4728325: RUNNING
  8601. job: 4728321: RUNNING
  8602. job: 4728286: RUNNING
  8603.  
  8604. Jul-28 01:24:28.923 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8605. Jul-28 01:24:33.920 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8606. job: 4713446: RUNNING
  8607. job: 4728323: RUNNING
  8608. job: 4728326: RUNNING
  8609. job: 4728325: RUNNING
  8610. job: 4728321: RUNNING
  8611. job: 4728286: RUNNING
  8612.  
  8613. Jul-28 01:24:33.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8614. Jul-28 01:24:33.922 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8615. job: 4713446: RUNNING
  8616. job: 4728323: RUNNING
  8617. job: 4728326: RUNNING
  8618. job: 4728325: RUNNING
  8619. job: 4728321: RUNNING
  8620. job: 4728286: RUNNING
  8621.  
  8622. Jul-28 01:24:33.922 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8623. Jul-28 01:24:38.920 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8624. job: 4713446: RUNNING
  8625. job: 4728323: RUNNING
  8626. job: 4728326: RUNNING
  8627. job: 4728325: RUNNING
  8628. job: 4728321: RUNNING
  8629. job: 4728286: RUNNING
  8630.  
  8631. Jul-28 01:24:38.920 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8632. Jul-28 01:24:38.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8633. job: 4713446: RUNNING
  8634. job: 4728323: RUNNING
  8635. job: 4728326: RUNNING
  8636. job: 4728325: RUNNING
  8637. job: 4728321: RUNNING
  8638. job: 4728286: RUNNING
  8639.  
  8640. Jul-28 01:24:38.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8641. Jul-28 01:24:43.920 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8642. job: 4713446: RUNNING
  8643. job: 4728323: RUNNING
  8644. job: 4728326: RUNNING
  8645. job: 4728325: RUNNING
  8646. job: 4728321: RUNNING
  8647. job: 4728286: RUNNING
  8648.  
  8649. Jul-28 01:24:43.920 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8650. Jul-28 01:24:43.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8651. job: 4713446: RUNNING
  8652. job: 4728323: RUNNING
  8653. job: 4728326: RUNNING
  8654. job: 4728325: RUNNING
  8655. job: 4728321: RUNNING
  8656. job: 4728286: RUNNING
  8657.  
  8658. Jul-28 01:24:43.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8659. Jul-28 01:24:48.920 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8660. job: 4713446: RUNNING
  8661. job: 4728323: RUNNING
  8662. job: 4728326: RUNNING
  8663. job: 4728325: RUNNING
  8664. job: 4728321: RUNNING
  8665. job: 4728286: RUNNING
  8666.  
  8667. Jul-28 01:24:48.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8668. Jul-28 01:24:48.922 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8669. job: 4713446: RUNNING
  8670. job: 4728323: RUNNING
  8671. job: 4728326: RUNNING
  8672. job: 4728325: RUNNING
  8673. job: 4728321: RUNNING
  8674. job: 4728286: RUNNING
  8675.  
  8676. Jul-28 01:24:48.922 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8677. Jul-28 01:24:53.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8678. job: 4713446: RUNNING
  8679. job: 4728323: RUNNING
  8680. job: 4728326: RUNNING
  8681. job: 4728325: RUNNING
  8682. job: 4728321: RUNNING
  8683. job: 4728286: RUNNING
  8684.  
  8685. Jul-28 01:24:53.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8686. Jul-28 01:24:53.922 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8687. job: 4713446: RUNNING
  8688. job: 4728323: RUNNING
  8689. job: 4728326: RUNNING
  8690. job: 4728325: RUNNING
  8691. job: 4728321: RUNNING
  8692. job: 4728286: RUNNING
  8693.  
  8694. Jul-28 01:24:53.922 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8695. Jul-28 01:24:58.920 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8696. job: 4713446: RUNNING
  8697. job: 4728323: RUNNING
  8698. job: 4728326: RUNNING
  8699. job: 4728325: RUNNING
  8700. job: 4728321: RUNNING
  8701. job: 4728286: RUNNING
  8702.  
  8703. Jul-28 01:24:58.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8704. Jul-28 01:24:58.922 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8705. job: 4713446: RUNNING
  8706. job: 4728323: RUNNING
  8707. job: 4728326: RUNNING
  8708. job: 4728325: RUNNING
  8709. job: 4728321: RUNNING
  8710. job: 4728286: RUNNING
  8711.  
  8712. Jul-28 01:24:58.922 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8713. Jul-28 01:25:03.920 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8714. job: 4713446: RUNNING
  8715. job: 4728323: RUNNING
  8716. job: 4728326: RUNNING
  8717. job: 4728325: RUNNING
  8718. job: 4728321: RUNNING
  8719. job: 4728286: RUNNING
  8720.  
  8721. Jul-28 01:25:03.920 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8722. Jul-28 01:25:03.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8723. job: 4713446: RUNNING
  8724. job: 4728323: RUNNING
  8725. job: 4728326: RUNNING
  8726. job: 4728325: RUNNING
  8727. job: 4728321: RUNNING
  8728. job: 4728286: RUNNING
  8729.  
  8730. Jul-28 01:25:03.921 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8731. Jul-28 01:25:08.922 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor lsf > tasks to be completed: 2 -- pending tasks are shown below
  8732. ~> TaskHandler[jobId: 4728325; id: 152; name: merge_pairsam_into_runs (library:HeLa1 run:lane1); status: RUNNING; exit: -; error: -; workDir: /farline/umw_job_dekker/HPCC/sv49w/distiller-nf/work/3b/37ed2fa2ba160f54196e762e24c83b started: 1501217508853; exited: -; ]
  8733. ~> TaskHandler[jobId: 4728326; id: 153; name: merge_pairsam_into_runs (library:HeLa1 run:lane3); status: RUNNING; exit: -; error: -; workDir: /farline/umw_job_dekker/HPCC/sv49w/distiller-nf/work/00/2633edc9da9ad5649d3d6dcffe5a44 started: 1501217938848; exited: -; ]
  8734. Jul-28 01:25:08.922 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  8735. Jul-28 01:25:09.054 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  8736. 4713446,RUN,Jul 27 11:44
  8737. 4728323,RUN,Jul 28 00:40
  8738. 4728326,RUN,Jul 28 00:51
  8739. 4728325,RUN,Jul 28 00:49
  8740. 4728321,RUN,Jul 28 00:34
  8741. 4728286,RUN,Jul 27 23:44
  8742.  
  8743.  
  8744. Jul-28 01:25:09.055 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8745. job: 4713446: RUNNING
  8746. job: 4728323: RUNNING
  8747. job: 4728326: RUNNING
  8748. job: 4728325: RUNNING
  8749. job: 4728321: RUNNING
  8750. job: 4728286: RUNNING
  8751.  
  8752. Jul-28 01:25:09.055 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8753. Jul-28 01:25:09.055 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8754. job: 4713446: RUNNING
  8755. job: 4728323: RUNNING
  8756. job: 4728326: RUNNING
  8757. job: 4728325: RUNNING
  8758. job: 4728321: RUNNING
  8759. job: 4728286: RUNNING
  8760.  
  8761. Jul-28 01:25:09.055 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8762. Jul-28 01:25:13.925 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8763. job: 4713446: RUNNING
  8764. job: 4728323: RUNNING
  8765. job: 4728326: RUNNING
  8766. job: 4728325: RUNNING
  8767. job: 4728321: RUNNING
  8768. job: 4728286: RUNNING
  8769.  
  8770. Jul-28 01:25:13.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8771. Jul-28 01:25:13.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8772. job: 4713446: RUNNING
  8773. job: 4728323: RUNNING
  8774. job: 4728326: RUNNING
  8775. job: 4728325: RUNNING
  8776. job: 4728321: RUNNING
  8777. job: 4728286: RUNNING
  8778.  
  8779. Jul-28 01:25:13.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8780. Jul-28 01:25:18.925 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8781. job: 4713446: RUNNING
  8782. job: 4728323: RUNNING
  8783. job: 4728326: RUNNING
  8784. job: 4728325: RUNNING
  8785. job: 4728321: RUNNING
  8786. job: 4728286: RUNNING
  8787.  
  8788. Jul-28 01:25:18.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8789. Jul-28 01:25:18.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8790. job: 4713446: RUNNING
  8791. job: 4728323: RUNNING
  8792. job: 4728326: RUNNING
  8793. job: 4728325: RUNNING
  8794. job: 4728321: RUNNING
  8795. job: 4728286: RUNNING
  8796.  
  8797. Jul-28 01:25:18.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8798. Jul-28 01:25:23.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8799. job: 4713446: RUNNING
  8800. job: 4728323: RUNNING
  8801. job: 4728326: RUNNING
  8802. job: 4728325: RUNNING
  8803. job: 4728321: RUNNING
  8804. job: 4728286: RUNNING
  8805.  
  8806. Jul-28 01:25:23.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8807. Jul-28 01:25:23.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8808. job: 4713446: RUNNING
  8809. job: 4728323: RUNNING
  8810. job: 4728326: RUNNING
  8811. job: 4728325: RUNNING
  8812. job: 4728321: RUNNING
  8813. job: 4728286: RUNNING
  8814.  
  8815. Jul-28 01:25:23.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8816. Jul-28 01:25:28.925 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8817. job: 4713446: RUNNING
  8818. job: 4728323: RUNNING
  8819. job: 4728326: RUNNING
  8820. job: 4728325: RUNNING
  8821. job: 4728321: RUNNING
  8822. job: 4728286: RUNNING
  8823.  
  8824. Jul-28 01:25:28.925 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8825. Jul-28 01:25:28.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8826. job: 4713446: RUNNING
  8827. job: 4728323: RUNNING
  8828. job: 4728326: RUNNING
  8829. job: 4728325: RUNNING
  8830. job: 4728321: RUNNING
  8831. job: 4728286: RUNNING
  8832.  
  8833. Jul-28 01:25:28.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8834. Jul-28 01:25:33.925 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8835. job: 4713446: RUNNING
  8836. job: 4728323: RUNNING
  8837. job: 4728326: RUNNING
  8838. job: 4728325: RUNNING
  8839. job: 4728321: RUNNING
  8840. job: 4728286: RUNNING
  8841.  
  8842. Jul-28 01:25:33.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8843. Jul-28 01:25:33.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8844. job: 4713446: RUNNING
  8845. job: 4728323: RUNNING
  8846. job: 4728326: RUNNING
  8847. job: 4728325: RUNNING
  8848. job: 4728321: RUNNING
  8849. job: 4728286: RUNNING
  8850.  
  8851. Jul-28 01:25:33.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8852. Jul-28 01:25:38.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8853. job: 4713446: RUNNING
  8854. job: 4728323: RUNNING
  8855. job: 4728326: RUNNING
  8856. job: 4728325: RUNNING
  8857. job: 4728321: RUNNING
  8858. job: 4728286: RUNNING
  8859.  
  8860. Jul-28 01:25:38.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8861. Jul-28 01:25:38.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8862. job: 4713446: RUNNING
  8863. job: 4728323: RUNNING
  8864. job: 4728326: RUNNING
  8865. job: 4728325: RUNNING
  8866. job: 4728321: RUNNING
  8867. job: 4728286: RUNNING
  8868.  
  8869. Jul-28 01:25:38.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8870. Jul-28 01:25:43.925 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8871. job: 4713446: RUNNING
  8872. job: 4728323: RUNNING
  8873. job: 4728326: RUNNING
  8874. job: 4728325: RUNNING
  8875. job: 4728321: RUNNING
  8876. job: 4728286: RUNNING
  8877.  
  8878. Jul-28 01:25:43.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8879. Jul-28 01:25:43.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8880. job: 4713446: RUNNING
  8881. job: 4728323: RUNNING
  8882. job: 4728326: RUNNING
  8883. job: 4728325: RUNNING
  8884. job: 4728321: RUNNING
  8885. job: 4728286: RUNNING
  8886.  
  8887. Jul-28 01:25:43.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8888. Jul-28 01:25:48.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8889. job: 4713446: RUNNING
  8890. job: 4728323: RUNNING
  8891. job: 4728326: RUNNING
  8892. job: 4728325: RUNNING
  8893. job: 4728321: RUNNING
  8894. job: 4728286: RUNNING
  8895.  
  8896. Jul-28 01:25:48.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8897. Jul-28 01:25:48.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8898. job: 4713446: RUNNING
  8899. job: 4728323: RUNNING
  8900. job: 4728326: RUNNING
  8901. job: 4728325: RUNNING
  8902. job: 4728321: RUNNING
  8903. job: 4728286: RUNNING
  8904.  
  8905. Jul-28 01:25:48.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8906. Jul-28 01:25:53.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8907. job: 4713446: RUNNING
  8908. job: 4728323: RUNNING
  8909. job: 4728326: RUNNING
  8910. job: 4728325: RUNNING
  8911. job: 4728321: RUNNING
  8912. job: 4728286: RUNNING
  8913.  
  8914. Jul-28 01:25:53.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8915. Jul-28 01:25:53.928 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8916. job: 4713446: RUNNING
  8917. job: 4728323: RUNNING
  8918. job: 4728326: RUNNING
  8919. job: 4728325: RUNNING
  8920. job: 4728321: RUNNING
  8921. job: 4728286: RUNNING
  8922.  
  8923. Jul-28 01:25:53.928 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8924. Jul-28 01:25:58.925 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8925. job: 4713446: RUNNING
  8926. job: 4728323: RUNNING
  8927. job: 4728326: RUNNING
  8928. job: 4728325: RUNNING
  8929. job: 4728321: RUNNING
  8930. job: 4728286: RUNNING
  8931.  
  8932. Jul-28 01:25:58.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8933. Jul-28 01:25:58.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8934. job: 4713446: RUNNING
  8935. job: 4728323: RUNNING
  8936. job: 4728326: RUNNING
  8937. job: 4728325: RUNNING
  8938. job: 4728321: RUNNING
  8939. job: 4728286: RUNNING
  8940.  
  8941. Jul-28 01:25:58.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8942. Jul-28 01:26:03.925 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8943. job: 4713446: RUNNING
  8944. job: 4728323: RUNNING
  8945. job: 4728326: RUNNING
  8946. job: 4728325: RUNNING
  8947. job: 4728321: RUNNING
  8948. job: 4728286: RUNNING
  8949.  
  8950. Jul-28 01:26:03.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8951. Jul-28 01:26:03.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8952. job: 4713446: RUNNING
  8953. job: 4728323: RUNNING
  8954. job: 4728326: RUNNING
  8955. job: 4728325: RUNNING
  8956. job: 4728321: RUNNING
  8957. job: 4728286: RUNNING
  8958.  
  8959. Jul-28 01:26:03.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8960. Jul-28 01:26:08.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  8961. Jul-28 01:26:09.048 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  8962. 4713446,RUN,Jul 27 11:44
  8963. 4728323,RUN,Jul 28 00:40
  8964. 4728326,RUN,Jul 28 00:51
  8965. 4728325,RUN,Jul 28 00:49
  8966. 4728321,RUN,Jul 28 00:34
  8967. 4728286,RUN,Jul 27 23:44
  8968.  
  8969.  
  8970. Jul-28 01:26:09.048 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8971. job: 4713446: RUNNING
  8972. job: 4728323: RUNNING
  8973. job: 4728326: RUNNING
  8974. job: 4728325: RUNNING
  8975. job: 4728321: RUNNING
  8976. job: 4728286: RUNNING
  8977.  
  8978. Jul-28 01:26:09.048 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8979. Jul-28 01:26:09.049 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8980. job: 4713446: RUNNING
  8981. job: 4728323: RUNNING
  8982. job: 4728326: RUNNING
  8983. job: 4728325: RUNNING
  8984. job: 4728321: RUNNING
  8985. job: 4728286: RUNNING
  8986.  
  8987. Jul-28 01:26:09.049 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  8988. Jul-28 01:26:13.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8989. job: 4713446: RUNNING
  8990. job: 4728323: RUNNING
  8991. job: 4728326: RUNNING
  8992. job: 4728325: RUNNING
  8993. job: 4728321: RUNNING
  8994. job: 4728286: RUNNING
  8995.  
  8996. Jul-28 01:26:13.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  8997. Jul-28 01:26:13.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  8998. job: 4713446: RUNNING
  8999. job: 4728323: RUNNING
  9000. job: 4728326: RUNNING
  9001. job: 4728325: RUNNING
  9002. job: 4728321: RUNNING
  9003. job: 4728286: RUNNING
  9004.  
  9005. Jul-28 01:26:13.928 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9006. Jul-28 01:26:18.924 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9007. job: 4713446: RUNNING
  9008. job: 4728323: RUNNING
  9009. job: 4728326: RUNNING
  9010. job: 4728325: RUNNING
  9011. job: 4728321: RUNNING
  9012. job: 4728286: RUNNING
  9013.  
  9014. Jul-28 01:26:18.924 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9015. Jul-28 01:26:18.925 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9016. job: 4713446: RUNNING
  9017. job: 4728323: RUNNING
  9018. job: 4728326: RUNNING
  9019. job: 4728325: RUNNING
  9020. job: 4728321: RUNNING
  9021. job: 4728286: RUNNING
  9022.  
  9023. Jul-28 01:26:18.925 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9024. Jul-28 01:26:23.925 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9025. job: 4713446: RUNNING
  9026. job: 4728323: RUNNING
  9027. job: 4728326: RUNNING
  9028. job: 4728325: RUNNING
  9029. job: 4728321: RUNNING
  9030. job: 4728286: RUNNING
  9031.  
  9032. Jul-28 01:26:23.925 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9033. Jul-28 01:26:23.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9034. job: 4713446: RUNNING
  9035. job: 4728323: RUNNING
  9036. job: 4728326: RUNNING
  9037. job: 4728325: RUNNING
  9038. job: 4728321: RUNNING
  9039. job: 4728286: RUNNING
  9040.  
  9041. Jul-28 01:26:23.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9042. Jul-28 01:26:28.925 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9043. job: 4713446: RUNNING
  9044. job: 4728323: RUNNING
  9045. job: 4728326: RUNNING
  9046. job: 4728325: RUNNING
  9047. job: 4728321: RUNNING
  9048. job: 4728286: RUNNING
  9049.  
  9050. Jul-28 01:26:28.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9051. Jul-28 01:26:28.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9052. job: 4713446: RUNNING
  9053. job: 4728323: RUNNING
  9054. job: 4728326: RUNNING
  9055. job: 4728325: RUNNING
  9056. job: 4728321: RUNNING
  9057. job: 4728286: RUNNING
  9058.  
  9059. Jul-28 01:26:28.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9060. Jul-28 01:26:33.925 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9061. job: 4713446: RUNNING
  9062. job: 4728323: RUNNING
  9063. job: 4728326: RUNNING
  9064. job: 4728325: RUNNING
  9065. job: 4728321: RUNNING
  9066. job: 4728286: RUNNING
  9067.  
  9068. Jul-28 01:26:33.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9069. Jul-28 01:26:33.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9070. job: 4713446: RUNNING
  9071. job: 4728323: RUNNING
  9072. job: 4728326: RUNNING
  9073. job: 4728325: RUNNING
  9074. job: 4728321: RUNNING
  9075. job: 4728286: RUNNING
  9076.  
  9077. Jul-28 01:26:33.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9078. Jul-28 01:26:38.925 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9079. job: 4713446: RUNNING
  9080. job: 4728323: RUNNING
  9081. job: 4728326: RUNNING
  9082. job: 4728325: RUNNING
  9083. job: 4728321: RUNNING
  9084. job: 4728286: RUNNING
  9085.  
  9086. Jul-28 01:26:38.925 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9087. Jul-28 01:26:38.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9088. job: 4713446: RUNNING
  9089. job: 4728323: RUNNING
  9090. job: 4728326: RUNNING
  9091. job: 4728325: RUNNING
  9092. job: 4728321: RUNNING
  9093. job: 4728286: RUNNING
  9094.  
  9095. Jul-28 01:26:38.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9096. Jul-28 01:26:43.925 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9097. job: 4713446: RUNNING
  9098. job: 4728323: RUNNING
  9099. job: 4728326: RUNNING
  9100. job: 4728325: RUNNING
  9101. job: 4728321: RUNNING
  9102. job: 4728286: RUNNING
  9103.  
  9104. Jul-28 01:26:43.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9105. Jul-28 01:26:43.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9106. job: 4713446: RUNNING
  9107. job: 4728323: RUNNING
  9108. job: 4728326: RUNNING
  9109. job: 4728325: RUNNING
  9110. job: 4728321: RUNNING
  9111. job: 4728286: RUNNING
  9112.  
  9113. Jul-28 01:26:43.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9114. Jul-28 01:26:48.925 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9115. job: 4713446: RUNNING
  9116. job: 4728323: RUNNING
  9117. job: 4728326: RUNNING
  9118. job: 4728325: RUNNING
  9119. job: 4728321: RUNNING
  9120. job: 4728286: RUNNING
  9121.  
  9122. Jul-28 01:26:48.925 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9123. Jul-28 01:26:48.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9124. job: 4713446: RUNNING
  9125. job: 4728323: RUNNING
  9126. job: 4728326: RUNNING
  9127. job: 4728325: RUNNING
  9128. job: 4728321: RUNNING
  9129. job: 4728286: RUNNING
  9130.  
  9131. Jul-28 01:26:48.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9132. Jul-28 01:26:53.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9133. job: 4713446: RUNNING
  9134. job: 4728323: RUNNING
  9135. job: 4728326: RUNNING
  9136. job: 4728325: RUNNING
  9137. job: 4728321: RUNNING
  9138. job: 4728286: RUNNING
  9139.  
  9140. Jul-28 01:26:53.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9141. Jul-28 01:26:53.928 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9142. job: 4713446: RUNNING
  9143. job: 4728323: RUNNING
  9144. job: 4728326: RUNNING
  9145. job: 4728325: RUNNING
  9146. job: 4728321: RUNNING
  9147. job: 4728286: RUNNING
  9148.  
  9149. Jul-28 01:26:53.928 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9150. Jul-28 01:26:58.925 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9151. job: 4713446: RUNNING
  9152. job: 4728323: RUNNING
  9153. job: 4728326: RUNNING
  9154. job: 4728325: RUNNING
  9155. job: 4728321: RUNNING
  9156. job: 4728286: RUNNING
  9157.  
  9158. Jul-28 01:26:58.925 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9159. Jul-28 01:26:58.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9160. job: 4713446: RUNNING
  9161. job: 4728323: RUNNING
  9162. job: 4728326: RUNNING
  9163. job: 4728325: RUNNING
  9164. job: 4728321: RUNNING
  9165. job: 4728286: RUNNING
  9166.  
  9167. Jul-28 01:26:58.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9168. Jul-28 01:27:03.925 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9169. job: 4713446: RUNNING
  9170. job: 4728323: RUNNING
  9171. job: 4728326: RUNNING
  9172. job: 4728325: RUNNING
  9173. job: 4728321: RUNNING
  9174. job: 4728286: RUNNING
  9175.  
  9176. Jul-28 01:27:03.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9177. Jul-28 01:27:03.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9178. job: 4713446: RUNNING
  9179. job: 4728323: RUNNING
  9180. job: 4728326: RUNNING
  9181. job: 4728325: RUNNING
  9182. job: 4728321: RUNNING
  9183. job: 4728286: RUNNING
  9184.  
  9185. Jul-28 01:27:03.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9186. Jul-28 01:27:08.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9187. job: 4713446: RUNNING
  9188. job: 4728323: RUNNING
  9189. job: 4728326: RUNNING
  9190. job: 4728325: RUNNING
  9191. job: 4728321: RUNNING
  9192. job: 4728286: RUNNING
  9193.  
  9194. Jul-28 01:27:08.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9195. Jul-28 01:27:08.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  9196. Jul-28 01:27:09.047 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  9197. 4713446,RUN,Jul 27 11:44
  9198. 4728323,RUN,Jul 28 00:40
  9199. 4728326,RUN,Jul 28 00:51
  9200. 4728325,RUN,Jul 28 00:49
  9201. 4728321,RUN,Jul 28 00:34
  9202. 4728286,RUN,Jul 27 23:44
  9203.  
  9204.  
  9205. Jul-28 01:27:09.047 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9206. job: 4713446: RUNNING
  9207. job: 4728323: RUNNING
  9208. job: 4728326: RUNNING
  9209. job: 4728325: RUNNING
  9210. job: 4728321: RUNNING
  9211. job: 4728286: RUNNING
  9212.  
  9213. Jul-28 01:27:09.047 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9214. Jul-28 01:27:13.925 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9215. job: 4713446: RUNNING
  9216. job: 4728323: RUNNING
  9217. job: 4728326: RUNNING
  9218. job: 4728325: RUNNING
  9219. job: 4728321: RUNNING
  9220. job: 4728286: RUNNING
  9221.  
  9222. Jul-28 01:27:13.925 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9223. Jul-28 01:27:13.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9224. job: 4713446: RUNNING
  9225. job: 4728323: RUNNING
  9226. job: 4728326: RUNNING
  9227. job: 4728325: RUNNING
  9228. job: 4728321: RUNNING
  9229. job: 4728286: RUNNING
  9230.  
  9231. Jul-28 01:27:13.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9232. Jul-28 01:27:18.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9233. job: 4713446: RUNNING
  9234. job: 4728323: RUNNING
  9235. job: 4728326: RUNNING
  9236. job: 4728325: RUNNING
  9237. job: 4728321: RUNNING
  9238. job: 4728286: RUNNING
  9239.  
  9240. Jul-28 01:27:18.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9241. Jul-28 01:27:18.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9242. job: 4713446: RUNNING
  9243. job: 4728323: RUNNING
  9244. job: 4728326: RUNNING
  9245. job: 4728325: RUNNING
  9246. job: 4728321: RUNNING
  9247. job: 4728286: RUNNING
  9248.  
  9249. Jul-28 01:27:18.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9250. Jul-28 01:27:23.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9251. job: 4713446: RUNNING
  9252. job: 4728323: RUNNING
  9253. job: 4728326: RUNNING
  9254. job: 4728325: RUNNING
  9255. job: 4728321: RUNNING
  9256. job: 4728286: RUNNING
  9257.  
  9258. Jul-28 01:27:23.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9259. Jul-28 01:27:23.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9260. job: 4713446: RUNNING
  9261. job: 4728323: RUNNING
  9262. job: 4728326: RUNNING
  9263. job: 4728325: RUNNING
  9264. job: 4728321: RUNNING
  9265. job: 4728286: RUNNING
  9266.  
  9267. Jul-28 01:27:23.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9268. Jul-28 01:27:28.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9269. job: 4713446: RUNNING
  9270. job: 4728323: RUNNING
  9271. job: 4728326: RUNNING
  9272. job: 4728325: RUNNING
  9273. job: 4728321: RUNNING
  9274. job: 4728286: RUNNING
  9275.  
  9276. Jul-28 01:27:28.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9277. Jul-28 01:27:28.929 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9278. job: 4713446: RUNNING
  9279. job: 4728323: RUNNING
  9280. job: 4728326: RUNNING
  9281. job: 4728325: RUNNING
  9282. job: 4728321: RUNNING
  9283. job: 4728286: RUNNING
  9284.  
  9285. Jul-28 01:27:28.929 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9286. Jul-28 01:27:33.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9287. job: 4713446: RUNNING
  9288. job: 4728323: RUNNING
  9289. job: 4728326: RUNNING
  9290. job: 4728325: RUNNING
  9291. job: 4728321: RUNNING
  9292. job: 4728286: RUNNING
  9293.  
  9294. Jul-28 01:27:33.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9295. Jul-28 01:27:33.929 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9296. job: 4713446: RUNNING
  9297. job: 4728323: RUNNING
  9298. job: 4728326: RUNNING
  9299. job: 4728325: RUNNING
  9300. job: 4728321: RUNNING
  9301. job: 4728286: RUNNING
  9302.  
  9303. Jul-28 01:27:33.929 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9304. Jul-28 01:27:38.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9305. job: 4713446: RUNNING
  9306. job: 4728323: RUNNING
  9307. job: 4728326: RUNNING
  9308. job: 4728325: RUNNING
  9309. job: 4728321: RUNNING
  9310. job: 4728286: RUNNING
  9311.  
  9312. Jul-28 01:27:38.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9313. Jul-28 01:27:38.929 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9314. job: 4713446: RUNNING
  9315. job: 4728323: RUNNING
  9316. job: 4728326: RUNNING
  9317. job: 4728325: RUNNING
  9318. job: 4728321: RUNNING
  9319. job: 4728286: RUNNING
  9320.  
  9321. Jul-28 01:27:38.929 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9322. Jul-28 01:27:43.925 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9323. job: 4713446: RUNNING
  9324. job: 4728323: RUNNING
  9325. job: 4728326: RUNNING
  9326. job: 4728325: RUNNING
  9327. job: 4728321: RUNNING
  9328. job: 4728286: RUNNING
  9329.  
  9330. Jul-28 01:27:43.925 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9331. Jul-28 01:27:43.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9332. job: 4713446: RUNNING
  9333. job: 4728323: RUNNING
  9334. job: 4728326: RUNNING
  9335. job: 4728325: RUNNING
  9336. job: 4728321: RUNNING
  9337. job: 4728286: RUNNING
  9338.  
  9339. Jul-28 01:27:43.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9340. Jul-28 01:27:48.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9341. job: 4713446: RUNNING
  9342. job: 4728323: RUNNING
  9343. job: 4728326: RUNNING
  9344. job: 4728325: RUNNING
  9345. job: 4728321: RUNNING
  9346. job: 4728286: RUNNING
  9347.  
  9348. Jul-28 01:27:48.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9349. Jul-28 01:27:48.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9350. job: 4713446: RUNNING
  9351. job: 4728323: RUNNING
  9352. job: 4728326: RUNNING
  9353. job: 4728325: RUNNING
  9354. job: 4728321: RUNNING
  9355. job: 4728286: RUNNING
  9356.  
  9357. Jul-28 01:27:48.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9358. Jul-28 01:27:53.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9359. job: 4713446: RUNNING
  9360. job: 4728323: RUNNING
  9361. job: 4728326: RUNNING
  9362. job: 4728325: RUNNING
  9363. job: 4728321: RUNNING
  9364. job: 4728286: RUNNING
  9365.  
  9366. Jul-28 01:27:53.926 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9367. Jul-28 01:27:53.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9368. job: 4713446: RUNNING
  9369. job: 4728323: RUNNING
  9370. job: 4728326: RUNNING
  9371. job: 4728325: RUNNING
  9372. job: 4728321: RUNNING
  9373. job: 4728286: RUNNING
  9374.  
  9375. Jul-28 01:27:53.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9376. Jul-28 01:27:58.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9377. job: 4713446: RUNNING
  9378. job: 4728323: RUNNING
  9379. job: 4728326: RUNNING
  9380. job: 4728325: RUNNING
  9381. job: 4728321: RUNNING
  9382. job: 4728286: RUNNING
  9383.  
  9384. Jul-28 01:27:58.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9385. Jul-28 01:27:58.928 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9386. job: 4713446: RUNNING
  9387. job: 4728323: RUNNING
  9388. job: 4728326: RUNNING
  9389. job: 4728325: RUNNING
  9390. job: 4728321: RUNNING
  9391. job: 4728286: RUNNING
  9392.  
  9393. Jul-28 01:27:58.928 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9394. Jul-28 01:28:03.931 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9395. job: 4713446: RUNNING
  9396. job: 4728323: RUNNING
  9397. job: 4728326: RUNNING
  9398. job: 4728325: RUNNING
  9399. job: 4728321: RUNNING
  9400. job: 4728286: RUNNING
  9401.  
  9402. Jul-28 01:28:03.932 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9403. Jul-28 01:28:03.933 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9404. job: 4713446: RUNNING
  9405. job: 4728323: RUNNING
  9406. job: 4728326: RUNNING
  9407. job: 4728325: RUNNING
  9408. job: 4728321: RUNNING
  9409. job: 4728286: RUNNING
  9410.  
  9411. Jul-28 01:28:03.933 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9412. Jul-28 01:28:08.928 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  9413. Jul-28 01:28:09.024 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  9414. 4713446,RUN,Jul 27 11:44
  9415. 4728323,RUN,Jul 28 00:40
  9416. 4728326,RUN,Jul 28 00:51
  9417. 4728325,RUN,Jul 28 00:49
  9418. 4728321,RUN,Jul 28 00:34
  9419. 4728286,RUN,Jul 27 23:44
  9420.  
  9421.  
  9422. Jul-28 01:28:09.024 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9423. job: 4713446: RUNNING
  9424. job: 4728323: RUNNING
  9425. job: 4728326: RUNNING
  9426. job: 4728325: RUNNING
  9427. job: 4728321: RUNNING
  9428. job: 4728286: RUNNING
  9429.  
  9430. Jul-28 01:28:09.024 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9431. Jul-28 01:28:09.025 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9432. job: 4713446: RUNNING
  9433. job: 4728323: RUNNING
  9434. job: 4728326: RUNNING
  9435. job: 4728325: RUNNING
  9436. job: 4728321: RUNNING
  9437. job: 4728286: RUNNING
  9438.  
  9439. Jul-28 01:28:09.025 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9440. Jul-28 01:28:13.928 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9441. job: 4713446: RUNNING
  9442. job: 4728323: RUNNING
  9443. job: 4728326: RUNNING
  9444. job: 4728325: RUNNING
  9445. job: 4728321: RUNNING
  9446. job: 4728286: RUNNING
  9447.  
  9448. Jul-28 01:28:13.928 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9449. Jul-28 01:28:13.929 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9450. job: 4713446: RUNNING
  9451. job: 4728323: RUNNING
  9452. job: 4728326: RUNNING
  9453. job: 4728325: RUNNING
  9454. job: 4728321: RUNNING
  9455. job: 4728286: RUNNING
  9456.  
  9457. Jul-28 01:28:13.929 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9458. Jul-28 01:28:18.928 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9459. job: 4713446: RUNNING
  9460. job: 4728323: RUNNING
  9461. job: 4728326: RUNNING
  9462. job: 4728325: RUNNING
  9463. job: 4728321: RUNNING
  9464. job: 4728286: RUNNING
  9465.  
  9466. Jul-28 01:28:18.929 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9467. Jul-28 01:28:18.930 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9468. job: 4713446: RUNNING
  9469. job: 4728323: RUNNING
  9470. job: 4728326: RUNNING
  9471. job: 4728325: RUNNING
  9472. job: 4728321: RUNNING
  9473. job: 4728286: RUNNING
  9474.  
  9475. Jul-28 01:28:18.930 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9476. Jul-28 01:28:23.928 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9477. job: 4713446: RUNNING
  9478. job: 4728323: RUNNING
  9479. job: 4728326: RUNNING
  9480. job: 4728325: RUNNING
  9481. job: 4728321: RUNNING
  9482. job: 4728286: RUNNING
  9483.  
  9484. Jul-28 01:28:23.929 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9485. Jul-28 01:28:23.930 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9486. job: 4713446: RUNNING
  9487. job: 4728323: RUNNING
  9488. job: 4728326: RUNNING
  9489. job: 4728325: RUNNING
  9490. job: 4728321: RUNNING
  9491. job: 4728286: RUNNING
  9492.  
  9493. Jul-28 01:28:23.930 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9494. Jul-28 01:28:28.929 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9495. job: 4713446: RUNNING
  9496. job: 4728323: RUNNING
  9497. job: 4728326: RUNNING
  9498. job: 4728325: RUNNING
  9499. job: 4728321: RUNNING
  9500. job: 4728286: RUNNING
  9501.  
  9502. Jul-28 01:28:28.930 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9503. Jul-28 01:28:28.931 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9504. job: 4713446: RUNNING
  9505. job: 4728323: RUNNING
  9506. job: 4728326: RUNNING
  9507. job: 4728325: RUNNING
  9508. job: 4728321: RUNNING
  9509. job: 4728286: RUNNING
  9510.  
  9511. Jul-28 01:28:28.931 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9512. Jul-28 01:28:33.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9513. job: 4713446: RUNNING
  9514. job: 4728323: RUNNING
  9515. job: 4728326: RUNNING
  9516. job: 4728325: RUNNING
  9517. job: 4728321: RUNNING
  9518. job: 4728286: RUNNING
  9519.  
  9520. Jul-28 01:28:33.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9521. Jul-28 01:28:33.928 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9522. job: 4713446: RUNNING
  9523. job: 4728323: RUNNING
  9524. job: 4728326: RUNNING
  9525. job: 4728325: RUNNING
  9526. job: 4728321: RUNNING
  9527. job: 4728286: RUNNING
  9528.  
  9529. Jul-28 01:28:33.928 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9530. Jul-28 01:28:38.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9531. job: 4713446: RUNNING
  9532. job: 4728323: RUNNING
  9533. job: 4728326: RUNNING
  9534. job: 4728325: RUNNING
  9535. job: 4728321: RUNNING
  9536. job: 4728286: RUNNING
  9537.  
  9538. Jul-28 01:28:38.928 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9539. Jul-28 01:28:38.928 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9540. job: 4713446: RUNNING
  9541. job: 4728323: RUNNING
  9542. job: 4728326: RUNNING
  9543. job: 4728325: RUNNING
  9544. job: 4728321: RUNNING
  9545. job: 4728286: RUNNING
  9546.  
  9547. Jul-28 01:28:38.928 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9548. Jul-28 01:28:43.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9549. job: 4713446: RUNNING
  9550. job: 4728323: RUNNING
  9551. job: 4728326: RUNNING
  9552. job: 4728325: RUNNING
  9553. job: 4728321: RUNNING
  9554. job: 4728286: RUNNING
  9555.  
  9556. Jul-28 01:28:43.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9557. Jul-28 01:28:43.928 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9558. job: 4713446: RUNNING
  9559. job: 4728323: RUNNING
  9560. job: 4728326: RUNNING
  9561. job: 4728325: RUNNING
  9562. job: 4728321: RUNNING
  9563. job: 4728286: RUNNING
  9564.  
  9565. Jul-28 01:28:43.928 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9566. Jul-28 01:28:48.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9567. job: 4713446: RUNNING
  9568. job: 4728323: RUNNING
  9569. job: 4728326: RUNNING
  9570. job: 4728325: RUNNING
  9571. job: 4728321: RUNNING
  9572. job: 4728286: RUNNING
  9573.  
  9574. Jul-28 01:28:48.928 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9575. Jul-28 01:28:48.929 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9576. job: 4713446: RUNNING
  9577. job: 4728323: RUNNING
  9578. job: 4728326: RUNNING
  9579. job: 4728325: RUNNING
  9580. job: 4728321: RUNNING
  9581. job: 4728286: RUNNING
  9582.  
  9583. Jul-28 01:28:48.929 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9584. Jul-28 01:28:53.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9585. job: 4713446: RUNNING
  9586. job: 4728323: RUNNING
  9587. job: 4728326: RUNNING
  9588. job: 4728325: RUNNING
  9589. job: 4728321: RUNNING
  9590. job: 4728286: RUNNING
  9591.  
  9592. Jul-28 01:28:53.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9593. Jul-28 01:28:53.928 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9594. job: 4713446: RUNNING
  9595. job: 4728323: RUNNING
  9596. job: 4728326: RUNNING
  9597. job: 4728325: RUNNING
  9598. job: 4728321: RUNNING
  9599. job: 4728286: RUNNING
  9600.  
  9601. Jul-28 01:28:53.928 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9602. Jul-28 01:28:58.927 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9603. job: 4713446: RUNNING
  9604. job: 4728323: RUNNING
  9605. job: 4728326: RUNNING
  9606. job: 4728325: RUNNING
  9607. job: 4728321: RUNNING
  9608. job: 4728286: RUNNING
  9609.  
  9610. Jul-28 01:28:58.928 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9611. Jul-28 01:28:58.928 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9612. job: 4713446: RUNNING
  9613. job: 4728323: RUNNING
  9614. job: 4728326: RUNNING
  9615. job: 4728325: RUNNING
  9616. job: 4728321: RUNNING
  9617. job: 4728286: RUNNING
  9618.  
  9619. Jul-28 01:28:58.928 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9620. Jul-28 01:29:03.928 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9621. job: 4713446: RUNNING
  9622. job: 4728323: RUNNING
  9623. job: 4728326: RUNNING
  9624. job: 4728325: RUNNING
  9625. job: 4728321: RUNNING
  9626. job: 4728286: RUNNING
  9627.  
  9628. Jul-28 01:29:03.928 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9629. Jul-28 01:29:03.929 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9630. job: 4713446: RUNNING
  9631. job: 4728323: RUNNING
  9632. job: 4728326: RUNNING
  9633. job: 4728325: RUNNING
  9634. job: 4728321: RUNNING
  9635. job: 4728286: RUNNING
  9636.  
  9637. Jul-28 01:29:03.929 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9638. Jul-28 01:29:08.929 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  9639. Jul-28 01:29:09.029 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  9640. 4713446,RUN,Jul 27 11:44
  9641. 4728323,RUN,Jul 28 00:40
  9642. 4728326,RUN,Jul 28 00:51
  9643. 4728325,RUN,Jul 28 00:49
  9644. 4728321,RUN,Jul 28 00:34
  9645. 4728286,RUN,Jul 27 23:44
  9646.  
  9647.  
  9648. Jul-28 01:29:09.029 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9649. job: 4713446: RUNNING
  9650. job: 4728323: RUNNING
  9651. job: 4728326: RUNNING
  9652. job: 4728325: RUNNING
  9653. job: 4728321: RUNNING
  9654. job: 4728286: RUNNING
  9655.  
  9656. Jul-28 01:29:09.029 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9657. Jul-28 01:29:09.031 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9658. job: 4713446: RUNNING
  9659. job: 4728323: RUNNING
  9660. job: 4728326: RUNNING
  9661. job: 4728325: RUNNING
  9662. job: 4728321: RUNNING
  9663. job: 4728286: RUNNING
  9664.  
  9665. Jul-28 01:29:09.031 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9666. Jul-28 01:29:13.932 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9667. job: 4713446: RUNNING
  9668. job: 4728323: RUNNING
  9669. job: 4728326: RUNNING
  9670. job: 4728325: RUNNING
  9671. job: 4728321: RUNNING
  9672. job: 4728286: RUNNING
  9673.  
  9674. Jul-28 01:29:13.933 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9675. Jul-28 01:29:13.934 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9676. job: 4713446: RUNNING
  9677. job: 4728323: RUNNING
  9678. job: 4728326: RUNNING
  9679. job: 4728325: RUNNING
  9680. job: 4728321: RUNNING
  9681. job: 4728286: RUNNING
  9682.  
  9683. Jul-28 01:29:13.934 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9684. Jul-28 01:29:18.929 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9685. job: 4713446: RUNNING
  9686. job: 4728323: RUNNING
  9687. job: 4728326: RUNNING
  9688. job: 4728325: RUNNING
  9689. job: 4728321: RUNNING
  9690. job: 4728286: RUNNING
  9691.  
  9692. Jul-28 01:29:18.929 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9693. Jul-28 01:29:18.930 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9694. job: 4713446: RUNNING
  9695. job: 4728323: RUNNING
  9696. job: 4728326: RUNNING
  9697. job: 4728325: RUNNING
  9698. job: 4728321: RUNNING
  9699. job: 4728286: RUNNING
  9700.  
  9701. Jul-28 01:29:18.930 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9702. Jul-28 01:29:23.929 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9703. job: 4713446: RUNNING
  9704. job: 4728323: RUNNING
  9705. job: 4728326: RUNNING
  9706. job: 4728325: RUNNING
  9707. job: 4728321: RUNNING
  9708. job: 4728286: RUNNING
  9709.  
  9710. Jul-28 01:29:23.930 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9711. Jul-28 01:29:23.931 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9712. job: 4713446: RUNNING
  9713. job: 4728323: RUNNING
  9714. job: 4728326: RUNNING
  9715. job: 4728325: RUNNING
  9716. job: 4728321: RUNNING
  9717. job: 4728286: RUNNING
  9718.  
  9719. Jul-28 01:29:23.931 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9720. Jul-28 01:29:28.929 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9721. job: 4713446: RUNNING
  9722. job: 4728323: RUNNING
  9723. job: 4728326: RUNNING
  9724. job: 4728325: RUNNING
  9725. job: 4728321: RUNNING
  9726. job: 4728286: RUNNING
  9727.  
  9728. Jul-28 01:29:28.929 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9729. Jul-28 01:29:28.930 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9730. job: 4713446: RUNNING
  9731. job: 4728323: RUNNING
  9732. job: 4728326: RUNNING
  9733. job: 4728325: RUNNING
  9734. job: 4728321: RUNNING
  9735. job: 4728286: RUNNING
  9736.  
  9737. Jul-28 01:29:28.930 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9738. Jul-28 01:29:33.930 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9739. job: 4713446: RUNNING
  9740. job: 4728323: RUNNING
  9741. job: 4728326: RUNNING
  9742. job: 4728325: RUNNING
  9743. job: 4728321: RUNNING
  9744. job: 4728286: RUNNING
  9745.  
  9746. Jul-28 01:29:33.930 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9747. Jul-28 01:29:33.931 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9748. job: 4713446: RUNNING
  9749. job: 4728323: RUNNING
  9750. job: 4728326: RUNNING
  9751. job: 4728325: RUNNING
  9752. job: 4728321: RUNNING
  9753. job: 4728286: RUNNING
  9754.  
  9755. Jul-28 01:29:33.931 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9756. Jul-28 01:29:38.929 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9757. job: 4713446: RUNNING
  9758. job: 4728323: RUNNING
  9759. job: 4728326: RUNNING
  9760. job: 4728325: RUNNING
  9761. job: 4728321: RUNNING
  9762. job: 4728286: RUNNING
  9763.  
  9764. Jul-28 01:29:38.929 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9765. Jul-28 01:29:38.930 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9766. job: 4713446: RUNNING
  9767. job: 4728323: RUNNING
  9768. job: 4728326: RUNNING
  9769. job: 4728325: RUNNING
  9770. job: 4728321: RUNNING
  9771. job: 4728286: RUNNING
  9772.  
  9773. Jul-28 01:29:38.930 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9774. Jul-28 01:29:43.928 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9775. job: 4713446: RUNNING
  9776. job: 4728323: RUNNING
  9777. job: 4728326: RUNNING
  9778. job: 4728325: RUNNING
  9779. job: 4728321: RUNNING
  9780. job: 4728286: RUNNING
  9781.  
  9782. Jul-28 01:29:43.928 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9783. Jul-28 01:29:43.929 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9784. job: 4713446: RUNNING
  9785. job: 4728323: RUNNING
  9786. job: 4728326: RUNNING
  9787. job: 4728325: RUNNING
  9788. job: 4728321: RUNNING
  9789. job: 4728286: RUNNING
  9790.  
  9791. Jul-28 01:29:43.929 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9792. Jul-28 01:29:48.932 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9793. job: 4713446: RUNNING
  9794. job: 4728323: RUNNING
  9795. job: 4728326: RUNNING
  9796. job: 4728325: RUNNING
  9797. job: 4728321: RUNNING
  9798. job: 4728286: RUNNING
  9799.  
  9800. Jul-28 01:29:48.932 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9801. Jul-28 01:29:48.933 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9802. job: 4713446: RUNNING
  9803. job: 4728323: RUNNING
  9804. job: 4728326: RUNNING
  9805. job: 4728325: RUNNING
  9806. job: 4728321: RUNNING
  9807. job: 4728286: RUNNING
  9808.  
  9809. Jul-28 01:29:48.933 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9810. Jul-28 01:29:53.932 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9811. job: 4713446: RUNNING
  9812. job: 4728323: RUNNING
  9813. job: 4728326: RUNNING
  9814. job: 4728325: RUNNING
  9815. job: 4728321: RUNNING
  9816. job: 4728286: RUNNING
  9817.  
  9818. Jul-28 01:29:53.932 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9819. Jul-28 01:29:53.935 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9820. job: 4713446: RUNNING
  9821. job: 4728323: RUNNING
  9822. job: 4728326: RUNNING
  9823. job: 4728325: RUNNING
  9824. job: 4728321: RUNNING
  9825. job: 4728286: RUNNING
  9826.  
  9827. Jul-28 01:29:53.935 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9828. Jul-28 01:29:58.930 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9829. job: 4713446: RUNNING
  9830. job: 4728323: RUNNING
  9831. job: 4728326: RUNNING
  9832. job: 4728325: RUNNING
  9833. job: 4728321: RUNNING
  9834. job: 4728286: RUNNING
  9835.  
  9836. Jul-28 01:29:58.930 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9837. Jul-28 01:29:58.931 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9838. job: 4713446: RUNNING
  9839. job: 4728323: RUNNING
  9840. job: 4728326: RUNNING
  9841. job: 4728325: RUNNING
  9842. job: 4728321: RUNNING
  9843. job: 4728286: RUNNING
  9844.  
  9845. Jul-28 01:29:58.931 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9846. Jul-28 01:30:03.933 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9847. job: 4713446: RUNNING
  9848. job: 4728323: RUNNING
  9849. job: 4728326: RUNNING
  9850. job: 4728325: RUNNING
  9851. job: 4728321: RUNNING
  9852. job: 4728286: RUNNING
  9853.  
  9854. Jul-28 01:30:03.933 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9855. Jul-28 01:30:03.935 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9856. job: 4713446: RUNNING
  9857. job: 4728323: RUNNING
  9858. job: 4728326: RUNNING
  9859. job: 4728325: RUNNING
  9860. job: 4728321: RUNNING
  9861. job: 4728286: RUNNING
  9862.  
  9863. Jul-28 01:30:03.935 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9864. Jul-28 01:30:08.936 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor lsf > tasks to be completed: 2 -- pending tasks are shown below
  9865. ~> TaskHandler[jobId: 4728325; id: 152; name: merge_pairsam_into_runs (library:HeLa1 run:lane1); status: RUNNING; exit: -; error: -; workDir: /farline/umw_job_dekker/HPCC/sv49w/distiller-nf/work/3b/37ed2fa2ba160f54196e762e24c83b started: 1501217508853; exited: -; ]
  9866. ~> TaskHandler[jobId: 4728326; id: 153; name: merge_pairsam_into_runs (library:HeLa1 run:lane3); status: RUNNING; exit: -; error: -; workDir: /farline/umw_job_dekker/HPCC/sv49w/distiller-nf/work/00/2633edc9da9ad5649d3d6dcffe5a44 started: 1501217938848; exited: -; ]
  9867. Jul-28 01:30:08.937 [Task monitor] TRACE n.executor.AbstractGridExecutor - Getting grid queue status: bjobs -o JOBID STAT SUBMIT_TIME delimiter=',' -noheader -q long
  9868. Jul-28 01:30:09.052 [Task monitor] TRACE n.executor.AbstractGridExecutor - LSF status result > exit: 0
  9869. 4713446,RUN,Jul 27 11:44
  9870. 4728323,RUN,Jul 28 00:40
  9871. 4728326,RUN,Jul 28 00:51
  9872. 4728325,RUN,Jul 28 00:49
  9873.  
  9874.  
  9875. Jul-28 01:30:09.052 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9876. job: 4713446: RUNNING
  9877. job: 4728323: RUNNING
  9878. job: 4728326: RUNNING
  9879. job: 4728325: RUNNING
  9880.  
  9881. Jul-28 01:30:09.053 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9882. Jul-28 01:30:09.053 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9883. job: 4713446: RUNNING
  9884. job: 4728323: RUNNING
  9885. job: 4728326: RUNNING
  9886. job: 4728325: RUNNING
  9887.  
  9888. Jul-28 01:30:09.053 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9889. Jul-28 01:30:13.942 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9890. job: 4713446: RUNNING
  9891. job: 4728323: RUNNING
  9892. job: 4728326: RUNNING
  9893. job: 4728325: RUNNING
  9894.  
  9895. Jul-28 01:30:13.942 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9896. Jul-28 01:30:13.943 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9897. job: 4713446: RUNNING
  9898. job: 4728323: RUNNING
  9899. job: 4728326: RUNNING
  9900. job: 4728325: RUNNING
  9901.  
  9902. Jul-28 01:30:13.943 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9903. Jul-28 01:30:18.941 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9904. job: 4713446: RUNNING
  9905. job: 4728323: RUNNING
  9906. job: 4728326: RUNNING
  9907. job: 4728325: RUNNING
  9908.  
  9909. Jul-28 01:30:18.941 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9910. Jul-28 01:30:18.942 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9911. job: 4713446: RUNNING
  9912. job: 4728323: RUNNING
  9913. job: 4728326: RUNNING
  9914. job: 4728325: RUNNING
  9915.  
  9916. Jul-28 01:30:18.942 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9917. Jul-28 01:30:23.941 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9918. job: 4713446: RUNNING
  9919. job: 4728323: RUNNING
  9920. job: 4728326: RUNNING
  9921. job: 4728325: RUNNING
  9922.  
  9923. Jul-28 01:30:23.942 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9924. Jul-28 01:30:23.943 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9925. job: 4713446: RUNNING
  9926. job: 4728323: RUNNING
  9927. job: 4728326: RUNNING
  9928. job: 4728325: RUNNING
  9929.  
  9930. Jul-28 01:30:23.943 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9931. Jul-28 01:30:28.941 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9932. job: 4713446: RUNNING
  9933. job: 4728323: RUNNING
  9934. job: 4728326: RUNNING
  9935. job: 4728325: RUNNING
  9936.  
  9937. Jul-28 01:30:28.942 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9938. Jul-28 01:30:28.942 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9939. job: 4713446: RUNNING
  9940. job: 4728323: RUNNING
  9941. job: 4728326: RUNNING
  9942. job: 4728325: RUNNING
  9943.  
  9944. Jul-28 01:30:28.942 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9945. Jul-28 01:30:33.945 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9946. job: 4713446: RUNNING
  9947. job: 4728323: RUNNING
  9948. job: 4728326: RUNNING
  9949. job: 4728325: RUNNING
  9950.  
  9951. Jul-28 01:30:33.946 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9952. Jul-28 01:30:33.947 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9953. job: 4713446: RUNNING
  9954. job: 4728323: RUNNING
  9955. job: 4728326: RUNNING
  9956. job: 4728325: RUNNING
  9957.  
  9958. Jul-28 01:30:33.947 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9959. Jul-28 01:30:38.944 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9960. job: 4713446: RUNNING
  9961. job: 4728323: RUNNING
  9962. job: 4728326: RUNNING
  9963. job: 4728325: RUNNING
  9964.  
  9965. Jul-28 01:30:38.945 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9966. Jul-28 01:30:38.945 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9967. job: 4713446: RUNNING
  9968. job: 4728323: RUNNING
  9969. job: 4728326: RUNNING
  9970. job: 4728325: RUNNING
  9971.  
  9972. Jul-28 01:30:38.945 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9973. Jul-28 01:30:43.942 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9974. job: 4713446: RUNNING
  9975. job: 4728323: RUNNING
  9976. job: 4728326: RUNNING
  9977. job: 4728325: RUNNING
  9978.  
  9979. Jul-28 01:30:43.942 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9980. Jul-28 01:30:43.943 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9981. job: 4713446: RUNNING
  9982. job: 4728323: RUNNING
  9983. job: 4728326: RUNNING
  9984. job: 4728325: RUNNING
  9985.  
  9986. Jul-28 01:30:43.943 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
  9987. Jul-28 01:30:48.948 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9988. job: 4713446: RUNNING
  9989. job: 4728323: RUNNING
  9990. job: 4728326: RUNNING
  9991. job: 4728325: RUNNING
  9992.  
  9993. Jul-28 01:30:48.949 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728325` active status: true
  9994. Jul-28 01:30:48.951 [Task monitor] TRACE n.executor.AbstractGridExecutor - Queue status:
  9995. job: 4713446: RUNNING
  9996. job: 4728323: RUNNING
  9997. job: 4728326: RUNNING
  9998. job: 4728325: RUNNING
  9999.  
  10000. Jul-28 01:30:48.951 [Task monitor] TRACE n.executor.AbstractGridExecutor - JobId `4728326` active status: true
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!