API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Mar 22nd, 2019
237
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 367.35 KB | None | 0 0
raw download clone embed print report
  1. INFO:cactus.progressive.projectWrapper:Using config from path /export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/cactus_progressive_config.xml.
  2. INFO:toil.lib.bioio:xmlRoot = <multi_cactus inputSequences="CA_SC5314.fna SC_YJM555.fna SC_YPS163.fna SC_sacCer3.fna CG_CBS138.fna">
  3. <tree>(CA_SC5314:1.0,SC_YJM555:1.0,SC_YPS163:1.0,SC_sacCer3:1.0,CG_CBS138:1.0)Anc0;</tree>
  4. <cactus experiment_path="/scratch/tmp/1073389.1.campus-haswell.q/tmpN4Ky31/progressiveAlignment/Anc0/Anc0_experiment.xml" name="Anc0" />
  5. </multi_cactus>
  6. WARNING:toil.batchSystems.singleMachine:Limiting maxMemory to physically available memory (134158856192).
  7. WARNING:toil.batchSystems.singleMachine:Limiting maxDisk to physically available disk (2431064735744).
  8. WARNING:toil.common:Batch system does not support auto-deployment. The user script ModuleDescriptor(dirPath='/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages', name='cactus.progressive.cactus_progressive', fromVirtualEnv=False) will have to be present at the same location on every worker.
  9. INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  10. INFO:toil.leader:Issued job 'RunCactusPreprocessorThenProgressiveDown' S/o/job51DlDr with job batch system ID: 0 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  11. INFO:toil.statsAndLogging:Got message from job at time 03-22-2019 18:06:05: Using the following configuration:
  12. <cactusWorkflowConfig distanceToAddToRootAlignment="0.1">
  13. <constants defaultCpu="1" defaultMemory="3500000000" defaultOverlargeCpu="1" defaultOverlargeMemory="3500000000">
  14.  
  15. <divergences five="0.35" four="0.25" one="0.1" three="0.2" two="0.15" useDefault="0" />
  16. </constants>
  17. <preprocessor check="1" checkAssemblyHub="1" memory="2000000000" preprocessJob="checkUniqueHeaders" />
  18. <preprocessor chunkSize="3000000" lastzOpts="--step=3 --ambiguous=iupac,100,100 --ungapped --queryhsplimit=keep,nowarn:1500" memory="2000000000" minPeriod="50" preprocessJob="lastzRepeatMask" proportionToSample="0.2" unmask="0" />
  19.  
  20. <trimBlast doTrimStrategy="1" keepParalogs="0" trimFlanking="10" trimMinSize="100" trimOutgroupDepth="1" trimOutgroupFlanking="2000" trimThreshold="1.0" trimWindowSize="1" />
  21. <ktserver memory="3500000000" />
  22. <setup makeEventHeadersAlphaNumeric="0" />
  23.  
  24. <caf alignmentFilter="filterSecondariesByMultipleSpecies" alpha="0.001" annealingRounds="128" blockTrim="5" chunkSize="25000000" compressFiles="1" deannealingRounds="2 8" doPhylogeny="0" filterByIdentity="0" identityRatio="3" lastzDisk="mediumDisk" lastzMemory="2000000000" maxAdjacencyComponentSizeRatio="50" maxAlignmentsPerSite="5" maxRecoverableChainLength="500000" maxRecoverableChainsIterations="5" maximumMedianSequenceLengthBetweenLinkedEnds="1000" minLengthForChromosome="1000000" minimumBlockDegree="2" minimumBlockDegreeToCheckSupport="10" minimumBlockHomologySupport="0.05" minimumDistance="0.01" minimumIngroupDegree="1" minimumMapQValue="0.0" minimumNumberOfSpecies="1" minimumOutgroupDegree="0" minimumSequenceLengthForBlast="30" minimumTreeCoverage="0.0" numTreeBuildingThreads="2" overlapSize="10000" phylogenyBreakpointScalingFactor="1.0" phylogenyCostPerDupPerBase="0.00" phylogenyCostPerLossPerBase="0.02" phylogenyDistanceCorrectionMethod="jukesCantor" phylogenyDoSplitsWithSupportHigherThanThisAllAtOnce="0.44" phylogenyHomologyUnitType="chain" phylogenyKeepSingleDegreeBlocks="0" phylogenyMaxBaseDistance="100" phylogenyMaxBlockDistance="50" phylogenyNumTrees="30" phylogenyRootingMethod="bestRecon" phylogenyScoringMethod="reconCost" phylogenySkipSingleCopyBlocks="1" phylogenyTreeBuildingMethod="guidedNeighborJoining,splitDecomposition" proportionOfUnalignedBasesForNewChromosome="0.8" realign="1" realignArguments="--gapGamma 0.0 --matchGamma 0.9 --diagonalExpansion 4 --splitMatrixBiggerThanThis 10 --constraintDiagonalTrim 0 --alignAmbiguityCharacters --splitIndelsLongerThanThis 99" removeRecoverableChains="unequalNumberOfIngroupCopies" runMapQFiltering="1" trim="0 0">
  25. <divergence argName="lastzArguments" default="--step=1 --ambiguous=iupac,100,100 --ydrop=3000" five="--step=2 --ambiguous=iupac,100,100 --ydrop=3000" four="--step=3 --ambiguous=iupac,100,100 --ydrop=3000" one="--step=2 --ambiguous=iupac,100,100 --ydrop=3000 --notransition" three="--step=4 --ambiguous=iupac,100,100 --ydrop=3000" two="--step=5 --ambiguous=iupac,100,100 --ydrop=3000" />
  26. <CactusCafRecursion maxFlowerGroupSize="100000000" />
  27. <CactusCafWrapper maxFlowerGroupSize="25000000" minFlowerSize="1" />
  28. <CactusCafWrapperLarge2 overlargeMemory="5000000000" />
  29. </caf>
  30.  
  31. <bar alignAmbiguityCharacters="1" anchorMatrixBiggerThanThis="500" bandingLimit="1000000" constraintDiagonalTrim="14" diagonalExpansion="20" gapGamma="0.0" largeEndSize="5000" matchGamma="0.2" minimumBlockDegree="2" minimumCoverageToRescue="0.5" minimumIngroupDegree="1" minimumNumberOfSpecies="1" minimumOutgroupDegree="0" minimumSizeToRescue="100" pruneOutStubAlignments="1" repeatMaskMatrixBiggerThanThis="500" rescue="0" runBar="1" spanningTrees="5" splitMatrixBiggerThanThis="3000" useBanding="1" useProgressiveMerging="1" veryLargeEndSize="2000000">
  32. <CactusBarRecursion maxFlowerGroupSize="100000000" />
  33. <CactusBarWrapper maxFlowerGroupSize="2000000" memory="2000000000" />
  34. <CactusBarWrapperLarge maxFlowerGroupSize="2000000" />
  35. <CactusBarEndAlignerWrapper memory="2000000000" />
  36. </bar>
  37. <normal iterations="0">
  38. <CactusNormalRecursion maxFlowerGroupSize="100000000" maxFlowerWrapperGroupSize="10000000" />
  39. <CactusNormalWrapper />
  40. </normal>
  41. <avg buildAvgs="0">
  42. <CactusAVGRecursion maxFlowerGroupSize="100000000" maxFlowerWrapperGroupSize="10000000" />
  43. <CactusAVGWrapper />
  44. </avg>
  45. <reference buildReference="1" ignoreUnalignedGaps="1" makeScaffolds="1" matchingAlgorithm="blossom5" maxWalkForCalculatingZ="100000" minNumberOfSequencesToSupportAdjacency="1" numberOfNs="10" permutations="10" phi="1.0" reference="Anc0" theta="0.0001" useSimulatedAnnealing="1" wiggle="0.9999">
  46. <CactusReferenceRecursion maxFlowerGroupSize="100000000" maxFlowerWrapperGroupSize="2000000" />
  47.  
  48. <CactusReferenceWrapper />
  49.  
  50. <CactusSetReferenceCoordinatesUpWrapper />
  51.  
  52. <CactusSetReferenceCoordinatesDownRecursion maxFlowerGroupSize="100000000" maxFlowerWrapperGroupSize="2000000" />
  53.  
  54. <CactusSetReferenceCoordinatesDownWrapper />
  55. </reference>
  56. <check runCheck="0">
  57. <CactusCheckRecursion maxFlowerGroupSize="100000000" maxFlowerWrapperGroupSize="2000000" />
  58. <CactusCheckWrapper />
  59. </check>
  60. <hal buildFasta="1" buildHal="1">
  61. <CactusHalGeneratorRecursion maxFlowerGroupSize="2000000" />
  62. <CactusHalGeneratorUpWrapper />
  63. </hal>
  64.  
  65. <multi_cactus>
  66. <outgroup ancestor_quality_fraction="0.75" max_num_outgroups="3" strategy="greedyPreference" threshold="0" />
  67.  
  68. <decomposition default_internal_node_prefix="Anc" max_parallel_subtrees="50000" self_alignment="false" subtree_size="2" />
  69.  
  70. </multi_cactus>
  71.  
  72. <ProgressiveNext />
  73.  
  74. <ProgressiveUp />
  75.  
  76. <ProgressiveDown />
  77.  
  78. <ProgressiveOut />
  79.  
  80. <RunCactusPreprocessorThenProgressiveDown />
  81.  
  82. <RunCactusPreprocessorThenProgressiveDown2 />
  83.  
  84. <exportHal disk="2000000000" />
  85. </cactusWorkflowConfig>
  86. INFO:toil.statsAndLogging:Got message from job at time 03-22-2019 18:06:05: Leaf names = ['CA_SC5314', 'SC_YJM555', 'SC_YPS163', 'SC_sacCer3', 'CG_CBS138']
  87. INFO:toil.leader:Job ended successfully: 'RunCactusPreprocessorThenProgressiveDown' S/o/job51DlDr
  88. INFO:toil.leader:Issued job 'logAssemblyStats' q/8/jobPFdgkO with job batch system ID: 1 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  89. INFO:toil.leader:Issued job 'logAssemblyStats' n/B/jobFeLOGj with job batch system ID: 2 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  90. INFO:toil.leader:Issued job 'logAssemblyStats' N/p/jobPk6LCN with job batch system ID: 3 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  91. INFO:toil.leader:Issued job 'logAssemblyStats' x/T/jobGR9d6i with job batch system ID: 4 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  92. INFO:toil.leader:Issued job 'logAssemblyStats' r/r/jobFbnvaq with job batch system ID: 5 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  93. INFO:toil.leader:Issued job 'CactusPreprocessor' m/w/jobwthaWH with job batch system ID: 6 and cores: 1, disk: 100.0 M, and memory: 30.0 G
  94. INFO:toil.statsAndLogging:Got message from job at time 03-22-2019 18:07:53: Before preprocessing, got assembly stats for genome SC_YPS163: Input-sample: /export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpdDG9eX/3be745d4-5d25-4da5-aaf5-2107b9108a80/tmpABqp5P.tmp Total-sequences: 607 Total-length: 11646928 Proportion-repeat-masked: 0.112232 ProportionNs: 0.001426 Total-Ns: 16608 N50: 39925 Median-sequence-length: 10020 Max-sequence-length: 170627 Min-sequence-length: 202
  95.  
  96. INFO:toil.statsAndLogging:Got message from job at time 03-22-2019 18:07:53: Before preprocessing, got assembly stats for genome SC_YJM555: Input-sample: /export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-906375ca-4612-4b9c-aace-ad64276eee7f/tmpxli04L/645a4dd5-2a7f-42ff-b1c4-205132aafa72/tmp2xj2AJ.tmp Total-sequences: 18 Total-length: 13295507 Proportion-repeat-masked: 0.109809 ProportionNs: 0.000429 Total-Ns: 5702 N50: 909661 Median-sequence-length: 730916 Max-sequence-length: 2586410 Min-sequence-length: 6117
  97.  
  98. INFO:toil.statsAndLogging:Got message from job at time 03-22-2019 18:07:55: Before preprocessing, got assembly stats for genome CG_CBS138: Input-sample: /export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpDWJSrW/141a6dda-d8c1-4ef1-86e6-29d936402055/tmpqa6o6T.tmp Total-sequences: 13 Total-length: 12318245 Proportion-repeat-masked: 0.104675 ProportionNs: 0.000024 Total-Ns: 300 N50: 1100349 Median-sequence-length: 992211 Max-sequence-length: 1455689 Min-sequence-length: 491328
  99.  
  100. INFO:toil.statsAndLogging:Got message from job at time 03-22-2019 18:07:55: Before preprocessing, got assembly stats for genome CA_SC5314: Input-sample: /export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-88711676-5948-47d0-acd3-569974301115/tmp9JNaQH/2c53c17e-0a51-4ab6-aff8-26f9935aed35/tmpcN28gw.tmp Total-sequences: 8 Total-length: 14282666 Proportion-repeat-masked: 0.215774 ProportionNs: 0.000446 Total-Ns: 6366 N50: 2231883 Median-sequence-length: 1799298 Max-sequence-length: 3188341 Min-sequence-length: 949511
  101.  
  102. INFO:toil.statsAndLogging:Got message from job at time 03-22-2019 18:07:57: Before preprocessing, got assembly stats for genome SC_sacCer3: Input-sample: /export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-b02a3811-2b63-4208-851d-7815af46a62d/tmphVdZmh/912df3be-ad63-4076-9d47-891b0343d341/tmp0_rZsf.tmp Total-sequences: 16 Total-length: 12071326 Proportion-repeat-masked: 0.127145 ProportionNs: 0.000000 Total-Ns: 0 N50: 924431 Median-sequence-length: 784333 Max-sequence-length: 1531933 Min-sequence-length: 230218
  103.  
  104. INFO:toil.leader:Job ended successfully: 'logAssemblyStats' n/B/jobFeLOGj
  105. INFO:toil.leader:Job ended successfully: 'logAssemblyStats' N/p/jobPk6LCN
  106. INFO:toil.leader:Job ended successfully: 'logAssemblyStats' x/T/jobGR9d6i
  107. INFO:toil.leader:Job ended successfully: 'logAssemblyStats' r/r/jobFbnvaq
  108. INFO:toil.leader:Job ended successfully: 'CactusPreprocessor' m/w/jobwthaWH
  109. INFO:toil.leader:Issued job 'CactusPreprocessor2' o/n/job2fJYU_ with job batch system ID: 7 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  110. INFO:toil.leader:Issued job 'CactusPreprocessor2' 8/w/joblU67kI with job batch system ID: 8 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  111. INFO:toil.leader:Issued job 'CactusPreprocessor2' 4/S/job_14EQ2 with job batch system ID: 9 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  112. INFO:toil.leader:Issued job 'CactusPreprocessor2' J/m/jobGiCpgv with job batch system ID: 10 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  113. INFO:toil.leader:Issued job 'CactusPreprocessor2' f/g/jobFgQ_K2 with job batch system ID: 11 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  114. INFO:toil.leader:Job ended successfully: 'logAssemblyStats' q/8/jobPFdgkO
  115. INFO:toil.leader:Job ended successfully: 'CactusPreprocessor2' f/g/jobFgQ_K2
  116. INFO:toil.leader:Issued job 'BatchPreprocessor' V/M/jobTLFgx0 with job batch system ID: 12 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  117. INFO:toil.leader:Job ended successfully: 'CactusPreprocessor2' o/n/job2fJYU_
  118. INFO:toil.leader:Issued job 'BatchPreprocessor' P/z/jobMssUGS with job batch system ID: 13 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  119. INFO:toil.leader:Job ended successfully: 'CactusPreprocessor2' 8/w/joblU67kI
  120. INFO:toil.leader:Issued job 'BatchPreprocessor' t/i/job7Uefxo with job batch system ID: 14 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  121. INFO:toil.leader:Job ended successfully: 'CactusPreprocessor2' 4/S/job_14EQ2
  122. INFO:toil.leader:Issued job 'BatchPreprocessor' f/E/jobnWxtEQ with job batch system ID: 15 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  123. INFO:toil.leader:Job ended successfully: 'CactusPreprocessor2' J/m/jobGiCpgv
  124. INFO:toil.leader:Issued job 'BatchPreprocessor' R/8/jobyfCzJt with job batch system ID: 16 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  125. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' t/i/job7Uefxo
  126. INFO:toil.leader:Issued job 'PreprocessSequence' q/C/job3oVHVz with job batch system ID: 17 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  127. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' f/E/jobnWxtEQ
  128. INFO:toil.leader:Issued job 'PreprocessSequence' 1/u/job6rZvIn with job batch system ID: 18 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  129. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' R/8/jobyfCzJt
  130. INFO:toil.leader:Issued job 'PreprocessSequence' j/2/job4paabJ with job batch system ID: 19 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  131. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' V/M/jobTLFgx0
  132. INFO:toil.leader:Issued job 'PreprocessSequence' M/D/jobXd0oRn with job batch system ID: 20 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  133. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' P/z/jobMssUGS
  134. INFO:toil.leader:Issued job 'PreprocessSequence' l/j/job3uxssM with job batch system ID: 21 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  135. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' j/2/job4paabJ
  136. INFO:toil.leader:Issued job 'CheckUniqueHeaders' a/D/jobKDuUyE with job batch system ID: 22 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  137. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' M/D/jobXd0oRn
  138. INFO:toil.leader:Issued job 'CheckUniqueHeaders' q/X/jobxfeQr9 with job batch system ID: 23 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  139. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' l/j/job3uxssM
  140. INFO:toil.leader:Issued job 'CheckUniqueHeaders' u/c/jobVh2c5J with job batch system ID: 24 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  141. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' q/C/job3oVHVz
  142. INFO:toil.leader:Issued job 'CheckUniqueHeaders' P/x/jobmlUldC with job batch system ID: 25 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  143. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' 1/u/job6rZvIn
  144. INFO:toil.leader:Issued job 'CheckUniqueHeaders' 9/Y/jobbaEMFw with job batch system ID: 26 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  145. INFO:toil.leader:Job ended successfully: 'CheckUniqueHeaders' a/D/jobKDuUyE
  146. INFO:toil.leader:Issued job 'PreprocessSequence' j/2/job4paabJ with job batch system ID: 27 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  147. INFO:toil.leader:Job ended successfully: 'CheckUniqueHeaders' q/X/jobxfeQr9
  148. INFO:toil.leader:Issued job 'PreprocessSequence' M/D/jobXd0oRn with job batch system ID: 28 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  149. INFO:toil.leader:Job ended successfully: 'CheckUniqueHeaders' u/c/jobVh2c5J
  150. INFO:toil.leader:Issued job 'PreprocessSequence' l/j/job3uxssM with job batch system ID: 29 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  151. INFO:toil.leader:Job ended successfully: 'CheckUniqueHeaders' P/x/jobmlUldC
  152. INFO:toil.leader:Issued job 'PreprocessSequence' q/C/job3oVHVz with job batch system ID: 30 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  153. INFO:toil.leader:Job ended successfully: 'CheckUniqueHeaders' 9/Y/jobbaEMFw
  154. INFO:toil.leader:Issued job 'PreprocessSequence' 1/u/job6rZvIn with job batch system ID: 31 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  155. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' j/2/job4paabJ
  156. INFO:toil.leader:Issued job 'BatchPreprocessor' f/c/jobSvXPTx with job batch system ID: 32 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  157. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' M/D/jobXd0oRn
  158. INFO:toil.leader:Issued job 'BatchPreprocessor' F/7/jobHY7oei with job batch system ID: 33 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  159. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' l/j/job3uxssM
  160. INFO:toil.leader:Issued job 'BatchPreprocessor' A/i/jobPbKLFQ with job batch system ID: 34 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  161. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' q/C/job3oVHVz
  162. INFO:toil.leader:Issued job 'BatchPreprocessor' C/r/jobVXmWzQ with job batch system ID: 35 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  163. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' 1/u/job6rZvIn
  164. INFO:toil.leader:Issued job 'BatchPreprocessor' m/P/job2_VgsV with job batch system ID: 36 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  165. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' f/c/jobSvXPTx
  166. INFO:toil.leader:Issued job 'PreprocessSequence' b/l/jobz2dp0e with job batch system ID: 37 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  167. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' F/7/jobHY7oei
  168. INFO:toil.leader:Issued job 'PreprocessSequence' J/d/jobqfiAMw with job batch system ID: 38 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  169. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' A/i/jobPbKLFQ
  170. INFO:toil.leader:Issued job 'PreprocessSequence' P/M/jobTtd_mC with job batch system ID: 39 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  171. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' C/r/jobVXmWzQ
  172. INFO:toil.leader:Issued job 'PreprocessSequence' 3/K/jobg9Fzpi with job batch system ID: 40 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  173. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' m/P/job2_VgsV
  174. INFO:toil.leader:Issued job 'PreprocessSequence' z/p/jobrbRL6o with job batch system ID: 41 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  175. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' J/d/jobqfiAMw
  176. INFO:toil.leader:Issued job 'LastzRepeatMaskJob' Q/b/jobJovEl7 with job batch system ID: 42 and cores: 1, disk: 100.0 M, and memory: 4.0 G
  177. INFO:toil.leader:Issued job 'LastzRepeatMaskJob' U/s/jobtQBUPU with job batch system ID: 43 and cores: 1, disk: 100.0 M, and memory: 4.0 G
  178. INFO:toil.leader:Issued job 'LastzRepeatMaskJob' A/G/jobXfghmL with job batch system ID: 44 and cores: 1, disk: 100.0 M, and memory: 4.0 G
  179. INFO:toil.leader:Issued job 'LastzRepeatMaskJob' I/H/job4cQCj3 with job batch system ID: 45 and cores: 1, disk: 100.0 M, and memory: 4.0 G
  180. INFO:toil.leader:Issued job 'LastzRepeatMaskJob' n/a/job7oxerF with job batch system ID: 46 and cores: 1, disk: 100.0 M, and memory: 4.0 G
  181. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' P/M/jobTtd_mC
  182. INFO:toil.leader:Issued job 'LastzRepeatMaskJob' O/A/job7vfZmV with job batch system ID: 47 and cores: 1, disk: 100.0 M, and memory: 4.0 G
  183. INFO:toil.leader:Issued job 'LastzRepeatMaskJob' Q/B/jobqqsoWf with job batch system ID: 48 and cores: 1, disk: 100.0 M, and memory: 4.0 G
  184. INFO:toil.leader:Issued job 'LastzRepeatMaskJob' V/I/jobNTK1Ns with job batch system ID: 49 and cores: 1, disk: 100.0 M, and memory: 4.0 G
  185. INFO:toil.leader:Issued job 'LastzRepeatMaskJob' N/U/joboa7JGC with job batch system ID: 50 and cores: 1, disk: 100.0 M, and memory: 4.0 G
  186. INFO:toil.leader:Issued job 'LastzRepeatMaskJob' 3/Z/jobydncTd with job batch system ID: 51 and cores: 1, disk: 100.0 M, and memory: 4.0 G
  187. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' 3/K/jobg9Fzpi
  188. INFO:toil.leader:Issued job 'LastzRepeatMaskJob' a/J/jobdsjDp1 with job batch system ID: 52 and cores: 1, disk: 100.0 M, and memory: 4.0 G
  189. INFO:toil.leader:Issued job 'LastzRepeatMaskJob' n/t/jobvE4jLI with job batch system ID: 53 and cores: 1, disk: 100.0 M, and memory: 4.0 G
  190. INFO:toil.leader:Issued job 'LastzRepeatMaskJob' 4/R/job4QAvXX with job batch system ID: 54 and cores: 1, disk: 100.0 M, and memory: 4.0 G
  191. INFO:toil.leader:Issued job 'LastzRepeatMaskJob' Q/i/job7QDnRC with job batch system ID: 55 and cores: 1, disk: 100.0 M, and memory: 4.0 G
  192. INFO:toil.leader:Issued job 'LastzRepeatMaskJob' F/b/jobzIeD9T with job batch system ID: 56 and cores: 1, disk: 100.0 M, and memory: 4.0 G
  193. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' z/p/jobrbRL6o
  194. INFO:toil.leader:Issued job 'LastzRepeatMaskJob' 0/1/jobr1ate4 with job batch system ID: 57 and cores: 1, disk: 100.0 M, and memory: 4.0 G
  195. INFO:toil.leader:Issued job 'LastzRepeatMaskJob' H/I/job3QQQrL with job batch system ID: 58 and cores: 1, disk: 100.0 M, and memory: 4.0 G
  196. INFO:toil.leader:Issued job 'LastzRepeatMaskJob' J/l/jobzvyaQK with job batch system ID: 59 and cores: 1, disk: 100.0 M, and memory: 4.0 G
  197. INFO:toil.leader:Issued job 'LastzRepeatMaskJob' J/o/job92O_79 with job batch system ID: 60 and cores: 1, disk: 100.0 M, and memory: 4.0 G
  198. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' b/l/jobz2dp0e
  199. INFO:toil.leader:Issued job 'LastzRepeatMaskJob' s/E/jobOsyU5P with job batch system ID: 61 and cores: 1, disk: 100.0 M, and memory: 4.0 G
  200. INFO:toil.leader:Issued job 'LastzRepeatMaskJob' V/D/jobqD65b5 with job batch system ID: 62 and cores: 1, disk: 100.0 M, and memory: 4.0 G
  201. INFO:toil.leader:Issued job 'LastzRepeatMaskJob' a/I/jobGYhaxc with job batch system ID: 63 and cores: 1, disk: 100.0 M, and memory: 4.0 G
  202. INFO:toil.leader:Issued job 'LastzRepeatMaskJob' C/p/jobdFSZFS with job batch system ID: 64 and cores: 1, disk: 100.0 M, and memory: 4.0 G
  203. INFO:toil.leader:Issued job 'LastzRepeatMaskJob' F/8/jobf2aY9K with job batch system ID: 65 and cores: 1, disk: 100.0 M, and memory: 4.0 G
  204. WARNING:toil.statsAndLogging:Got message from job at time 03-22-2019 18:27:49: Job used more disk than requested. Consider modifying the user script to avoid the chance of failure due to incorrectly requested resources. Job 3/K/jobg9Fzpi/g/tmp27_rp0-_serialiseJob-stream used 152.25% (152.3 MB [159649792B] used, 100.0 MB [104857600B] requested) at the end of its run.
  205. INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' Q/b/jobJovEl7
  206. INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' U/s/jobtQBUPU
  207. INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' A/G/jobXfghmL
  208. INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' I/H/job4cQCj3
  209. INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' n/a/job7oxerF
  210. INFO:toil.leader:Issued job 'MergeChunks' T/I/jobR7za68 with job batch system ID: 66 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  211. INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' O/A/job7vfZmV
  212. INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' Q/B/jobqqsoWf
  213. INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' V/I/jobNTK1Ns
  214. INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' N/U/joboa7JGC
  215. INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' 3/Z/jobydncTd
  216. INFO:toil.leader:Issued job 'MergeChunks' K/l/jobmgkWxZ with job batch system ID: 67 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  217. INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' a/J/jobdsjDp1
  218. INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' n/t/jobvE4jLI
  219. INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' 4/R/job4QAvXX
  220. INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' Q/i/job7QDnRC
  221. INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' F/b/jobzIeD9T
  222. INFO:toil.leader:Issued job 'MergeChunks' W/e/jobBlDQXh with job batch system ID: 68 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  223. INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' 0/1/jobr1ate4
  224. INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' H/I/job3QQQrL
  225. INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' J/l/jobzvyaQK
  226. INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' J/o/job92O_79
  227. INFO:toil.leader:Issued job 'MergeChunks' m/D/jobIswNxR with job batch system ID: 69 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  228. INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' s/E/jobOsyU5P
  229. INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' V/D/jobqD65b5
  230. INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' a/I/jobGYhaxc
  231. INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' C/p/jobdFSZFS
  232. INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' F/8/jobf2aY9K
  233. INFO:toil.leader:Issued job 'MergeChunks' 5/M/jobiZXatL with job batch system ID: 70 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  234. INFO:toil.leader:Job ended successfully: 'MergeChunks' W/e/jobBlDQXh
  235. INFO:toil.leader:Issued job 'MergeChunks2' 6/B/jobYSx9FK with job batch system ID: 71 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  236. INFO:toil.leader:Job ended successfully: 'MergeChunks' m/D/jobIswNxR
  237. INFO:toil.leader:Issued job 'MergeChunks2' h/R/jobfPv4QG with job batch system ID: 72 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  238. INFO:toil.leader:Job ended successfully: 'MergeChunks' 5/M/jobiZXatL
  239. INFO:toil.leader:Issued job 'MergeChunks2' z/G/jobFTmlWe with job batch system ID: 73 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  240. INFO:toil.leader:Job ended successfully: 'MergeChunks' T/I/jobR7za68
  241. INFO:toil.leader:Issued job 'MergeChunks2' 3/D/jobEqbMFz with job batch system ID: 74 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  242. INFO:toil.leader:Job ended successfully: 'MergeChunks' K/l/jobmgkWxZ
  243. INFO:toil.leader:Issued job 'MergeChunks2' z/L/jobxhxZti with job batch system ID: 75 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  244. INFO:toil.leader:Job ended successfully: 'MergeChunks2' h/R/jobfPv4QG
  245. INFO:toil.leader:Issued job 'MergeChunks' m/D/jobIswNxR with job batch system ID: 76 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  246. INFO:toil.leader:Job ended successfully: 'MergeChunks2' z/G/jobFTmlWe
  247. INFO:toil.leader:Issued job 'MergeChunks' 5/M/jobiZXatL with job batch system ID: 77 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  248. INFO:toil.leader:Job ended successfully: 'MergeChunks2' 3/D/jobEqbMFz
  249. INFO:toil.leader:Issued job 'MergeChunks' T/I/jobR7za68 with job batch system ID: 78 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  250. INFO:toil.leader:Job ended successfully: 'MergeChunks2' z/L/jobxhxZti
  251. INFO:toil.leader:Issued job 'MergeChunks' K/l/jobmgkWxZ with job batch system ID: 79 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  252. INFO:toil.leader:Job ended successfully: 'MergeChunks2' 6/B/jobYSx9FK
  253. INFO:toil.leader:Issued job 'MergeChunks' W/e/jobBlDQXh with job batch system ID: 80 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  254. INFO:toil.leader:Job ended successfully: 'MergeChunks' m/D/jobIswNxR
  255. INFO:toil.leader:Issued job 'PreprocessSequence' z/p/jobrbRL6o with job batch system ID: 81 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  256. INFO:toil.leader:Job ended successfully: 'MergeChunks' 5/M/jobiZXatL
  257. INFO:toil.leader:Issued job 'PreprocessSequence' b/l/jobz2dp0e with job batch system ID: 82 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  258. INFO:toil.leader:Job ended successfully: 'MergeChunks' T/I/jobR7za68
  259. INFO:toil.leader:Issued job 'PreprocessSequence' J/d/jobqfiAMw with job batch system ID: 83 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  260. INFO:toil.leader:Job ended successfully: 'MergeChunks' K/l/jobmgkWxZ
  261. INFO:toil.leader:Issued job 'PreprocessSequence' P/M/jobTtd_mC with job batch system ID: 84 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  262. INFO:toil.leader:Job ended successfully: 'MergeChunks' W/e/jobBlDQXh
  263. INFO:toil.leader:Issued job 'PreprocessSequence' 3/K/jobg9Fzpi with job batch system ID: 85 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  264. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' 3/K/jobg9Fzpi
  265. INFO:toil.leader:Issued job 'BatchPreprocessor' C/r/jobVXmWzQ with job batch system ID: 86 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  266. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' z/p/jobrbRL6o
  267. INFO:toil.leader:Issued job 'BatchPreprocessor' m/P/job2_VgsV with job batch system ID: 87 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  268. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' b/l/jobz2dp0e
  269. INFO:toil.leader:Issued job 'BatchPreprocessor' f/c/jobSvXPTx with job batch system ID: 88 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  270. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' J/d/jobqfiAMw
  271. INFO:toil.leader:Issued job 'BatchPreprocessor' F/7/jobHY7oei with job batch system ID: 89 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  272. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' P/M/jobTtd_mC
  273. INFO:toil.leader:Issued job 'BatchPreprocessor' A/i/jobPbKLFQ with job batch system ID: 90 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  274. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' F/7/jobHY7oei
  275. INFO:toil.leader:Issued job 'BatchPreprocessor' V/M/jobTLFgx0 with job batch system ID: 91 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  276. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' A/i/jobPbKLFQ
  277. INFO:toil.leader:Issued job 'BatchPreprocessor' P/z/jobMssUGS with job batch system ID: 92 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  278. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' C/r/jobVXmWzQ
  279. INFO:toil.leader:Issued job 'BatchPreprocessor' t/i/job7Uefxo with job batch system ID: 93 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  280. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' m/P/job2_VgsV
  281. INFO:toil.leader:Issued job 'BatchPreprocessor' f/E/jobnWxtEQ with job batch system ID: 94 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  282. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' f/c/jobSvXPTx
  283. INFO:toil.leader:Issued job 'BatchPreprocessor' R/8/jobyfCzJt with job batch system ID: 95 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  284. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' t/i/job7Uefxo
  285. INFO:toil.leader:Issued job 'CactusPreprocessor2' 8/w/joblU67kI with job batch system ID: 96 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  286. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' f/E/jobnWxtEQ
  287. INFO:toil.leader:Issued job 'CactusPreprocessor2' 4/S/job_14EQ2 with job batch system ID: 97 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  288. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' R/8/jobyfCzJt
  289. INFO:toil.leader:Issued job 'CactusPreprocessor2' J/m/jobGiCpgv with job batch system ID: 98 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  290. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' V/M/jobTLFgx0
  291. INFO:toil.leader:Issued job 'CactusPreprocessor2' f/g/jobFgQ_K2 with job batch system ID: 99 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  292. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' P/z/jobMssUGS
  293. INFO:toil.leader:Issued job 'CactusPreprocessor2' o/n/job2fJYU_ with job batch system ID: 100 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  294. INFO:toil.leader:Job ended successfully: 'CactusPreprocessor2' f/g/jobFgQ_K2
  295. INFO:toil.leader:Job ended successfully: 'CactusPreprocessor2' o/n/job2fJYU_
  296. INFO:toil.leader:Job ended successfully: 'CactusPreprocessor2' 8/w/joblU67kI
  297. INFO:toil.leader:Job ended successfully: 'CactusPreprocessor2' 4/S/job_14EQ2
  298. INFO:toil.leader:Job ended successfully: 'CactusPreprocessor2' J/m/jobGiCpgv
  299. INFO:toil.leader:Issued job 'CactusPreprocessor' m/w/jobwthaWH with job batch system ID: 101 and cores: 1, disk: 100.0 M, and memory: 30.0 G
  300. INFO:toil.leader:Job ended successfully: 'CactusPreprocessor' m/w/jobwthaWH
  301. INFO:toil.leader:Issued job 'RunCactusPreprocessorThenProgressiveDown2' t/h/jobMDYFld with job batch system ID: 102 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  302. INFO:toil.leader:Job ended successfully: 'RunCactusPreprocessorThenProgressiveDown2' t/h/jobMDYFld
  303. INFO:toil.leader:Issued job 'logAssemblyStats' b/v/job6GcRsc with job batch system ID: 103 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  304. INFO:toil.leader:Issued job 'logAssemblyStats' l/S/joblfXbvh with job batch system ID: 104 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  305. INFO:toil.leader:Issued job 'logAssemblyStats' F/r/job5FseXs with job batch system ID: 105 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  306. INFO:toil.leader:Issued job 'logAssemblyStats' X/j/jobH9ac3W with job batch system ID: 106 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  307. INFO:toil.leader:Issued job 'logAssemblyStats' 2/s/jobhmhKHS with job batch system ID: 107 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  308. INFO:toil.leader:Issued job 'ProgressiveDown' g/l/jobFGVVE1 with job batch system ID: 108 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  309. INFO:toil.statsAndLogging:Got message from job at time 03-22-2019 18:53:06: After preprocessing, got assembly stats for genome SC_sacCer3: Input-sample: /export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpm3_q5a/d7d1d92a-58aa-49bb-a518-9a05d4e3bccc/tmpUocHdR.tmp Total-sequences: 16 Total-length: 12071326 Proportion-repeat-masked: 0.129184 ProportionNs: 0.000000 Total-Ns: 0 N50: 924431 Median-sequence-length: 784333 Max-sequence-length: 1531933 Min-sequence-length: 230218
  310.  
  311. INFO:toil.statsAndLogging:Got message from job at time 03-22-2019 18:53:10: There are 0 dependent projects
  312. INFO:toil.statsAndLogging:Got message from job at time 03-22-2019 18:53:10: After preprocessing, got assembly stats for genome CG_CBS138: Input-sample: /export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpu7OExG/50f8b4e5-ba73-4e47-a123-4dd85b69d798/tmpeYx6kh.tmp Total-sequences: 13 Total-length: 12318245 Proportion-repeat-masked: 0.106714 ProportionNs: 0.000024 Total-Ns: 300 N50: 1100349 Median-sequence-length: 992211 Max-sequence-length: 1455689 Min-sequence-length: 491328
  313.  
  314. INFO:toil.statsAndLogging:Got message from job at time 03-22-2019 18:53:11: After preprocessing, got assembly stats for genome SC_YPS163: Input-sample: /export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-88711676-5948-47d0-acd3-569974301115/tmp8muoK2/4c8db99b-63e5-49d3-87aa-2bb2a86bae1b/tmpL2gIko.tmp Total-sequences: 607 Total-length: 11646928 Proportion-repeat-masked: 0.112651 ProportionNs: 0.001426 Total-Ns: 16608 N50: 39925 Median-sequence-length: 10020 Max-sequence-length: 170627 Min-sequence-length: 202
  315.  
  316. INFO:toil.statsAndLogging:Got message from job at time 03-22-2019 18:53:11: After preprocessing, got assembly stats for genome CA_SC5314: Input-sample: /export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpKlC2Do/e7f60581-4dc9-4423-95c2-d11b89d816cc/tmpFUVOPl.tmp Total-sequences: 8 Total-length: 14282666 Proportion-repeat-masked: 0.217227 ProportionNs: 0.000446 Total-Ns: 6366 N50: 2231883 Median-sequence-length: 1799298 Max-sequence-length: 3188341 Min-sequence-length: 949511
  317.  
  318. INFO:toil.statsAndLogging:Got message from job at time 03-22-2019 18:53:11: After preprocessing, got assembly stats for genome SC_YJM555: Input-sample: /export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpOKvLWG/0f1c54b3-2bd4-4306-97f1-601ab020ac60/tmpzSk5qY.tmp Total-sequences: 18 Total-length: 13295507 Proportion-repeat-masked: 0.217636 ProportionNs: 0.000429 Total-Ns: 5702 N50: 909661 Median-sequence-length: 730916 Max-sequence-length: 2586410 Min-sequence-length: 6117
  319.  
  320. INFO:toil.leader:Job ended successfully: 'logAssemblyStats' b/v/job6GcRsc
  321. INFO:toil.leader:Job ended successfully: 'logAssemblyStats' l/S/joblfXbvh
  322. INFO:toil.leader:Job ended successfully: 'logAssemblyStats' F/r/job5FseXs
  323. INFO:toil.leader:Job ended successfully: 'logAssemblyStats' X/j/jobH9ac3W
  324. INFO:toil.leader:Job ended successfully: 'logAssemblyStats' 2/s/jobhmhKHS
  325. INFO:toil.leader:Job ended successfully: 'ProgressiveDown' g/l/jobFGVVE1
  326. INFO:toil.leader:Issued job 'ProgressiveNext' P/3/jobUyelVB with job batch system ID: 109 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  327. INFO:toil.statsAndLogging:Got message from job at time 03-22-2019 18:55:49: Project has 0 dependencies
  328. INFO:toil.leader:Job ended successfully: 'ProgressiveNext' P/3/jobUyelVB
  329. INFO:toil.leader:Issued job 'ProgressiveUp' x/L/jobMNjZYG with job batch system ID: 110 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  330. INFO:toil.leader:Job ended successfully: 'ProgressiveUp' x/L/jobMNjZYG
  331. INFO:toil.leader:Issued job 'CactusTrimmingBlastPhase' u/S/jobhfwRMO with job batch system ID: 111 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  332. INFO:toil.statsAndLogging:Got message from job at time 03-22-2019 19:00:24: Running blast using the trimming strategy
  333. INFO:toil.statsAndLogging:Got message from job at time 03-22-2019 19:00:24: Ingroup sequences: [('SC_YJM555', 'u/S/jobhfwRMO/g/tmpUKp_RA-run-tmpZjdmjg.tmp'), ('SC_YPS163', 'u/S/jobhfwRMO/g/tmpgVPKkt-run-tmpDqlcDZ.tmp'), ('CA_SC5314', 'u/S/jobhfwRMO/g/tmp9DCB7A-run-tmp52q85q.tmp'), ('CG_CBS138', 'u/S/jobhfwRMO/g/tmpIvjfoC-run-tmpbTkgbi.tmp'), ('SC_sacCer3', 'u/S/jobhfwRMO/g/tmpKlnm7r-run-tmpdpoUbs.tmp')]
  334. INFO:toil.statsAndLogging:Got message from job at time 03-22-2019 19:00:24: Outgroup sequences: []
  335. INFO:toil.statsAndLogging:Got message from job at time 03-22-2019 19:00:24: Running mapQ uniquifying with parameters, minimumMapQValue: 0.0, maxAlignmentsPerSite 5, alpha: 0.001
  336. INFO:toil.leader:Job ended successfully: 'CactusTrimmingBlastPhase' u/S/jobhfwRMO
  337. INFO:toil.leader:Issued job 'EncapsulatedJob' T/v/jobnTxoOq with job batch system ID: 112 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  338. INFO:toil.leader:Job ended successfully: 'EncapsulatedJob' T/v/jobnTxoOq
  339. INFO:toil.leader:Issued job 'BlastIngroupsAndOutgroups' 7/R/jobcYz6Jp with job batch system ID: 113 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  340. INFO:toil.leader:Reissued any over long jobs
  341. INFO:toil.statsAndLogging:Got message from job at time 03-22-2019 19:05:06: Blasting ingroups vs outgroups. Ingroup genomes: SC_YJM555, SC_YPS163, CA_SC5314, CG_CBS138, SC_sacCer3, outgroup genomes:
  342. INFO:toil.leader:Job ended successfully: 'BlastIngroupsAndOutgroups' 7/R/jobcYz6Jp
  343. INFO:toil.leader:Issued job 'BlastSequencesAllAgainstAll' l/9/jobt3u71J with job batch system ID: 114 and cores: 1, disk: 300.0 M, and memory: 2.0 G
  344. INFO:toil.leader:Job ended successfully: 'BlastSequencesAllAgainstAll' l/9/jobt3u71J
  345. INFO:toil.leader:Issued job 'MakeSelfBlasts' p/D/job4ps_2w with job batch system ID: 115 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  346. INFO:toil.leader:Issued job 'MakeOffDiagonalBlasts' V/F/jobJZwizU with job batch system ID: 116 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  347. INFO:toil.leader:Job ended successfully: 'MakeSelfBlasts' p/D/job4ps_2w
  348. INFO:toil.leader:Issued job 'RunSelfBlast' a/v/jobQU2x0P with job batch system ID: 117 and cores: 1, disk: 100.0 M, and memory: 5.1 G
  349. INFO:toil.leader:Issued job 'RunSelfBlast' U/T/jobNrUYTO with job batch system ID: 118 and cores: 1, disk: 100.0 M, and memory: 5.1 G
  350. INFO:toil.leader:Issued job 'RunSelfBlast' i/P/jobi4Cx2N with job batch system ID: 119 and cores: 1, disk: 100.0 M, and memory: 5.1 G
  351. INFO:toil.leader:Job ended successfully: 'MakeOffDiagonalBlasts' V/F/jobJZwizU
  352. INFO:toil.leader:Issued job 'RunBlast' B/V/jobp893JN with job batch system ID: 120 and cores: 1, disk: 100.0 M, and memory: 5.2 G
  353. INFO:toil.leader:Issued job 'RunBlast' W/2/job3_8PUa with job batch system ID: 121 and cores: 1, disk: 100.0 M, and memory: 5.1 G
  354. INFO:toil.leader:Issued job 'RunBlast' Z/Y/jobbZVnqJ with job batch system ID: 122 and cores: 1, disk: 100.0 M, and memory: 5.1 G
  355. INFO:toil.leader:Job ended successfully: 'RunSelfBlast' i/P/jobi4Cx2N
  356. INFO:toil.leader:Job ended successfully: 'RunBlast' Z/Y/jobbZVnqJ
  357. INFO:toil.leader:Job ended successfully: 'RunSelfBlast' U/T/jobNrUYTO
  358. INFO:toil.leader:Job ended successfully: 'RunBlast' B/V/jobp893JN
  359. INFO:toil.leader:Job ended successfully: 'RunBlast' W/2/job3_8PUa
  360. INFO:toil.leader:Issued job 'CollateBlasts' B/d/job3TMcRO with job batch system ID: 123 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  361. INFO:toil.leader:Job ended successfully: 'CollateBlasts' B/d/job3TMcRO
  362. INFO:toil.leader:Issued job 'CollateBlasts2' 1/1/jobiEgXIj with job batch system ID: 124 and cores: 1, disk: 200.0 M, and memory: 2.0 G
  363. INFO:toil.leader:Job ended successfully: 'RunSelfBlast' a/v/jobQU2x0P
  364. INFO:toil.leader:Issued job 'CollateBlasts' V/L/jobHnqXNx with job batch system ID: 125 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  365. INFO:toil.leader:Job ended successfully: 'CollateBlasts2' 1/1/jobiEgXIj
  366. INFO:toil.leader:Issued job 'CollateBlasts' B/d/job3TMcRO with job batch system ID: 126 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  367. INFO:toil.leader:Job ended successfully: 'CollateBlasts' B/d/job3TMcRO
  368. INFO:toil.leader:Issued job 'MakeOffDiagonalBlasts' V/F/jobJZwizU with job batch system ID: 127 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  369. INFO:toil.leader:Job ended successfully: 'CollateBlasts' V/L/jobHnqXNx
  370. INFO:toil.leader:Issued job 'CollateBlasts2' M/A/jobnTA6Nc with job batch system ID: 128 and cores: 1, disk: 200.0 M, and memory: 2.0 G
  371. INFO:toil.leader:Job ended successfully: 'CollateBlasts2' M/A/jobnTA6Nc
  372. INFO:toil.leader:Issued job 'CollateBlasts' V/L/jobHnqXNx with job batch system ID: 129 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  373. INFO:toil.leader:Job ended successfully: 'MakeOffDiagonalBlasts' V/F/jobJZwizU
  374. INFO:toil.leader:Job ended successfully: 'CollateBlasts' V/L/jobHnqXNx
  375. INFO:toil.leader:Issued job 'MakeSelfBlasts' p/D/job4ps_2w with job batch system ID: 130 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  376. INFO:toil.leader:Job ended successfully: 'MakeSelfBlasts' p/D/job4ps_2w
  377. INFO:toil.leader:Issued job 'CollateBlasts' V/E/jobfk0qU7 with job batch system ID: 131 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  378. INFO:toil.leader:Job ended successfully: 'CollateBlasts' V/E/jobfk0qU7
  379. INFO:toil.leader:Issued job 'CollateBlasts2' C/c/jobS00wnv with job batch system ID: 132 and cores: 1, disk: 300.0 M, and memory: 2.0 G
  380. INFO:toil.leader:Job ended successfully: 'CollateBlasts2' C/c/jobS00wnv
  381. INFO:toil.leader:Issued job 'CollateBlasts' V/E/jobfk0qU7 with job batch system ID: 133 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  382. INFO:toil.leader:Job ended successfully: 'CollateBlasts' V/E/jobfk0qU7
  383. INFO:toil.leader:Issued job 'BlastSequencesAllAgainstAll' l/9/jobt3u71J with job batch system ID: 134 and cores: 1, disk: 300.0 M, and memory: 2.0 G
  384. INFO:toil.leader:Job ended successfully: 'BlastSequencesAllAgainstAll' l/9/jobt3u71J
  385. INFO:toil.leader:Issued job 'BlastIngroupsAndOutgroups' 7/R/jobcYz6Jp with job batch system ID: 135 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  386. INFO:toil.leader:Job ended successfully: 'BlastIngroupsAndOutgroups' 7/R/jobcYz6Jp
  387. INFO:toil.leader:Issued job 'Job' N/g/jobXSKAfc with job batch system ID: 136 and cores: 0, disk: 1.0 M, and memory: 32.0 M
  388. WARNING:toil.leader:Job failed with exit value 1: 'Job' N/g/jobXSKAfc
  389. WARNING:toil.leader:No log file is present, despite job failing: 'Job' N/g/jobXSKAfc
  390. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'Job' N/g/jobXSKAfc with ID N/g/jobXSKAfc to 5
  391. WARNING:toil.jobGraph:We have increased the default memory of the failed job 'Job' N/g/jobXSKAfc to 32212254720 bytes
  392. INFO:toil.leader:Issued job 'Job' N/g/jobXSKAfc with job batch system ID: 137 and cores: 0, disk: 1.0 M, and memory: 30.0 G
  393. INFO:toil.leader:Job ended successfully: 'Job' N/g/jobXSKAfc
  394. INFO:toil.leader:Issued job 'mappingQualityRescoring' h/5/jobME7sh5 with job batch system ID: 138 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  395. INFO:toil.statsAndLogging:Got message from job at time 03-22-2019 19:53:44: Input cigar file has 196067 lines
  396. INFO:toil.statsAndLogging:Got message from job at time 03-22-2019 19:53:44: Filtered, non-overlapping primary cigar file has 304032 lines
  397. INFO:toil.statsAndLogging:Got message from job at time 03-22-2019 19:53:44: Filtered, non-overlapping secondary cigar file has 1036896 lines
  398. INFO:toil.leader:Job ended successfully: 'mappingQualityRescoring' h/5/jobME7sh5
  399. INFO:toil.leader:Issued job 'Job' N/g/jobXSKAfc with job batch system ID: 139 and cores: 0, disk: 1.0 M, and memory: 30.0 G
  400. INFO:toil.leader:Job ended successfully: 'Job' N/g/jobXSKAfc
  401. INFO:toil.leader:Issued job 'EncapsulatedJob' T/v/jobnTxoOq with job batch system ID: 140 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  402. INFO:toil.leader:Job ended successfully: 'EncapsulatedJob' T/v/jobnTxoOq
  403. INFO:toil.leader:Issued job 'CactusSetupCheckpoint' y/U/jobCVFeXz with job batch system ID: 141 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  404. INFO:toil.leader:Job ended successfully: 'CactusSetupCheckpoint' y/U/jobCVFeXz
  405. INFO:toil.leader:Issued job 'StartPrimaryDB' t/F/jobmGZ5E8 with job batch system ID: 142 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  406. INFO:toil.leader:Job ended successfully: 'StartPrimaryDB' t/F/jobmGZ5E8
  407. INFO:toil.leader:Issued job 'KtServerService' w/W/joblafimE with job batch system ID: 143 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  408. INFO:toil.leader:Reissued any over long jobs
  409. INFO:toil.leader:Issued job 'CactusSetupPhase' 9/9/job2BqGvp with job batch system ID: 144 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  410. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' 9/9/job2BqGvp
  411. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' 9/9/job2BqGvp
  412. WARNING:toil.leader:9/9/job2BqGvp INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  413. WARNING:toil.leader:9/9/job2BqGvp INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  414. WARNING:toil.leader:9/9/job2BqGvp WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  415. WARNING:toil.leader:9/9/job2BqGvp WARNING:toil.fileStore:Detected that job (t/F/jobmGZ5E8/g/tmpWHssLX-_serialiseJob-stream) prematurely terminated. Fixing the state of the job on disk.
  416. WARNING:toil.leader:9/9/job2BqGvp INFO:toil.lib.bioio:Sequences in cactus setup: ['CA_SC5314', 'SC_YJM555', 'SC_YPS163', 'SC_sacCer3', 'CG_CBS138']
  417. WARNING:toil.leader:9/9/job2BqGvp INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=2|1\n', '>id=0|I\n', '>id=1|unplaced-422\n', '>id=4|I\n', '>id=3|A\n']
  418. WARNING:toil.leader:9/9/job2BqGvp INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '(CA_SC5314:1.0,SC_YJM555:1.0,SC_YPS163:1.0,SC_sacCer3:1.0,CG_CBS138:1.0)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="11106" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-88711676-5948-47d0-acd3-569974301115/tmp5tjGzQ/d7276378-4272-4d85-b3c1-758af7a21b95/tmpw2ke4T.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-88711676-5948-47d0-acd3-569974301115/tmp5tjGzQ/d7276378-4272-4d85-b3c1-758af7a21b95/tmpeOu1HW.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-88711676-5948-47d0-acd3-569974301115/tmp5tjGzQ/d7276378-4272-4d85-b3c1-758af7a21b95/tmpf3gOv0.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-88711676-5948-47d0-acd3-569974301115/tmp5tjGzQ/d7276378-4272-4d85-b3c1-758af7a21b95/tmpHgaq_i.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-88711676-5948-47d0-acd3-569974301115/tmp5tjGzQ/d7276378-4272-4d85-b3c1-758af7a21b95/tmpHlfLHS.tmp']
  419. WARNING:toil.leader:9/9/job2BqGvp Set log level to INFO
  420. WARNING:toil.leader:9/9/job2BqGvp Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  421. WARNING:toil.leader:9/9/job2BqGvp <kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="11106" />
  422. WARNING:toil.leader:9/9/job2BqGvp </st_kv_database_conf>
  423. WARNING:toil.leader:9/9/job2BqGvp
  424. WARNING:toil.leader:9/9/job2BqGvp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-88711676-5948-47d0-acd3-569974301115/tmp5tjGzQ/d7276378-4272-4d85-b3c1-758af7a21b95/tmpw2ke4T.tmp
  425. WARNING:toil.leader:9/9/job2BqGvp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-88711676-5948-47d0-acd3-569974301115/tmp5tjGzQ/d7276378-4272-4d85-b3c1-758af7a21b95/tmpeOu1HW.tmp
  426. WARNING:toil.leader:9/9/job2BqGvp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-88711676-5948-47d0-acd3-569974301115/tmp5tjGzQ/d7276378-4272-4d85-b3c1-758af7a21b95/tmpf3gOv0.tmp
  427. WARNING:toil.leader:9/9/job2BqGvp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-88711676-5948-47d0-acd3-569974301115/tmp5tjGzQ/d7276378-4272-4d85-b3c1-758af7a21b95/tmpHgaq_i.tmp
  428. WARNING:toil.leader:9/9/job2BqGvp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-88711676-5948-47d0-acd3-569974301115/tmp5tjGzQ/d7276378-4272-4d85-b3c1-758af7a21b95/tmpHlfLHS.tmp
  429. WARNING:toil.leader:9/9/job2BqGvp Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.35 with error: network error
  430. WARNING:toil.leader:9/9/job2BqGvp Uncaught exception
  431. WARNING:toil.leader:9/9/job2BqGvp Traceback (most recent call last):
  432. WARNING:toil.leader:9/9/job2BqGvp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  433. WARNING:toil.leader:9/9/job2BqGvp job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  434. WARNING:toil.leader:9/9/job2BqGvp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  435. WARNING:toil.leader:9/9/job2BqGvp super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  436. WARNING:toil.leader:9/9/job2BqGvp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  437. WARNING:toil.leader:9/9/job2BqGvp returnValues = self._run(jobGraph, fileStore)
  438. WARNING:toil.leader:9/9/job2BqGvp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  439. WARNING:toil.leader:9/9/job2BqGvp return self.run(fileStore)
  440. WARNING:toil.leader:9/9/job2BqGvp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  441. WARNING:toil.leader:9/9/job2BqGvp makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  442. WARNING:toil.leader:9/9/job2BqGvp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  443. WARNING:toil.leader:9/9/job2BqGvp parameters=["cactus_setup"] + args + sequences)
  444. WARNING:toil.leader:9/9/job2BqGvp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  445. WARNING:toil.leader:9/9/job2BqGvp raise RuntimeError("Command %s failed with output: %s" % (call, output))
  446. WARNING:toil.leader:9/9/job2BqGvp RuntimeError: Command ['cactus_setup', '--speciesTree', '(CA_SC5314:1.0,SC_YJM555:1.0,SC_YPS163:1.0,SC_sacCer3:1.0,CG_CBS138:1.0)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="11106" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-88711676-5948-47d0-acd3-569974301115/tmp5tjGzQ/d7276378-4272-4d85-b3c1-758af7a21b95/tmpw2ke4T.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-88711676-5948-47d0-acd3-569974301115/tmp5tjGzQ/d7276378-4272-4d85-b3c1-758af7a21b95/tmpeOu1HW.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-88711676-5948-47d0-acd3-569974301115/tmp5tjGzQ/d7276378-4272-4d85-b3c1-758af7a21b95/tmpf3gOv0.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-88711676-5948-47d0-acd3-569974301115/tmp5tjGzQ/d7276378-4272-4d85-b3c1-758af7a21b95/tmpHgaq_i.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-88711676-5948-47d0-acd3-569974301115/tmp5tjGzQ/d7276378-4272-4d85-b3c1-758af7a21b95/tmpHlfLHS.tmp'] failed with output:
  447. WARNING:toil.leader:9/9/job2BqGvp ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn35.grid.pub.ro
  448. WARNING:toil.leader:9/9/job2BqGvp WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' 9/9/job2BqGvp with ID 9/9/job2BqGvp to 5
  449. WARNING:toil.leader:9/9/job2BqGvp WARNING:toil.jobGraph:We have increased the default memory of the failed job 'CactusSetupPhase' 9/9/job2BqGvp to 32212254720 bytes
  450. INFO:toil.leader:Issued job 'CactusSetupPhase' 9/9/job2BqGvp with job batch system ID: 145 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  451. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' w/W/joblafimE
  452. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' w/W/joblafimE
  453. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' w/W/joblafimE with ID w/W/joblafimE to 5
  454. WARNING:toil.jobGraph:We have increased the default memory of the failed job 'KtServerService' w/W/joblafimE to 32212254720 bytes
  455. INFO:toil.leader:Issued job 'KtServerService' w/W/joblafimE with job batch system ID: 146 and cores: 0, disk: 2.0 G, and memory: 30.0 G
  456. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' 9/9/job2BqGvp
  457. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' 9/9/job2BqGvp
  458. WARNING:toil.leader:9/9/job2BqGvp INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  459. WARNING:toil.leader:9/9/job2BqGvp INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  460. WARNING:toil.leader:9/9/job2BqGvp WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  461. WARNING:toil.leader:9/9/job2BqGvp INFO:toil.lib.bioio:Sequences in cactus setup: ['CA_SC5314', 'SC_YJM555', 'SC_YPS163', 'SC_sacCer3', 'CG_CBS138']
  462. WARNING:toil.leader:9/9/job2BqGvp INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=2|1\n', '>id=0|I\n', '>id=1|unplaced-422\n', '>id=4|I\n', '>id=3|A\n']
  463. WARNING:toil.leader:9/9/job2BqGvp INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '(CA_SC5314:1.0,SC_YJM555:1.0,SC_YPS163:1.0,SC_sacCer3:1.0,CG_CBS138:1.0)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="11106" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-b02a3811-2b63-4208-851d-7815af46a62d/tmp1TMnNb/106e34c9-450d-4403-ae84-5cd4d4b94a00/tmpdgsok4.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-b02a3811-2b63-4208-851d-7815af46a62d/tmp1TMnNb/106e34c9-450d-4403-ae84-5cd4d4b94a00/tmpUd_Yvr.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-b02a3811-2b63-4208-851d-7815af46a62d/tmp1TMnNb/106e34c9-450d-4403-ae84-5cd4d4b94a00/tmpvQ4u31.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-b02a3811-2b63-4208-851d-7815af46a62d/tmp1TMnNb/106e34c9-450d-4403-ae84-5cd4d4b94a00/tmp645n1s.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-b02a3811-2b63-4208-851d-7815af46a62d/tmp1TMnNb/106e34c9-450d-4403-ae84-5cd4d4b94a00/tmpqlMzFa.tmp']
  464. WARNING:toil.leader:9/9/job2BqGvp Set log level to INFO
  465. WARNING:toil.leader:9/9/job2BqGvp Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  466. WARNING:toil.leader:9/9/job2BqGvp <kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="11106" />
  467. WARNING:toil.leader:9/9/job2BqGvp </st_kv_database_conf>
  468. WARNING:toil.leader:9/9/job2BqGvp
  469. WARNING:toil.leader:9/9/job2BqGvp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-b02a3811-2b63-4208-851d-7815af46a62d/tmp1TMnNb/106e34c9-450d-4403-ae84-5cd4d4b94a00/tmpdgsok4.tmp
  470. WARNING:toil.leader:9/9/job2BqGvp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-b02a3811-2b63-4208-851d-7815af46a62d/tmp1TMnNb/106e34c9-450d-4403-ae84-5cd4d4b94a00/tmpUd_Yvr.tmp
  471. WARNING:toil.leader:9/9/job2BqGvp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-b02a3811-2b63-4208-851d-7815af46a62d/tmp1TMnNb/106e34c9-450d-4403-ae84-5cd4d4b94a00/tmpvQ4u31.tmp
  472. WARNING:toil.leader:9/9/job2BqGvp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-b02a3811-2b63-4208-851d-7815af46a62d/tmp1TMnNb/106e34c9-450d-4403-ae84-5cd4d4b94a00/tmp645n1s.tmp
  473. WARNING:toil.leader:9/9/job2BqGvp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-b02a3811-2b63-4208-851d-7815af46a62d/tmp1TMnNb/106e34c9-450d-4403-ae84-5cd4d4b94a00/tmpqlMzFa.tmp
  474. WARNING:toil.leader:9/9/job2BqGvp Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.35 with error: network error
  475. WARNING:toil.leader:9/9/job2BqGvp Uncaught exception
  476. WARNING:toil.leader:9/9/job2BqGvp Traceback (most recent call last):
  477. WARNING:toil.leader:9/9/job2BqGvp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  478. WARNING:toil.leader:9/9/job2BqGvp job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  479. WARNING:toil.leader:9/9/job2BqGvp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  480. WARNING:toil.leader:9/9/job2BqGvp super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  481. WARNING:toil.leader:9/9/job2BqGvp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  482. WARNING:toil.leader:9/9/job2BqGvp returnValues = self._run(jobGraph, fileStore)
  483. WARNING:toil.leader:9/9/job2BqGvp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  484. WARNING:toil.leader:9/9/job2BqGvp return self.run(fileStore)
  485. WARNING:toil.leader:9/9/job2BqGvp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  486. WARNING:toil.leader:9/9/job2BqGvp makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  487. WARNING:toil.leader:9/9/job2BqGvp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  488. WARNING:toil.leader:9/9/job2BqGvp parameters=["cactus_setup"] + args + sequences)
  489. WARNING:toil.leader:9/9/job2BqGvp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  490. WARNING:toil.leader:9/9/job2BqGvp raise RuntimeError("Command %s failed with output: %s" % (call, output))
  491. WARNING:toil.leader:9/9/job2BqGvp RuntimeError: Command ['cactus_setup', '--speciesTree', '(CA_SC5314:1.0,SC_YJM555:1.0,SC_YPS163:1.0,SC_sacCer3:1.0,CG_CBS138:1.0)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="11106" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-b02a3811-2b63-4208-851d-7815af46a62d/tmp1TMnNb/106e34c9-450d-4403-ae84-5cd4d4b94a00/tmpdgsok4.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-b02a3811-2b63-4208-851d-7815af46a62d/tmp1TMnNb/106e34c9-450d-4403-ae84-5cd4d4b94a00/tmpUd_Yvr.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-b02a3811-2b63-4208-851d-7815af46a62d/tmp1TMnNb/106e34c9-450d-4403-ae84-5cd4d4b94a00/tmpvQ4u31.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-b02a3811-2b63-4208-851d-7815af46a62d/tmp1TMnNb/106e34c9-450d-4403-ae84-5cd4d4b94a00/tmp645n1s.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-b02a3811-2b63-4208-851d-7815af46a62d/tmp1TMnNb/106e34c9-450d-4403-ae84-5cd4d4b94a00/tmpqlMzFa.tmp'] failed with output:
  492. WARNING:toil.leader:9/9/job2BqGvp ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn39.grid.pub.ro
  493. WARNING:toil.leader:9/9/job2BqGvp WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' 9/9/job2BqGvp with ID 9/9/job2BqGvp to 4
  494. INFO:toil.leader:Issued job 'CactusSetupPhase' 9/9/job2BqGvp with job batch system ID: 147 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  495. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' 9/9/job2BqGvp
  496. INFO:toil.leader:Issued job 'CactusCafPhase' A/d/jobSxWsF9 with job batch system ID: 148 and cores: 1, disk: 2.0 G, and memory: 600.0 M
  497. WARNING:toil.leader:Job failed with exit value 137: 'CactusCafPhase' A/d/jobSxWsF9
  498. WARNING:toil.leader:No log file is present, despite job failing: 'CactusCafPhase' A/d/jobSxWsF9
  499. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusCafPhase' A/d/jobSxWsF9 with ID A/d/jobSxWsF9 to 5
  500. WARNING:toil.jobGraph:We have increased the default memory of the failed job 'CactusCafPhase' A/d/jobSxWsF9 to 32212254720 bytes
  501. INFO:toil.leader:Issued job 'CactusCafPhase' A/d/jobSxWsF9 with job batch system ID: 149 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  502. INFO:toil.statsAndLogging:Got message from job at time 03-22-2019 20:15:57: Converted headers of cigar file e/H/tmpf9bBtA-mappingQualityRescoring-tmppQ4aAR.tmp to internal names, new file /export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpIaUJ_n/fa4ea6d9-d537-41d0-aed3-fc2ca56f8a45/tmpGbM36F.tmp
  503. INFO:toil.statsAndLogging:Got message from job at time 03-22-2019 20:15:57: Converted headers of secondary cigar file K/K/tmpLekoYw-mappingQualityRescoring-tmpdRtl8V.tmp to internal names, new file /export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpIaUJ_n/fa4ea6d9-d537-41d0-aed3-fc2ca56f8a45/tmpKBbpuD.tmp
  504. INFO:toil.leader:Job ended successfully: 'CactusCafPhase' A/d/jobSxWsF9
  505. INFO:toil.leader:Issued job 'CactusCafWrapper' 5/4/jobgM1Z3h with job batch system ID: 150 and cores: 1, disk: 2.0 G, and memory: 1.2 G
  506. INFO:toil.leader:Job ended successfully: 'CactusCafWrapper' 5/4/jobgM1Z3h
  507. INFO:toil.leader:Issued job 'SavePrimaryDB' B/1/jobR0HiTQ with job batch system ID: 151 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  508. INFO:toil.statsAndLogging:Got message from job at time 03-22-2019 20:21:05: At end of caf phase, got stats {
  509. "flowerName": 0,
  510. "totalBases": 63614672,
  511. "totalEnds": 68470,
  512. "totalCaps": 320266,
  513. "maxEndDegree": 46,
  514. "maxAdjacencyLength": 1594858,
  515. "totalBlocks": 33573,
  516. "totalGroups": 29151,
  517. "totalEdges": 79290,
  518. "totalFreeEnds": 1256,
  519. "totalAttachedEnds": 68,
  520. "totalChains": 2984,
  521. "totalLinkGroups": 29105
  522. }
  523. flower name: 0 total bases: 63614672 total-ends: 68470 total-caps: 320266 max-end-degree: 46 max-adjacency-length: 1594858 total-blocks: 33573 total-groups: 29151 total-edges: 39645 total-free-ends: 1256 total-attached-ends: 68 total-chains: 2984 total-link groups: 29105
  524.  
  525. INFO:toil.leader:Job ended successfully: 'SavePrimaryDB' B/1/jobR0HiTQ
  526. INFO:toil.leader:Issued job 'CactusCafPhase' A/d/jobSxWsF9 with job batch system ID: 152 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  527. INFO:toil.leader:Job ended successfully: 'CactusCafPhase' A/d/jobSxWsF9
  528. INFO:toil.leader:Issued job 'CactusSetupPhase' 9/9/job2BqGvp with job batch system ID: 153 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  529. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' 9/9/job2BqGvp
  530. INFO:toil.leader:Job ended successfully: 'KtServerService' w/W/joblafimE
  531. INFO:toil.leader:Issued job 'StartPrimaryDB' t/F/jobmGZ5E8 with job batch system ID: 154 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  532. INFO:toil.leader:Job ended successfully: 'StartPrimaryDB' t/F/jobmGZ5E8
  533. INFO:toil.leader:Issued job 'CactusBarCheckpoint' J/O/jobgSECJU with job batch system ID: 155 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  534. INFO:toil.leader:Job ended successfully: 'CactusBarCheckpoint' J/O/jobgSECJU
  535. INFO:toil.leader:Issued job 'StartPrimaryDB' o/H/jobK044oZ with job batch system ID: 156 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  536. INFO:toil.leader:Job ended successfully: 'StartPrimaryDB' o/H/jobK044oZ
  537. INFO:toil.leader:Issued job 'KtServerService' v/j/jobrpRlEi with job batch system ID: 157 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  538. INFO:toil.leader:Issued job 'CactusBarPhase' w/9/jobvXUh4l with job batch system ID: 158 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  539. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' v/j/jobrpRlEi
  540. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' v/j/jobrpRlEi
  541. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' v/j/jobrpRlEi with ID v/j/jobrpRlEi to 5
  542. WARNING:toil.jobGraph:We have increased the default memory of the failed job 'KtServerService' v/j/jobrpRlEi to 32212254720 bytes
  543. INFO:toil.leader:Issued job 'KtServerService' v/j/jobrpRlEi with job batch system ID: 159 and cores: 0, disk: 2.0 G, and memory: 30.0 G
  544. INFO:toil.leader:Job ended successfully: 'CactusBarPhase' w/9/jobvXUh4l
  545. INFO:toil.leader:Issued job 'CactusBarRecursion' t/p/joba_2BHK with job batch system ID: 160 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  546. INFO:toil.leader:Job ended successfully: 'CactusBarRecursion' t/p/joba_2BHK
  547. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusBarRecursion' t/p/joba_2BHK
  548. WARNING:toil.leader:t/p/joba_2BHK INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  549. WARNING:toil.leader:t/p/joba_2BHK INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  550. WARNING:toil.leader:t/p/joba_2BHK WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  551. WARNING:toil.leader:t/p/joba_2BHK INFO:cactus.shared.common:Running the command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="22034" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000']
  552. WARNING:toil.leader:t/p/joba_2BHK Set log level to INFO
  553. WARNING:toil.leader:t/p/joba_2BHK Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.41 with error: network error
  554. WARNING:toil.leader:t/p/joba_2BHK Uncaught exception
  555. WARNING:toil.leader:t/p/joba_2BHK Traceback (most recent call last):
  556. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  557. WARNING:toil.leader:t/p/joba_2BHK job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  558. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  559. WARNING:toil.leader:t/p/joba_2BHK super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  560. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  561. WARNING:toil.leader:t/p/joba_2BHK returnValues = self._run(jobGraph, fileStore)
  562. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  563. WARNING:toil.leader:t/p/joba_2BHK return self.run(fileStore)
  564. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 831, in run
  565. WARNING:toil.leader:t/p/joba_2BHK self.makeRecursiveJobs(fileStore=fileStore)
  566. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 417, in makeRecursiveJobs
  567. WARNING:toil.leader:t/p/joba_2BHK default=CactusRecursionJob.maxSequenceSizeOfFlowerGroupingDefault))
  568. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 138, in runCactusGetFlowers
  569. WARNING:toil.leader:t/p/joba_2BHK fileStore=fileStore)
  570. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  571. WARNING:toil.leader:t/p/joba_2BHK raise RuntimeError("Command %s failed with output: %s" % (call, output))
  572. WARNING:toil.leader:t/p/joba_2BHK RuntimeError: Command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="22034" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000'] failed with output:
  573. WARNING:toil.leader:t/p/joba_2BHK ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn39.grid.pub.ro
  574. WARNING:toil.leader:t/p/joba_2BHK WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusBarRecursion' t/p/joba_2BHK with ID t/p/joba_2BHK to 5
  575. WARNING:toil.leader:t/p/joba_2BHK WARNING:toil.jobGraph:We have increased the default memory of the failed job 'CactusBarRecursion' t/p/joba_2BHK to 32212254720 bytes
  576. INFO:toil.leader:Issued job 'CactusBarRecursion' t/p/joba_2BHK with job batch system ID: 161 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  577. INFO:toil.leader:Job ended successfully: 'CactusBarRecursion' t/p/joba_2BHK
  578. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusBarRecursion' t/p/joba_2BHK
  579. WARNING:toil.leader:t/p/joba_2BHK INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  580. WARNING:toil.leader:t/p/joba_2BHK INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  581. WARNING:toil.leader:t/p/joba_2BHK WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  582. WARNING:toil.leader:t/p/joba_2BHK INFO:cactus.shared.common:Running the command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="22034" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000']
  583. WARNING:toil.leader:t/p/joba_2BHK Set log level to INFO
  584. WARNING:toil.leader:t/p/joba_2BHK Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.41 with error: network error
  585. WARNING:toil.leader:t/p/joba_2BHK Uncaught exception
  586. WARNING:toil.leader:t/p/joba_2BHK Traceback (most recent call last):
  587. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  588. WARNING:toil.leader:t/p/joba_2BHK job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  589. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  590. WARNING:toil.leader:t/p/joba_2BHK super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  591. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  592. WARNING:toil.leader:t/p/joba_2BHK returnValues = self._run(jobGraph, fileStore)
  593. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  594. WARNING:toil.leader:t/p/joba_2BHK return self.run(fileStore)
  595. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 831, in run
  596. WARNING:toil.leader:t/p/joba_2BHK self.makeRecursiveJobs(fileStore=fileStore)
  597. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 417, in makeRecursiveJobs
  598. WARNING:toil.leader:t/p/joba_2BHK default=CactusRecursionJob.maxSequenceSizeOfFlowerGroupingDefault))
  599. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 138, in runCactusGetFlowers
  600. WARNING:toil.leader:t/p/joba_2BHK fileStore=fileStore)
  601. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  602. WARNING:toil.leader:t/p/joba_2BHK raise RuntimeError("Command %s failed with output: %s" % (call, output))
  603. WARNING:toil.leader:t/p/joba_2BHK RuntimeError: Command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="22034" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000'] failed with output:
  604. WARNING:toil.leader:t/p/joba_2BHK ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn35.grid.pub.ro
  605. WARNING:toil.leader:t/p/joba_2BHK WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusBarRecursion' t/p/joba_2BHK with ID t/p/joba_2BHK to 4
  606. INFO:toil.leader:Issued job 'CactusBarRecursion' t/p/joba_2BHK with job batch system ID: 162 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  607. INFO:toil.leader:Job ended successfully: 'CactusBarRecursion' t/p/joba_2BHK
  608. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusBarRecursion' t/p/joba_2BHK
  609. WARNING:toil.leader:t/p/joba_2BHK INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  610. WARNING:toil.leader:t/p/joba_2BHK INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  611. WARNING:toil.leader:t/p/joba_2BHK WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  612. WARNING:toil.leader:t/p/joba_2BHK WARNING:toil.fileStore:Detected that job (9/9/job2BqGvp/g/tmpMQqMti-_serialiseJob-stream) prematurely terminated. Fixing the state of the job on disk.
  613. WARNING:toil.leader:t/p/joba_2BHK INFO:cactus.shared.common:Running the command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="22034" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000']
  614. WARNING:toil.leader:t/p/joba_2BHK Set log level to INFO
  615. WARNING:toil.leader:t/p/joba_2BHK Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.41 with error: network error
  616. WARNING:toil.leader:t/p/joba_2BHK Uncaught exception
  617. WARNING:toil.leader:t/p/joba_2BHK Traceback (most recent call last):
  618. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  619. WARNING:toil.leader:t/p/joba_2BHK job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  620. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  621. WARNING:toil.leader:t/p/joba_2BHK super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  622. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  623. WARNING:toil.leader:t/p/joba_2BHK returnValues = self._run(jobGraph, fileStore)
  624. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  625. WARNING:toil.leader:t/p/joba_2BHK return self.run(fileStore)
  626. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 831, in run
  627. WARNING:toil.leader:t/p/joba_2BHK self.makeRecursiveJobs(fileStore=fileStore)
  628. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 417, in makeRecursiveJobs
  629. WARNING:toil.leader:t/p/joba_2BHK default=CactusRecursionJob.maxSequenceSizeOfFlowerGroupingDefault))
  630. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 138, in runCactusGetFlowers
  631. WARNING:toil.leader:t/p/joba_2BHK fileStore=fileStore)
  632. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  633. WARNING:toil.leader:t/p/joba_2BHK raise RuntimeError("Command %s failed with output: %s" % (call, output))
  634. WARNING:toil.leader:t/p/joba_2BHK RuntimeError: Command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="22034" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000'] failed with output:
  635. WARNING:toil.leader:t/p/joba_2BHK ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn29.grid.pub.ro
  636. WARNING:toil.leader:t/p/joba_2BHK WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusBarRecursion' t/p/joba_2BHK with ID t/p/joba_2BHK to 3
  637. INFO:toil.leader:Issued job 'CactusBarRecursion' t/p/joba_2BHK with job batch system ID: 163 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  638. INFO:toil.leader:Job ended successfully: 'CactusBarRecursion' t/p/joba_2BHK
  639. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusBarRecursion' t/p/joba_2BHK
  640. WARNING:toil.leader:t/p/joba_2BHK INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  641. WARNING:toil.leader:t/p/joba_2BHK INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  642. WARNING:toil.leader:t/p/joba_2BHK WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  643. WARNING:toil.leader:t/p/joba_2BHK INFO:cactus.shared.common:Running the command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="22034" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000']
  644. WARNING:toil.leader:t/p/joba_2BHK Set log level to INFO
  645. WARNING:toil.leader:t/p/joba_2BHK Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.41 with error: network error
  646. WARNING:toil.leader:t/p/joba_2BHK Uncaught exception
  647. WARNING:toil.leader:t/p/joba_2BHK Traceback (most recent call last):
  648. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  649. WARNING:toil.leader:t/p/joba_2BHK job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  650. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  651. WARNING:toil.leader:t/p/joba_2BHK super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  652. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  653. WARNING:toil.leader:t/p/joba_2BHK returnValues = self._run(jobGraph, fileStore)
  654. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  655. WARNING:toil.leader:t/p/joba_2BHK return self.run(fileStore)
  656. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 831, in run
  657. WARNING:toil.leader:t/p/joba_2BHK self.makeRecursiveJobs(fileStore=fileStore)
  658. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 417, in makeRecursiveJobs
  659. WARNING:toil.leader:t/p/joba_2BHK default=CactusRecursionJob.maxSequenceSizeOfFlowerGroupingDefault))
  660. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 138, in runCactusGetFlowers
  661. WARNING:toil.leader:t/p/joba_2BHK fileStore=fileStore)
  662. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  663. WARNING:toil.leader:t/p/joba_2BHK raise RuntimeError("Command %s failed with output: %s" % (call, output))
  664. WARNING:toil.leader:t/p/joba_2BHK RuntimeError: Command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="22034" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000'] failed with output:
  665. WARNING:toil.leader:t/p/joba_2BHK ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn35.grid.pub.ro
  666. WARNING:toil.leader:t/p/joba_2BHK WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusBarRecursion' t/p/joba_2BHK with ID t/p/joba_2BHK to 2
  667. INFO:toil.leader:Issued job 'CactusBarRecursion' t/p/joba_2BHK with job batch system ID: 164 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  668. INFO:toil.leader:Job ended successfully: 'CactusBarRecursion' t/p/joba_2BHK
  669. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusBarRecursion' t/p/joba_2BHK
  670. WARNING:toil.leader:t/p/joba_2BHK INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  671. WARNING:toil.leader:t/p/joba_2BHK INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  672. WARNING:toil.leader:t/p/joba_2BHK WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  673. WARNING:toil.leader:t/p/joba_2BHK WARNING:toil.fileStore:Detected that job (o/H/jobK044oZ/g/tmpY6LrnG-_serialiseJob-stream) prematurely terminated. Fixing the state of the job on disk.
  674. WARNING:toil.leader:t/p/joba_2BHK INFO:cactus.shared.common:Running the command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="22034" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000']
  675. WARNING:toil.leader:t/p/joba_2BHK Set log level to INFO
  676. WARNING:toil.leader:t/p/joba_2BHK Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.41 with error: network error
  677. WARNING:toil.leader:t/p/joba_2BHK Uncaught exception
  678. WARNING:toil.leader:t/p/joba_2BHK Traceback (most recent call last):
  679. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  680. WARNING:toil.leader:t/p/joba_2BHK job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  681. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  682. WARNING:toil.leader:t/p/joba_2BHK super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  683. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  684. WARNING:toil.leader:t/p/joba_2BHK returnValues = self._run(jobGraph, fileStore)
  685. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  686. WARNING:toil.leader:t/p/joba_2BHK return self.run(fileStore)
  687. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 831, in run
  688. WARNING:toil.leader:t/p/joba_2BHK self.makeRecursiveJobs(fileStore=fileStore)
  689. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 417, in makeRecursiveJobs
  690. WARNING:toil.leader:t/p/joba_2BHK default=CactusRecursionJob.maxSequenceSizeOfFlowerGroupingDefault))
  691. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 138, in runCactusGetFlowers
  692. WARNING:toil.leader:t/p/joba_2BHK fileStore=fileStore)
  693. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  694. WARNING:toil.leader:t/p/joba_2BHK raise RuntimeError("Command %s failed with output: %s" % (call, output))
  695. WARNING:toil.leader:t/p/joba_2BHK RuntimeError: Command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="22034" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000'] failed with output:
  696. WARNING:toil.leader:t/p/joba_2BHK ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn41.grid.pub.ro
  697. WARNING:toil.leader:t/p/joba_2BHK WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusBarRecursion' t/p/joba_2BHK with ID t/p/joba_2BHK to 1
  698. INFO:toil.leader:Issued job 'CactusBarRecursion' t/p/joba_2BHK with job batch system ID: 165 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  699. INFO:toil.leader:Job ended successfully: 'CactusBarRecursion' t/p/joba_2BHK
  700. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusBarRecursion' t/p/joba_2BHK
  701. WARNING:toil.leader:t/p/joba_2BHK INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  702. WARNING:toil.leader:t/p/joba_2BHK INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  703. WARNING:toil.leader:t/p/joba_2BHK WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  704. WARNING:toil.leader:t/p/joba_2BHK INFO:cactus.shared.common:Running the command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="22034" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000']
  705. WARNING:toil.leader:t/p/joba_2BHK Set log level to INFO
  706. WARNING:toil.leader:t/p/joba_2BHK Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.41 with error: network error
  707. WARNING:toil.leader:t/p/joba_2BHK Uncaught exception
  708. WARNING:toil.leader:t/p/joba_2BHK Traceback (most recent call last):
  709. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  710. WARNING:toil.leader:t/p/joba_2BHK job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  711. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  712. WARNING:toil.leader:t/p/joba_2BHK super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  713. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  714. WARNING:toil.leader:t/p/joba_2BHK returnValues = self._run(jobGraph, fileStore)
  715. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  716. WARNING:toil.leader:t/p/joba_2BHK return self.run(fileStore)
  717. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 831, in run
  718. WARNING:toil.leader:t/p/joba_2BHK self.makeRecursiveJobs(fileStore=fileStore)
  719. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 417, in makeRecursiveJobs
  720. WARNING:toil.leader:t/p/joba_2BHK default=CactusRecursionJob.maxSequenceSizeOfFlowerGroupingDefault))
  721. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 138, in runCactusGetFlowers
  722. WARNING:toil.leader:t/p/joba_2BHK fileStore=fileStore)
  723. WARNING:toil.leader:t/p/joba_2BHK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  724. WARNING:toil.leader:t/p/joba_2BHK raise RuntimeError("Command %s failed with output: %s" % (call, output))
  725. WARNING:toil.leader:t/p/joba_2BHK RuntimeError: Command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="22034" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000'] failed with output:
  726. WARNING:toil.leader:t/p/joba_2BHK ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn39.grid.pub.ro
  727. WARNING:toil.leader:t/p/joba_2BHK WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusBarRecursion' t/p/joba_2BHK with ID t/p/joba_2BHK to 0
  728. WARNING:toil.leader:Job 'CactusBarRecursion' t/p/joba_2BHK with ID t/p/joba_2BHK is completely failed
  729. INFO:toil.leader:Job ended successfully: 'KtServerService' v/j/jobrpRlEi
  730. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'KtServerService' v/j/jobrpRlEi
  731. WARNING:toil.leader:v/j/jobrpRlEi INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  732. WARNING:toil.leader:v/j/jobrpRlEi INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  733. WARNING:toil.leader:v/j/jobrpRlEi WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  734. WARNING:toil.leader:v/j/jobrpRlEi WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  735. WARNING:toil.leader:v/j/jobrpRlEi INFO:cactus.shared.common:Running the command ['netstat', '-tuplen']
  736. WARNING:toil.leader:v/j/jobrpRlEi (No info could be read for "-p": geteuid()=98354 but you should be root.)
  737. WARNING:toil.leader:v/j/jobrpRlEi INFO:cactus.shared.common:Running the command ['ktserver', '-port', '8664', '-ls', '-tout', '200000', '-th', '64', '-bgs', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-2bc2432e-0437-4e28-8030-69bdeb17d728/tmp_FMkFV/2d6fb3ff-190a-4ced-9d8e-20e323da01cc/teLImyP/snapshot', '-bgsc', 'lzo', '-bgsi', '1000000', '-log', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-2bc2432e-0437-4e28-8030-69bdeb17d728/tmp_FMkFV/2d6fb3ff-190a-4ced-9d8e-20e323da01cc/tmpNepz2s.tmp', ':#opts=ls#bnum=30m#msiz=50g#ktopts=p']
  738. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  739. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  740. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  741. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 513
  742. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  743. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  744. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  745. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 513
  746. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  747. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  748. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  749. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 514
  750. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  751. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  752. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  753. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 515
  754. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  755. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  756. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  757. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 516
  758. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  759. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  760. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  761. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 517
  762. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  763. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  764. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  765. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 518
  766. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  767. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  768. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  769. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 519
  770. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  771. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  772. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  773. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 520
  774. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  775. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  776. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  777. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 521
  778. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  779. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  780. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  781. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 522
  782. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  783. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  784. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  785. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 523
  786. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  787. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  788. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  789. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 524
  790. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  791. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  792. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  793. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 525
  794. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  795. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  796. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  797. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 526
  798. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  799. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  800. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  801. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 527
  802. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  803. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  804. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  805. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 528
  806. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  807. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  808. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  809. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 529
  810. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  811. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  812. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  813. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 530
  814. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  815. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  816. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  817. WARNING:toil.leader:v/j/jobrpRlEi INFO:toil.lib.bioio:Ktserver running.
  818. WARNING:toil.leader:v/j/jobrpRlEi INFO:toil.lib.bioio:Ktserver running.
  819. WARNING:toil.leader:v/j/jobrpRlEi INFO:toil.lib.bioio:Ktserver running.
  820. WARNING:toil.leader:v/j/jobrpRlEi INFO:cactus.shared.common:Running the command ['ktremotemgr', 'remove', '-port', '8664', '-host', '172.16.13.38', 'TERMINATE']
  821. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 531
  822. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  823. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  824. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  825. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 533
  826. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  827. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  828. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  829. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 534
  830. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  831. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  832. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  833. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 535
  834. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  835. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  836. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  837. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 536
  838. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  839. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  840. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  841. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 537
  842. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  843. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  844. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  845. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 538
  846. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  847. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  848. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  849. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 539
  850. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  851. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  852. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  853. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 540
  854. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  855. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  856. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  857. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 541
  858. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  859. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  860. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  861. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 542
  862. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  863. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  864. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  865. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 543
  866. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  867. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  868. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  869. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 544
  870. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  871. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  872. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  873. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 545
  874. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  875. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  876. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  877. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 546
  878. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  879. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  880. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  881. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 547
  882. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  883. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  884. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  885. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 548
  886. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  887. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  888. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  889. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 549
  890. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  891. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  892. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  893. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 550
  894. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  895. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  896. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  897. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 551
  898. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  899. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  900. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  901. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 552
  902. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  903. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  904. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  905. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 553
  906. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  907. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  908. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  909. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 554
  910. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  911. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  912. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  913. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 555
  914. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  915. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  916. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  917. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 556
  918. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  919. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  920. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  921. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 557
  922. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  923. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  924. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  925. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 558
  926. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  927. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  928. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  929. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 559
  930. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  931. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  932. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  933. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 560
  934. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  935. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  936. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  937. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 561
  938. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  939. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  940. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  941. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 562
  942. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  943. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  944. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  945. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 563
  946. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  947. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  948. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  949. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 564
  950. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  951. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  952. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  953. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 565
  954. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  955. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  956. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  957. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 566
  958. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  959. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  960. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  961. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 567
  962. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  963. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  964. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  965. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 568
  966. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  967. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  968. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  969. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 569
  970. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  971. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  972. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  973. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 570
  974. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  975. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  976. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  977. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 571
  978. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  979. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  980. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  981. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 572
  982. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  983. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  984. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  985. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 573
  986. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  987. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  988. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  989. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 574
  990. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  991. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  992. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  993. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 575
  994. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  995. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  996. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  997. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 576
  998. WARNING:toil.leader:v/j/jobrpRlEi Stack limits: Infinity Infinity
  999. WARNING:toil.leader:v/j/jobrpRlEi Process limits: 510853 510853
  1000. WARNING:toil.leader:v/j/jobrpRlEi Kernel thread cap: 1021706
  1001. WARNING:toil.leader:v/j/jobrpRlEi Current total threads running: 577
  1002. WARNING:toil.leader:v/j/jobrpRlEi INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '8664', '-host', '172.16.13.38', 'TERMINATE']
  1003. WARNING:toil.leader:v/j/jobrpRlEi INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '8664', '-host', '172.16.13.38', 'TERMINATE']
  1004. WARNING:toil.leader:v/j/jobrpRlEi INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '8664', '-host', '172.16.13.38', 'TERMINATE']
  1005. WARNING:toil.leader:v/j/jobrpRlEi INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '8664', '-host', '172.16.13.38', 'TERMINATE']
  1006. WARNING:toil.leader:v/j/jobrpRlEi INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '8664', '-host', '172.16.13.38', 'TERMINATE']
  1007. WARNING:toil.leader:v/j/jobrpRlEi INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '8664', '-host', '172.16.13.38', 'TERMINATE']
  1008. WARNING:toil.leader:v/j/jobrpRlEi INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '8664', '-host', '172.16.13.38', 'TERMINATE']
  1009. WARNING:toil.leader:v/j/jobrpRlEi INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '8664', '-host', '172.16.13.38', 'TERMINATE']
  1010. WARNING:toil.leader:v/j/jobrpRlEi INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '8664', '-host', '172.16.13.38', 'TERMINATE']
  1011. WARNING:toil.leader:v/j/jobrpRlEi INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '8664', '-host', '172.16.13.38', 'TERMINATE']
  1012. WARNING:toil.leader:v/j/jobrpRlEi INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '8664', '-host', '172.16.13.38', 'TERMINATE']
  1013. WARNING:toil.leader:v/j/jobrpRlEi INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '8664', '-host', '172.16.13.38', 'TERMINATE']
  1014. WARNING:toil.leader:v/j/jobrpRlEi INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '8664', '-host', '172.16.13.38', 'TERMINATE']
  1015. WARNING:toil.leader:v/j/jobrpRlEi INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '8664', '-host', '172.16.13.38', 'TERMINATE']
  1016. WARNING:toil.leader:v/j/jobrpRlEi INFO:cactus.shared.common:Running the command ['ktremotemgr', 'set', '-port', '8664', '-host', '172.16.13.38', 'TERMINATE', '1']
  1017. WARNING:toil.leader:v/j/jobrpRlEi INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '8664', '-host', '172.16.13.38', 'TERMINATE']
  1018. WARNING:toil.leader:v/j/jobrpRlEi 1
  1019. WARNING:toil.leader:v/j/jobrpRlEi Traceback (most recent call last):
  1020. WARNING:toil.leader:v/j/jobrpRlEi File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1021. WARNING:toil.leader:v/j/jobrpRlEi job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1022. WARNING:toil.leader:v/j/jobrpRlEi File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1023. WARNING:toil.leader:v/j/jobrpRlEi returnValues = self._run(jobGraph, fileStore)
  1024. WARNING:toil.leader:v/j/jobrpRlEi File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1694, in _run
  1025. WARNING:toil.leader:v/j/jobrpRlEi returnValues = self.run(fileStore)
  1026. WARNING:toil.leader:v/j/jobrpRlEi File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1668, in run
  1027. WARNING:toil.leader:v/j/jobrpRlEi raise RuntimeError("Detected the error jobStoreID has been removed so exiting with an error")
  1028. WARNING:toil.leader:v/j/jobrpRlEi RuntimeError: Detected the error jobStoreID has been removed so exiting with an error
  1029. WARNING:toil.leader:v/j/jobrpRlEi ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn38.grid.pub.ro
  1030. WARNING:toil.leader:v/j/jobrpRlEi WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' v/j/jobrpRlEi with ID v/j/jobrpRlEi to 4
  1031. WARNING:toil.leader:Job 'KtServerService' v/j/jobrpRlEi with ID v/j/jobrpRlEi is completely failed
  1032. WARNING:toil.leader:Job: o/H/jobK044oZ is being restarted as a checkpoint after the total failure of jobs in its subtree.
  1033. INFO:toil.leader:Issued job 'StartPrimaryDB' o/H/jobK044oZ with job batch system ID: 166 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1034. INFO:toil.leader:Job ended successfully: 'StartPrimaryDB' o/H/jobK044oZ
  1035. INFO:toil.leader:Issued job 'KtServerService' k/O/jobbZcLhL with job batch system ID: 167 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1036. INFO:toil.leader:Issued job 'CactusBarPhase' i/c/jobII471V with job batch system ID: 168 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1037. INFO:toil.leader:Job ended successfully: 'CactusBarPhase' i/c/jobII471V
  1038. INFO:toil.leader:Issued job 'CactusBarRecursion' A/4/jobBus8Bf with job batch system ID: 169 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  1039. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' k/O/jobbZcLhL
  1040. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' k/O/jobbZcLhL
  1041. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' k/O/jobbZcLhL with ID k/O/jobbZcLhL to 5
  1042. WARNING:toil.jobGraph:We have increased the default memory of the failed job 'KtServerService' k/O/jobbZcLhL to 32212254720 bytes
  1043. INFO:toil.leader:Issued job 'KtServerService' k/O/jobbZcLhL with job batch system ID: 170 and cores: 0, disk: 2.0 G, and memory: 30.0 G
  1044. INFO:toil.leader:Reissued any over long jobs
  1045. INFO:toil.leader:Job ended successfully: 'CactusBarRecursion' A/4/jobBus8Bf
  1046. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusBarRecursion' A/4/jobBus8Bf
  1047. WARNING:toil.leader:A/4/jobBus8Bf INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1048. WARNING:toil.leader:A/4/jobBus8Bf INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1049. WARNING:toil.leader:A/4/jobBus8Bf WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1050. WARNING:toil.leader:A/4/jobBus8Bf INFO:cactus.shared.common:Running the command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="4707" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000']
  1051. WARNING:toil.leader:A/4/jobBus8Bf Set log level to INFO
  1052. WARNING:toil.leader:A/4/jobBus8Bf Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.35 with error: network error
  1053. WARNING:toil.leader:A/4/jobBus8Bf Uncaught exception
  1054. WARNING:toil.leader:A/4/jobBus8Bf Traceback (most recent call last):
  1055. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1056. WARNING:toil.leader:A/4/jobBus8Bf job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1057. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1058. WARNING:toil.leader:A/4/jobBus8Bf super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1059. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1060. WARNING:toil.leader:A/4/jobBus8Bf returnValues = self._run(jobGraph, fileStore)
  1061. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1062. WARNING:toil.leader:A/4/jobBus8Bf return self.run(fileStore)
  1063. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 831, in run
  1064. WARNING:toil.leader:A/4/jobBus8Bf self.makeRecursiveJobs(fileStore=fileStore)
  1065. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 417, in makeRecursiveJobs
  1066. WARNING:toil.leader:A/4/jobBus8Bf default=CactusRecursionJob.maxSequenceSizeOfFlowerGroupingDefault))
  1067. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 138, in runCactusGetFlowers
  1068. WARNING:toil.leader:A/4/jobBus8Bf fileStore=fileStore)
  1069. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1070. WARNING:toil.leader:A/4/jobBus8Bf raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1071. WARNING:toil.leader:A/4/jobBus8Bf RuntimeError: Command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="4707" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000'] failed with output:
  1072. WARNING:toil.leader:A/4/jobBus8Bf ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn34.grid.pub.ro
  1073. WARNING:toil.leader:A/4/jobBus8Bf WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusBarRecursion' A/4/jobBus8Bf with ID A/4/jobBus8Bf to 5
  1074. WARNING:toil.leader:A/4/jobBus8Bf WARNING:toil.jobGraph:We have increased the default memory of the failed job 'CactusBarRecursion' A/4/jobBus8Bf to 32212254720 bytes
  1075. INFO:toil.leader:Issued job 'CactusBarRecursion' A/4/jobBus8Bf with job batch system ID: 171 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  1076. INFO:toil.leader:Job ended successfully: 'CactusBarRecursion' A/4/jobBus8Bf
  1077. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusBarRecursion' A/4/jobBus8Bf
  1078. WARNING:toil.leader:A/4/jobBus8Bf INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1079. WARNING:toil.leader:A/4/jobBus8Bf INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1080. WARNING:toil.leader:A/4/jobBus8Bf WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1081. WARNING:toil.leader:A/4/jobBus8Bf WARNING:toil.fileStore:Detected that job (o/H/jobK044oZ/g/tmpTxmonp-_serialiseJob-stream) prematurely terminated. Fixing the state of the job on disk.
  1082. WARNING:toil.leader:A/4/jobBus8Bf INFO:cactus.shared.common:Running the command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="4707" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000']
  1083. WARNING:toil.leader:A/4/jobBus8Bf Set log level to INFO
  1084. WARNING:toil.leader:A/4/jobBus8Bf Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.35 with error: network error
  1085. WARNING:toil.leader:A/4/jobBus8Bf Uncaught exception
  1086. WARNING:toil.leader:A/4/jobBus8Bf Traceback (most recent call last):
  1087. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1088. WARNING:toil.leader:A/4/jobBus8Bf job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1089. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1090. WARNING:toil.leader:A/4/jobBus8Bf super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1091. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1092. WARNING:toil.leader:A/4/jobBus8Bf returnValues = self._run(jobGraph, fileStore)
  1093. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1094. WARNING:toil.leader:A/4/jobBus8Bf return self.run(fileStore)
  1095. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 831, in run
  1096. WARNING:toil.leader:A/4/jobBus8Bf self.makeRecursiveJobs(fileStore=fileStore)
  1097. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 417, in makeRecursiveJobs
  1098. WARNING:toil.leader:A/4/jobBus8Bf default=CactusRecursionJob.maxSequenceSizeOfFlowerGroupingDefault))
  1099. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 138, in runCactusGetFlowers
  1100. WARNING:toil.leader:A/4/jobBus8Bf fileStore=fileStore)
  1101. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1102. WARNING:toil.leader:A/4/jobBus8Bf raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1103. WARNING:toil.leader:A/4/jobBus8Bf RuntimeError: Command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="4707" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000'] failed with output:
  1104. WARNING:toil.leader:A/4/jobBus8Bf ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn35.grid.pub.ro
  1105. WARNING:toil.leader:A/4/jobBus8Bf WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusBarRecursion' A/4/jobBus8Bf with ID A/4/jobBus8Bf to 4
  1106. INFO:toil.leader:Issued job 'CactusBarRecursion' A/4/jobBus8Bf with job batch system ID: 172 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  1107. INFO:toil.leader:Job ended successfully: 'CactusBarRecursion' A/4/jobBus8Bf
  1108. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusBarRecursion' A/4/jobBus8Bf
  1109. WARNING:toil.leader:A/4/jobBus8Bf INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1110. WARNING:toil.leader:A/4/jobBus8Bf INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1111. WARNING:toil.leader:A/4/jobBus8Bf WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1112. WARNING:toil.leader:A/4/jobBus8Bf INFO:cactus.shared.common:Running the command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="4707" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000']
  1113. WARNING:toil.leader:A/4/jobBus8Bf Set log level to INFO
  1114. WARNING:toil.leader:A/4/jobBus8Bf Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.35 with error: network error
  1115. WARNING:toil.leader:A/4/jobBus8Bf Uncaught exception
  1116. WARNING:toil.leader:A/4/jobBus8Bf Traceback (most recent call last):
  1117. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1118. WARNING:toil.leader:A/4/jobBus8Bf job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1119. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1120. WARNING:toil.leader:A/4/jobBus8Bf super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1121. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1122. WARNING:toil.leader:A/4/jobBus8Bf returnValues = self._run(jobGraph, fileStore)
  1123. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1124. WARNING:toil.leader:A/4/jobBus8Bf return self.run(fileStore)
  1125. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 831, in run
  1126. WARNING:toil.leader:A/4/jobBus8Bf self.makeRecursiveJobs(fileStore=fileStore)
  1127. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 417, in makeRecursiveJobs
  1128. WARNING:toil.leader:A/4/jobBus8Bf default=CactusRecursionJob.maxSequenceSizeOfFlowerGroupingDefault))
  1129. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 138, in runCactusGetFlowers
  1130. WARNING:toil.leader:A/4/jobBus8Bf fileStore=fileStore)
  1131. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1132. WARNING:toil.leader:A/4/jobBus8Bf raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1133. WARNING:toil.leader:A/4/jobBus8Bf RuntimeError: Command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="4707" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000'] failed with output:
  1134. WARNING:toil.leader:A/4/jobBus8Bf ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn38.grid.pub.ro
  1135. WARNING:toil.leader:A/4/jobBus8Bf WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusBarRecursion' A/4/jobBus8Bf with ID A/4/jobBus8Bf to 3
  1136. INFO:toil.leader:Issued job 'CactusBarRecursion' A/4/jobBus8Bf with job batch system ID: 173 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  1137. INFO:toil.leader:Job ended successfully: 'CactusBarRecursion' A/4/jobBus8Bf
  1138. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusBarRecursion' A/4/jobBus8Bf
  1139. WARNING:toil.leader:A/4/jobBus8Bf INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1140. WARNING:toil.leader:A/4/jobBus8Bf INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1141. WARNING:toil.leader:A/4/jobBus8Bf WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1142. WARNING:toil.leader:A/4/jobBus8Bf INFO:cactus.shared.common:Running the command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="4707" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000']
  1143. WARNING:toil.leader:A/4/jobBus8Bf Set log level to INFO
  1144. WARNING:toil.leader:A/4/jobBus8Bf Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.35 with error: network error
  1145. WARNING:toil.leader:A/4/jobBus8Bf Uncaught exception
  1146. WARNING:toil.leader:A/4/jobBus8Bf Traceback (most recent call last):
  1147. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1148. WARNING:toil.leader:A/4/jobBus8Bf job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1149. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1150. WARNING:toil.leader:A/4/jobBus8Bf super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1151. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1152. WARNING:toil.leader:A/4/jobBus8Bf returnValues = self._run(jobGraph, fileStore)
  1153. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1154. WARNING:toil.leader:A/4/jobBus8Bf return self.run(fileStore)
  1155. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 831, in run
  1156. WARNING:toil.leader:A/4/jobBus8Bf self.makeRecursiveJobs(fileStore=fileStore)
  1157. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 417, in makeRecursiveJobs
  1158. WARNING:toil.leader:A/4/jobBus8Bf default=CactusRecursionJob.maxSequenceSizeOfFlowerGroupingDefault))
  1159. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 138, in runCactusGetFlowers
  1160. WARNING:toil.leader:A/4/jobBus8Bf fileStore=fileStore)
  1161. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1162. WARNING:toil.leader:A/4/jobBus8Bf raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1163. WARNING:toil.leader:A/4/jobBus8Bf RuntimeError: Command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="4707" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000'] failed with output:
  1164. WARNING:toil.leader:A/4/jobBus8Bf ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn39.grid.pub.ro
  1165. WARNING:toil.leader:A/4/jobBus8Bf WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusBarRecursion' A/4/jobBus8Bf with ID A/4/jobBus8Bf to 2
  1166. INFO:toil.leader:Issued job 'CactusBarRecursion' A/4/jobBus8Bf with job batch system ID: 174 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  1167. INFO:toil.leader:Job ended successfully: 'CactusBarRecursion' A/4/jobBus8Bf
  1168. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusBarRecursion' A/4/jobBus8Bf
  1169. WARNING:toil.leader:A/4/jobBus8Bf INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1170. WARNING:toil.leader:A/4/jobBus8Bf INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1171. WARNING:toil.leader:A/4/jobBus8Bf WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1172. WARNING:toil.leader:A/4/jobBus8Bf INFO:cactus.shared.common:Running the command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="4707" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000']
  1173. WARNING:toil.leader:A/4/jobBus8Bf Set log level to INFO
  1174. WARNING:toil.leader:A/4/jobBus8Bf Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.35 with error: network error
  1175. WARNING:toil.leader:A/4/jobBus8Bf Uncaught exception
  1176. WARNING:toil.leader:A/4/jobBus8Bf Traceback (most recent call last):
  1177. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1178. WARNING:toil.leader:A/4/jobBus8Bf job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1179. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1180. WARNING:toil.leader:A/4/jobBus8Bf super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1181. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1182. WARNING:toil.leader:A/4/jobBus8Bf returnValues = self._run(jobGraph, fileStore)
  1183. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1184. WARNING:toil.leader:A/4/jobBus8Bf return self.run(fileStore)
  1185. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 831, in run
  1186. WARNING:toil.leader:A/4/jobBus8Bf self.makeRecursiveJobs(fileStore=fileStore)
  1187. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 417, in makeRecursiveJobs
  1188. WARNING:toil.leader:A/4/jobBus8Bf default=CactusRecursionJob.maxSequenceSizeOfFlowerGroupingDefault))
  1189. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 138, in runCactusGetFlowers
  1190. WARNING:toil.leader:A/4/jobBus8Bf fileStore=fileStore)
  1191. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1192. WARNING:toil.leader:A/4/jobBus8Bf raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1193. WARNING:toil.leader:A/4/jobBus8Bf RuntimeError: Command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="4707" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000'] failed with output:
  1194. WARNING:toil.leader:A/4/jobBus8Bf ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn29.grid.pub.ro
  1195. WARNING:toil.leader:A/4/jobBus8Bf WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusBarRecursion' A/4/jobBus8Bf with ID A/4/jobBus8Bf to 1
  1196. INFO:toil.leader:Issued job 'CactusBarRecursion' A/4/jobBus8Bf with job batch system ID: 175 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  1197. INFO:toil.leader:Job ended successfully: 'CactusBarRecursion' A/4/jobBus8Bf
  1198. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusBarRecursion' A/4/jobBus8Bf
  1199. WARNING:toil.leader:A/4/jobBus8Bf INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1200. WARNING:toil.leader:A/4/jobBus8Bf INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1201. WARNING:toil.leader:A/4/jobBus8Bf WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1202. WARNING:toil.leader:A/4/jobBus8Bf INFO:cactus.shared.common:Running the command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="4707" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000']
  1203. WARNING:toil.leader:A/4/jobBus8Bf Set log level to INFO
  1204. WARNING:toil.leader:A/4/jobBus8Bf Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.35 with error: network error
  1205. WARNING:toil.leader:A/4/jobBus8Bf Uncaught exception
  1206. WARNING:toil.leader:A/4/jobBus8Bf Traceback (most recent call last):
  1207. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1208. WARNING:toil.leader:A/4/jobBus8Bf job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1209. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1210. WARNING:toil.leader:A/4/jobBus8Bf super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1211. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1212. WARNING:toil.leader:A/4/jobBus8Bf returnValues = self._run(jobGraph, fileStore)
  1213. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1214. WARNING:toil.leader:A/4/jobBus8Bf return self.run(fileStore)
  1215. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 831, in run
  1216. WARNING:toil.leader:A/4/jobBus8Bf self.makeRecursiveJobs(fileStore=fileStore)
  1217. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 417, in makeRecursiveJobs
  1218. WARNING:toil.leader:A/4/jobBus8Bf default=CactusRecursionJob.maxSequenceSizeOfFlowerGroupingDefault))
  1219. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 138, in runCactusGetFlowers
  1220. WARNING:toil.leader:A/4/jobBus8Bf fileStore=fileStore)
  1221. WARNING:toil.leader:A/4/jobBus8Bf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1222. WARNING:toil.leader:A/4/jobBus8Bf raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1223. WARNING:toil.leader:A/4/jobBus8Bf RuntimeError: Command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="4707" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000'] failed with output:
  1224. WARNING:toil.leader:A/4/jobBus8Bf ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn35.grid.pub.ro
  1225. WARNING:toil.leader:A/4/jobBus8Bf WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusBarRecursion' A/4/jobBus8Bf with ID A/4/jobBus8Bf to 0
  1226. WARNING:toil.leader:Job 'CactusBarRecursion' A/4/jobBus8Bf with ID A/4/jobBus8Bf is completely failed
  1227. INFO:toil.leader:Job ended successfully: 'KtServerService' k/O/jobbZcLhL
  1228. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'KtServerService' k/O/jobbZcLhL
  1229. WARNING:toil.leader:k/O/jobbZcLhL INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1230. WARNING:toil.leader:k/O/jobbZcLhL INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1231. WARNING:toil.leader:k/O/jobbZcLhL WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1232. WARNING:toil.leader:k/O/jobbZcLhL WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1233. WARNING:toil.leader:k/O/jobbZcLhL INFO:cactus.shared.common:Running the command ['netstat', '-tuplen']
  1234. WARNING:toil.leader:k/O/jobbZcLhL (No info could be read for "-p": geteuid()=98354 but you should be root.)
  1235. WARNING:toil.leader:k/O/jobbZcLhL INFO:cactus.shared.common:Running the command ['ktserver', '-port', '28296', '-ls', '-tout', '200000', '-th', '64', '-bgs', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-6386a5c9-5d92-486a-9720-412b1ca610f6/tmprhEh3g/07bb9495-008c-4274-81f6-2a31b4e9f904/t2c06li/snapshot', '-bgsc', 'lzo', '-bgsi', '1000000', '-log', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-6386a5c9-5d92-486a-9720-412b1ca610f6/tmprhEh3g/07bb9495-008c-4274-81f6-2a31b4e9f904/tmpPhUJKf.tmp', ':#opts=ls#bnum=30m#msiz=50g#ktopts=p']
  1236. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1237. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1238. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1239. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 550
  1240. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1241. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1242. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1243. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 550
  1244. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1245. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1246. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1247. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 551
  1248. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1249. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1250. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1251. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 552
  1252. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1253. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1254. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1255. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 553
  1256. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1257. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1258. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1259. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 554
  1260. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1261. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1262. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1263. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 555
  1264. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1265. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1266. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1267. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 556
  1268. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1269. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1270. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1271. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 557
  1272. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1273. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1274. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1275. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 558
  1276. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1277. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1278. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1279. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 559
  1280. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1281. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1282. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1283. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 560
  1284. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1285. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1286. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1287. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 561
  1288. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1289. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1290. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1291. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 562
  1292. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1293. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1294. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1295. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 563
  1296. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1297. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1298. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1299. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 564
  1300. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1301. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1302. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1303. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 565
  1304. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1305. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1306. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1307. WARNING:toil.leader:k/O/jobbZcLhL INFO:toil.lib.bioio:Ktserver running.
  1308. WARNING:toil.leader:k/O/jobbZcLhL INFO:toil.lib.bioio:Ktserver running.
  1309. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 566
  1310. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1311. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1312. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1313. WARNING:toil.leader:k/O/jobbZcLhL INFO:toil.lib.bioio:Ktserver running.
  1314. WARNING:toil.leader:k/O/jobbZcLhL INFO:cactus.shared.common:Running the command ['ktremotemgr', 'remove', '-port', '28296', '-host', '172.16.13.37', 'TERMINATE']
  1315. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 568
  1316. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1317. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1318. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1319. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 569
  1320. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1321. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1322. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1323. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 570
  1324. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1325. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1326. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1327. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 571
  1328. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1329. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1330. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1331. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 572
  1332. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1333. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1334. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1335. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 573
  1336. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1337. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1338. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1339. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 574
  1340. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1341. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1342. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1343. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 575
  1344. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1345. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1346. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1347. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 576
  1348. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1349. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1350. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1351. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 577
  1352. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1353. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1354. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1355. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 578
  1356. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1357. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1358. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1359. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 579
  1360. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1361. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1362. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1363. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 580
  1364. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1365. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1366. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1367. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 581
  1368. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1369. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1370. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1371. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 582
  1372. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1373. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1374. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1375. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 583
  1376. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1377. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1378. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1379. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 584
  1380. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1381. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1382. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1383. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 585
  1384. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1385. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1386. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1387. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 586
  1388. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1389. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1390. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1391. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 587
  1392. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1393. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1394. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1395. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 588
  1396. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1397. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1398. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1399. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 589
  1400. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1401. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1402. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1403. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 590
  1404. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1405. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1406. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1407. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 591
  1408. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1409. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1410. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1411. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 592
  1412. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1413. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1414. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1415. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 593
  1416. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1417. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1418. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1419. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 594
  1420. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1421. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1422. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1423. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 595
  1424. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1425. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1426. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1427. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 596
  1428. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1429. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1430. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1431. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 597
  1432. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1433. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1434. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1435. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 598
  1436. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1437. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1438. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1439. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 599
  1440. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1441. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1442. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1443. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 600
  1444. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1445. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1446. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1447. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 601
  1448. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1449. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1450. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1451. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 602
  1452. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1453. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1454. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1455. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 603
  1456. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1457. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1458. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1459. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 604
  1460. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1461. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1462. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1463. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 605
  1464. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1465. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1466. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1467. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 606
  1468. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1469. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1470. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1471. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 607
  1472. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1473. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1474. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1475. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 608
  1476. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1477. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1478. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1479. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 609
  1480. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1481. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1482. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1483. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 610
  1484. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1485. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1486. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1487. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 611
  1488. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1489. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1490. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1491. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 612
  1492. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1493. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1494. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1495. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 613
  1496. WARNING:toil.leader:k/O/jobbZcLhL Stack limits: Infinity Infinity
  1497. WARNING:toil.leader:k/O/jobbZcLhL Process limits: 511096 511096
  1498. WARNING:toil.leader:k/O/jobbZcLhL Kernel thread cap: 1022192
  1499. WARNING:toil.leader:k/O/jobbZcLhL Current total threads running: 614
  1500. WARNING:toil.leader:k/O/jobbZcLhL INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '28296', '-host', '172.16.13.37', 'TERMINATE']
  1501. WARNING:toil.leader:k/O/jobbZcLhL INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '28296', '-host', '172.16.13.37', 'TERMINATE']
  1502. WARNING:toil.leader:k/O/jobbZcLhL INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '28296', '-host', '172.16.13.37', 'TERMINATE']
  1503. WARNING:toil.leader:k/O/jobbZcLhL INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '28296', '-host', '172.16.13.37', 'TERMINATE']
  1504. WARNING:toil.leader:k/O/jobbZcLhL INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '28296', '-host', '172.16.13.37', 'TERMINATE']
  1505. WARNING:toil.leader:k/O/jobbZcLhL INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '28296', '-host', '172.16.13.37', 'TERMINATE']
  1506. WARNING:toil.leader:k/O/jobbZcLhL INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '28296', '-host', '172.16.13.37', 'TERMINATE']
  1507. WARNING:toil.leader:k/O/jobbZcLhL INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '28296', '-host', '172.16.13.37', 'TERMINATE']
  1508. WARNING:toil.leader:k/O/jobbZcLhL INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '28296', '-host', '172.16.13.37', 'TERMINATE']
  1509. WARNING:toil.leader:k/O/jobbZcLhL INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '28296', '-host', '172.16.13.37', 'TERMINATE']
  1510. WARNING:toil.leader:k/O/jobbZcLhL INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '28296', '-host', '172.16.13.37', 'TERMINATE']
  1511. WARNING:toil.leader:k/O/jobbZcLhL INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '28296', '-host', '172.16.13.37', 'TERMINATE']
  1512. WARNING:toil.leader:k/O/jobbZcLhL INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '28296', '-host', '172.16.13.37', 'TERMINATE']
  1513. WARNING:toil.leader:k/O/jobbZcLhL INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '28296', '-host', '172.16.13.37', 'TERMINATE']
  1514. WARNING:toil.leader:k/O/jobbZcLhL INFO:cactus.shared.common:Running the command ['ktremotemgr', 'set', '-port', '28296', '-host', '172.16.13.37', 'TERMINATE', '1']
  1515. WARNING:toil.leader:k/O/jobbZcLhL INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '28296', '-host', '172.16.13.37', 'TERMINATE']
  1516. WARNING:toil.leader:k/O/jobbZcLhL 1
  1517. WARNING:toil.leader:k/O/jobbZcLhL Traceback (most recent call last):
  1518. WARNING:toil.leader:k/O/jobbZcLhL File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1519. WARNING:toil.leader:k/O/jobbZcLhL job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1520. WARNING:toil.leader:k/O/jobbZcLhL File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1521. WARNING:toil.leader:k/O/jobbZcLhL returnValues = self._run(jobGraph, fileStore)
  1522. WARNING:toil.leader:k/O/jobbZcLhL File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1694, in _run
  1523. WARNING:toil.leader:k/O/jobbZcLhL returnValues = self.run(fileStore)
  1524. WARNING:toil.leader:k/O/jobbZcLhL File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1668, in run
  1525. WARNING:toil.leader:k/O/jobbZcLhL raise RuntimeError("Detected the error jobStoreID has been removed so exiting with an error")
  1526. WARNING:toil.leader:k/O/jobbZcLhL RuntimeError: Detected the error jobStoreID has been removed so exiting with an error
  1527. WARNING:toil.leader:k/O/jobbZcLhL ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn37.grid.pub.ro
  1528. WARNING:toil.leader:k/O/jobbZcLhL WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' k/O/jobbZcLhL with ID k/O/jobbZcLhL to 4
  1529. WARNING:toil.leader:Job 'KtServerService' k/O/jobbZcLhL with ID k/O/jobbZcLhL is completely failed
  1530. WARNING:toil.leader:Job: o/H/jobK044oZ is being restarted as a checkpoint after the total failure of jobs in its subtree.
  1531. INFO:toil.leader:Issued job 'StartPrimaryDB' o/H/jobK044oZ with job batch system ID: 176 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1532. INFO:toil.leader:Job ended successfully: 'StartPrimaryDB' o/H/jobK044oZ
  1533. INFO:toil.leader:Issued job 'KtServerService' K/3/jobZoBnBp with job batch system ID: 177 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1534. INFO:toil.leader:Issued job 'CactusBarPhase' h/X/jobw7N0DT with job batch system ID: 178 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1535. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' K/3/jobZoBnBp
  1536. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' K/3/jobZoBnBp
  1537. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' K/3/jobZoBnBp with ID K/3/jobZoBnBp to 5
  1538. WARNING:toil.jobGraph:We have increased the default memory of the failed job 'KtServerService' K/3/jobZoBnBp to 32212254720 bytes
  1539. INFO:toil.leader:Issued job 'KtServerService' K/3/jobZoBnBp with job batch system ID: 179 and cores: 0, disk: 2.0 G, and memory: 30.0 G
  1540. INFO:toil.leader:Job ended successfully: 'CactusBarPhase' h/X/jobw7N0DT
  1541. INFO:toil.leader:Issued job 'CactusBarRecursion' 9/o/job5VmMi9 with job batch system ID: 180 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  1542. INFO:toil.leader:Job ended successfully: 'CactusBarRecursion' 9/o/job5VmMi9
  1543. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusBarRecursion' 9/o/job5VmMi9
  1544. WARNING:toil.leader:9/o/job5VmMi9 INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1545. WARNING:toil.leader:9/o/job5VmMi9 INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1546. WARNING:toil.leader:9/o/job5VmMi9 WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1547. WARNING:toil.leader:9/o/job5VmMi9 INFO:cactus.shared.common:Running the command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="17032" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000']
  1548. WARNING:toil.leader:9/o/job5VmMi9 Set log level to INFO
  1549. WARNING:toil.leader:9/o/job5VmMi9 Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.29 with error: network error
  1550. WARNING:toil.leader:9/o/job5VmMi9 Uncaught exception
  1551. WARNING:toil.leader:9/o/job5VmMi9 Traceback (most recent call last):
  1552. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1553. WARNING:toil.leader:9/o/job5VmMi9 job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1554. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1555. WARNING:toil.leader:9/o/job5VmMi9 super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1556. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1557. WARNING:toil.leader:9/o/job5VmMi9 returnValues = self._run(jobGraph, fileStore)
  1558. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1559. WARNING:toil.leader:9/o/job5VmMi9 return self.run(fileStore)
  1560. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 831, in run
  1561. WARNING:toil.leader:9/o/job5VmMi9 self.makeRecursiveJobs(fileStore=fileStore)
  1562. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 417, in makeRecursiveJobs
  1563. WARNING:toil.leader:9/o/job5VmMi9 default=CactusRecursionJob.maxSequenceSizeOfFlowerGroupingDefault))
  1564. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 138, in runCactusGetFlowers
  1565. WARNING:toil.leader:9/o/job5VmMi9 fileStore=fileStore)
  1566. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1567. WARNING:toil.leader:9/o/job5VmMi9 raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1568. WARNING:toil.leader:9/o/job5VmMi9 RuntimeError: Command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="17032" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000'] failed with output:
  1569. WARNING:toil.leader:9/o/job5VmMi9 ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn40.grid.pub.ro
  1570. WARNING:toil.leader:9/o/job5VmMi9 WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusBarRecursion' 9/o/job5VmMi9 with ID 9/o/job5VmMi9 to 5
  1571. WARNING:toil.leader:9/o/job5VmMi9 WARNING:toil.jobGraph:We have increased the default memory of the failed job 'CactusBarRecursion' 9/o/job5VmMi9 to 32212254720 bytes
  1572. INFO:toil.leader:Issued job 'CactusBarRecursion' 9/o/job5VmMi9 with job batch system ID: 181 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  1573. INFO:toil.leader:Job ended successfully: 'CactusBarRecursion' 9/o/job5VmMi9
  1574. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusBarRecursion' 9/o/job5VmMi9
  1575. WARNING:toil.leader:9/o/job5VmMi9 INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1576. WARNING:toil.leader:9/o/job5VmMi9 INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1577. WARNING:toil.leader:9/o/job5VmMi9 WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1578. WARNING:toil.leader:9/o/job5VmMi9 INFO:cactus.shared.common:Running the command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="17032" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000']
  1579. WARNING:toil.leader:9/o/job5VmMi9 Set log level to INFO
  1580. WARNING:toil.leader:9/o/job5VmMi9 Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.29 with error: network error
  1581. WARNING:toil.leader:9/o/job5VmMi9 Uncaught exception
  1582. WARNING:toil.leader:9/o/job5VmMi9 Traceback (most recent call last):
  1583. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1584. WARNING:toil.leader:9/o/job5VmMi9 job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1585. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1586. WARNING:toil.leader:9/o/job5VmMi9 super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1587. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1588. WARNING:toil.leader:9/o/job5VmMi9 returnValues = self._run(jobGraph, fileStore)
  1589. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1590. WARNING:toil.leader:9/o/job5VmMi9 return self.run(fileStore)
  1591. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 831, in run
  1592. WARNING:toil.leader:9/o/job5VmMi9 self.makeRecursiveJobs(fileStore=fileStore)
  1593. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 417, in makeRecursiveJobs
  1594. WARNING:toil.leader:9/o/job5VmMi9 default=CactusRecursionJob.maxSequenceSizeOfFlowerGroupingDefault))
  1595. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 138, in runCactusGetFlowers
  1596. WARNING:toil.leader:9/o/job5VmMi9 fileStore=fileStore)
  1597. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1598. WARNING:toil.leader:9/o/job5VmMi9 raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1599. WARNING:toil.leader:9/o/job5VmMi9 RuntimeError: Command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="17032" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000'] failed with output:
  1600. WARNING:toil.leader:9/o/job5VmMi9 ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn37.grid.pub.ro
  1601. WARNING:toil.leader:9/o/job5VmMi9 WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusBarRecursion' 9/o/job5VmMi9 with ID 9/o/job5VmMi9 to 4
  1602. INFO:toil.leader:Issued job 'CactusBarRecursion' 9/o/job5VmMi9 with job batch system ID: 182 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  1603. INFO:toil.leader:Job ended successfully: 'CactusBarRecursion' 9/o/job5VmMi9
  1604. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusBarRecursion' 9/o/job5VmMi9
  1605. WARNING:toil.leader:9/o/job5VmMi9 INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1606. WARNING:toil.leader:9/o/job5VmMi9 INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1607. WARNING:toil.leader:9/o/job5VmMi9 WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1608. WARNING:toil.leader:9/o/job5VmMi9 INFO:cactus.shared.common:Running the command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="17032" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000']
  1609. WARNING:toil.leader:9/o/job5VmMi9 Set log level to INFO
  1610. WARNING:toil.leader:9/o/job5VmMi9 Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.29 with error: network error
  1611. WARNING:toil.leader:9/o/job5VmMi9 Uncaught exception
  1612. WARNING:toil.leader:9/o/job5VmMi9 Traceback (most recent call last):
  1613. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1614. WARNING:toil.leader:9/o/job5VmMi9 job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1615. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1616. WARNING:toil.leader:9/o/job5VmMi9 super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1617. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1618. WARNING:toil.leader:9/o/job5VmMi9 returnValues = self._run(jobGraph, fileStore)
  1619. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1620. WARNING:toil.leader:9/o/job5VmMi9 return self.run(fileStore)
  1621. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 831, in run
  1622. WARNING:toil.leader:9/o/job5VmMi9 self.makeRecursiveJobs(fileStore=fileStore)
  1623. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 417, in makeRecursiveJobs
  1624. WARNING:toil.leader:9/o/job5VmMi9 default=CactusRecursionJob.maxSequenceSizeOfFlowerGroupingDefault))
  1625. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 138, in runCactusGetFlowers
  1626. WARNING:toil.leader:9/o/job5VmMi9 fileStore=fileStore)
  1627. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1628. WARNING:toil.leader:9/o/job5VmMi9 raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1629. WARNING:toil.leader:9/o/job5VmMi9 RuntimeError: Command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="17032" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000'] failed with output:
  1630. WARNING:toil.leader:9/o/job5VmMi9 ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn41.grid.pub.ro
  1631. WARNING:toil.leader:9/o/job5VmMi9 WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusBarRecursion' 9/o/job5VmMi9 with ID 9/o/job5VmMi9 to 3
  1632. INFO:toil.leader:Issued job 'CactusBarRecursion' 9/o/job5VmMi9 with job batch system ID: 183 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  1633. INFO:toil.leader:Job ended successfully: 'CactusBarRecursion' 9/o/job5VmMi9
  1634. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusBarRecursion' 9/o/job5VmMi9
  1635. WARNING:toil.leader:9/o/job5VmMi9 INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1636. WARNING:toil.leader:9/o/job5VmMi9 INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1637. WARNING:toil.leader:9/o/job5VmMi9 WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1638. WARNING:toil.leader:9/o/job5VmMi9 INFO:cactus.shared.common:Running the command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="17032" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000']
  1639. WARNING:toil.leader:9/o/job5VmMi9 Set log level to INFO
  1640. WARNING:toil.leader:9/o/job5VmMi9 Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.29 with error: network error
  1641. WARNING:toil.leader:9/o/job5VmMi9 Uncaught exception
  1642. WARNING:toil.leader:9/o/job5VmMi9 Traceback (most recent call last):
  1643. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1644. WARNING:toil.leader:9/o/job5VmMi9 job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1645. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1646. WARNING:toil.leader:9/o/job5VmMi9 super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1647. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1648. WARNING:toil.leader:9/o/job5VmMi9 returnValues = self._run(jobGraph, fileStore)
  1649. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1650. WARNING:toil.leader:9/o/job5VmMi9 return self.run(fileStore)
  1651. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 831, in run
  1652. WARNING:toil.leader:9/o/job5VmMi9 self.makeRecursiveJobs(fileStore=fileStore)
  1653. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 417, in makeRecursiveJobs
  1654. WARNING:toil.leader:9/o/job5VmMi9 default=CactusRecursionJob.maxSequenceSizeOfFlowerGroupingDefault))
  1655. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 138, in runCactusGetFlowers
  1656. WARNING:toil.leader:9/o/job5VmMi9 fileStore=fileStore)
  1657. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1658. WARNING:toil.leader:9/o/job5VmMi9 raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1659. WARNING:toil.leader:9/o/job5VmMi9 RuntimeError: Command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="17032" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000'] failed with output:
  1660. WARNING:toil.leader:9/o/job5VmMi9 ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn34.grid.pub.ro
  1661. WARNING:toil.leader:9/o/job5VmMi9 WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusBarRecursion' 9/o/job5VmMi9 with ID 9/o/job5VmMi9 to 2
  1662. INFO:toil.leader:Issued job 'CactusBarRecursion' 9/o/job5VmMi9 with job batch system ID: 184 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  1663. INFO:toil.leader:Job ended successfully: 'CactusBarRecursion' 9/o/job5VmMi9
  1664. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusBarRecursion' 9/o/job5VmMi9
  1665. WARNING:toil.leader:9/o/job5VmMi9 INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1666. WARNING:toil.leader:9/o/job5VmMi9 INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1667. WARNING:toil.leader:9/o/job5VmMi9 WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1668. WARNING:toil.leader:9/o/job5VmMi9 INFO:cactus.shared.common:Running the command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="17032" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000']
  1669. WARNING:toil.leader:9/o/job5VmMi9 Set log level to INFO
  1670. WARNING:toil.leader:9/o/job5VmMi9 Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.29 with error: network error
  1671. WARNING:toil.leader:9/o/job5VmMi9 Uncaught exception
  1672. WARNING:toil.leader:9/o/job5VmMi9 Traceback (most recent call last):
  1673. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1674. WARNING:toil.leader:9/o/job5VmMi9 job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1675. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1676. WARNING:toil.leader:9/o/job5VmMi9 super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1677. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1678. WARNING:toil.leader:9/o/job5VmMi9 returnValues = self._run(jobGraph, fileStore)
  1679. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1680. WARNING:toil.leader:9/o/job5VmMi9 return self.run(fileStore)
  1681. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 831, in run
  1682. WARNING:toil.leader:9/o/job5VmMi9 self.makeRecursiveJobs(fileStore=fileStore)
  1683. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 417, in makeRecursiveJobs
  1684. WARNING:toil.leader:9/o/job5VmMi9 default=CactusRecursionJob.maxSequenceSizeOfFlowerGroupingDefault))
  1685. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 138, in runCactusGetFlowers
  1686. WARNING:toil.leader:9/o/job5VmMi9 fileStore=fileStore)
  1687. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1688. WARNING:toil.leader:9/o/job5VmMi9 raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1689. WARNING:toil.leader:9/o/job5VmMi9 RuntimeError: Command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="17032" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000'] failed with output:
  1690. WARNING:toil.leader:9/o/job5VmMi9 ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn40.grid.pub.ro
  1691. WARNING:toil.leader:9/o/job5VmMi9 WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusBarRecursion' 9/o/job5VmMi9 with ID 9/o/job5VmMi9 to 1
  1692. INFO:toil.leader:Issued job 'CactusBarRecursion' 9/o/job5VmMi9 with job batch system ID: 185 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  1693. INFO:toil.leader:Job ended successfully: 'CactusBarRecursion' 9/o/job5VmMi9
  1694. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusBarRecursion' 9/o/job5VmMi9
  1695. WARNING:toil.leader:9/o/job5VmMi9 INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1696. WARNING:toil.leader:9/o/job5VmMi9 INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1697. WARNING:toil.leader:9/o/job5VmMi9 WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1698. WARNING:toil.leader:9/o/job5VmMi9 INFO:cactus.shared.common:Running the command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="17032" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000']
  1699. WARNING:toil.leader:9/o/job5VmMi9 Set log level to INFO
  1700. WARNING:toil.leader:9/o/job5VmMi9 Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.29 with error: network error
  1701. WARNING:toil.leader:9/o/job5VmMi9 Uncaught exception
  1702. WARNING:toil.leader:9/o/job5VmMi9 Traceback (most recent call last):
  1703. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1704. WARNING:toil.leader:9/o/job5VmMi9 job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1705. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1706. WARNING:toil.leader:9/o/job5VmMi9 super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1707. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1708. WARNING:toil.leader:9/o/job5VmMi9 returnValues = self._run(jobGraph, fileStore)
  1709. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1710. WARNING:toil.leader:9/o/job5VmMi9 return self.run(fileStore)
  1711. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 831, in run
  1712. WARNING:toil.leader:9/o/job5VmMi9 self.makeRecursiveJobs(fileStore=fileStore)
  1713. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 417, in makeRecursiveJobs
  1714. WARNING:toil.leader:9/o/job5VmMi9 default=CactusRecursionJob.maxSequenceSizeOfFlowerGroupingDefault))
  1715. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 138, in runCactusGetFlowers
  1716. WARNING:toil.leader:9/o/job5VmMi9 fileStore=fileStore)
  1717. WARNING:toil.leader:9/o/job5VmMi9 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1718. WARNING:toil.leader:9/o/job5VmMi9 raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1719. WARNING:toil.leader:9/o/job5VmMi9 RuntimeError: Command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="17032" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000'] failed with output:
  1720. WARNING:toil.leader:9/o/job5VmMi9 ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn38.grid.pub.ro
  1721. WARNING:toil.leader:9/o/job5VmMi9 WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusBarRecursion' 9/o/job5VmMi9 with ID 9/o/job5VmMi9 to 0
  1722. WARNING:toil.leader:Job 'CactusBarRecursion' 9/o/job5VmMi9 with ID 9/o/job5VmMi9 is completely failed
  1723. INFO:toil.leader:Job ended successfully: 'KtServerService' K/3/jobZoBnBp
  1724. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'KtServerService' K/3/jobZoBnBp
  1725. WARNING:toil.leader:K/3/jobZoBnBp INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1726. WARNING:toil.leader:K/3/jobZoBnBp INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1727. WARNING:toil.leader:K/3/jobZoBnBp WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1728. WARNING:toil.leader:K/3/jobZoBnBp WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1729. WARNING:toil.leader:K/3/jobZoBnBp INFO:cactus.shared.common:Running the command ['netstat', '-tuplen']
  1730. WARNING:toil.leader:K/3/jobZoBnBp (No info could be read for "-p": geteuid()=98354 but you should be root.)
  1731. WARNING:toil.leader:K/3/jobZoBnBp INFO:cactus.shared.common:Running the command ['ktserver', '-port', '20468', '-ls', '-tout', '200000', '-th', '64', '-bgs', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-b02a3811-2b63-4208-851d-7815af46a62d/tmpCWDNW1/7e4e1c99-e696-494d-9865-473277b3a1a7/tZ0iLVN/snapshot', '-bgsc', 'lzo', '-bgsi', '1000000', '-log', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-b02a3811-2b63-4208-851d-7815af46a62d/tmpCWDNW1/7e4e1c99-e696-494d-9865-473277b3a1a7/tmp2H1g0s.tmp', ':#opts=ls#bnum=30m#msiz=50g#ktopts=p']
  1732. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1733. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1734. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1735. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 556
  1736. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1737. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1738. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1739. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 556
  1740. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1741. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1742. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1743. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 557
  1744. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1745. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1746. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1747. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 558
  1748. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1749. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1750. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1751. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 559
  1752. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1753. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1754. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1755. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 560
  1756. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1757. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1758. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1759. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 561
  1760. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1761. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1762. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1763. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 562
  1764. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1765. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1766. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1767. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 563
  1768. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1769. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1770. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1771. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 564
  1772. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1773. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1774. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1775. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 565
  1776. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1777. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1778. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1779. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 566
  1780. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1781. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1782. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1783. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 567
  1784. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1785. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1786. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1787. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 568
  1788. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1789. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1790. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1791. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 569
  1792. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1793. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1794. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1795. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 570
  1796. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1797. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1798. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1799. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 571
  1800. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1801. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1802. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1803. WARNING:toil.leader:K/3/jobZoBnBp INFO:toil.lib.bioio:Ktserver running.
  1804. WARNING:toil.leader:K/3/jobZoBnBp INFO:toil.lib.bioio:Ktserver running.
  1805. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 572
  1806. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1807. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1808. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1809. WARNING:toil.leader:K/3/jobZoBnBp INFO:toil.lib.bioio:Ktserver running.
  1810. WARNING:toil.leader:K/3/jobZoBnBp INFO:cactus.shared.common:Running the command ['ktremotemgr', 'remove', '-port', '20468', '-host', '172.16.13.39', 'TERMINATE']
  1811. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 573
  1812. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1813. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1814. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1815. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 575
  1816. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1817. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1818. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1819. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 576
  1820. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1821. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1822. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1823. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 577
  1824. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1825. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1826. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1827. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 578
  1828. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1829. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1830. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1831. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 579
  1832. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1833. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1834. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1835. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 580
  1836. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1837. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1838. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1839. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 581
  1840. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1841. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1842. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1843. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 582
  1844. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1845. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1846. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1847. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 583
  1848. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1849. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1850. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1851. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 584
  1852. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1853. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1854. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1855. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 585
  1856. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1857. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1858. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1859. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 586
  1860. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1861. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1862. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1863. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 587
  1864. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1865. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1866. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1867. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 588
  1868. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1869. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1870. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1871. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 589
  1872. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1873. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1874. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1875. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 590
  1876. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1877. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1878. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1879. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 591
  1880. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1881. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1882. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1883. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 592
  1884. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1885. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1886. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1887. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 593
  1888. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1889. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1890. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1891. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 594
  1892. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1893. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1894. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1895. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 595
  1896. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1897. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1898. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1899. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 596
  1900. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1901. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1902. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1903. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 597
  1904. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1905. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1906. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1907. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 598
  1908. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1909. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1910. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1911. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 599
  1912. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1913. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1914. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1915. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 600
  1916. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1917. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1918. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1919. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 601
  1920. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1921. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1922. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1923. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 602
  1924. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1925. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1926. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1927. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 603
  1928. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1929. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1930. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1931. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 604
  1932. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1933. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1934. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1935. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 605
  1936. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1937. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1938. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1939. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 606
  1940. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1941. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1942. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1943. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 607
  1944. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1945. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1946. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1947. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 608
  1948. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1949. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1950. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1951. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 609
  1952. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1953. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1954. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1955. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 610
  1956. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1957. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1958. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1959. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 611
  1960. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1961. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1962. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1963. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 612
  1964. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1965. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1966. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1967. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 613
  1968. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1969. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1970. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1971. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 614
  1972. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1973. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1974. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1975. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 615
  1976. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1977. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1978. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1979. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 616
  1980. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1981. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1982. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1983. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 617
  1984. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1985. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1986. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1987. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 618
  1988. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1989. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1990. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1991. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 619
  1992. WARNING:toil.leader:K/3/jobZoBnBp Stack limits: Infinity Infinity
  1993. WARNING:toil.leader:K/3/jobZoBnBp Process limits: 510853 510853
  1994. WARNING:toil.leader:K/3/jobZoBnBp Kernel thread cap: 1021706
  1995. WARNING:toil.leader:K/3/jobZoBnBp Current total threads running: 620
  1996. WARNING:toil.leader:K/3/jobZoBnBp INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '20468', '-host', '172.16.13.39', 'TERMINATE']
  1997. WARNING:toil.leader:K/3/jobZoBnBp INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '20468', '-host', '172.16.13.39', 'TERMINATE']
  1998. WARNING:toil.leader:K/3/jobZoBnBp INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '20468', '-host', '172.16.13.39', 'TERMINATE']
  1999. WARNING:toil.leader:K/3/jobZoBnBp INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '20468', '-host', '172.16.13.39', 'TERMINATE']
  2000. WARNING:toil.leader:K/3/jobZoBnBp INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '20468', '-host', '172.16.13.39', 'TERMINATE']
  2001. WARNING:toil.leader:K/3/jobZoBnBp INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '20468', '-host', '172.16.13.39', 'TERMINATE']
  2002. WARNING:toil.leader:K/3/jobZoBnBp INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '20468', '-host', '172.16.13.39', 'TERMINATE']
  2003. WARNING:toil.leader:K/3/jobZoBnBp INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '20468', '-host', '172.16.13.39', 'TERMINATE']
  2004. WARNING:toil.leader:K/3/jobZoBnBp INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '20468', '-host', '172.16.13.39', 'TERMINATE']
  2005. WARNING:toil.leader:K/3/jobZoBnBp INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '20468', '-host', '172.16.13.39', 'TERMINATE']
  2006. WARNING:toil.leader:K/3/jobZoBnBp INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '20468', '-host', '172.16.13.39', 'TERMINATE']
  2007. WARNING:toil.leader:K/3/jobZoBnBp INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '20468', '-host', '172.16.13.39', 'TERMINATE']
  2008. WARNING:toil.leader:K/3/jobZoBnBp INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '20468', '-host', '172.16.13.39', 'TERMINATE']
  2009. WARNING:toil.leader:K/3/jobZoBnBp INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '20468', '-host', '172.16.13.39', 'TERMINATE']
  2010. WARNING:toil.leader:K/3/jobZoBnBp INFO:cactus.shared.common:Running the command ['ktremotemgr', 'set', '-port', '20468', '-host', '172.16.13.39', 'TERMINATE', '1']
  2011. WARNING:toil.leader:K/3/jobZoBnBp INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '20468', '-host', '172.16.13.39', 'TERMINATE']
  2012. WARNING:toil.leader:K/3/jobZoBnBp 1
  2013. WARNING:toil.leader:K/3/jobZoBnBp Traceback (most recent call last):
  2014. WARNING:toil.leader:K/3/jobZoBnBp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  2015. WARNING:toil.leader:K/3/jobZoBnBp job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2016. WARNING:toil.leader:K/3/jobZoBnBp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  2017. WARNING:toil.leader:K/3/jobZoBnBp returnValues = self._run(jobGraph, fileStore)
  2018. WARNING:toil.leader:K/3/jobZoBnBp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1694, in _run
  2019. WARNING:toil.leader:K/3/jobZoBnBp returnValues = self.run(fileStore)
  2020. WARNING:toil.leader:K/3/jobZoBnBp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1668, in run
  2021. WARNING:toil.leader:K/3/jobZoBnBp raise RuntimeError("Detected the error jobStoreID has been removed so exiting with an error")
  2022. WARNING:toil.leader:K/3/jobZoBnBp RuntimeError: Detected the error jobStoreID has been removed so exiting with an error
  2023. WARNING:toil.leader:K/3/jobZoBnBp ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn39.grid.pub.ro
  2024. WARNING:toil.leader:K/3/jobZoBnBp WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' K/3/jobZoBnBp with ID K/3/jobZoBnBp to 4
  2025. WARNING:toil.leader:Job 'KtServerService' K/3/jobZoBnBp with ID K/3/jobZoBnBp is completely failed
  2026. WARNING:toil.leader:Job: o/H/jobK044oZ is being restarted as a checkpoint after the total failure of jobs in its subtree.
  2027. INFO:toil.leader:Issued job 'StartPrimaryDB' o/H/jobK044oZ with job batch system ID: 186 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  2028. INFO:toil.leader:Job ended successfully: 'StartPrimaryDB' o/H/jobK044oZ
  2029. INFO:toil.leader:Issued job 'KtServerService' Q/J/jobdnGWgD with job batch system ID: 187 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  2030. INFO:toil.leader:Issued job 'CactusBarPhase' m/B/jobEH6tAv with job batch system ID: 188 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  2031. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' Q/J/jobdnGWgD
  2032. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' Q/J/jobdnGWgD
  2033. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' Q/J/jobdnGWgD with ID Q/J/jobdnGWgD to 5
  2034. WARNING:toil.jobGraph:We have increased the default memory of the failed job 'KtServerService' Q/J/jobdnGWgD to 32212254720 bytes
  2035. INFO:toil.leader:Issued job 'KtServerService' Q/J/jobdnGWgD with job batch system ID: 189 and cores: 0, disk: 2.0 G, and memory: 30.0 G
  2036. INFO:toil.leader:Job ended successfully: 'CactusBarPhase' m/B/jobEH6tAv
  2037. INFO:toil.leader:Issued job 'CactusBarRecursion' 7/G/jobwxuc5q with job batch system ID: 190 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  2038. INFO:toil.leader:Job ended successfully: 'CactusBarRecursion' 7/G/jobwxuc5q
  2039. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusBarRecursion' 7/G/jobwxuc5q
  2040. WARNING:toil.leader:7/G/jobwxuc5q INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  2041. WARNING:toil.leader:7/G/jobwxuc5q INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  2042. WARNING:toil.leader:7/G/jobwxuc5q WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  2043. WARNING:toil.leader:7/G/jobwxuc5q INFO:cactus.shared.common:Running the command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="8790" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000']
  2044. WARNING:toil.leader:7/G/jobwxuc5q Set log level to INFO
  2045. WARNING:toil.leader:7/G/jobwxuc5q Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.29 with error: network error
  2046. WARNING:toil.leader:7/G/jobwxuc5q Uncaught exception
  2047. WARNING:toil.leader:7/G/jobwxuc5q Traceback (most recent call last):
  2048. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  2049. WARNING:toil.leader:7/G/jobwxuc5q job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2050. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  2051. WARNING:toil.leader:7/G/jobwxuc5q super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2052. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  2053. WARNING:toil.leader:7/G/jobwxuc5q returnValues = self._run(jobGraph, fileStore)
  2054. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  2055. WARNING:toil.leader:7/G/jobwxuc5q return self.run(fileStore)
  2056. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 831, in run
  2057. WARNING:toil.leader:7/G/jobwxuc5q self.makeRecursiveJobs(fileStore=fileStore)
  2058. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 417, in makeRecursiveJobs
  2059. WARNING:toil.leader:7/G/jobwxuc5q default=CactusRecursionJob.maxSequenceSizeOfFlowerGroupingDefault))
  2060. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 138, in runCactusGetFlowers
  2061. WARNING:toil.leader:7/G/jobwxuc5q fileStore=fileStore)
  2062. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  2063. WARNING:toil.leader:7/G/jobwxuc5q raise RuntimeError("Command %s failed with output: %s" % (call, output))
  2064. WARNING:toil.leader:7/G/jobwxuc5q RuntimeError: Command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="8790" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000'] failed with output:
  2065. WARNING:toil.leader:7/G/jobwxuc5q ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn40.grid.pub.ro
  2066. WARNING:toil.leader:7/G/jobwxuc5q WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusBarRecursion' 7/G/jobwxuc5q with ID 7/G/jobwxuc5q to 5
  2067. WARNING:toil.leader:7/G/jobwxuc5q WARNING:toil.jobGraph:We have increased the default memory of the failed job 'CactusBarRecursion' 7/G/jobwxuc5q to 32212254720 bytes
  2068. INFO:toil.leader:Issued job 'CactusBarRecursion' 7/G/jobwxuc5q with job batch system ID: 191 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  2069. INFO:toil.leader:Job ended successfully: 'CactusBarRecursion' 7/G/jobwxuc5q
  2070. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusBarRecursion' 7/G/jobwxuc5q
  2071. WARNING:toil.leader:7/G/jobwxuc5q INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  2072. WARNING:toil.leader:7/G/jobwxuc5q INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  2073. WARNING:toil.leader:7/G/jobwxuc5q WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  2074. WARNING:toil.leader:7/G/jobwxuc5q INFO:cactus.shared.common:Running the command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="8790" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000']
  2075. WARNING:toil.leader:7/G/jobwxuc5q Set log level to INFO
  2076. WARNING:toil.leader:7/G/jobwxuc5q Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.29 with error: network error
  2077. WARNING:toil.leader:7/G/jobwxuc5q Uncaught exception
  2078. WARNING:toil.leader:7/G/jobwxuc5q Traceback (most recent call last):
  2079. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  2080. WARNING:toil.leader:7/G/jobwxuc5q job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2081. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  2082. WARNING:toil.leader:7/G/jobwxuc5q super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2083. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  2084. WARNING:toil.leader:7/G/jobwxuc5q returnValues = self._run(jobGraph, fileStore)
  2085. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  2086. WARNING:toil.leader:7/G/jobwxuc5q return self.run(fileStore)
  2087. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 831, in run
  2088. WARNING:toil.leader:7/G/jobwxuc5q self.makeRecursiveJobs(fileStore=fileStore)
  2089. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 417, in makeRecursiveJobs
  2090. WARNING:toil.leader:7/G/jobwxuc5q default=CactusRecursionJob.maxSequenceSizeOfFlowerGroupingDefault))
  2091. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 138, in runCactusGetFlowers
  2092. WARNING:toil.leader:7/G/jobwxuc5q fileStore=fileStore)
  2093. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  2094. WARNING:toil.leader:7/G/jobwxuc5q raise RuntimeError("Command %s failed with output: %s" % (call, output))
  2095. WARNING:toil.leader:7/G/jobwxuc5q RuntimeError: Command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="8790" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000'] failed with output:
  2096. WARNING:toil.leader:7/G/jobwxuc5q ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn38.grid.pub.ro
  2097. WARNING:toil.leader:7/G/jobwxuc5q WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusBarRecursion' 7/G/jobwxuc5q with ID 7/G/jobwxuc5q to 4
  2098. INFO:toil.leader:Issued job 'CactusBarRecursion' 7/G/jobwxuc5q with job batch system ID: 192 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  2099. INFO:toil.leader:Job ended successfully: 'CactusBarRecursion' 7/G/jobwxuc5q
  2100. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusBarRecursion' 7/G/jobwxuc5q
  2101. WARNING:toil.leader:7/G/jobwxuc5q INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  2102. WARNING:toil.leader:7/G/jobwxuc5q INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  2103. WARNING:toil.leader:7/G/jobwxuc5q WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  2104. WARNING:toil.leader:7/G/jobwxuc5q WARNING:toil.fileStore:Detected that job (o/H/jobK044oZ/g/tmphUEhpW-_serialiseJob-stream) prematurely terminated. Fixing the state of the job on disk.
  2105. WARNING:toil.leader:7/G/jobwxuc5q INFO:cactus.shared.common:Running the command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="8790" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000']
  2106. WARNING:toil.leader:7/G/jobwxuc5q Set log level to INFO
  2107. WARNING:toil.leader:7/G/jobwxuc5q Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.29 with error: network error
  2108. WARNING:toil.leader:7/G/jobwxuc5q Uncaught exception
  2109. WARNING:toil.leader:7/G/jobwxuc5q Traceback (most recent call last):
  2110. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  2111. WARNING:toil.leader:7/G/jobwxuc5q job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2112. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  2113. WARNING:toil.leader:7/G/jobwxuc5q super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2114. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  2115. WARNING:toil.leader:7/G/jobwxuc5q returnValues = self._run(jobGraph, fileStore)
  2116. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  2117. WARNING:toil.leader:7/G/jobwxuc5q return self.run(fileStore)
  2118. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 831, in run
  2119. WARNING:toil.leader:7/G/jobwxuc5q self.makeRecursiveJobs(fileStore=fileStore)
  2120. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 417, in makeRecursiveJobs
  2121. WARNING:toil.leader:7/G/jobwxuc5q default=CactusRecursionJob.maxSequenceSizeOfFlowerGroupingDefault))
  2122. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 138, in runCactusGetFlowers
  2123. WARNING:toil.leader:7/G/jobwxuc5q fileStore=fileStore)
  2124. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  2125. WARNING:toil.leader:7/G/jobwxuc5q raise RuntimeError("Command %s failed with output: %s" % (call, output))
  2126. WARNING:toil.leader:7/G/jobwxuc5q RuntimeError: Command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="8790" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000'] failed with output:
  2127. WARNING:toil.leader:7/G/jobwxuc5q ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn29.grid.pub.ro
  2128. WARNING:toil.leader:7/G/jobwxuc5q WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusBarRecursion' 7/G/jobwxuc5q with ID 7/G/jobwxuc5q to 3
  2129. INFO:toil.leader:Issued job 'CactusBarRecursion' 7/G/jobwxuc5q with job batch system ID: 193 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  2130. INFO:toil.leader:Job ended successfully: 'CactusBarRecursion' 7/G/jobwxuc5q
  2131. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusBarRecursion' 7/G/jobwxuc5q
  2132. WARNING:toil.leader:7/G/jobwxuc5q INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  2133. WARNING:toil.leader:7/G/jobwxuc5q INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  2134. WARNING:toil.leader:7/G/jobwxuc5q WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  2135. WARNING:toil.leader:7/G/jobwxuc5q INFO:cactus.shared.common:Running the command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="8790" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000']
  2136. WARNING:toil.leader:7/G/jobwxuc5q Set log level to INFO
  2137. WARNING:toil.leader:7/G/jobwxuc5q Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.29 with error: network error
  2138. WARNING:toil.leader:7/G/jobwxuc5q Uncaught exception
  2139. WARNING:toil.leader:7/G/jobwxuc5q Traceback (most recent call last):
  2140. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  2141. WARNING:toil.leader:7/G/jobwxuc5q job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2142. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  2143. WARNING:toil.leader:7/G/jobwxuc5q super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2144. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  2145. WARNING:toil.leader:7/G/jobwxuc5q returnValues = self._run(jobGraph, fileStore)
  2146. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  2147. WARNING:toil.leader:7/G/jobwxuc5q return self.run(fileStore)
  2148. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 831, in run
  2149. WARNING:toil.leader:7/G/jobwxuc5q self.makeRecursiveJobs(fileStore=fileStore)
  2150. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 417, in makeRecursiveJobs
  2151. WARNING:toil.leader:7/G/jobwxuc5q default=CactusRecursionJob.maxSequenceSizeOfFlowerGroupingDefault))
  2152. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 138, in runCactusGetFlowers
  2153. WARNING:toil.leader:7/G/jobwxuc5q fileStore=fileStore)
  2154. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  2155. WARNING:toil.leader:7/G/jobwxuc5q raise RuntimeError("Command %s failed with output: %s" % (call, output))
  2156. WARNING:toil.leader:7/G/jobwxuc5q RuntimeError: Command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="8790" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000'] failed with output:
  2157. WARNING:toil.leader:7/G/jobwxuc5q ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn40.grid.pub.ro
  2158. WARNING:toil.leader:7/G/jobwxuc5q WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusBarRecursion' 7/G/jobwxuc5q with ID 7/G/jobwxuc5q to 2
  2159. INFO:toil.leader:Issued job 'CactusBarRecursion' 7/G/jobwxuc5q with job batch system ID: 194 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  2160. INFO:toil.leader:Job ended successfully: 'CactusBarRecursion' 7/G/jobwxuc5q
  2161. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusBarRecursion' 7/G/jobwxuc5q
  2162. WARNING:toil.leader:7/G/jobwxuc5q INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  2163. WARNING:toil.leader:7/G/jobwxuc5q INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  2164. WARNING:toil.leader:7/G/jobwxuc5q WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  2165. WARNING:toil.leader:7/G/jobwxuc5q INFO:cactus.shared.common:Running the command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="8790" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000']
  2166. WARNING:toil.leader:7/G/jobwxuc5q Set log level to INFO
  2167. WARNING:toil.leader:7/G/jobwxuc5q Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.29 with error: network error
  2168. WARNING:toil.leader:7/G/jobwxuc5q Uncaught exception
  2169. WARNING:toil.leader:7/G/jobwxuc5q Traceback (most recent call last):
  2170. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  2171. WARNING:toil.leader:7/G/jobwxuc5q job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2172. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  2173. WARNING:toil.leader:7/G/jobwxuc5q super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2174. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  2175. WARNING:toil.leader:7/G/jobwxuc5q returnValues = self._run(jobGraph, fileStore)
  2176. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  2177. WARNING:toil.leader:7/G/jobwxuc5q return self.run(fileStore)
  2178. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 831, in run
  2179. WARNING:toil.leader:7/G/jobwxuc5q self.makeRecursiveJobs(fileStore=fileStore)
  2180. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 417, in makeRecursiveJobs
  2181. WARNING:toil.leader:7/G/jobwxuc5q default=CactusRecursionJob.maxSequenceSizeOfFlowerGroupingDefault))
  2182. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 138, in runCactusGetFlowers
  2183. WARNING:toil.leader:7/G/jobwxuc5q fileStore=fileStore)
  2184. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  2185. WARNING:toil.leader:7/G/jobwxuc5q raise RuntimeError("Command %s failed with output: %s" % (call, output))
  2186. WARNING:toil.leader:7/G/jobwxuc5q RuntimeError: Command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="8790" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000'] failed with output:
  2187. WARNING:toil.leader:7/G/jobwxuc5q ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn39.grid.pub.ro
  2188. WARNING:toil.leader:7/G/jobwxuc5q WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusBarRecursion' 7/G/jobwxuc5q with ID 7/G/jobwxuc5q to 1
  2189. INFO:toil.leader:Issued job 'CactusBarRecursion' 7/G/jobwxuc5q with job batch system ID: 195 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  2190. INFO:toil.leader:Job ended successfully: 'CactusBarRecursion' 7/G/jobwxuc5q
  2191. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusBarRecursion' 7/G/jobwxuc5q
  2192. WARNING:toil.leader:7/G/jobwxuc5q INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  2193. WARNING:toil.leader:7/G/jobwxuc5q INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  2194. WARNING:toil.leader:7/G/jobwxuc5q WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  2195. WARNING:toil.leader:7/G/jobwxuc5q INFO:cactus.shared.common:Running the command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="8790" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000']
  2196. WARNING:toil.leader:7/G/jobwxuc5q Set log level to INFO
  2197. WARNING:toil.leader:7/G/jobwxuc5q Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.29 with error: network error
  2198. WARNING:toil.leader:7/G/jobwxuc5q Uncaught exception
  2199. WARNING:toil.leader:7/G/jobwxuc5q Traceback (most recent call last):
  2200. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  2201. WARNING:toil.leader:7/G/jobwxuc5q job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2202. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  2203. WARNING:toil.leader:7/G/jobwxuc5q super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2204. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  2205. WARNING:toil.leader:7/G/jobwxuc5q returnValues = self._run(jobGraph, fileStore)
  2206. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  2207. WARNING:toil.leader:7/G/jobwxuc5q return self.run(fileStore)
  2208. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 831, in run
  2209. WARNING:toil.leader:7/G/jobwxuc5q self.makeRecursiveJobs(fileStore=fileStore)
  2210. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 417, in makeRecursiveJobs
  2211. WARNING:toil.leader:7/G/jobwxuc5q default=CactusRecursionJob.maxSequenceSizeOfFlowerGroupingDefault))
  2212. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 138, in runCactusGetFlowers
  2213. WARNING:toil.leader:7/G/jobwxuc5q fileStore=fileStore)
  2214. WARNING:toil.leader:7/G/jobwxuc5q File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  2215. WARNING:toil.leader:7/G/jobwxuc5q raise RuntimeError("Command %s failed with output: %s" % (call, output))
  2216. WARNING:toil.leader:7/G/jobwxuc5q RuntimeError: Command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="8790" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000'] failed with output:
  2217. WARNING:toil.leader:7/G/jobwxuc5q ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn38.grid.pub.ro
  2218. WARNING:toil.leader:7/G/jobwxuc5q WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusBarRecursion' 7/G/jobwxuc5q with ID 7/G/jobwxuc5q to 0
  2219. WARNING:toil.leader:Job 'CactusBarRecursion' 7/G/jobwxuc5q with ID 7/G/jobwxuc5q is completely failed
  2220. INFO:toil.leader:Reissued any over long jobs
  2221. INFO:toil.leader:Job ended successfully: 'KtServerService' Q/J/jobdnGWgD
  2222. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'KtServerService' Q/J/jobdnGWgD
  2223. WARNING:toil.leader:Q/J/jobdnGWgD INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  2224. WARNING:toil.leader:Q/J/jobdnGWgD INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  2225. WARNING:toil.leader:Q/J/jobdnGWgD WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  2226. WARNING:toil.leader:Q/J/jobdnGWgD WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  2227. WARNING:toil.leader:Q/J/jobdnGWgD INFO:cactus.shared.common:Running the command ['netstat', '-tuplen']
  2228. WARNING:toil.leader:Q/J/jobdnGWgD (No info could be read for "-p": geteuid()=98354 but you should be root.)
  2229. WARNING:toil.leader:Q/J/jobdnGWgD INFO:cactus.shared.common:Running the command ['ktserver', '-port', '1884', '-ls', '-tout', '200000', '-th', '64', '-bgs', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-b02a3811-2b63-4208-851d-7815af46a62d/tmpEsR7JD/e22b9011-f2ad-4f48-a271-ecf7ac0ce299/tNVDKS9/snapshot', '-bgsc', 'lzo', '-bgsi', '1000000', '-log', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-b02a3811-2b63-4208-851d-7815af46a62d/tmpEsR7JD/e22b9011-f2ad-4f48-a271-ecf7ac0ce299/tmpWBiHLX.tmp', ':#opts=ls#bnum=30m#msiz=50g#ktopts=p']
  2230. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2231. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2232. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2233. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 556
  2234. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2235. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2236. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2237. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 556
  2238. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2239. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2240. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2241. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 557
  2242. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2243. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2244. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2245. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 558
  2246. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2247. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2248. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2249. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 559
  2250. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2251. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2252. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2253. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 560
  2254. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2255. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2256. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2257. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 561
  2258. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2259. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2260. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2261. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 562
  2262. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2263. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2264. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2265. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 563
  2266. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2267. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2268. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2269. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 564
  2270. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2271. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2272. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2273. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 565
  2274. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2275. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2276. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2277. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 566
  2278. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2279. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2280. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2281. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 567
  2282. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2283. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2284. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2285. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 568
  2286. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2287. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2288. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2289. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 569
  2290. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2291. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2292. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2293. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 570
  2294. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2295. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2296. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2297. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 571
  2298. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2299. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2300. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2301. WARNING:toil.leader:Q/J/jobdnGWgD INFO:toil.lib.bioio:Ktserver running.
  2302. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 572
  2303. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2304. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 INFO:toil.lib.bioio:Ktserver running.
  2305. WARNING:toil.leader:Q/J/jobdnGWgD 510853
  2306. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2307. WARNING:toil.leader:Q/J/jobdnGWgD INFO:toil.lib.bioio:Ktserver running.
  2308. WARNING:toil.leader:Q/J/jobdnGWgD INFO:cactus.shared.common:Running the command ['ktremotemgr', 'remove', '-port', '1884', '-host', '172.16.13.39', 'TERMINATE']
  2309. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 573
  2310. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2311. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2312. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2313. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 575
  2314. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2315. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2316. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2317. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 576
  2318. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2319. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2320. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2321. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 577
  2322. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2323. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2324. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2325. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 578
  2326. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2327. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2328. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2329. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 579
  2330. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2331. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2332. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2333. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 580
  2334. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2335. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2336. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2337. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 581
  2338. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2339. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2340. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2341. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 582
  2342. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2343. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2344. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2345. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 583
  2346. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2347. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2348. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2349. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 584
  2350. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2351. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2352. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2353. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 585
  2354. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2355. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2356. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2357. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 586
  2358. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2359. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2360. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2361. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 587
  2362. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2363. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2364. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2365. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 588
  2366. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2367. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2368. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2369. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 589
  2370. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2371. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2372. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2373. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 590
  2374. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2375. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2376. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2377. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 591
  2378. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2379. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2380. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2381. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 592
  2382. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2383. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2384. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2385. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 593
  2386. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2387. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2388. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2389. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 594
  2390. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2391. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2392. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2393. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 595
  2394. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2395. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2396. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2397. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 596
  2398. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2399. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2400. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2401. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 597
  2402. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2403. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2404. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2405. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 598
  2406. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2407. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2408. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2409. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 599
  2410. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2411. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2412. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2413. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 600
  2414. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2415. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2416. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2417. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 601
  2418. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2419. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2420. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2421. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 602
  2422. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2423. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2424. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2425. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 603
  2426. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2427. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2428. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2429. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 604
  2430. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2431. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2432. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2433. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 605
  2434. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2435. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2436. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2437. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 606
  2438. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2439. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2440. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2441. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 607
  2442. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2443. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2444. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2445. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 608
  2446. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2447. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2448. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2449. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 609
  2450. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2451. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2452. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2453. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 610
  2454. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2455. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2456. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2457. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 611
  2458. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2459. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2460. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2461. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 612
  2462. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2463. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2464. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2465. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 613
  2466. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2467. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2468. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2469. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 614
  2470. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2471. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2472. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2473. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 615
  2474. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2475. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2476. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2477. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 616
  2478. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2479. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2480. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2481. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 617
  2482. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2483. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2484. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2485. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 618
  2486. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2487. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2488. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2489. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 619
  2490. WARNING:toil.leader:Q/J/jobdnGWgD Stack limits: Infinity Infinity
  2491. WARNING:toil.leader:Q/J/jobdnGWgD Process limits: 510853 510853
  2492. WARNING:toil.leader:Q/J/jobdnGWgD Kernel thread cap: 1021706
  2493. WARNING:toil.leader:Q/J/jobdnGWgD Current total threads running: 620
  2494. WARNING:toil.leader:Q/J/jobdnGWgD INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '1884', '-host', '172.16.13.39', 'TERMINATE']
  2495. WARNING:toil.leader:Q/J/jobdnGWgD INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '1884', '-host', '172.16.13.39', 'TERMINATE']
  2496. WARNING:toil.leader:Q/J/jobdnGWgD INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '1884', '-host', '172.16.13.39', 'TERMINATE']
  2497. WARNING:toil.leader:Q/J/jobdnGWgD INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '1884', '-host', '172.16.13.39', 'TERMINATE']
  2498. WARNING:toil.leader:Q/J/jobdnGWgD INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '1884', '-host', '172.16.13.39', 'TERMINATE']
  2499. WARNING:toil.leader:Q/J/jobdnGWgD INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '1884', '-host', '172.16.13.39', 'TERMINATE']
  2500. WARNING:toil.leader:Q/J/jobdnGWgD INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '1884', '-host', '172.16.13.39', 'TERMINATE']
  2501. WARNING:toil.leader:Q/J/jobdnGWgD INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '1884', '-host', '172.16.13.39', 'TERMINATE']
  2502. WARNING:toil.leader:Q/J/jobdnGWgD INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '1884', '-host', '172.16.13.39', 'TERMINATE']
  2503. WARNING:toil.leader:Q/J/jobdnGWgD INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '1884', '-host', '172.16.13.39', 'TERMINATE']
  2504. WARNING:toil.leader:Q/J/jobdnGWgD INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '1884', '-host', '172.16.13.39', 'TERMINATE']
  2505. WARNING:toil.leader:Q/J/jobdnGWgD INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '1884', '-host', '172.16.13.39', 'TERMINATE']
  2506. WARNING:toil.leader:Q/J/jobdnGWgD INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '1884', '-host', '172.16.13.39', 'TERMINATE']
  2507. WARNING:toil.leader:Q/J/jobdnGWgD INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '1884', '-host', '172.16.13.39', 'TERMINATE']
  2508. WARNING:toil.leader:Q/J/jobdnGWgD INFO:cactus.shared.common:Running the command ['ktremotemgr', 'set', '-port', '1884', '-host', '172.16.13.39', 'TERMINATE', '1']
  2509. WARNING:toil.leader:Q/J/jobdnGWgD INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '1884', '-host', '172.16.13.39', 'TERMINATE']
  2510. WARNING:toil.leader:Q/J/jobdnGWgD 1
  2511. WARNING:toil.leader:Q/J/jobdnGWgD Traceback (most recent call last):
  2512. WARNING:toil.leader:Q/J/jobdnGWgD File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  2513. WARNING:toil.leader:Q/J/jobdnGWgD job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2514. WARNING:toil.leader:Q/J/jobdnGWgD File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  2515. WARNING:toil.leader:Q/J/jobdnGWgD returnValues = self._run(jobGraph, fileStore)
  2516. WARNING:toil.leader:Q/J/jobdnGWgD File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1694, in _run
  2517. WARNING:toil.leader:Q/J/jobdnGWgD returnValues = self.run(fileStore)
  2518. WARNING:toil.leader:Q/J/jobdnGWgD File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1668, in run
  2519. WARNING:toil.leader:Q/J/jobdnGWgD raise RuntimeError("Detected the error jobStoreID has been removed so exiting with an error")
  2520. WARNING:toil.leader:Q/J/jobdnGWgD RuntimeError: Detected the error jobStoreID has been removed so exiting with an error
  2521. WARNING:toil.leader:Q/J/jobdnGWgD ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn39.grid.pub.ro
  2522. WARNING:toil.leader:Q/J/jobdnGWgD WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' Q/J/jobdnGWgD with ID Q/J/jobdnGWgD to 4
  2523. WARNING:toil.leader:Job 'KtServerService' Q/J/jobdnGWgD with ID Q/J/jobdnGWgD is completely failed
  2524. WARNING:toil.leader:Job: o/H/jobK044oZ is being restarted as a checkpoint after the total failure of jobs in its subtree.
  2525. INFO:toil.leader:Issued job 'StartPrimaryDB' o/H/jobK044oZ with job batch system ID: 196 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  2526. INFO:toil.leader:Job ended successfully: 'StartPrimaryDB' o/H/jobK044oZ
  2527. INFO:toil.leader:Issued job 'KtServerService' j/A/jobFLBkqS with job batch system ID: 197 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  2528. INFO:toil.leader:Issued job 'CactusBarPhase' C/A/jobHZPztk with job batch system ID: 198 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  2529. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' j/A/jobFLBkqS
  2530. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' j/A/jobFLBkqS
  2531. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' j/A/jobFLBkqS with ID j/A/jobFLBkqS to 5
  2532. WARNING:toil.jobGraph:We have increased the default memory of the failed job 'KtServerService' j/A/jobFLBkqS to 32212254720 bytes
  2533. INFO:toil.leader:Issued job 'KtServerService' j/A/jobFLBkqS with job batch system ID: 199 and cores: 0, disk: 2.0 G, and memory: 30.0 G
  2534. INFO:toil.leader:Job ended successfully: 'CactusBarPhase' C/A/jobHZPztk
  2535. INFO:toil.leader:Issued job 'CactusBarRecursion' 5/t/jobdy7gqr with job batch system ID: 200 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  2536. INFO:toil.leader:Job ended successfully: 'CactusBarRecursion' 5/t/jobdy7gqr
  2537. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusBarRecursion' 5/t/jobdy7gqr
  2538. WARNING:toil.leader:5/t/jobdy7gqr INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  2539. WARNING:toil.leader:5/t/jobdy7gqr INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  2540. WARNING:toil.leader:5/t/jobdy7gqr WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  2541. WARNING:toil.leader:5/t/jobdy7gqr INFO:cactus.shared.common:Running the command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.37" port="27564" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000']
  2542. WARNING:toil.leader:5/t/jobdy7gqr Set log level to INFO
  2543. WARNING:toil.leader:5/t/jobdy7gqr Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.37 with error: network error
  2544. WARNING:toil.leader:5/t/jobdy7gqr Uncaught exception
  2545. WARNING:toil.leader:5/t/jobdy7gqr Traceback (most recent call last):
  2546. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  2547. WARNING:toil.leader:5/t/jobdy7gqr job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2548. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  2549. WARNING:toil.leader:5/t/jobdy7gqr super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2550. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  2551. WARNING:toil.leader:5/t/jobdy7gqr returnValues = self._run(jobGraph, fileStore)
  2552. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  2553. WARNING:toil.leader:5/t/jobdy7gqr return self.run(fileStore)
  2554. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 831, in run
  2555. WARNING:toil.leader:5/t/jobdy7gqr self.makeRecursiveJobs(fileStore=fileStore)
  2556. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 417, in makeRecursiveJobs
  2557. WARNING:toil.leader:5/t/jobdy7gqr default=CactusRecursionJob.maxSequenceSizeOfFlowerGroupingDefault))
  2558. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 138, in runCactusGetFlowers
  2559. WARNING:toil.leader:5/t/jobdy7gqr fileStore=fileStore)
  2560. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  2561. WARNING:toil.leader:5/t/jobdy7gqr raise RuntimeError("Command %s failed with output: %s" % (call, output))
  2562. WARNING:toil.leader:5/t/jobdy7gqr RuntimeError: Command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.37" port="27564" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000'] failed with output:
  2563. WARNING:toil.leader:5/t/jobdy7gqr ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn39.grid.pub.ro
  2564. WARNING:toil.leader:5/t/jobdy7gqr WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusBarRecursion' 5/t/jobdy7gqr with ID 5/t/jobdy7gqr to 5
  2565. WARNING:toil.leader:5/t/jobdy7gqr WARNING:toil.jobGraph:We have increased the default memory of the failed job 'CactusBarRecursion' 5/t/jobdy7gqr to 32212254720 bytes
  2566. INFO:toil.leader:Issued job 'CactusBarRecursion' 5/t/jobdy7gqr with job batch system ID: 201 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  2567. INFO:toil.leader:Job ended successfully: 'CactusBarRecursion' 5/t/jobdy7gqr
  2568. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusBarRecursion' 5/t/jobdy7gqr
  2569. WARNING:toil.leader:5/t/jobdy7gqr INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  2570. WARNING:toil.leader:5/t/jobdy7gqr INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  2571. WARNING:toil.leader:5/t/jobdy7gqr WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  2572. WARNING:toil.leader:5/t/jobdy7gqr INFO:cactus.shared.common:Running the command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.37" port="27564" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000']
  2573. WARNING:toil.leader:5/t/jobdy7gqr Set log level to INFO
  2574. WARNING:toil.leader:5/t/jobdy7gqr Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.37 with error: network error
  2575. WARNING:toil.leader:5/t/jobdy7gqr Uncaught exception
  2576. WARNING:toil.leader:5/t/jobdy7gqr Traceback (most recent call last):
  2577. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  2578. WARNING:toil.leader:5/t/jobdy7gqr job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2579. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  2580. WARNING:toil.leader:5/t/jobdy7gqr super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2581. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  2582. WARNING:toil.leader:5/t/jobdy7gqr returnValues = self._run(jobGraph, fileStore)
  2583. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  2584. WARNING:toil.leader:5/t/jobdy7gqr return self.run(fileStore)
  2585. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 831, in run
  2586. WARNING:toil.leader:5/t/jobdy7gqr self.makeRecursiveJobs(fileStore=fileStore)
  2587. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 417, in makeRecursiveJobs
  2588. WARNING:toil.leader:5/t/jobdy7gqr default=CactusRecursionJob.maxSequenceSizeOfFlowerGroupingDefault))
  2589. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 138, in runCactusGetFlowers
  2590. WARNING:toil.leader:5/t/jobdy7gqr fileStore=fileStore)
  2591. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  2592. WARNING:toil.leader:5/t/jobdy7gqr raise RuntimeError("Command %s failed with output: %s" % (call, output))
  2593. WARNING:toil.leader:5/t/jobdy7gqr RuntimeError: Command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.37" port="27564" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000'] failed with output:
  2594. WARNING:toil.leader:5/t/jobdy7gqr ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn39.grid.pub.ro
  2595. WARNING:toil.leader:5/t/jobdy7gqr WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusBarRecursion' 5/t/jobdy7gqr with ID 5/t/jobdy7gqr to 4
  2596. INFO:toil.leader:Issued job 'CactusBarRecursion' 5/t/jobdy7gqr with job batch system ID: 202 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  2597. INFO:toil.leader:Job ended successfully: 'CactusBarRecursion' 5/t/jobdy7gqr
  2598. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusBarRecursion' 5/t/jobdy7gqr
  2599. WARNING:toil.leader:5/t/jobdy7gqr INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  2600. WARNING:toil.leader:5/t/jobdy7gqr INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  2601. WARNING:toil.leader:5/t/jobdy7gqr WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  2602. WARNING:toil.leader:5/t/jobdy7gqr INFO:cactus.shared.common:Running the command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.37" port="27564" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000']
  2603. WARNING:toil.leader:5/t/jobdy7gqr Set log level to INFO
  2604. WARNING:toil.leader:5/t/jobdy7gqr Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.37 with error: network error
  2605. WARNING:toil.leader:5/t/jobdy7gqr Uncaught exception
  2606. WARNING:toil.leader:5/t/jobdy7gqr Traceback (most recent call last):
  2607. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  2608. WARNING:toil.leader:5/t/jobdy7gqr job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2609. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  2610. WARNING:toil.leader:5/t/jobdy7gqr super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2611. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  2612. WARNING:toil.leader:5/t/jobdy7gqr returnValues = self._run(jobGraph, fileStore)
  2613. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  2614. WARNING:toil.leader:5/t/jobdy7gqr return self.run(fileStore)
  2615. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 831, in run
  2616. WARNING:toil.leader:5/t/jobdy7gqr self.makeRecursiveJobs(fileStore=fileStore)
  2617. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 417, in makeRecursiveJobs
  2618. WARNING:toil.leader:5/t/jobdy7gqr default=CactusRecursionJob.maxSequenceSizeOfFlowerGroupingDefault))
  2619. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 138, in runCactusGetFlowers
  2620. WARNING:toil.leader:5/t/jobdy7gqr fileStore=fileStore)
  2621. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  2622. WARNING:toil.leader:5/t/jobdy7gqr raise RuntimeError("Command %s failed with output: %s" % (call, output))
  2623. WARNING:toil.leader:5/t/jobdy7gqr RuntimeError: Command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.37" port="27564" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000'] failed with output:
  2624. WARNING:toil.leader:5/t/jobdy7gqr ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn29.grid.pub.ro
  2625. WARNING:toil.leader:5/t/jobdy7gqr WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusBarRecursion' 5/t/jobdy7gqr with ID 5/t/jobdy7gqr to 3
  2626. INFO:toil.leader:Issued job 'CactusBarRecursion' 5/t/jobdy7gqr with job batch system ID: 203 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  2627. INFO:toil.leader:Job ended successfully: 'CactusBarRecursion' 5/t/jobdy7gqr
  2628. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusBarRecursion' 5/t/jobdy7gqr
  2629. WARNING:toil.leader:5/t/jobdy7gqr INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  2630. WARNING:toil.leader:5/t/jobdy7gqr INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  2631. WARNING:toil.leader:5/t/jobdy7gqr WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  2632. WARNING:toil.leader:5/t/jobdy7gqr INFO:cactus.shared.common:Running the command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.37" port="27564" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000']
  2633. WARNING:toil.leader:5/t/jobdy7gqr Set log level to INFO
  2634. WARNING:toil.leader:5/t/jobdy7gqr Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.37 with error: network error
  2635. WARNING:toil.leader:5/t/jobdy7gqr Uncaught exception
  2636. WARNING:toil.leader:5/t/jobdy7gqr Traceback (most recent call last):
  2637. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  2638. WARNING:toil.leader:5/t/jobdy7gqr job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2639. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  2640. WARNING:toil.leader:5/t/jobdy7gqr super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2641. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  2642. WARNING:toil.leader:5/t/jobdy7gqr returnValues = self._run(jobGraph, fileStore)
  2643. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  2644. WARNING:toil.leader:5/t/jobdy7gqr return self.run(fileStore)
  2645. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 831, in run
  2646. WARNING:toil.leader:5/t/jobdy7gqr self.makeRecursiveJobs(fileStore=fileStore)
  2647. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 417, in makeRecursiveJobs
  2648. WARNING:toil.leader:5/t/jobdy7gqr default=CactusRecursionJob.maxSequenceSizeOfFlowerGroupingDefault))
  2649. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 138, in runCactusGetFlowers
  2650. WARNING:toil.leader:5/t/jobdy7gqr fileStore=fileStore)
  2651. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  2652. WARNING:toil.leader:5/t/jobdy7gqr raise RuntimeError("Command %s failed with output: %s" % (call, output))
  2653. WARNING:toil.leader:5/t/jobdy7gqr RuntimeError: Command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.37" port="27564" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000'] failed with output:
  2654. WARNING:toil.leader:5/t/jobdy7gqr ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn41.grid.pub.ro
  2655. WARNING:toil.leader:5/t/jobdy7gqr WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusBarRecursion' 5/t/jobdy7gqr with ID 5/t/jobdy7gqr to 2
  2656. INFO:toil.leader:Issued job 'CactusBarRecursion' 5/t/jobdy7gqr with job batch system ID: 204 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  2657. INFO:toil.leader:Job ended successfully: 'CactusBarRecursion' 5/t/jobdy7gqr
  2658. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusBarRecursion' 5/t/jobdy7gqr
  2659. WARNING:toil.leader:5/t/jobdy7gqr INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  2660. WARNING:toil.leader:5/t/jobdy7gqr INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  2661. WARNING:toil.leader:5/t/jobdy7gqr WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  2662. WARNING:toil.leader:5/t/jobdy7gqr WARNING:toil.fileStore:Detected that job (o/H/jobK044oZ/g/tmpYapNeo-_serialiseJob-stream) prematurely terminated. Fixing the state of the job on disk.
  2663. WARNING:toil.leader:5/t/jobdy7gqr INFO:cactus.shared.common:Running the command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.37" port="27564" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000']
  2664. WARNING:toil.leader:5/t/jobdy7gqr Set log level to INFO
  2665. WARNING:toil.leader:5/t/jobdy7gqr Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.37 with error: network error
  2666. WARNING:toil.leader:5/t/jobdy7gqr Uncaught exception
  2667. WARNING:toil.leader:5/t/jobdy7gqr Traceback (most recent call last):
  2668. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  2669. WARNING:toil.leader:5/t/jobdy7gqr job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2670. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  2671. WARNING:toil.leader:5/t/jobdy7gqr super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2672. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  2673. WARNING:toil.leader:5/t/jobdy7gqr returnValues = self._run(jobGraph, fileStore)
  2674. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  2675. WARNING:toil.leader:5/t/jobdy7gqr return self.run(fileStore)
  2676. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 831, in run
  2677. WARNING:toil.leader:5/t/jobdy7gqr self.makeRecursiveJobs(fileStore=fileStore)
  2678. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 417, in makeRecursiveJobs
  2679. WARNING:toil.leader:5/t/jobdy7gqr default=CactusRecursionJob.maxSequenceSizeOfFlowerGroupingDefault))
  2680. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 138, in runCactusGetFlowers
  2681. WARNING:toil.leader:5/t/jobdy7gqr fileStore=fileStore)
  2682. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  2683. WARNING:toil.leader:5/t/jobdy7gqr raise RuntimeError("Command %s failed with output: %s" % (call, output))
  2684. WARNING:toil.leader:5/t/jobdy7gqr RuntimeError: Command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.37" port="27564" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000'] failed with output:
  2685. WARNING:toil.leader:5/t/jobdy7gqr ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn37.grid.pub.ro
  2686. WARNING:toil.leader:5/t/jobdy7gqr WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusBarRecursion' 5/t/jobdy7gqr with ID 5/t/jobdy7gqr to 1
  2687. INFO:toil.leader:Issued job 'CactusBarRecursion' 5/t/jobdy7gqr with job batch system ID: 205 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  2688. INFO:toil.leader:Job ended successfully: 'CactusBarRecursion' 5/t/jobdy7gqr
  2689. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusBarRecursion' 5/t/jobdy7gqr
  2690. WARNING:toil.leader:5/t/jobdy7gqr INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  2691. WARNING:toil.leader:5/t/jobdy7gqr INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  2692. WARNING:toil.leader:5/t/jobdy7gqr WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  2693. WARNING:toil.leader:5/t/jobdy7gqr INFO:cactus.shared.common:Running the command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.37" port="27564" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000']
  2694. WARNING:toil.leader:5/t/jobdy7gqr Set log level to INFO
  2695. WARNING:toil.leader:5/t/jobdy7gqr Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.37 with error: network error
  2696. WARNING:toil.leader:5/t/jobdy7gqr Uncaught exception
  2697. WARNING:toil.leader:5/t/jobdy7gqr Traceback (most recent call last):
  2698. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  2699. WARNING:toil.leader:5/t/jobdy7gqr job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2700. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  2701. WARNING:toil.leader:5/t/jobdy7gqr super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2702. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  2703. WARNING:toil.leader:5/t/jobdy7gqr returnValues = self._run(jobGraph, fileStore)
  2704. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  2705. WARNING:toil.leader:5/t/jobdy7gqr return self.run(fileStore)
  2706. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 831, in run
  2707. WARNING:toil.leader:5/t/jobdy7gqr self.makeRecursiveJobs(fileStore=fileStore)
  2708. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 417, in makeRecursiveJobs
  2709. WARNING:toil.leader:5/t/jobdy7gqr default=CactusRecursionJob.maxSequenceSizeOfFlowerGroupingDefault))
  2710. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 138, in runCactusGetFlowers
  2711. WARNING:toil.leader:5/t/jobdy7gqr fileStore=fileStore)
  2712. WARNING:toil.leader:5/t/jobdy7gqr File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  2713. WARNING:toil.leader:5/t/jobdy7gqr raise RuntimeError("Command %s failed with output: %s" % (call, output))
  2714. WARNING:toil.leader:5/t/jobdy7gqr RuntimeError: Command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.37" port="27564" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000'] failed with output:
  2715. WARNING:toil.leader:5/t/jobdy7gqr ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn34.grid.pub.ro
  2716. WARNING:toil.leader:5/t/jobdy7gqr WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusBarRecursion' 5/t/jobdy7gqr with ID 5/t/jobdy7gqr to 0
  2717. WARNING:toil.leader:Job 'CactusBarRecursion' 5/t/jobdy7gqr with ID 5/t/jobdy7gqr is completely failed
  2718. INFO:toil.leader:Job ended successfully: 'KtServerService' j/A/jobFLBkqS
  2719. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'KtServerService' j/A/jobFLBkqS
  2720. WARNING:toil.leader:j/A/jobFLBkqS INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  2721. WARNING:toil.leader:j/A/jobFLBkqS INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  2722. WARNING:toil.leader:j/A/jobFLBkqS WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  2723. WARNING:toil.leader:j/A/jobFLBkqS WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  2724. WARNING:toil.leader:j/A/jobFLBkqS INFO:cactus.shared.common:Running the command ['netstat', '-tuplen']
  2725. WARNING:toil.leader:j/A/jobFLBkqS (No info could be read for "-p": geteuid()=98354 but you should be root.)
  2726. WARNING:toil.leader:j/A/jobFLBkqS INFO:cactus.shared.common:Running the command ['ktserver', '-port', '2511', '-ls', '-tout', '200000', '-th', '64', '-bgs', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpO3AhCs/f1f98f57-0ade-4158-b29c-27e0234f32ea/t59c4fi/snapshot', '-bgsc', 'lzo', '-bgsi', '1000000', '-log', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpO3AhCs/f1f98f57-0ade-4158-b29c-27e0234f32ea/tmpaGQJoZ.tmp', ':#opts=ls#bnum=30m#msiz=50g#ktopts=p']
  2727. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2728. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2729. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2730. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 514
  2731. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2732. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2733. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2734. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 514
  2735. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2736. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2737. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2738. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 515
  2739. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2740. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2741. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2742. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 516
  2743. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2744. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2745. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2746. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 517
  2747. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2748. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2749. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2750. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 518
  2751. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2752. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2753. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2754. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 519
  2755. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2756. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2757. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2758. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 520
  2759. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2760. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2761. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2762. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 521
  2763. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2764. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2765. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2766. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 522
  2767. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2768. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2769. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2770. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 523
  2771. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2772. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2773. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2774. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 524
  2775. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2776. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2777. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2778. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 525
  2779. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2780. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2781. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2782. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 526
  2783. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2784. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2785. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2786. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 527
  2787. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2788. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2789. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2790. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 528
  2791. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2792. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2793. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2794. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 529
  2795. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2796. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2797. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2798. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 530
  2799. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2800. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2801. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2802. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 531
  2803. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2804. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2805. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2806. WARNING:toil.leader:j/A/jobFLBkqS INFO:toil.lib.bioio:Ktserver running.
  2807. WARNING:toil.leader:j/A/jobFLBkqS INFO:toil.lib.bioio:Ktserver running.
  2808. WARNING:toil.leader:j/A/jobFLBkqS INFO:toil.lib.bioio:Ktserver running.
  2809. WARNING:toil.leader:j/A/jobFLBkqS INFO:cactus.shared.common:Running the command ['ktremotemgr', 'remove', '-port', '2511', '-host', '172.16.13.38', 'TERMINATE']
  2810. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 532
  2811. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2812. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2813. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2814. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 534
  2815. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2816. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2817. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2818. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 535
  2819. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2820. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2821. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2822. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 536
  2823. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2824. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2825. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2826. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 537
  2827. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2828. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2829. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2830. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 538
  2831. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2832. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2833. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2834. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 539
  2835. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2836. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2837. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2838. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 540
  2839. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2840. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2841. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2842. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 541
  2843. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2844. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2845. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2846. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 542
  2847. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2848. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2849. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2850. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 543
  2851. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2852. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2853. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2854. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 544
  2855. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2856. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2857. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2858. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 545
  2859. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2860. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2861. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2862. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 546
  2863. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2864. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2865. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2866. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 547
  2867. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2868. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2869. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2870. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 548
  2871. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2872. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2873. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2874. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 549
  2875. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2876. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2877. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2878. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 550
  2879. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2880. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2881. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2882. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 551
  2883. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2884. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2885. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2886. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 552
  2887. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2888. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2889. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2890. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 553
  2891. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2892. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2893. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2894. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 554
  2895. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2896. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2897. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2898. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 555
  2899. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2900. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2901. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2902. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 556
  2903. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2904. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2905. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2906. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 557
  2907. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2908. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2909. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2910. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 558
  2911. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2912. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2913. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2914. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 559
  2915. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2916. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2917. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2918. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 560
  2919. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2920. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2921. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2922. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 561
  2923. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2924. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2925. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2926. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 562
  2927. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2928. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2929. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2930. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 563
  2931. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2932. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2933. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2934. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 564
  2935. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2936. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2937. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2938. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 565
  2939. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2940. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2941. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2942. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 566
  2943. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2944. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2945. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2946. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 567
  2947. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2948. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2949. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2950. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 568
  2951. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2952. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2953. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2954. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 569
  2955. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2956. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2957. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2958. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 570
  2959. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2960. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2961. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2962. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 571
  2963. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2964. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2965. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2966. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 572
  2967. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2968. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2969. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2970. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 573
  2971. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2972. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2973. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2974. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 574
  2975. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2976. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2977. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2978. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 575
  2979. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2980. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2981. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2982. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 576
  2983. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2984. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2985. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2986. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 577
  2987. WARNING:toil.leader:j/A/jobFLBkqS Stack limits: Infinity Infinity
  2988. WARNING:toil.leader:j/A/jobFLBkqS Process limits: 510853 510853
  2989. WARNING:toil.leader:j/A/jobFLBkqS Kernel thread cap: 1021706
  2990. WARNING:toil.leader:j/A/jobFLBkqS Current total threads running: 578
  2991. WARNING:toil.leader:j/A/jobFLBkqS INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '2511', '-host', '172.16.13.38', 'TERMINATE']
  2992. WARNING:toil.leader:j/A/jobFLBkqS INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '2511', '-host', '172.16.13.38', 'TERMINATE']
  2993. WARNING:toil.leader:j/A/jobFLBkqS INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '2511', '-host', '172.16.13.38', 'TERMINATE']
  2994. WARNING:toil.leader:j/A/jobFLBkqS INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '2511', '-host', '172.16.13.38', 'TERMINATE']
  2995. WARNING:toil.leader:j/A/jobFLBkqS INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '2511', '-host', '172.16.13.38', 'TERMINATE']
  2996. WARNING:toil.leader:j/A/jobFLBkqS INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '2511', '-host', '172.16.13.38', 'TERMINATE']
  2997. WARNING:toil.leader:j/A/jobFLBkqS INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '2511', '-host', '172.16.13.38', 'TERMINATE']
  2998. WARNING:toil.leader:j/A/jobFLBkqS INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '2511', '-host', '172.16.13.38', 'TERMINATE']
  2999. WARNING:toil.leader:j/A/jobFLBkqS INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '2511', '-host', '172.16.13.38', 'TERMINATE']
  3000. WARNING:toil.leader:j/A/jobFLBkqS INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '2511', '-host', '172.16.13.38', 'TERMINATE']
  3001. WARNING:toil.leader:j/A/jobFLBkqS INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '2511', '-host', '172.16.13.38', 'TERMINATE']
  3002. WARNING:toil.leader:j/A/jobFLBkqS INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '2511', '-host', '172.16.13.38', 'TERMINATE']
  3003. WARNING:toil.leader:j/A/jobFLBkqS INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '2511', '-host', '172.16.13.38', 'TERMINATE']
  3004. WARNING:toil.leader:j/A/jobFLBkqS INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '2511', '-host', '172.16.13.38', 'TERMINATE']
  3005. WARNING:toil.leader:j/A/jobFLBkqS INFO:cactus.shared.common:Running the command ['ktremotemgr', 'set', '-port', '2511', '-host', '172.16.13.38', 'TERMINATE', '1']
  3006. WARNING:toil.leader:j/A/jobFLBkqS INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '2511', '-host', '172.16.13.38', 'TERMINATE']
  3007. WARNING:toil.leader:j/A/jobFLBkqS 1
  3008. WARNING:toil.leader:j/A/jobFLBkqS Traceback (most recent call last):
  3009. WARNING:toil.leader:j/A/jobFLBkqS File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  3010. WARNING:toil.leader:j/A/jobFLBkqS job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  3011. WARNING:toil.leader:j/A/jobFLBkqS File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  3012. WARNING:toil.leader:j/A/jobFLBkqS returnValues = self._run(jobGraph, fileStore)
  3013. WARNING:toil.leader:j/A/jobFLBkqS File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1694, in _run
  3014. WARNING:toil.leader:j/A/jobFLBkqS returnValues = self.run(fileStore)
  3015. WARNING:toil.leader:j/A/jobFLBkqS File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1668, in run
  3016. WARNING:toil.leader:j/A/jobFLBkqS raise RuntimeError("Detected the error jobStoreID has been removed so exiting with an error")
  3017. WARNING:toil.leader:j/A/jobFLBkqS RuntimeError: Detected the error jobStoreID has been removed so exiting with an error
  3018. WARNING:toil.leader:j/A/jobFLBkqS ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn38.grid.pub.ro
  3019. WARNING:toil.leader:j/A/jobFLBkqS WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' j/A/jobFLBkqS with ID j/A/jobFLBkqS to 4
  3020. WARNING:toil.leader:Job 'KtServerService' j/A/jobFLBkqS with ID j/A/jobFLBkqS is completely failed
  3021. WARNING:toil.leader:Job: o/H/jobK044oZ is being restarted as a checkpoint after the total failure of jobs in its subtree.
  3022. INFO:toil.leader:Issued job 'StartPrimaryDB' o/H/jobK044oZ with job batch system ID: 206 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  3023. INFO:toil.leader:Job ended successfully: 'StartPrimaryDB' o/H/jobK044oZ
  3024. INFO:toil.leader:Issued job 'KtServerService' N/g/job9uS5DS with job batch system ID: 207 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  3025. INFO:toil.leader:Issued job 'CactusBarPhase' C/J/jobXK2yjT with job batch system ID: 208 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  3026. INFO:toil.leader:Job ended successfully: 'CactusBarPhase' C/J/jobXK2yjT
  3027. INFO:toil.leader:Issued job 'CactusBarRecursion' 2/R/jobsDixkf with job batch system ID: 209 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  3028. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' N/g/job9uS5DS
  3029. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' N/g/job9uS5DS
  3030. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' N/g/job9uS5DS with ID N/g/job9uS5DS to 5
  3031. WARNING:toil.jobGraph:We have increased the default memory of the failed job 'KtServerService' N/g/job9uS5DS to 32212254720 bytes
  3032. INFO:toil.leader:Issued job 'KtServerService' N/g/job9uS5DS with job batch system ID: 210 and cores: 0, disk: 2.0 G, and memory: 30.0 G
  3033. INFO:toil.leader:Job ended successfully: 'CactusBarRecursion' 2/R/jobsDixkf
  3034. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusBarRecursion' 2/R/jobsDixkf
  3035. WARNING:toil.leader:2/R/jobsDixkf INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  3036. WARNING:toil.leader:2/R/jobsDixkf INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  3037. WARNING:toil.leader:2/R/jobsDixkf WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  3038. WARNING:toil.leader:2/R/jobsDixkf INFO:cactus.shared.common:Running the command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.39" port="24584" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000']
  3039. WARNING:toil.leader:2/R/jobsDixkf Set log level to INFO
  3040. WARNING:toil.leader:2/R/jobsDixkf Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.39 with error: network error
  3041. WARNING:toil.leader:2/R/jobsDixkf Uncaught exception
  3042. WARNING:toil.leader:2/R/jobsDixkf Traceback (most recent call last):
  3043. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  3044. WARNING:toil.leader:2/R/jobsDixkf job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  3045. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  3046. WARNING:toil.leader:2/R/jobsDixkf super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  3047. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  3048. WARNING:toil.leader:2/R/jobsDixkf returnValues = self._run(jobGraph, fileStore)
  3049. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  3050. WARNING:toil.leader:2/R/jobsDixkf return self.run(fileStore)
  3051. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 831, in run
  3052. WARNING:toil.leader:2/R/jobsDixkf self.makeRecursiveJobs(fileStore=fileStore)
  3053. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 417, in makeRecursiveJobs
  3054. WARNING:toil.leader:2/R/jobsDixkf default=CactusRecursionJob.maxSequenceSizeOfFlowerGroupingDefault))
  3055. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 138, in runCactusGetFlowers
  3056. WARNING:toil.leader:2/R/jobsDixkf fileStore=fileStore)
  3057. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  3058. WARNING:toil.leader:2/R/jobsDixkf raise RuntimeError("Command %s failed with output: %s" % (call, output))
  3059. WARNING:toil.leader:2/R/jobsDixkf RuntimeError: Command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.39" port="24584" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000'] failed with output:
  3060. WARNING:toil.leader:2/R/jobsDixkf ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn38.grid.pub.ro
  3061. WARNING:toil.leader:2/R/jobsDixkf WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusBarRecursion' 2/R/jobsDixkf with ID 2/R/jobsDixkf to 5
  3062. WARNING:toil.leader:2/R/jobsDixkf WARNING:toil.jobGraph:We have increased the default memory of the failed job 'CactusBarRecursion' 2/R/jobsDixkf to 32212254720 bytes
  3063. INFO:toil.leader:Issued job 'CactusBarRecursion' 2/R/jobsDixkf with job batch system ID: 211 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  3064. INFO:toil.leader:Job ended successfully: 'CactusBarRecursion' 2/R/jobsDixkf
  3065. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusBarRecursion' 2/R/jobsDixkf
  3066. WARNING:toil.leader:2/R/jobsDixkf INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  3067. WARNING:toil.leader:2/R/jobsDixkf INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  3068. WARNING:toil.leader:2/R/jobsDixkf WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  3069. WARNING:toil.leader:2/R/jobsDixkf INFO:cactus.shared.common:Running the command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.39" port="24584" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000']
  3070. WARNING:toil.leader:2/R/jobsDixkf Set log level to INFO
  3071. WARNING:toil.leader:2/R/jobsDixkf Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.39 with error: network error
  3072. WARNING:toil.leader:2/R/jobsDixkf Uncaught exception
  3073. WARNING:toil.leader:2/R/jobsDixkf Traceback (most recent call last):
  3074. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  3075. WARNING:toil.leader:2/R/jobsDixkf job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  3076. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  3077. WARNING:toil.leader:2/R/jobsDixkf super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  3078. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  3079. WARNING:toil.leader:2/R/jobsDixkf returnValues = self._run(jobGraph, fileStore)
  3080. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  3081. WARNING:toil.leader:2/R/jobsDixkf return self.run(fileStore)
  3082. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 831, in run
  3083. WARNING:toil.leader:2/R/jobsDixkf self.makeRecursiveJobs(fileStore=fileStore)
  3084. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 417, in makeRecursiveJobs
  3085. WARNING:toil.leader:2/R/jobsDixkf default=CactusRecursionJob.maxSequenceSizeOfFlowerGroupingDefault))
  3086. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 138, in runCactusGetFlowers
  3087. WARNING:toil.leader:2/R/jobsDixkf fileStore=fileStore)
  3088. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  3089. WARNING:toil.leader:2/R/jobsDixkf raise RuntimeError("Command %s failed with output: %s" % (call, output))
  3090. WARNING:toil.leader:2/R/jobsDixkf RuntimeError: Command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.39" port="24584" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000'] failed with output:
  3091. WARNING:toil.leader:2/R/jobsDixkf ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn35.grid.pub.ro
  3092. WARNING:toil.leader:2/R/jobsDixkf WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusBarRecursion' 2/R/jobsDixkf with ID 2/R/jobsDixkf to 4
  3093. INFO:toil.leader:Issued job 'CactusBarRecursion' 2/R/jobsDixkf with job batch system ID: 212 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  3094. INFO:toil.leader:Job ended successfully: 'CactusBarRecursion' 2/R/jobsDixkf
  3095. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusBarRecursion' 2/R/jobsDixkf
  3096. WARNING:toil.leader:2/R/jobsDixkf INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  3097. WARNING:toil.leader:2/R/jobsDixkf INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  3098. WARNING:toil.leader:2/R/jobsDixkf WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  3099. WARNING:toil.leader:2/R/jobsDixkf INFO:cactus.shared.common:Running the command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.39" port="24584" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000']
  3100. WARNING:toil.leader:2/R/jobsDixkf Set log level to INFO
  3101. WARNING:toil.leader:2/R/jobsDixkf Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.39 with error: network error
  3102. WARNING:toil.leader:2/R/jobsDixkf Uncaught exception
  3103. WARNING:toil.leader:2/R/jobsDixkf Traceback (most recent call last):
  3104. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  3105. WARNING:toil.leader:2/R/jobsDixkf job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  3106. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  3107. WARNING:toil.leader:2/R/jobsDixkf super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  3108. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  3109. WARNING:toil.leader:2/R/jobsDixkf returnValues = self._run(jobGraph, fileStore)
  3110. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  3111. WARNING:toil.leader:2/R/jobsDixkf return self.run(fileStore)
  3112. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 831, in run
  3113. WARNING:toil.leader:2/R/jobsDixkf self.makeRecursiveJobs(fileStore=fileStore)
  3114. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 417, in makeRecursiveJobs
  3115. WARNING:toil.leader:2/R/jobsDixkf default=CactusRecursionJob.maxSequenceSizeOfFlowerGroupingDefault))
  3116. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 138, in runCactusGetFlowers
  3117. WARNING:toil.leader:2/R/jobsDixkf fileStore=fileStore)
  3118. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  3119. WARNING:toil.leader:2/R/jobsDixkf raise RuntimeError("Command %s failed with output: %s" % (call, output))
  3120. WARNING:toil.leader:2/R/jobsDixkf RuntimeError: Command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.39" port="24584" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000'] failed with output:
  3121. WARNING:toil.leader:2/R/jobsDixkf ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn40.grid.pub.ro
  3122. WARNING:toil.leader:2/R/jobsDixkf WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusBarRecursion' 2/R/jobsDixkf with ID 2/R/jobsDixkf to 3
  3123. INFO:toil.leader:Issued job 'CactusBarRecursion' 2/R/jobsDixkf with job batch system ID: 213 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  3124. INFO:toil.leader:Job ended successfully: 'CactusBarRecursion' 2/R/jobsDixkf
  3125. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusBarRecursion' 2/R/jobsDixkf
  3126. WARNING:toil.leader:2/R/jobsDixkf INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  3127. WARNING:toil.leader:2/R/jobsDixkf INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  3128. WARNING:toil.leader:2/R/jobsDixkf WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  3129. WARNING:toil.leader:2/R/jobsDixkf WARNING:toil.fileStore:Detected that job (o/H/jobK044oZ/g/tmpoYUmMV-_serialiseJob-stream) prematurely terminated. Fixing the state of the job on disk.
  3130. WARNING:toil.leader:2/R/jobsDixkf INFO:cactus.shared.common:Running the command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.39" port="24584" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000']
  3131. WARNING:toil.leader:2/R/jobsDixkf Set log level to INFO
  3132. WARNING:toil.leader:2/R/jobsDixkf Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.39 with error: network error
  3133. WARNING:toil.leader:2/R/jobsDixkf Uncaught exception
  3134. WARNING:toil.leader:2/R/jobsDixkf Traceback (most recent call last):
  3135. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  3136. WARNING:toil.leader:2/R/jobsDixkf job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  3137. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  3138. WARNING:toil.leader:2/R/jobsDixkf super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  3139. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  3140. WARNING:toil.leader:2/R/jobsDixkf returnValues = self._run(jobGraph, fileStore)
  3141. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  3142. WARNING:toil.leader:2/R/jobsDixkf return self.run(fileStore)
  3143. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 831, in run
  3144. WARNING:toil.leader:2/R/jobsDixkf self.makeRecursiveJobs(fileStore=fileStore)
  3145. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 417, in makeRecursiveJobs
  3146. WARNING:toil.leader:2/R/jobsDixkf default=CactusRecursionJob.maxSequenceSizeOfFlowerGroupingDefault))
  3147. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 138, in runCactusGetFlowers
  3148. WARNING:toil.leader:2/R/jobsDixkf fileStore=fileStore)
  3149. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  3150. WARNING:toil.leader:2/R/jobsDixkf raise RuntimeError("Command %s failed with output: %s" % (call, output))
  3151. WARNING:toil.leader:2/R/jobsDixkf RuntimeError: Command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.39" port="24584" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000'] failed with output:
  3152. WARNING:toil.leader:2/R/jobsDixkf ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn39.grid.pub.ro
  3153. WARNING:toil.leader:2/R/jobsDixkf WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusBarRecursion' 2/R/jobsDixkf with ID 2/R/jobsDixkf to 2
  3154. INFO:toil.leader:Issued job 'CactusBarRecursion' 2/R/jobsDixkf with job batch system ID: 214 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  3155. INFO:toil.leader:Job ended successfully: 'CactusBarRecursion' 2/R/jobsDixkf
  3156. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusBarRecursion' 2/R/jobsDixkf
  3157. WARNING:toil.leader:2/R/jobsDixkf INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  3158. WARNING:toil.leader:2/R/jobsDixkf INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  3159. WARNING:toil.leader:2/R/jobsDixkf WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  3160. WARNING:toil.leader:2/R/jobsDixkf INFO:cactus.shared.common:Running the command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.39" port="24584" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000']
  3161. WARNING:toil.leader:2/R/jobsDixkf Set log level to INFO
  3162. WARNING:toil.leader:2/R/jobsDixkf Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.39 with error: network error
  3163. WARNING:toil.leader:2/R/jobsDixkf Uncaught exception
  3164. WARNING:toil.leader:2/R/jobsDixkf Traceback (most recent call last):
  3165. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  3166. WARNING:toil.leader:2/R/jobsDixkf job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  3167. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  3168. WARNING:toil.leader:2/R/jobsDixkf super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  3169. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  3170. WARNING:toil.leader:2/R/jobsDixkf returnValues = self._run(jobGraph, fileStore)
  3171. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  3172. WARNING:toil.leader:2/R/jobsDixkf return self.run(fileStore)
  3173. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 831, in run
  3174. WARNING:toil.leader:2/R/jobsDixkf self.makeRecursiveJobs(fileStore=fileStore)
  3175. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 417, in makeRecursiveJobs
  3176. WARNING:toil.leader:2/R/jobsDixkf default=CactusRecursionJob.maxSequenceSizeOfFlowerGroupingDefault))
  3177. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 138, in runCactusGetFlowers
  3178. WARNING:toil.leader:2/R/jobsDixkf fileStore=fileStore)
  3179. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  3180. WARNING:toil.leader:2/R/jobsDixkf raise RuntimeError("Command %s failed with output: %s" % (call, output))
  3181. WARNING:toil.leader:2/R/jobsDixkf RuntimeError: Command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.39" port="24584" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000'] failed with output:
  3182. WARNING:toil.leader:2/R/jobsDixkf ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn37.grid.pub.ro
  3183. WARNING:toil.leader:2/R/jobsDixkf WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusBarRecursion' 2/R/jobsDixkf with ID 2/R/jobsDixkf to 1
  3184. INFO:toil.leader:Issued job 'CactusBarRecursion' 2/R/jobsDixkf with job batch system ID: 215 and cores: 1, disk: 2.0 G, and memory: 30.0 G
  3185. INFO:toil.leader:Job ended successfully: 'CactusBarRecursion' 2/R/jobsDixkf
  3186. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusBarRecursion' 2/R/jobsDixkf
  3187. WARNING:toil.leader:2/R/jobsDixkf INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  3188. WARNING:toil.leader:2/R/jobsDixkf INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  3189. WARNING:toil.leader:2/R/jobsDixkf WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  3190. WARNING:toil.leader:2/R/jobsDixkf INFO:cactus.shared.common:Running the command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.39" port="24584" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000']
  3191. WARNING:toil.leader:2/R/jobsDixkf Set log level to INFO
  3192. WARNING:toil.leader:2/R/jobsDixkf Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.39 with error: network error
  3193. WARNING:toil.leader:2/R/jobsDixkf Uncaught exception
  3194. WARNING:toil.leader:2/R/jobsDixkf Traceback (most recent call last):
  3195. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  3196. WARNING:toil.leader:2/R/jobsDixkf job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  3197. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  3198. WARNING:toil.leader:2/R/jobsDixkf super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  3199. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  3200. WARNING:toil.leader:2/R/jobsDixkf returnValues = self._run(jobGraph, fileStore)
  3201. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  3202. WARNING:toil.leader:2/R/jobsDixkf return self.run(fileStore)
  3203. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 831, in run
  3204. WARNING:toil.leader:2/R/jobsDixkf self.makeRecursiveJobs(fileStore=fileStore)
  3205. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 417, in makeRecursiveJobs
  3206. WARNING:toil.leader:2/R/jobsDixkf default=CactusRecursionJob.maxSequenceSizeOfFlowerGroupingDefault))
  3207. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 138, in runCactusGetFlowers
  3208. WARNING:toil.leader:2/R/jobsDixkf fileStore=fileStore)
  3209. WARNING:toil.leader:2/R/jobsDixkf File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  3210. WARNING:toil.leader:2/R/jobsDixkf raise RuntimeError("Command %s failed with output: %s" % (call, output))
  3211. WARNING:toil.leader:2/R/jobsDixkf RuntimeError: Command ['cactus_workflow_getFlowers', 'INFO', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.39" port="24584" />\n\t\t</st_kv_database_conf>\n\t', '0', '100000000', '1000000'] failed with output:
  3212. WARNING:toil.leader:2/R/jobsDixkf ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn41.grid.pub.ro
  3213. WARNING:toil.leader:2/R/jobsDixkf WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusBarRecursion' 2/R/jobsDixkf with ID 2/R/jobsDixkf to 0
  3214. WARNING:toil.leader:Job 'CactusBarRecursion' 2/R/jobsDixkf with ID 2/R/jobsDixkf is completely failed
  3215. INFO:toil.leader:Job ended successfully: 'KtServerService' N/g/job9uS5DS
  3216. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'KtServerService' N/g/job9uS5DS
  3217. WARNING:toil.leader:N/g/job9uS5DS INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  3218. WARNING:toil.leader:N/g/job9uS5DS INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  3219. WARNING:toil.leader:N/g/job9uS5DS WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  3220. WARNING:toil.leader:N/g/job9uS5DS WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  3221. WARNING:toil.leader:N/g/job9uS5DS INFO:cactus.shared.common:Running the command ['netstat', '-tuplen']
  3222. WARNING:toil.leader:N/g/job9uS5DS (No info could be read for "-p": geteuid()=98354 but you should be root.)
  3223. WARNING:toil.leader:N/g/job9uS5DS INFO:cactus.shared.common:Running the command ['ktserver', '-port', '18263', '-ls', '-tout', '200000', '-th', '64', '-bgs', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-906375ca-4612-4b9c-aace-ad64276eee7f/tmpk9C_Wl/be03937f-2dd2-4ab1-8a0c-1e2f0bd428e1/tLls7dz/snapshot', '-bgsc', 'lzo', '-bgsi', '1000000', '-log', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test-yeasts/cactusYeastsTemp/toil-41b954e7-f42f-4e6a-bb56-685f856e252e-906375ca-4612-4b9c-aace-ad64276eee7f/tmpk9C_Wl/be03937f-2dd2-4ab1-8a0c-1e2f0bd428e1/tmpxmX28I.tmp', ':#opts=ls#bnum=30m#msiz=50g#ktopts=p']
  3224. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3225. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3226. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3227. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 555
  3228. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3229. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3230. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3231. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 555
  3232. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3233. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3234. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3235. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 556
  3236. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3237. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3238. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3239. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 557
  3240. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3241. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3242. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3243. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 558
  3244. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3245. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3246. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3247. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 559
  3248. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3249. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3250. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3251. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 560
  3252. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3253. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3254. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3255. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 561
  3256. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3257. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3258. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3259. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 562
  3260. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3261. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3262. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3263. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 563
  3264. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3265. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3266. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3267. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 564
  3268. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3269. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3270. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3271. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 565
  3272. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3273. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3274. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3275. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 566
  3276. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3277. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3278. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3279. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 567
  3280. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3281. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3282. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3283. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 568
  3284. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3285. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3286. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3287. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 569
  3288. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3289. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3290. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3291. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 570
  3292. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3293. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3294. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3295. WARNING:toil.leader:N/g/job9uS5DS INFO:toil.lib.bioio:Ktserver running.
  3296. WARNING:toil.leader:N/g/job9uS5DS INFO:toil.lib.bioio:Ktserver running.
  3297. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 571
  3298. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3299. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3300. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3301. WARNING:toil.leader:N/g/job9uS5DS INFO:toil.lib.bioio:Ktserver running.
  3302. WARNING:toil.leader:N/g/job9uS5DS INFO:cactus.shared.common:Running the command ['ktremotemgr', 'remove', '-port', '18263', '-host', '172.16.13.34', 'TERMINATE']
  3303. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 573
  3304. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3305. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3306. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3307. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 574
  3308. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3309. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3310. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3311. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 575
  3312. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3313. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3314. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3315. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 576
  3316. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3317. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3318. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3319. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 577
  3320. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3321. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3322. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3323. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 578
  3324. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3325. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3326. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3327. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 579
  3328. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3329. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3330. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3331. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 580
  3332. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3333. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3334. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3335. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 581
  3336. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3337. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3338. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3339. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 582
  3340. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3341. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3342. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3343. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 583
  3344. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3345. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3346. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3347. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 584
  3348. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3349. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3350. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3351. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 585
  3352. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3353. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3354. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3355. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 586
  3356. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3357. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3358. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3359. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 587
  3360. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3361. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3362. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3363. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 588
  3364. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3365. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3366. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3367. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 589
  3368. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3369. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3370. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3371. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 590
  3372. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3373. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3374. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3375. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 591
  3376. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3377. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3378. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3379. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 592
  3380. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3381. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3382. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3383. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 593
  3384. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3385. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3386. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3387. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 594
  3388. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3389. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3390. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3391. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 595
  3392. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3393. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3394. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3395. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 596
  3396. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3397. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3398. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3399. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 597
  3400. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3401. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3402. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3403. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 598
  3404. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3405. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3406. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3407. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 599
  3408. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3409. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3410. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3411. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 600
  3412. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3413. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3414. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3415. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 601
  3416. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3417. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3418. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3419. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 602
  3420. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3421. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3422. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3423. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 603
  3424. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3425. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3426. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3427. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 604
  3428. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3429. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3430. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3431. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 605
  3432. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3433. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3434. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3435. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 606
  3436. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3437. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3438. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3439. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 607
  3440. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3441. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3442. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3443. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 608
  3444. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3445. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3446. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3447. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 609
  3448. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3449. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3450. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3451. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 610
  3452. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3453. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3454. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3455. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 611
  3456. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3457. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3458. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3459. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 612
  3460. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3461. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3462. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3463. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 613
  3464. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3465. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3466. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3467. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 614
  3468. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3469. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3470. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3471. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 615
  3472. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3473. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3474. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3475. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 616
  3476. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3477. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3478. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3479. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 617
  3480. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3481. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3482. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3483. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 618
  3484. WARNING:toil.leader:N/g/job9uS5DS Stack limits: Infinity Infinity
  3485. WARNING:toil.leader:N/g/job9uS5DS Process limits: 511096 511096
  3486. WARNING:toil.leader:N/g/job9uS5DS Kernel thread cap: 1022192
  3487. WARNING:toil.leader:N/g/job9uS5DS Current total threads running: 619
  3488. WARNING:toil.leader:N/g/job9uS5DS INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '18263', '-host', '172.16.13.34', 'TERMINATE']
  3489. WARNING:toil.leader:N/g/job9uS5DS INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '18263', '-host', '172.16.13.34', 'TERMINATE']
  3490. WARNING:toil.leader:N/g/job9uS5DS INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '18263', '-host', '172.16.13.34', 'TERMINATE']
  3491. WARNING:toil.leader:N/g/job9uS5DS INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '18263', '-host', '172.16.13.34', 'TERMINATE']
  3492. WARNING:toil.leader:N/g/job9uS5DS INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '18263', '-host', '172.16.13.34', 'TERMINATE']
  3493. WARNING:toil.leader:N/g/job9uS5DS INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '18263', '-host', '172.16.13.34', 'TERMINATE']
  3494. WARNING:toil.leader:N/g/job9uS5DS INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '18263', '-host', '172.16.13.34', 'TERMINATE']
  3495. WARNING:toil.leader:N/g/job9uS5DS INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '18263', '-host', '172.16.13.34', 'TERMINATE']
  3496. WARNING:toil.leader:N/g/job9uS5DS INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '18263', '-host', '172.16.13.34', 'TERMINATE']
  3497. WARNING:toil.leader:N/g/job9uS5DS INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '18263', '-host', '172.16.13.34', 'TERMINATE']
  3498. WARNING:toil.leader:N/g/job9uS5DS INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '18263', '-host', '172.16.13.34', 'TERMINATE']
  3499. WARNING:toil.leader:N/g/job9uS5DS INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '18263', '-host', '172.16.13.34', 'TERMINATE']
  3500. WARNING:toil.leader:N/g/job9uS5DS INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '18263', '-host', '172.16.13.34', 'TERMINATE']
  3501. WARNING:toil.leader:N/g/job9uS5DS INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '18263', '-host', '172.16.13.34', 'TERMINATE']
  3502. WARNING:toil.leader:N/g/job9uS5DS INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '18263', '-host', '172.16.13.34', 'TERMINATE']
  3503. WARNING:toil.leader:N/g/job9uS5DS INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '18263', '-host', '172.16.13.34', 'TERMINATE']
  3504. WARNING:toil.leader:N/g/job9uS5DS INFO:cactus.shared.common:Running the command ['ktremotemgr', 'set', '-port', '18263', '-host', '172.16.13.34', 'TERMINATE', '1']
  3505. WARNING:toil.leader:N/g/job9uS5DS INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '18263', '-host', '172.16.13.34', 'TERMINATE']
  3506. WARNING:toil.leader:N/g/job9uS5DS 1
  3507. WARNING:toil.leader:N/g/job9uS5DS Traceback (most recent call last):
  3508. WARNING:toil.leader:N/g/job9uS5DS File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  3509. WARNING:toil.leader:N/g/job9uS5DS job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  3510. WARNING:toil.leader:N/g/job9uS5DS File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  3511. WARNING:toil.leader:N/g/job9uS5DS returnValues = self._run(jobGraph, fileStore)
  3512. WARNING:toil.leader:N/g/job9uS5DS File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1694, in _run
  3513. WARNING:toil.leader:N/g/job9uS5DS returnValues = self.run(fileStore)
  3514. WARNING:toil.leader:N/g/job9uS5DS File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1668, in run
  3515. WARNING:toil.leader:N/g/job9uS5DS raise RuntimeError("Detected the error jobStoreID has been removed so exiting with an error")
  3516. WARNING:toil.leader:N/g/job9uS5DS RuntimeError: Detected the error jobStoreID has been removed so exiting with an error
  3517. WARNING:toil.leader:N/g/job9uS5DS ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn34.grid.pub.ro
  3518. WARNING:toil.leader:N/g/job9uS5DS WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' N/g/job9uS5DS with ID N/g/job9uS5DS to 4
  3519. WARNING:toil.leader:Job 'KtServerService' N/g/job9uS5DS with ID N/g/job9uS5DS is completely failed
  3520. INFO:toil.leader:Finished toil run with 12 failed jobs.
  3521. INFO:toil.leader:Failed jobs at end of the run: 'RunCactusPreprocessorThenProgressiveDown' S/o/job51DlDr 'CactusBarRecursion' 2/R/jobsDixkf 'ProgressiveNext' P/3/jobUyelVB 'CactusBarPhase' C/J/jobXK2yjT 'KtServerService' N/g/job9uS5DS 'CactusSetupCheckpoint' y/U/jobCVFeXz 'ProgressiveUp' x/L/jobMNjZYG 'CactusTrimmingBlastPhase' u/S/jobhfwRMO 'StartPrimaryDB' o/H/jobK044oZ 'ProgressiveDown' g/l/jobFGVVE1 'CactusBarCheckpoint' J/O/jobgSECJU 'RunCactusPreprocessorThenProgressiveDown2' t/h/jobMDYFld
  3522. Traceback (most recent call last):
  3523. File "/export/home/ncit/external/a.mizeranschi/toil_conda/bin/cactus", line 11, in <module>
  3524. sys.exit(main())
  3525. File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/progressive/cactus_progressive.py", line 520, in main
  3526. halID = toil.start(RunCactusPreprocessorThenProgressiveDown(options, project, memory=configWrapper.getDefaultMemory()))
  3527. File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/common.py", line 784, in start
  3528. return self._runMainLoop(rootJobGraph)
  3529. File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/common.py", line 1059, in _runMainLoop
  3530. jobCache=self._jobCache).run()
  3531. File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/leader.py", line 237, in run
  3532. raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore)
  3533. toil.leader.FailedJobsException
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!