API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Mar 18th, 2019
127
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 460.18 KB | None | 0 0
raw download clone embed print report
  1. INFO:cactus.progressive.projectWrapper:Using config from path /export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/cactus_progressive_config.xml.
  2. INFO:toil.lib.bioio:xmlRoot = <multi_cactus inputSequences="simMouse.chr6 simRat.chr6 simHuman.chr6 simDog.chr6 simCow.chr6">
  3. <tree>(simMouse_chr6:0.084509,simRat_chr6:0.091589,simHuman_chr6:0.415992,simDog_chr6:0.488495,simCow_chr6:0.514545)mr;</tree>
  4. <cactus experiment_path="/scratch/tmp/1067775.1.campus-haswell.q/tmppDMoKf/progressiveAlignment/mr/mr_experiment.xml" name="mr" />
  5. </multi_cactus>
  6. WARNING:toil.batchSystems.singleMachine:Limiting maxMemory to physically available memory (134158856192).
  7. WARNING:toil.batchSystems.singleMachine:Limiting maxDisk to physically available disk (2466665988096).
  8. WARNING:toil.common:Batch system does not support auto-deployment. The user script ModuleDescriptor(dirPath='/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages', name='cactus.progressive.cactus_progressive', fromVirtualEnv=False) will have to be present at the same location on every worker.
  9. INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  10. INFO:toil.leader:Issued job 'RunCactusPreprocessorThenProgressiveDown' n/R/jobJ7SRhl with job batch system ID: 0 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  11. INFO:toil.statsAndLogging:Got message from job at time 03-16-2019 17:42:21: Using the following configuration:
  12. <cactusWorkflowConfig distanceToAddToRootAlignment="0.1">
  13. <constants defaultCpu="1" defaultMemory="3500000000" defaultOverlargeCpu="1" defaultOverlargeMemory="3500000000">
  14.  
  15. <divergences five="0.35" four="0.25" one="0.1" three="0.2" two="0.15" useDefault="0" />
  16. </constants>
  17. <preprocessor check="1" checkAssemblyHub="1" memory="2000000000" preprocessJob="checkUniqueHeaders" />
  18. <preprocessor chunkSize="3000000" lastzOpts="--step=3 --ambiguous=iupac,100,100 --ungapped --queryhsplimit=keep,nowarn:1500" memory="2000000000" minPeriod="50" preprocessJob="lastzRepeatMask" proportionToSample="0.2" unmask="0" />
  19.  
  20. <trimBlast doTrimStrategy="1" keepParalogs="0" trimFlanking="10" trimMinSize="100" trimOutgroupDepth="1" trimOutgroupFlanking="2000" trimThreshold="1.0" trimWindowSize="1" />
  21. <ktserver memory="3500000000" />
  22. <setup makeEventHeadersAlphaNumeric="0" />
  23.  
  24. <caf alignmentFilter="filterSecondariesByMultipleSpecies" alpha="0.001" annealingRounds="128" blockTrim="5" chunkSize="25000000" compressFiles="1" deannealingRounds="2 8" doPhylogeny="0" filterByIdentity="0" identityRatio="3" lastzDisk="mediumDisk" lastzMemory="2000000000" maxAdjacencyComponentSizeRatio="50" maxAlignmentsPerSite="5" maxRecoverableChainLength="500000" maxRecoverableChainsIterations="5" maximumMedianSequenceLengthBetweenLinkedEnds="1000" minLengthForChromosome="1000000" minimumBlockDegree="2" minimumBlockDegreeToCheckSupport="10" minimumBlockHomologySupport="0.05" minimumDistance="0.01" minimumIngroupDegree="1" minimumMapQValue="0.0" minimumNumberOfSpecies="1" minimumOutgroupDegree="0" minimumSequenceLengthForBlast="30" minimumTreeCoverage="0.0" numTreeBuildingThreads="2" overlapSize="10000" phylogenyBreakpointScalingFactor="1.0" phylogenyCostPerDupPerBase="0.00" phylogenyCostPerLossPerBase="0.02" phylogenyDistanceCorrectionMethod="jukesCantor" phylogenyDoSplitsWithSupportHigherThanThisAllAtOnce="0.44" phylogenyHomologyUnitType="chain" phylogenyKeepSingleDegreeBlocks="0" phylogenyMaxBaseDistance="100" phylogenyMaxBlockDistance="50" phylogenyNumTrees="30" phylogenyRootingMethod="bestRecon" phylogenyScoringMethod="reconCost" phylogenySkipSingleCopyBlocks="1" phylogenyTreeBuildingMethod="guidedNeighborJoining,splitDecomposition" proportionOfUnalignedBasesForNewChromosome="0.8" realign="1" realignArguments="--gapGamma 0.0 --matchGamma 0.9 --diagonalExpansion 4 --splitMatrixBiggerThanThis 10 --constraintDiagonalTrim 0 --alignAmbiguityCharacters --splitIndelsLongerThanThis 99" removeRecoverableChains="unequalNumberOfIngroupCopies" runMapQFiltering="1" trim="0 0">
  25. <divergence argName="lastzArguments" default="--step=1 --ambiguous=iupac,100,100 --ydrop=3000" five="--step=2 --ambiguous=iupac,100,100 --ydrop=3000" four="--step=3 --ambiguous=iupac,100,100 --ydrop=3000" one="--step=2 --ambiguous=iupac,100,100 --ydrop=3000 --notransition" three="--step=4 --ambiguous=iupac,100,100 --ydrop=3000" two="--step=5 --ambiguous=iupac,100,100 --ydrop=3000" />
  26. <CactusCafRecursion maxFlowerGroupSize="100000000" />
  27. <CactusCafWrapper maxFlowerGroupSize="25000000" minFlowerSize="1" />
  28. <CactusCafWrapperLarge2 overlargeMemory="5000000000" />
  29. </caf>
  30.  
  31. <bar alignAmbiguityCharacters="1" anchorMatrixBiggerThanThis="500" bandingLimit="1000000" constraintDiagonalTrim="14" diagonalExpansion="20" gapGamma="0.0" largeEndSize="5000" matchGamma="0.2" minimumBlockDegree="2" minimumCoverageToRescue="0.5" minimumIngroupDegree="1" minimumNumberOfSpecies="1" minimumOutgroupDegree="0" minimumSizeToRescue="100" pruneOutStubAlignments="1" repeatMaskMatrixBiggerThanThis="500" rescue="0" runBar="1" spanningTrees="5" splitMatrixBiggerThanThis="3000" useBanding="1" useProgressiveMerging="1" veryLargeEndSize="2000000">
  32. <CactusBarRecursion maxFlowerGroupSize="100000000" />
  33. <CactusBarWrapper maxFlowerGroupSize="2000000" memory="2000000000" />
  34. <CactusBarWrapperLarge maxFlowerGroupSize="2000000" />
  35. <CactusBarEndAlignerWrapper memory="2000000000" />
  36. </bar>
  37. <normal iterations="0">
  38. <CactusNormalRecursion maxFlowerGroupSize="100000000" maxFlowerWrapperGroupSize="10000000" />
  39. <CactusNormalWrapper />
  40. </normal>
  41. <avg buildAvgs="0">
  42. <CactusAVGRecursion maxFlowerGroupSize="100000000" maxFlowerWrapperGroupSize="10000000" />
  43. <CactusAVGWrapper />
  44. </avg>
  45. <reference buildReference="1" ignoreUnalignedGaps="1" makeScaffolds="1" matchingAlgorithm="blossom5" maxWalkForCalculatingZ="100000" minNumberOfSequencesToSupportAdjacency="1" numberOfNs="10" permutations="10" phi="1.0" reference="mr" theta="0.0001" useSimulatedAnnealing="1" wiggle="0.9999">
  46. <CactusReferenceRecursion maxFlowerGroupSize="100000000" maxFlowerWrapperGroupSize="2000000" />
  47.  
  48. <CactusReferenceWrapper />
  49.  
  50. <CactusSetReferenceCoordinatesUpWrapper />
  51.  
  52. <CactusSetReferenceCoordinatesDownRecursion maxFlowerGroupSize="100000000" maxFlowerWrapperGroupSize="2000000" />
  53.  
  54. <CactusSetReferenceCoordinatesDownWrapper />
  55. </reference>
  56. <check runCheck="0">
  57. <CactusCheckRecursion maxFlowerGroupSize="100000000" maxFlowerWrapperGroupSize="2000000" />
  58. <CactusCheckWrapper />
  59. </check>
  60. <hal buildFasta="1" buildHal="1">
  61. <CactusHalGeneratorRecursion maxFlowerGroupSize="2000000" />
  62. <CactusHalGeneratorUpWrapper />
  63. </hal>
  64.  
  65. <multi_cactus>
  66. <outgroup ancestor_quality_fraction="0.75" max_num_outgroups="3" strategy="greedyPreference" threshold="0" />
  67.  
  68. <decomposition default_internal_node_prefix="Anc" max_parallel_subtrees="50000" self_alignment="false" subtree_size="2" />
  69.  
  70. </multi_cactus>
  71.  
  72. <ProgressiveNext />
  73.  
  74. <ProgressiveUp />
  75.  
  76. <ProgressiveDown />
  77.  
  78. <ProgressiveOut />
  79.  
  80. <RunCactusPreprocessorThenProgressiveDown />
  81.  
  82. <RunCactusPreprocessorThenProgressiveDown2 />
  83.  
  84. <exportHal disk="2000000000" />
  85. </cactusWorkflowConfig>
  86. INFO:toil.statsAndLogging:Got message from job at time 03-16-2019 17:42:21: Leaf names = ['simMouse_chr6', 'simRat_chr6', 'simHuman_chr6', 'simDog_chr6', 'simCow_chr6']
  87. INFO:toil.leader:Job ended successfully: 'RunCactusPreprocessorThenProgressiveDown' n/R/jobJ7SRhl
  88. INFO:toil.leader:Issued job 'logAssemblyStats' N/b/jobOV8hzO with job batch system ID: 1 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  89. INFO:toil.leader:Issued job 'logAssemblyStats' J/c/job3zI7pk with job batch system ID: 2 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  90. INFO:toil.leader:Issued job 'logAssemblyStats' H/9/jobBhtCUa with job batch system ID: 3 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  91. INFO:toil.leader:Issued job 'logAssemblyStats' F/Q/jobwskdLi with job batch system ID: 4 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  92. INFO:toil.leader:Issued job 'logAssemblyStats' 4/Z/jobGqzd5V with job batch system ID: 5 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  93. INFO:toil.leader:Issued job 'CactusPreprocessor' Z/9/jobzDRgr6 with job batch system ID: 6 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  94. INFO:toil.statsAndLogging:Got message from job at time 03-16-2019 17:44:26: Before preprocessing, got assembly stats for genome simRat_chr6: Input-sample: /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpd7v4xW/f4487a79-83bc-4c04-943c-071837d21e17/tmp0KsL4u.tmp Total-sequences: 1 Total-length: 647215 Proportion-repeat-masked: 0.075276 ProportionNs: 0.000000 Total-Ns: 0 N50: 647215 Median-sequence-length: 647215 Max-sequence-length: 647215 Min-sequence-length: 647215
  95.  
  96. INFO:toil.statsAndLogging:Got message from job at time 03-16-2019 17:44:28: Before preprocessing, got assembly stats for genome simHuman_chr6: Input-sample: /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmps1Oi59/1664c96e-f285-4c54-b3ab-2627633882a5/tmpcKZxYL.tmp Total-sequences: 1 Total-length: 601863 Proportion-repeat-masked: 0.074751 ProportionNs: 0.000000 Total-Ns: 0 N50: 601863 Median-sequence-length: 601863 Max-sequence-length: 601863 Min-sequence-length: 601863
  97.  
  98. INFO:toil.statsAndLogging:Got message from job at time 03-16-2019 17:44:32: Before preprocessing, got assembly stats for genome simCow_chr6: Input-sample: /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpvGOsIV/495cd453-3fe2-4318-bebc-81a4d6e59b4a/tmpGbxmt1.tmp Total-sequences: 1 Total-length: 602619 Proportion-repeat-masked: 0.054570 ProportionNs: 0.000000 Total-Ns: 0 N50: 602619 Median-sequence-length: 602619 Max-sequence-length: 602619 Min-sequence-length: 602619
  99.  
  100. INFO:toil.statsAndLogging:Got message from job at time 03-16-2019 17:44:32: Before preprocessing, got assembly stats for genome simMouse_chr6: Input-sample: /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpySEMCF/61bc91e3-d775-4adf-b67d-ed8513e2c50f/tmpHOdeKx.tmp Total-sequences: 1 Total-length: 636262 Proportion-repeat-masked: 0.056477 ProportionNs: 0.000000 Total-Ns: 0 N50: 636262 Median-sequence-length: 636262 Max-sequence-length: 636262 Min-sequence-length: 636262
  101.  
  102. INFO:toil.statsAndLogging:Got message from job at time 03-16-2019 17:44:34: Before preprocessing, got assembly stats for genome simDog_chr6: Input-sample: /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpUJXkCs/7787ad9b-0ee0-49e2-86b8-2fc58430049c/tmpLHWpPV.tmp Total-sequences: 1 Total-length: 593897 Proportion-repeat-masked: 0.045009 ProportionNs: 0.000000 Total-Ns: 0 N50: 593897 Median-sequence-length: 593897 Max-sequence-length: 593897 Min-sequence-length: 593897
  103.  
  104. INFO:toil.leader:Job ended successfully: 'logAssemblyStats' J/c/job3zI7pk
  105. INFO:toil.leader:Job ended successfully: 'logAssemblyStats' H/9/jobBhtCUa
  106. INFO:toil.leader:Job ended successfully: 'logAssemblyStats' F/Q/jobwskdLi
  107. INFO:toil.leader:Job ended successfully: 'logAssemblyStats' 4/Z/jobGqzd5V
  108. INFO:toil.leader:Job ended successfully: 'CactusPreprocessor' Z/9/jobzDRgr6
  109. INFO:toil.leader:Issued job 'CactusPreprocessor2' 6/S/joboXOEmF with job batch system ID: 7 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  110. INFO:toil.leader:Issued job 'CactusPreprocessor2' w/l/jobZAs1jS with job batch system ID: 8 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  111. INFO:toil.leader:Issued job 'CactusPreprocessor2' Y/Q/jobC7VGWH with job batch system ID: 9 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  112. INFO:toil.leader:Issued job 'CactusPreprocessor2' H/a/jobfDAMTR with job batch system ID: 10 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  113. INFO:toil.leader:Issued job 'CactusPreprocessor2' I/w/jobDw5m1r with job batch system ID: 11 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  114. INFO:toil.leader:Job ended successfully: 'logAssemblyStats' N/b/jobOV8hzO
  115. INFO:toil.leader:Job ended successfully: 'CactusPreprocessor2' 6/S/joboXOEmF
  116. INFO:toil.leader:Issued job 'BatchPreprocessor' H/t/jobG1dhHr with job batch system ID: 12 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  117. INFO:toil.leader:Job ended successfully: 'CactusPreprocessor2' w/l/jobZAs1jS
  118. INFO:toil.leader:Issued job 'BatchPreprocessor' k/E/jobRXBPkc with job batch system ID: 13 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  119. INFO:toil.leader:Job ended successfully: 'CactusPreprocessor2' Y/Q/jobC7VGWH
  120. INFO:toil.leader:Issued job 'BatchPreprocessor' K/c/job2zduWb with job batch system ID: 14 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  121. INFO:toil.leader:Job ended successfully: 'CactusPreprocessor2' H/a/jobfDAMTR
  122. INFO:toil.leader:Issued job 'BatchPreprocessor' r/x/job3fRie6 with job batch system ID: 15 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  123. INFO:toil.leader:Job ended successfully: 'CactusPreprocessor2' I/w/jobDw5m1r
  124. INFO:toil.leader:Issued job 'BatchPreprocessor' P/S/jobiGXqS_ with job batch system ID: 16 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  125. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' K/c/job2zduWb
  126. INFO:toil.leader:Issued job 'PreprocessSequence' p/f/job_8MJAD with job batch system ID: 17 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  127. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' r/x/job3fRie6
  128. INFO:toil.leader:Issued job 'PreprocessSequence' z/o/jobsAmTrq with job batch system ID: 18 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  129. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' P/S/jobiGXqS_
  130. INFO:toil.leader:Issued job 'PreprocessSequence' x/I/jobJQ7ARQ with job batch system ID: 19 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  131. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' H/t/jobG1dhHr
  132. INFO:toil.leader:Issued job 'PreprocessSequence' A/7/jobcrEjQz with job batch system ID: 20 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  133. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' k/E/jobRXBPkc
  134. INFO:toil.leader:Issued job 'PreprocessSequence' M/p/jobdGFf_F with job batch system ID: 21 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  135. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' A/7/jobcrEjQz
  136. INFO:toil.leader:Issued job 'CheckUniqueHeaders' y/Y/jobkWJa8O with job batch system ID: 22 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  137. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' M/p/jobdGFf_F
  138. INFO:toil.leader:Issued job 'CheckUniqueHeaders' h/K/jobxIw0jw with job batch system ID: 23 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  139. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' p/f/job_8MJAD
  140. INFO:toil.leader:Issued job 'CheckUniqueHeaders' B/F/jobpkWegn with job batch system ID: 24 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  141. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' z/o/jobsAmTrq
  142. INFO:toil.leader:Issued job 'CheckUniqueHeaders' N/u/job1esAP6 with job batch system ID: 25 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  143. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' x/I/jobJQ7ARQ
  144. INFO:toil.leader:Issued job 'CheckUniqueHeaders' v/s/job_TlyM3 with job batch system ID: 26 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  145. INFO:toil.leader:Job ended successfully: 'CheckUniqueHeaders' y/Y/jobkWJa8O
  146. INFO:toil.leader:Issued job 'PreprocessSequence' A/7/jobcrEjQz with job batch system ID: 27 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  147. INFO:toil.leader:Job ended successfully: 'CheckUniqueHeaders' h/K/jobxIw0jw
  148. INFO:toil.leader:Issued job 'PreprocessSequence' M/p/jobdGFf_F with job batch system ID: 28 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  149. INFO:toil.leader:Job ended successfully: 'CheckUniqueHeaders' B/F/jobpkWegn
  150. INFO:toil.leader:Issued job 'PreprocessSequence' p/f/job_8MJAD with job batch system ID: 29 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  151. INFO:toil.leader:Job ended successfully: 'CheckUniqueHeaders' N/u/job1esAP6
  152. INFO:toil.leader:Issued job 'PreprocessSequence' z/o/jobsAmTrq with job batch system ID: 30 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  153. INFO:toil.leader:Job ended successfully: 'CheckUniqueHeaders' v/s/job_TlyM3
  154. INFO:toil.leader:Issued job 'PreprocessSequence' x/I/jobJQ7ARQ with job batch system ID: 31 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  155. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' z/o/jobsAmTrq
  156. INFO:toil.leader:Issued job 'BatchPreprocessor' 9/o/jobkZWab1 with job batch system ID: 32 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  157. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' x/I/jobJQ7ARQ
  158. INFO:toil.leader:Issued job 'BatchPreprocessor' m/q/jobIQzPcW with job batch system ID: 33 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  159. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' A/7/jobcrEjQz
  160. INFO:toil.leader:Issued job 'BatchPreprocessor' 7/3/jobuxfBYT with job batch system ID: 34 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  161. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' M/p/jobdGFf_F
  162. INFO:toil.leader:Issued job 'BatchPreprocessor' l/e/jobG6Dx_L with job batch system ID: 35 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  163. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' p/f/job_8MJAD
  164. INFO:toil.leader:Issued job 'BatchPreprocessor' Q/R/jobEBIBfZ with job batch system ID: 36 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  165. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' m/q/jobIQzPcW
  166. INFO:toil.leader:Issued job 'PreprocessSequence' 9/P/jobp4MKJ0 with job batch system ID: 37 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  167. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' 7/3/jobuxfBYT
  168. INFO:toil.leader:Issued job 'PreprocessSequence' y/r/job_25dwF with job batch system ID: 38 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  169. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' l/e/jobG6Dx_L
  170. INFO:toil.leader:Issued job 'PreprocessSequence' L/A/jobwTWj3u with job batch system ID: 39 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  171. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' Q/R/jobEBIBfZ
  172. INFO:toil.leader:Issued job 'PreprocessSequence' p/W/jobNyhkpA with job batch system ID: 40 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  173. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' 9/o/jobkZWab1
  174. INFO:toil.leader:Issued job 'PreprocessSequence' Z/d/jobfcuGw8 with job batch system ID: 41 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  175. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' p/W/jobNyhkpA
  176. INFO:toil.leader:Issued job 'LastzRepeatMaskJob' b/q/jobMa49b7 with job batch system ID: 42 and cores: 1, disk: 100.0 M, and memory: 4.0 G
  177. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' Z/d/jobfcuGw8
  178. INFO:toil.leader:Issued job 'LastzRepeatMaskJob' K/P/jobv6ieGw with job batch system ID: 43 and cores: 1, disk: 100.0 M, and memory: 4.0 G
  179. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' 9/P/jobp4MKJ0
  180. INFO:toil.leader:Issued job 'LastzRepeatMaskJob' U/c/jobaQLA2Y with job batch system ID: 44 and cores: 1, disk: 100.0 M, and memory: 4.0 G
  181. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' y/r/job_25dwF
  182. INFO:toil.leader:Issued job 'LastzRepeatMaskJob' i/p/job6NfM8s with job batch system ID: 45 and cores: 1, disk: 100.0 M, and memory: 4.0 G
  183. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' L/A/jobwTWj3u
  184. INFO:toil.leader:Issued job 'LastzRepeatMaskJob' Q/S/jobf3fqRx with job batch system ID: 46 and cores: 1, disk: 100.0 M, and memory: 4.0 G
  185. INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' U/c/jobaQLA2Y
  186. INFO:toil.leader:Issued job 'MergeChunks' d/o/jobYp7KAc with job batch system ID: 47 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  187. INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' i/p/job6NfM8s
  188. INFO:toil.leader:Issued job 'MergeChunks' r/n/jobOSPIJD with job batch system ID: 48 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  189. INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' Q/S/jobf3fqRx
  190. INFO:toil.leader:Issued job 'MergeChunks' k/R/jobvj9yCM with job batch system ID: 49 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  191. INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' b/q/jobMa49b7
  192. INFO:toil.leader:Issued job 'MergeChunks' 6/i/jobyJAMGq with job batch system ID: 50 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  193. INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' K/P/jobv6ieGw
  194. INFO:toil.leader:Issued job 'MergeChunks' k/H/joby6WMso with job batch system ID: 51 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  195. INFO:toil.leader:Job ended successfully: 'MergeChunks' d/o/jobYp7KAc
  196. INFO:toil.leader:Issued job 'MergeChunks2' 6/3/jobsMoPHe with job batch system ID: 52 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  197. INFO:toil.leader:Job ended successfully: 'MergeChunks' r/n/jobOSPIJD
  198. INFO:toil.leader:Issued job 'MergeChunks2' W/8/job4OL1Al with job batch system ID: 53 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  199. INFO:toil.leader:Job ended successfully: 'MergeChunks' k/R/jobvj9yCM
  200. INFO:toil.leader:Issued job 'MergeChunks2' n/J/job2s8_uG with job batch system ID: 54 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  201. INFO:toil.leader:Job ended successfully: 'MergeChunks' 6/i/jobyJAMGq
  202. INFO:toil.leader:Issued job 'MergeChunks2' V/4/joborRjM4 with job batch system ID: 55 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  203. INFO:toil.leader:Job ended successfully: 'MergeChunks' k/H/joby6WMso
  204. INFO:toil.leader:Issued job 'MergeChunks2' F/0/jobxA6M9W with job batch system ID: 56 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  205. INFO:toil.leader:Job ended successfully: 'MergeChunks2' W/8/job4OL1Al
  206. INFO:toil.leader:Issued job 'MergeChunks' r/n/jobOSPIJD with job batch system ID: 57 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  207. INFO:toil.leader:Job ended successfully: 'MergeChunks2' n/J/job2s8_uG
  208. INFO:toil.leader:Issued job 'MergeChunks' k/R/jobvj9yCM with job batch system ID: 58 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  209. INFO:toil.leader:Job ended successfully: 'MergeChunks2' V/4/joborRjM4
  210. INFO:toil.leader:Issued job 'MergeChunks' 6/i/jobyJAMGq with job batch system ID: 59 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  211. INFO:toil.leader:Job ended successfully: 'MergeChunks2' F/0/jobxA6M9W
  212. INFO:toil.leader:Issued job 'MergeChunks' k/H/joby6WMso with job batch system ID: 60 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  213. INFO:toil.leader:Job ended successfully: 'MergeChunks2' 6/3/jobsMoPHe
  214. INFO:toil.leader:Issued job 'MergeChunks' d/o/jobYp7KAc with job batch system ID: 61 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  215. INFO:toil.leader:Job ended successfully: 'MergeChunks' k/H/joby6WMso
  216. INFO:toil.leader:Issued job 'PreprocessSequence' Z/d/jobfcuGw8 with job batch system ID: 62 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  217. INFO:toil.leader:Job ended successfully: 'MergeChunks' d/o/jobYp7KAc
  218. INFO:toil.leader:Issued job 'PreprocessSequence' 9/P/jobp4MKJ0 with job batch system ID: 63 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  219. INFO:toil.leader:Job ended successfully: 'MergeChunks' r/n/jobOSPIJD
  220. INFO:toil.leader:Issued job 'PreprocessSequence' y/r/job_25dwF with job batch system ID: 64 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  221. INFO:toil.leader:Job ended successfully: 'MergeChunks' k/R/jobvj9yCM
  222. INFO:toil.leader:Issued job 'PreprocessSequence' L/A/jobwTWj3u with job batch system ID: 65 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  223. INFO:toil.leader:Job ended successfully: 'MergeChunks' 6/i/jobyJAMGq
  224. INFO:toil.leader:Issued job 'PreprocessSequence' p/W/jobNyhkpA with job batch system ID: 66 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  225. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' L/A/jobwTWj3u
  226. INFO:toil.leader:Issued job 'BatchPreprocessor' l/e/jobG6Dx_L with job batch system ID: 67 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  227. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' p/W/jobNyhkpA
  228. INFO:toil.leader:Issued job 'BatchPreprocessor' Q/R/jobEBIBfZ with job batch system ID: 68 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  229. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' Z/d/jobfcuGw8
  230. INFO:toil.leader:Issued job 'BatchPreprocessor' 9/o/jobkZWab1 with job batch system ID: 69 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  231. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' 9/P/jobp4MKJ0
  232. INFO:toil.leader:Issued job 'BatchPreprocessor' m/q/jobIQzPcW with job batch system ID: 70 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  233. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' y/r/job_25dwF
  234. INFO:toil.leader:Issued job 'BatchPreprocessor' 7/3/jobuxfBYT with job batch system ID: 71 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  235. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' m/q/jobIQzPcW
  236. INFO:toil.leader:Issued job 'BatchPreprocessor' P/S/jobiGXqS_ with job batch system ID: 72 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  237. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' 7/3/jobuxfBYT
  238. INFO:toil.leader:Issued job 'BatchPreprocessor' H/t/jobG1dhHr with job batch system ID: 73 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  239. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' l/e/jobG6Dx_L
  240. INFO:toil.leader:Issued job 'BatchPreprocessor' k/E/jobRXBPkc with job batch system ID: 74 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  241. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' Q/R/jobEBIBfZ
  242. INFO:toil.leader:Issued job 'BatchPreprocessor' K/c/job2zduWb with job batch system ID: 75 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  243. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' 9/o/jobkZWab1
  244. INFO:toil.leader:Issued job 'BatchPreprocessor' r/x/job3fRie6 with job batch system ID: 76 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  245. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' H/t/jobG1dhHr
  246. INFO:toil.leader:Issued job 'CactusPreprocessor2' 6/S/joboXOEmF with job batch system ID: 77 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  247. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' k/E/jobRXBPkc
  248. INFO:toil.leader:Issued job 'CactusPreprocessor2' w/l/jobZAs1jS with job batch system ID: 78 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  249. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' K/c/job2zduWb
  250. INFO:toil.leader:Issued job 'CactusPreprocessor2' Y/Q/jobC7VGWH with job batch system ID: 79 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  251. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' r/x/job3fRie6
  252. INFO:toil.leader:Issued job 'CactusPreprocessor2' H/a/jobfDAMTR with job batch system ID: 80 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  253. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' P/S/jobiGXqS_
  254. INFO:toil.leader:Issued job 'CactusPreprocessor2' I/w/jobDw5m1r with job batch system ID: 81 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  255. INFO:toil.leader:Job ended successfully: 'CactusPreprocessor2' I/w/jobDw5m1r
  256. INFO:toil.leader:Job ended successfully: 'CactusPreprocessor2' 6/S/joboXOEmF
  257. INFO:toil.leader:Job ended successfully: 'CactusPreprocessor2' w/l/jobZAs1jS
  258. INFO:toil.leader:Job ended successfully: 'CactusPreprocessor2' Y/Q/jobC7VGWH
  259. INFO:toil.leader:Job ended successfully: 'CactusPreprocessor2' H/a/jobfDAMTR
  260. INFO:toil.leader:Issued job 'CactusPreprocessor' Z/9/jobzDRgr6 with job batch system ID: 82 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  261. INFO:toil.leader:Job ended successfully: 'CactusPreprocessor' Z/9/jobzDRgr6
  262. INFO:toil.leader:Issued job 'RunCactusPreprocessorThenProgressiveDown2' w/J/jobQeKsUS with job batch system ID: 83 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  263. INFO:toil.leader:Job ended successfully: 'RunCactusPreprocessorThenProgressiveDown2' w/J/jobQeKsUS
  264. INFO:toil.leader:Issued job 'logAssemblyStats' B/N/jobwtsoQN with job batch system ID: 84 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  265. INFO:toil.leader:Issued job 'logAssemblyStats' a/k/jobxT5dgh with job batch system ID: 85 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  266. INFO:toil.leader:Issued job 'logAssemblyStats' p/f/jobtrY_dV with job batch system ID: 86 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  267. INFO:toil.leader:Issued job 'logAssemblyStats' f/8/jobxBaq38 with job batch system ID: 87 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  268. INFO:toil.leader:Issued job 'logAssemblyStats' e/T/job0Mdwg6 with job batch system ID: 88 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  269. INFO:toil.leader:Issued job 'ProgressiveDown' R/B/jobHI0bbU with job batch system ID: 89 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  270. INFO:toil.statsAndLogging:Got message from job at time 03-16-2019 18:27:22: There are 0 dependent projects
  271. INFO:toil.statsAndLogging:Got message from job at time 03-16-2019 18:27:22: After preprocessing, got assembly stats for genome simRat_chr6: Input-sample: /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpCfQMUY/4a298f3d-754d-4d60-b0fc-70f2fe314120/tmp5J3yTQ.tmp Total-sequences: 1 Total-length: 647215 Proportion-repeat-masked: 0.075276 ProportionNs: 0.000000 Total-Ns: 0 N50: 647215 Median-sequence-length: 647215 Max-sequence-length: 647215 Min-sequence-length: 647215
  272.  
  273. INFO:toil.statsAndLogging:Got message from job at time 03-16-2019 18:27:27: After preprocessing, got assembly stats for genome simDog_chr6: Input-sample: /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpzLAxv4/a42d561d-c4b9-4090-8fda-7cedda1820d3/tmpQv_Sca.tmp Total-sequences: 1 Total-length: 593897 Proportion-repeat-masked: 0.045009 ProportionNs: 0.000000 Total-Ns: 0 N50: 593897 Median-sequence-length: 593897 Max-sequence-length: 593897 Min-sequence-length: 593897
  274.  
  275. INFO:toil.statsAndLogging:Got message from job at time 03-16-2019 18:27:27: After preprocessing, got assembly stats for genome simCow_chr6: Input-sample: /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpYWtNa7/09a3a50d-d49a-451b-8ab4-018dc6b5f56d/tmpciKoBZ.tmp Total-sequences: 1 Total-length: 602619 Proportion-repeat-masked: 0.054570 ProportionNs: 0.000000 Total-Ns: 0 N50: 602619 Median-sequence-length: 602619 Max-sequence-length: 602619 Min-sequence-length: 602619
  276.  
  277. INFO:toil.statsAndLogging:Got message from job at time 03-16-2019 18:27:27: After preprocessing, got assembly stats for genome simMouse_chr6: Input-sample: /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpdawOuf/b7c921bc-3d64-4604-a441-f2985166dbed/tmpmbrSiF.tmp Total-sequences: 1 Total-length: 636262 Proportion-repeat-masked: 0.056477 ProportionNs: 0.000000 Total-Ns: 0 N50: 636262 Median-sequence-length: 636262 Max-sequence-length: 636262 Min-sequence-length: 636262
  278.  
  279. INFO:toil.statsAndLogging:Got message from job at time 03-16-2019 18:27:28: After preprocessing, got assembly stats for genome simHuman_chr6: Input-sample: /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmp8WmB2o/9866d9a4-2637-4cd8-abee-2c3a0625189e/tmpZjrQ5o.tmp Total-sequences: 1 Total-length: 601863 Proportion-repeat-masked: 0.074751 ProportionNs: 0.000000 Total-Ns: 0 N50: 601863 Median-sequence-length: 601863 Max-sequence-length: 601863 Min-sequence-length: 601863
  280.  
  281. INFO:toil.leader:Job ended successfully: 'ProgressiveDown' R/B/jobHI0bbU
  282. INFO:toil.leader:Issued job 'ProgressiveNext' n/a/jobcxAFzs with job batch system ID: 90 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  283. INFO:toil.leader:Job ended successfully: 'logAssemblyStats' B/N/jobwtsoQN
  284. INFO:toil.leader:Job ended successfully: 'logAssemblyStats' a/k/jobxT5dgh
  285. INFO:toil.leader:Job ended successfully: 'logAssemblyStats' p/f/jobtrY_dV
  286. INFO:toil.leader:Job ended successfully: 'logAssemblyStats' f/8/jobxBaq38
  287. INFO:toil.leader:Job ended successfully: 'logAssemblyStats' e/T/job0Mdwg6
  288. INFO:toil.statsAndLogging:Got message from job at time 03-16-2019 18:30:08: Project has 0 dependencies
  289. INFO:toil.leader:Job ended successfully: 'ProgressiveNext' n/a/jobcxAFzs
  290. INFO:toil.leader:Issued job 'ProgressiveUp' H/4/job4fjXBN with job batch system ID: 91 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  291. INFO:toil.leader:Job ended successfully: 'ProgressiveUp' H/4/job4fjXBN
  292. INFO:toil.leader:Issued job 'CactusTrimmingBlastPhase' N/t/jobC7X2Fy with job batch system ID: 92 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  293. INFO:toil.statsAndLogging:Got message from job at time 03-16-2019 18:34:36: Running blast using the trimming strategy
  294. INFO:toil.statsAndLogging:Got message from job at time 03-16-2019 18:34:36: Ingroup sequences: [('simMouse_chr6', 'N/t/jobC7X2Fy/g/tmpi_a0zK-run-tmpLmf_EH.tmp'), ('simRat_chr6', 'N/t/jobC7X2Fy/g/tmpue5kLf-run-tmpaa1raO.tmp')]
  295. INFO:toil.statsAndLogging:Got message from job at time 03-16-2019 18:34:36: Outgroup sequences: [('simHuman_chr6', 'N/t/jobC7X2Fy/g/tmpW5fPig-run-tmpcdOTI1.tmp'), ('simDog_chr6', 'N/t/jobC7X2Fy/g/tmp8fjzn2-run-tmp0Xj4WI.tmp'), ('simCow_chr6', 'N/t/jobC7X2Fy/g/tmpRccxPs-run-tmp1uw9oE.tmp')]
  296. INFO:toil.statsAndLogging:Got message from job at time 03-16-2019 18:34:36: Running mapQ uniquifying with parameters, minimumMapQValue: 0.0, maxAlignmentsPerSite 5, alpha: 0.001
  297. INFO:toil.leader:Job ended successfully: 'CactusTrimmingBlastPhase' N/t/jobC7X2Fy
  298. INFO:toil.leader:Issued job 'EncapsulatedJob' i/y/jobbkaY07 with job batch system ID: 93 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  299. INFO:toil.leader:Job ended successfully: 'EncapsulatedJob' i/y/jobbkaY07
  300. INFO:toil.leader:Issued job 'BlastIngroupsAndOutgroups' A/N/joblWCR15 with job batch system ID: 94 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  301. INFO:toil.statsAndLogging:Got message from job at time 03-16-2019 18:40:13: Blasting ingroups vs outgroups. Ingroup genomes: simMouse_chr6, simRat_chr6, outgroup genomes: simHuman_chr6, simDog_chr6, simCow_chr6
  302. INFO:toil.leader:Job ended successfully: 'BlastIngroupsAndOutgroups' A/N/joblWCR15
  303. INFO:toil.leader:Issued job 'BlastSequencesAllAgainstAll' W/F/jobiaHBp3 with job batch system ID: 95 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  304. INFO:toil.leader:Issued job 'BlastFirstOutgroup' 1/G/jobDuPa5A with job batch system ID: 96 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  305. INFO:toil.leader:Reissued any over long jobs
  306. INFO:toil.leader:Job ended successfully: 'BlastFirstOutgroup' 1/G/jobDuPa5A
  307. INFO:toil.leader:Issued job 'BlastSequencesAgainstEachOther' n/N/jobaEr9bi with job batch system ID: 97 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  308. INFO:toil.leader:Job ended successfully: 'BlastSequencesAllAgainstAll' W/F/jobiaHBp3
  309. INFO:toil.leader:Issued job 'MakeSelfBlasts' 8/p/jobu7Di5b with job batch system ID: 98 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  310. INFO:toil.leader:Issued job 'MakeOffDiagonalBlasts' 0/k/jobrk_zNu with job batch system ID: 99 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  311. INFO:toil.leader:Job ended successfully: 'MakeOffDiagonalBlasts' 0/k/jobrk_zNu
  312. INFO:toil.leader:Issued job 'CollateBlasts' o/r/jobRCh6Uw with job batch system ID: 100 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  313. INFO:toil.leader:Job ended successfully: 'BlastSequencesAgainstEachOther' n/N/jobaEr9bi
  314. INFO:toil.leader:Issued job 'RunBlast' F/K/jobB_Ts4D with job batch system ID: 101 and cores: 1, disk: 100.0 M, and memory: 100.0 M
  315. INFO:toil.leader:Job ended successfully: 'MakeSelfBlasts' 8/p/jobu7Di5b
  316. INFO:toil.leader:Issued job 'RunSelfBlast' U/Z/jobIRtrle with job batch system ID: 102 and cores: 1, disk: 100.0 M, and memory: 100.0 M
  317. WARNING:toil.leader:Job failed with exit value 1: 'RunBlast' F/K/jobB_Ts4D
  318. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'RunBlast' F/K/jobB_Ts4D
  319. WARNING:toil.leader:F/K/jobB_Ts4D INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  320. WARNING:toil.leader:F/K/jobB_Ts4D INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  321. WARNING:toil.leader:F/K/jobB_Ts4D WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  322. WARNING:toil.leader:F/K/jobB_Ts4D INFO:cactus.shared.common:Running the command ['cPecanLastz', '--format=cigar', '--notrivial', '--step=1', '--ambiguous=iupac,100,100', '--ydrop=3000', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpZ8rJuy/8f4e1200-a9e7-4b91-a7af-c36efe2d9614/tmpbxvJz0.tmp[multiple][nameparse=darkspace]', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpZ8rJuy/8f4e1200-a9e7-4b91-a7af-c36efe2d9614/tmpIcAboC.tmp[nameparse=darkspace]']
  323. WARNING:toil.leader:F/K/jobB_Ts4D FAILURE: call to malloc failed to allocate 72,242,864 bytes, for new_position_table
  324. WARNING:toil.leader:F/K/jobB_Ts4D Traceback (most recent call last):
  325. WARNING:toil.leader:F/K/jobB_Ts4D File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  326. WARNING:toil.leader:F/K/jobB_Ts4D job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  327. WARNING:toil.leader:F/K/jobB_Ts4D File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  328. WARNING:toil.leader:F/K/jobB_Ts4D super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  329. WARNING:toil.leader:F/K/jobB_Ts4D File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  330. WARNING:toil.leader:F/K/jobB_Ts4D returnValues = self._run(jobGraph, fileStore)
  331. WARNING:toil.leader:F/K/jobB_Ts4D File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  332. WARNING:toil.leader:F/K/jobB_Ts4D return self.run(fileStore)
  333. WARNING:toil.leader:F/K/jobB_Ts4D File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/blast/blast.py", line 443, in run
  334. WARNING:toil.leader:F/K/jobB_Ts4D runLastz(seqFile1, seqFile2, blastResultsFile, lastzArguments = self.blastOptions.lastzArguments)
  335. WARNING:toil.leader:F/K/jobB_Ts4D File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 758, in runLastz
  336. WARNING:toil.leader:F/K/jobB_Ts4D soft_timeout=5400)
  337. WARNING:toil.leader:F/K/jobB_Ts4D File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  338. WARNING:toil.leader:F/K/jobB_Ts4D raise RuntimeError("Command %s failed with output: %s" % (call, output))
  339. WARNING:toil.leader:F/K/jobB_Ts4D RuntimeError: Command ['cPecanLastz', '--format=cigar', '--notrivial', '--step=1', '--ambiguous=iupac,100,100', '--ydrop=3000', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpZ8rJuy/8f4e1200-a9e7-4b91-a7af-c36efe2d9614/tmpbxvJz0.tmp[multiple][nameparse=darkspace]', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpZ8rJuy/8f4e1200-a9e7-4b91-a7af-c36efe2d9614/tmpIcAboC.tmp[nameparse=darkspace]'] failed with output: None
  340. WARNING:toil.leader:F/K/jobB_Ts4D ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn41.grid.pub.ro
  341. WARNING:toil.leader:F/K/jobB_Ts4D WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'RunBlast' F/K/jobB_Ts4D with ID F/K/jobB_Ts4D to 5
  342. WARNING:toil.leader:F/K/jobB_Ts4D WARNING:toil.jobGraph:We have increased the default memory of the failed job 'RunBlast' F/K/jobB_Ts4D to 2147483648 bytes
  343. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'RunBlast' F/K/jobB_Ts4D with ID F/K/jobB_Ts4D to 4
  344. INFO:toil.leader:Issued job 'RunBlast' F/K/jobB_Ts4D with job batch system ID: 103 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  345. WARNING:toil.leader:Job failed with exit value 1: 'RunSelfBlast' U/Z/jobIRtrle
  346. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'RunSelfBlast' U/Z/jobIRtrle
  347. WARNING:toil.leader:U/Z/jobIRtrle INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  348. WARNING:toil.leader:U/Z/jobIRtrle INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  349. WARNING:toil.leader:U/Z/jobIRtrle WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  350. WARNING:toil.leader:U/Z/jobIRtrle INFO:cactus.shared.common:Running the command ['cPecanLastz', '--format=cigar', '--notrivial', '--step=1', '--ambiguous=iupac,100,100', '--ydrop=3000', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpOkhoGm/d3318a14-1f9c-4a37-a36d-1fd2d9b6ef75/tmpq891IJ.tmp[multiple][nameparse=darkspace]', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpOkhoGm/d3318a14-1f9c-4a37-a36d-1fd2d9b6ef75/tmpq891IJ.tmp[nameparse=darkspace]']
  351. WARNING:toil.leader:U/Z/jobIRtrle FAILURE: call to malloc failed to allocate 72,242,864 bytes, for new_position_table
  352. WARNING:toil.leader:U/Z/jobIRtrle Traceback (most recent call last):
  353. WARNING:toil.leader:U/Z/jobIRtrle File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  354. WARNING:toil.leader:U/Z/jobIRtrle job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  355. WARNING:toil.leader:U/Z/jobIRtrle File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  356. WARNING:toil.leader:U/Z/jobIRtrle super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  357. WARNING:toil.leader:U/Z/jobIRtrle File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  358. WARNING:toil.leader:U/Z/jobIRtrle returnValues = self._run(jobGraph, fileStore)
  359. WARNING:toil.leader:U/Z/jobIRtrle File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  360. WARNING:toil.leader:U/Z/jobIRtrle return self.run(fileStore)
  361. WARNING:toil.leader:U/Z/jobIRtrle File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/blast/blast.py", line 402, in run
  362. WARNING:toil.leader:U/Z/jobIRtrle runSelfLastz(seqFile, blastResultsFile, lastzArguments=self.blastOptions.lastzArguments)
  363. WARNING:toil.leader:U/Z/jobIRtrle File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 769, in runSelfLastz
  364. WARNING:toil.leader:U/Z/jobIRtrle soft_timeout=5400)
  365. WARNING:toil.leader:U/Z/jobIRtrle File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  366. WARNING:toil.leader:U/Z/jobIRtrle raise RuntimeError("Command %s failed with output: %s" % (call, output))
  367. WARNING:toil.leader:U/Z/jobIRtrle RuntimeError: Command ['cPecanLastz', '--format=cigar', '--notrivial', '--step=1', '--ambiguous=iupac,100,100', '--ydrop=3000', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpOkhoGm/d3318a14-1f9c-4a37-a36d-1fd2d9b6ef75/tmpq891IJ.tmp[multiple][nameparse=darkspace]', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpOkhoGm/d3318a14-1f9c-4a37-a36d-1fd2d9b6ef75/tmpq891IJ.tmp[nameparse=darkspace]'] failed with output: None
  368. WARNING:toil.leader:U/Z/jobIRtrle ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn35.grid.pub.ro
  369. WARNING:toil.leader:U/Z/jobIRtrle WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'RunSelfBlast' U/Z/jobIRtrle with ID U/Z/jobIRtrle to 5
  370. WARNING:toil.leader:U/Z/jobIRtrle WARNING:toil.jobGraph:We have increased the default memory of the failed job 'RunSelfBlast' U/Z/jobIRtrle to 2147483648 bytes
  371. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'RunSelfBlast' U/Z/jobIRtrle with ID U/Z/jobIRtrle to 4
  372. INFO:toil.leader:Issued job 'RunSelfBlast' U/Z/jobIRtrle with job batch system ID: 104 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  373. INFO:toil.leader:Job ended successfully: 'CollateBlasts' o/r/jobRCh6Uw
  374. INFO:toil.leader:Issued job 'CollateBlasts2' q/i/jobRNhz0c with job batch system ID: 105 and cores: 1, disk: 0.0 , and memory: 2.0 G
  375. WARNING:toil.statsAndLogging:Got message from job at time 03-16-2019 18:48:54: Job used more disk than requested. Consider modifying the user script to avoid the chance of failure due to incorrectly requested resources. Job o/r/jobRCh6Uw/g/tmpsXGsfI-_serialiseJob-stream used 0.00% (4.0 KB [4096B] used, 0.0 B [0B] requested) at the end of its run.
  376. INFO:toil.leader:Job ended successfully: 'RunBlast' F/K/jobB_Ts4D
  377. INFO:toil.leader:Issued job 'CollateBlasts' G/h/jobDREf0G with job batch system ID: 106 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  378. INFO:toil.leader:Job ended successfully: 'RunSelfBlast' U/Z/jobIRtrle
  379. INFO:toil.leader:Issued job 'CollateBlasts' C/g/joby6AKG3 with job batch system ID: 107 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  380. INFO:toil.leader:Job ended successfully: 'CollateBlasts2' q/i/jobRNhz0c
  381. INFO:toil.leader:Issued job 'CollateBlasts' o/r/jobRCh6Uw with job batch system ID: 108 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  382. INFO:toil.leader:Job ended successfully: 'CollateBlasts' C/g/joby6AKG3
  383. INFO:toil.leader:Issued job 'CollateBlasts2' S/Z/jobS06z51 with job batch system ID: 109 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  384. INFO:toil.leader:Job ended successfully: 'CollateBlasts' o/r/jobRCh6Uw
  385. INFO:toil.leader:Issued job 'MakeOffDiagonalBlasts' 0/k/jobrk_zNu with job batch system ID: 110 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  386. INFO:toil.leader:Job ended successfully: 'CollateBlasts' G/h/jobDREf0G
  387. INFO:toil.leader:Issued job 'CollateBlasts2' Y/c/jobnvZiSJ with job batch system ID: 111 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  388. INFO:toil.leader:Job ended successfully: 'CollateBlasts2' Y/c/jobnvZiSJ
  389. INFO:toil.leader:Issued job 'CollateBlasts' G/h/jobDREf0G with job batch system ID: 112 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  390. INFO:toil.leader:Job ended successfully: 'CollateBlasts2' S/Z/jobS06z51
  391. INFO:toil.leader:Issued job 'CollateBlasts' C/g/joby6AKG3 with job batch system ID: 113 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  392. INFO:toil.leader:Job ended successfully: 'MakeOffDiagonalBlasts' 0/k/jobrk_zNu
  393. INFO:toil.leader:Job ended successfully: 'CollateBlasts' C/g/joby6AKG3
  394. INFO:toil.leader:Issued job 'MakeSelfBlasts' 8/p/jobu7Di5b with job batch system ID: 114 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  395. INFO:toil.leader:Job ended successfully: 'CollateBlasts' G/h/jobDREf0G
  396. INFO:toil.leader:Issued job 'BlastSequencesAgainstEachOther' n/N/jobaEr9bi with job batch system ID: 115 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  397. INFO:toil.leader:Job ended successfully: 'MakeSelfBlasts' 8/p/jobu7Di5b
  398. INFO:toil.leader:Issued job 'CollateBlasts' 3/6/jobU6i3Lz with job batch system ID: 116 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  399. INFO:toil.leader:Job ended successfully: 'CollateBlasts' 3/6/jobU6i3Lz
  400. INFO:toil.leader:Issued job 'CollateBlasts2' a/Y/jobU1oGX0 with job batch system ID: 117 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  401. INFO:toil.leader:Job ended successfully: 'BlastSequencesAgainstEachOther' n/N/jobaEr9bi
  402. INFO:toil.leader:Issued job 'TrimAndRecurseOnOutgroups' Z/r/jobGpwTCX with job batch system ID: 118 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  403. INFO:toil.statsAndLogging:Got message from job at time 03-16-2019 19:16:09: Coverage on simMouse_chr6 from outgroup #1, simHuman_chr6: 42.9126680518% (current ingroup length 636262, untrimmed length 636262). Outgroup trimmed to 593077 bp from 601863
  404. INFO:toil.statsAndLogging:Got message from job at time 03-16-2019 19:16:09: Coverage on simRat_chr6 from outgroup #1, simHuman_chr6: 41.0293333745% (current ingroup length 647215, untrimmed length 647215). Outgroup trimmed to 593077 bp from 601863
  405. INFO:toil.statsAndLogging:Got message from job at time 03-16-2019 19:16:09: Cumulative coverage of 1 outgroups on ingroup simMouse_chr6: 42.9126680518
  406. INFO:toil.statsAndLogging:Got message from job at time 03-16-2019 19:16:09: Cumulative coverage of 1 outgroups on ingroup simRat_chr6: 41.0293333745
  407. INFO:toil.leader:Job ended successfully: 'TrimAndRecurseOnOutgroups' Z/r/jobGpwTCX
  408. INFO:toil.leader:Issued job 'BlastFirstOutgroup' z/A/jobclGaav with job batch system ID: 119 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  409. INFO:toil.leader:Job ended successfully: 'CollateBlasts2' a/Y/jobU1oGX0
  410. INFO:toil.leader:Issued job 'CollateBlasts' 3/6/jobU6i3Lz with job batch system ID: 120 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  411. INFO:toil.leader:Job ended successfully: 'BlastFirstOutgroup' z/A/jobclGaav
  412. INFO:toil.leader:Issued job 'BlastSequencesAgainstEachOther' 4/M/jobMdBO7I with job batch system ID: 121 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  413. INFO:toil.leader:Job ended successfully: 'BlastSequencesAgainstEachOther' 4/M/jobMdBO7I
  414. INFO:toil.leader:Issued job 'RunBlast' 8/e/jobKSocqq with job batch system ID: 122 and cores: 1, disk: 100.0 M, and memory: 100.0 M
  415. WARNING:toil.leader:Job failed with exit value 1: 'RunBlast' 8/e/jobKSocqq
  416. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'RunBlast' 8/e/jobKSocqq
  417. WARNING:toil.leader:8/e/jobKSocqq INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  418. WARNING:toil.leader:8/e/jobKSocqq INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  419. WARNING:toil.leader:8/e/jobKSocqq WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  420. WARNING:toil.leader:8/e/jobKSocqq INFO:cactus.shared.common:Running the command ['cPecanLastz', '--format=cigar', '--notrivial', '--step=1', '--ambiguous=iupac,100,100', '--ydrop=3000', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpnmbIeE/f0f9e2e5-54df-460b-8429-61297babc0e1/tmpD78Oy6.tmp[multiple][nameparse=darkspace]', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpnmbIeE/f0f9e2e5-54df-460b-8429-61297babc0e1/tmpN6sBAV.tmp[nameparse=darkspace]']
  421. WARNING:toil.leader:8/e/jobKSocqq FAILURE: call to malloc failed to allocate 69,733,056 bytes, for new_position_table
  422. WARNING:toil.leader:8/e/jobKSocqq Traceback (most recent call last):
  423. WARNING:toil.leader:8/e/jobKSocqq File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  424. WARNING:toil.leader:8/e/jobKSocqq job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  425. WARNING:toil.leader:8/e/jobKSocqq File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  426. WARNING:toil.leader:8/e/jobKSocqq super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  427. WARNING:toil.leader:8/e/jobKSocqq File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  428. WARNING:toil.leader:8/e/jobKSocqq returnValues = self._run(jobGraph, fileStore)
  429. WARNING:toil.leader:8/e/jobKSocqq File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  430. WARNING:toil.leader:8/e/jobKSocqq return self.run(fileStore)
  431. WARNING:toil.leader:8/e/jobKSocqq File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/blast/blast.py", line 443, in run
  432. WARNING:toil.leader:8/e/jobKSocqq runLastz(seqFile1, seqFile2, blastResultsFile, lastzArguments = self.blastOptions.lastzArguments)
  433. WARNING:toil.leader:8/e/jobKSocqq File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 758, in runLastz
  434. WARNING:toil.leader:8/e/jobKSocqq soft_timeout=5400)
  435. WARNING:toil.leader:8/e/jobKSocqq File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  436. WARNING:toil.leader:8/e/jobKSocqq raise RuntimeError("Command %s failed with output: %s" % (call, output))
  437. WARNING:toil.leader:8/e/jobKSocqq RuntimeError: Command ['cPecanLastz', '--format=cigar', '--notrivial', '--step=1', '--ambiguous=iupac,100,100', '--ydrop=3000', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpnmbIeE/f0f9e2e5-54df-460b-8429-61297babc0e1/tmpD78Oy6.tmp[multiple][nameparse=darkspace]', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpnmbIeE/f0f9e2e5-54df-460b-8429-61297babc0e1/tmpN6sBAV.tmp[nameparse=darkspace]'] failed with output: None
  438. WARNING:toil.leader:8/e/jobKSocqq ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn38.grid.pub.ro
  439. WARNING:toil.leader:8/e/jobKSocqq WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'RunBlast' 8/e/jobKSocqq with ID 8/e/jobKSocqq to 5
  440. WARNING:toil.leader:8/e/jobKSocqq WARNING:toil.jobGraph:We have increased the default memory of the failed job 'RunBlast' 8/e/jobKSocqq to 2147483648 bytes
  441. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'RunBlast' 8/e/jobKSocqq with ID 8/e/jobKSocqq to 4
  442. INFO:toil.leader:Issued job 'RunBlast' 8/e/jobKSocqq with job batch system ID: 123 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  443. INFO:toil.leader:Job ended successfully: 'CollateBlasts' 3/6/jobU6i3Lz
  444. INFO:toil.leader:Issued job 'BlastSequencesAllAgainstAll' W/F/jobiaHBp3 with job batch system ID: 124 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  445. INFO:toil.leader:Reissued any over long jobs
  446. INFO:toil.leader:Job ended successfully: 'RunBlast' 8/e/jobKSocqq
  447. INFO:toil.leader:Issued job 'CollateBlasts' L/P/jobddzsEi with job batch system ID: 125 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  448. INFO:toil.leader:Job ended successfully: 'BlastSequencesAllAgainstAll' W/F/jobiaHBp3
  449. INFO:toil.leader:Job ended successfully: 'CollateBlasts' L/P/jobddzsEi
  450. INFO:toil.leader:Issued job 'CollateBlasts2' i/Q/jobcUfsVS with job batch system ID: 126 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  451. INFO:toil.leader:Job ended successfully: 'CollateBlasts2' i/Q/jobcUfsVS
  452. INFO:toil.leader:Issued job 'CollateBlasts' L/P/jobddzsEi with job batch system ID: 127 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  453. INFO:toil.leader:Job ended successfully: 'CollateBlasts' L/P/jobddzsEi
  454. INFO:toil.leader:Issued job 'BlastSequencesAgainstEachOther' 4/M/jobMdBO7I with job batch system ID: 128 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  455. INFO:toil.leader:Job ended successfully: 'BlastSequencesAgainstEachOther' 4/M/jobMdBO7I
  456. INFO:toil.leader:Issued job 'TrimAndRecurseOnOutgroups' i/b/jobOBuIL3 with job batch system ID: 129 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  457. INFO:toil.statsAndLogging:Got message from job at time 03-16-2019 19:54:23: Coverage on simMouse_chr6 from outgroup #2, simDog_chr6: 5.02493867797% (current ingroup length 317178, untrimmed length 636262). Outgroup trimmed to 269178 bp from 593897
  458. INFO:toil.statsAndLogging:Got message from job at time 03-16-2019 19:54:23: Coverage on simRat_chr6 from outgroup #2, simDog_chr6: 4.32875040364% (current ingroup length 337557, untrimmed length 647215). Outgroup trimmed to 269178 bp from 593897
  459. INFO:toil.statsAndLogging:Got message from job at time 03-16-2019 19:54:23: Cumulative coverage of 2 outgroups on ingroup simMouse_chr6: 45.3963933097
  460. INFO:toil.statsAndLogging:Got message from job at time 03-16-2019 19:54:23: Cumulative coverage of 2 outgroups on ingroup simRat_chr6: 43.2647574608
  461. INFO:toil.leader:Job ended successfully: 'TrimAndRecurseOnOutgroups' i/b/jobOBuIL3
  462. INFO:toil.leader:Issued job 'BlastFirstOutgroup' M/d/jobOEhjiP with job batch system ID: 130 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  463. INFO:toil.leader:Job ended successfully: 'BlastFirstOutgroup' M/d/jobOEhjiP
  464. INFO:toil.leader:Issued job 'BlastSequencesAgainstEachOther' y/X/jobkzeOyo with job batch system ID: 131 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  465. INFO:toil.leader:Job ended successfully: 'BlastSequencesAgainstEachOther' y/X/jobkzeOyo
  466. INFO:toil.leader:Issued job 'RunBlast' T/5/job0V3VBR with job batch system ID: 132 and cores: 1, disk: 100.0 M, and memory: 100.0 M
  467. WARNING:toil.leader:Job failed with exit value 1: 'RunBlast' T/5/job0V3VBR
  468. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'RunBlast' T/5/job0V3VBR
  469. WARNING:toil.leader:T/5/job0V3VBR INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  470. WARNING:toil.leader:T/5/job0V3VBR INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  471. WARNING:toil.leader:T/5/job0V3VBR WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  472. WARNING:toil.leader:T/5/job0V3VBR INFO:cactus.shared.common:Running the command ['cPecanLastz', '--format=cigar', '--notrivial', '--step=1', '--ambiguous=iupac,100,100', '--ydrop=3000', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpNNfVhB/9624c9e9-a48f-4a7f-8046-9f5c03e09bc3/tmpH7rBhc.tmp[multiple][nameparse=darkspace]', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpNNfVhB/9624c9e9-a48f-4a7f-8046-9f5c03e09bc3/tmpeBCz3f.tmp[nameparse=darkspace]']
  473. WARNING:toil.leader:T/5/job0V3VBR FAILURE: call to malloc failed to allocate 69,592,544 bytes, for new_position_table
  474. WARNING:toil.leader:T/5/job0V3VBR Traceback (most recent call last):
  475. WARNING:toil.leader:T/5/job0V3VBR File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  476. WARNING:toil.leader:T/5/job0V3VBR job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  477. WARNING:toil.leader:T/5/job0V3VBR File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  478. WARNING:toil.leader:T/5/job0V3VBR super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  479. WARNING:toil.leader:T/5/job0V3VBR File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  480. WARNING:toil.leader:T/5/job0V3VBR returnValues = self._run(jobGraph, fileStore)
  481. WARNING:toil.leader:T/5/job0V3VBR File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  482. WARNING:toil.leader:T/5/job0V3VBR return self.run(fileStore)
  483. WARNING:toil.leader:T/5/job0V3VBR File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/blast/blast.py", line 443, in run
  484. WARNING:toil.leader:T/5/job0V3VBR runLastz(seqFile1, seqFile2, blastResultsFile, lastzArguments = self.blastOptions.lastzArguments)
  485. WARNING:toil.leader:T/5/job0V3VBR File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 758, in runLastz
  486. WARNING:toil.leader:T/5/job0V3VBR soft_timeout=5400)
  487. WARNING:toil.leader:T/5/job0V3VBR File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  488. WARNING:toil.leader:T/5/job0V3VBR raise RuntimeError("Command %s failed with output: %s" % (call, output))
  489. WARNING:toil.leader:T/5/job0V3VBR RuntimeError: Command ['cPecanLastz', '--format=cigar', '--notrivial', '--step=1', '--ambiguous=iupac,100,100', '--ydrop=3000', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpNNfVhB/9624c9e9-a48f-4a7f-8046-9f5c03e09bc3/tmpH7rBhc.tmp[multiple][nameparse=darkspace]', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpNNfVhB/9624c9e9-a48f-4a7f-8046-9f5c03e09bc3/tmpeBCz3f.tmp[nameparse=darkspace]'] failed with output: None
  490. WARNING:toil.leader:T/5/job0V3VBR ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn39.grid.pub.ro
  491. WARNING:toil.leader:T/5/job0V3VBR WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'RunBlast' T/5/job0V3VBR with ID T/5/job0V3VBR to 5
  492. WARNING:toil.leader:T/5/job0V3VBR WARNING:toil.jobGraph:We have increased the default memory of the failed job 'RunBlast' T/5/job0V3VBR to 2147483648 bytes
  493. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'RunBlast' T/5/job0V3VBR with ID T/5/job0V3VBR to 4
  494. INFO:toil.leader:Issued job 'RunBlast' T/5/job0V3VBR with job batch system ID: 133 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  495. INFO:toil.leader:Job ended successfully: 'RunBlast' T/5/job0V3VBR
  496. INFO:toil.leader:Issued job 'CollateBlasts' G/e/jobZYoQt0 with job batch system ID: 134 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  497. INFO:toil.leader:Job ended successfully: 'CollateBlasts' G/e/jobZYoQt0
  498. INFO:toil.leader:Issued job 'CollateBlasts2' Q/0/jobfoygd5 with job batch system ID: 135 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  499. INFO:toil.leader:Job ended successfully: 'CollateBlasts2' Q/0/jobfoygd5
  500. INFO:toil.leader:Issued job 'CollateBlasts' G/e/jobZYoQt0 with job batch system ID: 136 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  501. INFO:toil.leader:Job ended successfully: 'CollateBlasts' G/e/jobZYoQt0
  502. INFO:toil.leader:Issued job 'BlastSequencesAgainstEachOther' y/X/jobkzeOyo with job batch system ID: 137 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  503. INFO:toil.leader:Job ended successfully: 'BlastSequencesAgainstEachOther' y/X/jobkzeOyo
  504. INFO:toil.leader:Issued job 'TrimAndRecurseOnOutgroups' 1/9/job7p96t0 with job batch system ID: 138 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  505. INFO:toil.statsAndLogging:Got message from job at time 03-16-2019 20:16:06: Coverage on simMouse_chr6 from outgroup #3, simCow_chr6: 1.45565380493% (current ingroup length 298560, untrimmed length 636262). Outgroup trimmed to 108335 bp from 602619
  506. INFO:toil.statsAndLogging:Got message from job at time 03-16-2019 20:16:06: Coverage on simRat_chr6 from outgroup #3, simCow_chr6: 1.14062280934% (current ingroup length 320965, untrimmed length 647215). Outgroup trimmed to 108335 bp from 602619
  507. INFO:toil.statsAndLogging:Got message from job at time 03-16-2019 20:16:06: Cumulative coverage of 3 outgroups on ingroup simMouse_chr6: 46.0489546759
  508. INFO:toil.statsAndLogging:Got message from job at time 03-16-2019 20:16:06: Cumulative coverage of 3 outgroups on ingroup simRat_chr6: 43.8301028252
  509. INFO:toil.leader:Job ended successfully: 'TrimAndRecurseOnOutgroups' 1/9/job7p96t0
  510. INFO:toil.leader:Issued job 'BlastFirstOutgroup' M/d/jobOEhjiP with job batch system ID: 139 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  511. INFO:toil.leader:Job ended successfully: 'BlastFirstOutgroup' M/d/jobOEhjiP
  512. INFO:toil.leader:Issued job 'TrimAndRecurseOnOutgroups' i/b/jobOBuIL3 with job batch system ID: 140 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  513. INFO:toil.leader:Job ended successfully: 'TrimAndRecurseOnOutgroups' i/b/jobOBuIL3
  514. INFO:toil.leader:Issued job 'BlastFirstOutgroup' z/A/jobclGaav with job batch system ID: 141 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  515. INFO:toil.leader:Job ended successfully: 'BlastFirstOutgroup' z/A/jobclGaav
  516. INFO:toil.leader:Issued job 'TrimAndRecurseOnOutgroups' Z/r/jobGpwTCX with job batch system ID: 142 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  517. INFO:toil.leader:Job ended successfully: 'TrimAndRecurseOnOutgroups' Z/r/jobGpwTCX
  518. INFO:toil.leader:Issued job 'BlastFirstOutgroup' 1/G/jobDuPa5A with job batch system ID: 143 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  519. INFO:toil.leader:Job ended successfully: 'BlastFirstOutgroup' 1/G/jobDuPa5A
  520. INFO:toil.leader:Issued job 'CollateBlasts' 6/W/jobBGQIXF with job batch system ID: 144 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  521. INFO:toil.leader:Job ended successfully: 'CollateBlasts' 6/W/jobBGQIXF
  522. INFO:toil.leader:Issued job 'CollateBlasts2' y/K/jobdiJfsT with job batch system ID: 145 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  523. INFO:toil.leader:Job ended successfully: 'CollateBlasts2' y/K/jobdiJfsT
  524. INFO:toil.leader:Issued job 'CollateBlasts' 6/W/jobBGQIXF with job batch system ID: 146 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  525. INFO:toil.leader:Job ended successfully: 'CollateBlasts' 6/W/jobBGQIXF
  526. INFO:toil.leader:Issued job 'BlastIngroupsAndOutgroups' A/N/joblWCR15 with job batch system ID: 147 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  527. INFO:toil.leader:Job ended successfully: 'BlastIngroupsAndOutgroups' A/N/joblWCR15
  528. INFO:toil.leader:Issued job 'Job' h/2/jobPiMlb4 with job batch system ID: 148 and cores: 0, disk: 1.0 M, and memory: 32.0 M
  529. WARNING:toil.leader:Job failed with exit value 1: 'Job' h/2/jobPiMlb4
  530. WARNING:toil.leader:No log file is present, despite job failing: 'Job' h/2/jobPiMlb4
  531. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'Job' h/2/jobPiMlb4 with ID h/2/jobPiMlb4 to 5
  532. WARNING:toil.jobGraph:We have increased the default memory of the failed job 'Job' h/2/jobPiMlb4 to 2147483648 bytes
  533. INFO:toil.leader:Issued job 'Job' h/2/jobPiMlb4 with job batch system ID: 149 and cores: 0, disk: 1.0 M, and memory: 2.0 G
  534. INFO:toil.leader:Reissued any over long jobs
  535. INFO:toil.leader:Job ended successfully: 'Job' h/2/jobPiMlb4
  536. INFO:toil.leader:Issued job 'mappingQualityRescoring' z/L/jobigQDay with job batch system ID: 150 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  537. INFO:toil.statsAndLogging:Got message from job at time 03-16-2019 20:43:39: Input cigar file has 7362 lines
  538. INFO:toil.statsAndLogging:Got message from job at time 03-16-2019 20:43:39: Filtered, non-overlapping primary cigar file has 11273 lines
  539. INFO:toil.statsAndLogging:Got message from job at time 03-16-2019 20:43:39: Filtered, non-overlapping secondary cigar file has 31711 lines
  540. INFO:toil.leader:Job ended successfully: 'mappingQualityRescoring' z/L/jobigQDay
  541. INFO:toil.leader:Issued job 'Job' h/2/jobPiMlb4 with job batch system ID: 151 and cores: 0, disk: 1.0 M, and memory: 2.0 G
  542. INFO:toil.leader:Job ended successfully: 'Job' h/2/jobPiMlb4
  543. INFO:toil.leader:Issued job 'EncapsulatedJob' i/y/jobbkaY07 with job batch system ID: 152 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  544. INFO:toil.leader:Job ended successfully: 'EncapsulatedJob' i/y/jobbkaY07
  545. INFO:toil.leader:Issued job 'CactusSetupCheckpoint' m/g/jobFH953E with job batch system ID: 153 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  546. INFO:toil.leader:Job ended successfully: 'CactusSetupCheckpoint' m/g/jobFH953E
  547. INFO:toil.leader:Issued job 'StartPrimaryDB' N/8/job2cN94i with job batch system ID: 154 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  548. INFO:toil.leader:Job ended successfully: 'StartPrimaryDB' N/8/job2cN94i
  549. INFO:toil.leader:Issued job 'KtServerService' 4/i/jobKW0T3L with job batch system ID: 155 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  550. INFO:toil.leader:Issued job 'CactusSetupPhase' W/k/job7syrZa with job batch system ID: 156 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  551. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' 4/i/jobKW0T3L
  552. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' 4/i/jobKW0T3L
  553. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' 4/i/jobKW0T3L with ID 4/i/jobKW0T3L to 5
  554. INFO:toil.leader:Issued job 'KtServerService' 4/i/jobKW0T3L with job batch system ID: 157 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  555. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' W/k/job7syrZa
  556. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' W/k/job7syrZa
  557. WARNING:toil.leader:W/k/job7syrZa INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  558. WARNING:toil.leader:W/k/job7syrZa INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  559. WARNING:toil.leader:W/k/job7syrZa WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  560. WARNING:toil.leader:W/k/job7syrZa INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  561. WARNING:toil.leader:W/k/job7syrZa INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  562. WARNING:toil.leader:W/k/job7syrZa INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="12731" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpet4xvN/6597f03b-73b0-4f87-9e8d-a0a568695b00/tmpUTVzxA.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpet4xvN/6597f03b-73b0-4f87-9e8d-a0a568695b00/tmpVjjQYS.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpet4xvN/6597f03b-73b0-4f87-9e8d-a0a568695b00/tmpSRCHZ1.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpet4xvN/6597f03b-73b0-4f87-9e8d-a0a568695b00/tmpMUUbgG.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpet4xvN/6597f03b-73b0-4f87-9e8d-a0a568695b00/tmppzuOH5.tmp']
  563. WARNING:toil.leader:W/k/job7syrZa Set log level to INFO
  564. WARNING:toil.leader:W/k/job7syrZa Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  565. WARNING:toil.leader:W/k/job7syrZa <kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="12731" />
  566. WARNING:toil.leader:W/k/job7syrZa </st_kv_database_conf>
  567. WARNING:toil.leader:W/k/job7syrZa
  568. WARNING:toil.leader:W/k/job7syrZa Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpet4xvN/6597f03b-73b0-4f87-9e8d-a0a568695b00/tmpUTVzxA.tmp
  569. WARNING:toil.leader:W/k/job7syrZa Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpet4xvN/6597f03b-73b0-4f87-9e8d-a0a568695b00/tmpVjjQYS.tmp
  570. WARNING:toil.leader:W/k/job7syrZa Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpet4xvN/6597f03b-73b0-4f87-9e8d-a0a568695b00/tmpSRCHZ1.tmp
  571. WARNING:toil.leader:W/k/job7syrZa Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpet4xvN/6597f03b-73b0-4f87-9e8d-a0a568695b00/tmpMUUbgG.tmp
  572. WARNING:toil.leader:W/k/job7syrZa Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpet4xvN/6597f03b-73b0-4f87-9e8d-a0a568695b00/tmppzuOH5.tmp
  573. WARNING:toil.leader:W/k/job7syrZa Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.29 with error: network error
  574. WARNING:toil.leader:W/k/job7syrZa Uncaught exception
  575. WARNING:toil.leader:W/k/job7syrZa Traceback (most recent call last):
  576. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  577. WARNING:toil.leader:W/k/job7syrZa job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  578. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  579. WARNING:toil.leader:W/k/job7syrZa super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  580. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  581. WARNING:toil.leader:W/k/job7syrZa returnValues = self._run(jobGraph, fileStore)
  582. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  583. WARNING:toil.leader:W/k/job7syrZa return self.run(fileStore)
  584. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  585. WARNING:toil.leader:W/k/job7syrZa makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  586. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  587. WARNING:toil.leader:W/k/job7syrZa parameters=["cactus_setup"] + args + sequences)
  588. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  589. WARNING:toil.leader:W/k/job7syrZa raise RuntimeError("Command %s failed with output: %s" % (call, output))
  590. WARNING:toil.leader:W/k/job7syrZa RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="12731" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpet4xvN/6597f03b-73b0-4f87-9e8d-a0a568695b00/tmpUTVzxA.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpet4xvN/6597f03b-73b0-4f87-9e8d-a0a568695b00/tmpVjjQYS.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpet4xvN/6597f03b-73b0-4f87-9e8d-a0a568695b00/tmpSRCHZ1.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpet4xvN/6597f03b-73b0-4f87-9e8d-a0a568695b00/tmpMUUbgG.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpet4xvN/6597f03b-73b0-4f87-9e8d-a0a568695b00/tmppzuOH5.tmp'] failed with output:
  591. WARNING:toil.leader:W/k/job7syrZa ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn35.grid.pub.ro
  592. WARNING:toil.leader:W/k/job7syrZa WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' W/k/job7syrZa with ID W/k/job7syrZa to 5
  593. INFO:toil.leader:Issued job 'CactusSetupPhase' W/k/job7syrZa with job batch system ID: 158 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  594. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' 4/i/jobKW0T3L
  595. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' 4/i/jobKW0T3L
  596. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' 4/i/jobKW0T3L with ID 4/i/jobKW0T3L to 4
  597. INFO:toil.leader:Issued job 'KtServerService' 4/i/jobKW0T3L with job batch system ID: 159 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  598. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' W/k/job7syrZa
  599. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' W/k/job7syrZa
  600. WARNING:toil.leader:W/k/job7syrZa INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  601. WARNING:toil.leader:W/k/job7syrZa INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  602. WARNING:toil.leader:W/k/job7syrZa WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  603. WARNING:toil.leader:W/k/job7syrZa INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  604. WARNING:toil.leader:W/k/job7syrZa INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  605. WARNING:toil.leader:W/k/job7syrZa INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.40" port="17589" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpm0TM67/5bee7032-6700-4f59-840d-cdc2a2e9a26a/tmpxyRuIy.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpm0TM67/5bee7032-6700-4f59-840d-cdc2a2e9a26a/tmpl3YLt0.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpm0TM67/5bee7032-6700-4f59-840d-cdc2a2e9a26a/tmptKHWPX.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpm0TM67/5bee7032-6700-4f59-840d-cdc2a2e9a26a/tmpnA1JHS.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpm0TM67/5bee7032-6700-4f59-840d-cdc2a2e9a26a/tmp9fEypc.tmp']
  606. WARNING:toil.leader:W/k/job7syrZa Set log level to INFO
  607. WARNING:toil.leader:W/k/job7syrZa Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  608. WARNING:toil.leader:W/k/job7syrZa <kyoto_tycoon database_dir="fakepath" host="172.16.13.40" port="17589" />
  609. WARNING:toil.leader:W/k/job7syrZa </st_kv_database_conf>
  610. WARNING:toil.leader:W/k/job7syrZa
  611. WARNING:toil.leader:W/k/job7syrZa Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpm0TM67/5bee7032-6700-4f59-840d-cdc2a2e9a26a/tmpxyRuIy.tmp
  612. WARNING:toil.leader:W/k/job7syrZa Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpm0TM67/5bee7032-6700-4f59-840d-cdc2a2e9a26a/tmpl3YLt0.tmp
  613. WARNING:toil.leader:W/k/job7syrZa Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpm0TM67/5bee7032-6700-4f59-840d-cdc2a2e9a26a/tmptKHWPX.tmp
  614. WARNING:toil.leader:W/k/job7syrZa Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpm0TM67/5bee7032-6700-4f59-840d-cdc2a2e9a26a/tmpnA1JHS.tmp
  615. WARNING:toil.leader:W/k/job7syrZa Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpm0TM67/5bee7032-6700-4f59-840d-cdc2a2e9a26a/tmp9fEypc.tmp
  616. WARNING:toil.leader:W/k/job7syrZa Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.40 with error: network error
  617. WARNING:toil.leader:W/k/job7syrZa Uncaught exception
  618. WARNING:toil.leader:W/k/job7syrZa Traceback (most recent call last):
  619. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  620. WARNING:toil.leader:W/k/job7syrZa job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  621. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  622. WARNING:toil.leader:W/k/job7syrZa super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  623. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  624. WARNING:toil.leader:W/k/job7syrZa returnValues = self._run(jobGraph, fileStore)
  625. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  626. WARNING:toil.leader:W/k/job7syrZa return self.run(fileStore)
  627. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  628. WARNING:toil.leader:W/k/job7syrZa makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  629. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  630. WARNING:toil.leader:W/k/job7syrZa parameters=["cactus_setup"] + args + sequences)
  631. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  632. WARNING:toil.leader:W/k/job7syrZa raise RuntimeError("Command %s failed with output: %s" % (call, output))
  633. WARNING:toil.leader:W/k/job7syrZa RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.40" port="17589" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpm0TM67/5bee7032-6700-4f59-840d-cdc2a2e9a26a/tmpxyRuIy.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpm0TM67/5bee7032-6700-4f59-840d-cdc2a2e9a26a/tmpl3YLt0.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpm0TM67/5bee7032-6700-4f59-840d-cdc2a2e9a26a/tmptKHWPX.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpm0TM67/5bee7032-6700-4f59-840d-cdc2a2e9a26a/tmpnA1JHS.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpm0TM67/5bee7032-6700-4f59-840d-cdc2a2e9a26a/tmp9fEypc.tmp'] failed with output:
  634. WARNING:toil.leader:W/k/job7syrZa ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn39.grid.pub.ro
  635. WARNING:toil.leader:W/k/job7syrZa WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' W/k/job7syrZa with ID W/k/job7syrZa to 4
  636. INFO:toil.leader:Issued job 'CactusSetupPhase' W/k/job7syrZa with job batch system ID: 160 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  637. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' W/k/job7syrZa
  638. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' W/k/job7syrZa
  639. WARNING:toil.leader:W/k/job7syrZa INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  640. WARNING:toil.leader:W/k/job7syrZa INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  641. WARNING:toil.leader:W/k/job7syrZa WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  642. WARNING:toil.leader:W/k/job7syrZa INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  643. WARNING:toil.leader:W/k/job7syrZa INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  644. WARNING:toil.leader:W/k/job7syrZa INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.40" port="17589" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpWwZuqJ/7a00aa86-3046-4fbf-8e8c-e7056396947e/tmpf1a4ge.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpWwZuqJ/7a00aa86-3046-4fbf-8e8c-e7056396947e/tmpWu9GAr.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpWwZuqJ/7a00aa86-3046-4fbf-8e8c-e7056396947e/tmpFAp40T.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpWwZuqJ/7a00aa86-3046-4fbf-8e8c-e7056396947e/tmpQvhpEa.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpWwZuqJ/7a00aa86-3046-4fbf-8e8c-e7056396947e/tmpwJwUam.tmp']
  645. WARNING:toil.leader:W/k/job7syrZa Set log level to INFO
  646. WARNING:toil.leader:W/k/job7syrZa Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  647. WARNING:toil.leader:W/k/job7syrZa <kyoto_tycoon database_dir="fakepath" host="172.16.13.40" port="17589" />
  648. WARNING:toil.leader:W/k/job7syrZa </st_kv_database_conf>
  649. WARNING:toil.leader:W/k/job7syrZa
  650. WARNING:toil.leader:W/k/job7syrZa Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpWwZuqJ/7a00aa86-3046-4fbf-8e8c-e7056396947e/tmpf1a4ge.tmp
  651. WARNING:toil.leader:W/k/job7syrZa Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpWwZuqJ/7a00aa86-3046-4fbf-8e8c-e7056396947e/tmpWu9GAr.tmp
  652. WARNING:toil.leader:W/k/job7syrZa Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpWwZuqJ/7a00aa86-3046-4fbf-8e8c-e7056396947e/tmpFAp40T.tmp
  653. WARNING:toil.leader:W/k/job7syrZa Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpWwZuqJ/7a00aa86-3046-4fbf-8e8c-e7056396947e/tmpQvhpEa.tmp
  654. WARNING:toil.leader:W/k/job7syrZa Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpWwZuqJ/7a00aa86-3046-4fbf-8e8c-e7056396947e/tmpwJwUam.tmp
  655. WARNING:toil.leader:W/k/job7syrZa Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.40 with error: network error
  656. WARNING:toil.leader:W/k/job7syrZa Uncaught exception
  657. WARNING:toil.leader:W/k/job7syrZa Traceback (most recent call last):
  658. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  659. WARNING:toil.leader:W/k/job7syrZa job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  660. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  661. WARNING:toil.leader:W/k/job7syrZa super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  662. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  663. WARNING:toil.leader:W/k/job7syrZa returnValues = self._run(jobGraph, fileStore)
  664. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  665. WARNING:toil.leader:W/k/job7syrZa return self.run(fileStore)
  666. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  667. WARNING:toil.leader:W/k/job7syrZa makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  668. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  669. WARNING:toil.leader:W/k/job7syrZa parameters=["cactus_setup"] + args + sequences)
  670. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  671. WARNING:toil.leader:W/k/job7syrZa raise RuntimeError("Command %s failed with output: %s" % (call, output))
  672. WARNING:toil.leader:W/k/job7syrZa RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.40" port="17589" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpWwZuqJ/7a00aa86-3046-4fbf-8e8c-e7056396947e/tmpf1a4ge.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpWwZuqJ/7a00aa86-3046-4fbf-8e8c-e7056396947e/tmpWu9GAr.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpWwZuqJ/7a00aa86-3046-4fbf-8e8c-e7056396947e/tmpFAp40T.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpWwZuqJ/7a00aa86-3046-4fbf-8e8c-e7056396947e/tmpQvhpEa.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpWwZuqJ/7a00aa86-3046-4fbf-8e8c-e7056396947e/tmpwJwUam.tmp'] failed with output:
  673. WARNING:toil.leader:W/k/job7syrZa ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn34.grid.pub.ro
  674. WARNING:toil.leader:W/k/job7syrZa WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' W/k/job7syrZa with ID W/k/job7syrZa to 3
  675. INFO:toil.leader:Issued job 'CactusSetupPhase' W/k/job7syrZa with job batch system ID: 161 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  676. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' 4/i/jobKW0T3L
  677. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' 4/i/jobKW0T3L
  678. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' 4/i/jobKW0T3L with ID 4/i/jobKW0T3L to 3
  679. INFO:toil.leader:Issued job 'KtServerService' 4/i/jobKW0T3L with job batch system ID: 162 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  680. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' W/k/job7syrZa
  681. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' W/k/job7syrZa
  682. WARNING:toil.leader:W/k/job7syrZa INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  683. WARNING:toil.leader:W/k/job7syrZa INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  684. WARNING:toil.leader:W/k/job7syrZa WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  685. WARNING:toil.leader:W/k/job7syrZa INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  686. WARNING:toil.leader:W/k/job7syrZa INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  687. WARNING:toil.leader:W/k/job7syrZa INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="10233" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpDqCzNK/aad144a7-92f0-42b5-b29c-e13449fd146e/tmph2xdv3.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpDqCzNK/aad144a7-92f0-42b5-b29c-e13449fd146e/tmpQyCvmT.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpDqCzNK/aad144a7-92f0-42b5-b29c-e13449fd146e/tmpO_Hlik.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpDqCzNK/aad144a7-92f0-42b5-b29c-e13449fd146e/tmpdmUdrB.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpDqCzNK/aad144a7-92f0-42b5-b29c-e13449fd146e/tmprVlYVC.tmp']
  688. WARNING:toil.leader:W/k/job7syrZa Set log level to INFO
  689. WARNING:toil.leader:W/k/job7syrZa Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  690. WARNING:toil.leader:W/k/job7syrZa <kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="10233" />
  691. WARNING:toil.leader:W/k/job7syrZa </st_kv_database_conf>
  692. WARNING:toil.leader:W/k/job7syrZa
  693. WARNING:toil.leader:W/k/job7syrZa Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpDqCzNK/aad144a7-92f0-42b5-b29c-e13449fd146e/tmph2xdv3.tmp
  694. WARNING:toil.leader:W/k/job7syrZa Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpDqCzNK/aad144a7-92f0-42b5-b29c-e13449fd146e/tmpQyCvmT.tmp
  695. WARNING:toil.leader:W/k/job7syrZa Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpDqCzNK/aad144a7-92f0-42b5-b29c-e13449fd146e/tmpO_Hlik.tmp
  696. WARNING:toil.leader:W/k/job7syrZa Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpDqCzNK/aad144a7-92f0-42b5-b29c-e13449fd146e/tmpdmUdrB.tmp
  697. WARNING:toil.leader:W/k/job7syrZa Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpDqCzNK/aad144a7-92f0-42b5-b29c-e13449fd146e/tmprVlYVC.tmp
  698. WARNING:toil.leader:W/k/job7syrZa Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.29 with error: network error
  699. WARNING:toil.leader:W/k/job7syrZa Uncaught exception
  700. WARNING:toil.leader:W/k/job7syrZa Traceback (most recent call last):
  701. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  702. WARNING:toil.leader:W/k/job7syrZa job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  703. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  704. WARNING:toil.leader:W/k/job7syrZa super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  705. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  706. WARNING:toil.leader:W/k/job7syrZa returnValues = self._run(jobGraph, fileStore)
  707. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  708. WARNING:toil.leader:W/k/job7syrZa return self.run(fileStore)
  709. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  710. WARNING:toil.leader:W/k/job7syrZa makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  711. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  712. WARNING:toil.leader:W/k/job7syrZa parameters=["cactus_setup"] + args + sequences)
  713. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  714. WARNING:toil.leader:W/k/job7syrZa raise RuntimeError("Command %s failed with output: %s" % (call, output))
  715. WARNING:toil.leader:W/k/job7syrZa RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="10233" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpDqCzNK/aad144a7-92f0-42b5-b29c-e13449fd146e/tmph2xdv3.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpDqCzNK/aad144a7-92f0-42b5-b29c-e13449fd146e/tmpQyCvmT.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpDqCzNK/aad144a7-92f0-42b5-b29c-e13449fd146e/tmpO_Hlik.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpDqCzNK/aad144a7-92f0-42b5-b29c-e13449fd146e/tmpdmUdrB.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpDqCzNK/aad144a7-92f0-42b5-b29c-e13449fd146e/tmprVlYVC.tmp'] failed with output:
  716. WARNING:toil.leader:W/k/job7syrZa ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn39.grid.pub.ro
  717. WARNING:toil.leader:W/k/job7syrZa WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' W/k/job7syrZa with ID W/k/job7syrZa to 2
  718. INFO:toil.leader:Issued job 'CactusSetupPhase' W/k/job7syrZa with job batch system ID: 163 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  719. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' 4/i/jobKW0T3L
  720. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' 4/i/jobKW0T3L
  721. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' 4/i/jobKW0T3L with ID 4/i/jobKW0T3L to 2
  722. INFO:toil.leader:Issued job 'KtServerService' 4/i/jobKW0T3L with job batch system ID: 164 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  723. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' 4/i/jobKW0T3L
  724. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' 4/i/jobKW0T3L
  725. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' 4/i/jobKW0T3L with ID 4/i/jobKW0T3L to 1
  726. INFO:toil.leader:Issued job 'KtServerService' 4/i/jobKW0T3L with job batch system ID: 165 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  727. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' W/k/job7syrZa
  728. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' W/k/job7syrZa
  729. WARNING:toil.leader:W/k/job7syrZa INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  730. WARNING:toil.leader:W/k/job7syrZa INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  731. WARNING:toil.leader:W/k/job7syrZa WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  732. WARNING:toil.leader:W/k/job7syrZa INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  733. WARNING:toil.leader:W/k/job7syrZa INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  734. WARNING:toil.leader:W/k/job7syrZa INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="2974" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpSljjqA/dff94741-6051-4075-b724-31bd8156348d/tmpao_scE.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpSljjqA/dff94741-6051-4075-b724-31bd8156348d/tmp7ly47f.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpSljjqA/dff94741-6051-4075-b724-31bd8156348d/tmp1XV02j.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpSljjqA/dff94741-6051-4075-b724-31bd8156348d/tmpVCvfIh.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpSljjqA/dff94741-6051-4075-b724-31bd8156348d/tmpeA4r9U.tmp']
  735. WARNING:toil.leader:W/k/job7syrZa Set log level to INFO
  736. WARNING:toil.leader:W/k/job7syrZa Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  737. WARNING:toil.leader:W/k/job7syrZa <kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="2974" />
  738. WARNING:toil.leader:W/k/job7syrZa </st_kv_database_conf>
  739. WARNING:toil.leader:W/k/job7syrZa
  740. WARNING:toil.leader:W/k/job7syrZa Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpSljjqA/dff94741-6051-4075-b724-31bd8156348d/tmpao_scE.tmp
  741. WARNING:toil.leader:W/k/job7syrZa Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpSljjqA/dff94741-6051-4075-b724-31bd8156348d/tmp7ly47f.tmp
  742. WARNING:toil.leader:W/k/job7syrZa Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpSljjqA/dff94741-6051-4075-b724-31bd8156348d/tmp1XV02j.tmp
  743. WARNING:toil.leader:W/k/job7syrZa Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpSljjqA/dff94741-6051-4075-b724-31bd8156348d/tmpVCvfIh.tmp
  744. WARNING:toil.leader:W/k/job7syrZa Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpSljjqA/dff94741-6051-4075-b724-31bd8156348d/tmpeA4r9U.tmp
  745. WARNING:toil.leader:W/k/job7syrZa Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.41 with error: network error
  746. WARNING:toil.leader:W/k/job7syrZa Uncaught exception
  747. WARNING:toil.leader:W/k/job7syrZa Traceback (most recent call last):
  748. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  749. WARNING:toil.leader:W/k/job7syrZa job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  750. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  751. WARNING:toil.leader:W/k/job7syrZa super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  752. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  753. WARNING:toil.leader:W/k/job7syrZa returnValues = self._run(jobGraph, fileStore)
  754. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  755. WARNING:toil.leader:W/k/job7syrZa return self.run(fileStore)
  756. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  757. WARNING:toil.leader:W/k/job7syrZa makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  758. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  759. WARNING:toil.leader:W/k/job7syrZa parameters=["cactus_setup"] + args + sequences)
  760. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  761. WARNING:toil.leader:W/k/job7syrZa raise RuntimeError("Command %s failed with output: %s" % (call, output))
  762. WARNING:toil.leader:W/k/job7syrZa RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="2974" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpSljjqA/dff94741-6051-4075-b724-31bd8156348d/tmpao_scE.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpSljjqA/dff94741-6051-4075-b724-31bd8156348d/tmp7ly47f.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpSljjqA/dff94741-6051-4075-b724-31bd8156348d/tmp1XV02j.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpSljjqA/dff94741-6051-4075-b724-31bd8156348d/tmpVCvfIh.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpSljjqA/dff94741-6051-4075-b724-31bd8156348d/tmpeA4r9U.tmp'] failed with output:
  763. WARNING:toil.leader:W/k/job7syrZa ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn34.grid.pub.ro
  764. WARNING:toil.leader:W/k/job7syrZa WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' W/k/job7syrZa with ID W/k/job7syrZa to 1
  765. INFO:toil.leader:Issued job 'CactusSetupPhase' W/k/job7syrZa with job batch system ID: 166 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  766. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' W/k/job7syrZa
  767. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' W/k/job7syrZa
  768. WARNING:toil.leader:W/k/job7syrZa INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  769. WARNING:toil.leader:W/k/job7syrZa INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  770. WARNING:toil.leader:W/k/job7syrZa WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  771. WARNING:toil.leader:W/k/job7syrZa INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  772. WARNING:toil.leader:W/k/job7syrZa INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  773. WARNING:toil.leader:W/k/job7syrZa INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.40" port="29336" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpT4fVSU/c3e5eb42-2c97-48cd-9565-0ce92512f9af/tmpqPoIqy.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpT4fVSU/c3e5eb42-2c97-48cd-9565-0ce92512f9af/tmpvscZoB.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpT4fVSU/c3e5eb42-2c97-48cd-9565-0ce92512f9af/tmpYEEHh5.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpT4fVSU/c3e5eb42-2c97-48cd-9565-0ce92512f9af/tmp9Kn4J4.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpT4fVSU/c3e5eb42-2c97-48cd-9565-0ce92512f9af/tmpTI5qDv.tmp']
  774. WARNING:toil.leader:W/k/job7syrZa Set log level to INFO
  775. WARNING:toil.leader:W/k/job7syrZa Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  776. WARNING:toil.leader:W/k/job7syrZa <kyoto_tycoon database_dir="fakepath" host="172.16.13.40" port="29336" />
  777. WARNING:toil.leader:W/k/job7syrZa </st_kv_database_conf>
  778. WARNING:toil.leader:W/k/job7syrZa
  779. WARNING:toil.leader:W/k/job7syrZa Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpT4fVSU/c3e5eb42-2c97-48cd-9565-0ce92512f9af/tmpqPoIqy.tmp
  780. WARNING:toil.leader:W/k/job7syrZa Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpT4fVSU/c3e5eb42-2c97-48cd-9565-0ce92512f9af/tmpvscZoB.tmp
  781. WARNING:toil.leader:W/k/job7syrZa Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpT4fVSU/c3e5eb42-2c97-48cd-9565-0ce92512f9af/tmpYEEHh5.tmp
  782. WARNING:toil.leader:W/k/job7syrZa Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpT4fVSU/c3e5eb42-2c97-48cd-9565-0ce92512f9af/tmp9Kn4J4.tmp
  783. WARNING:toil.leader:W/k/job7syrZa Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpT4fVSU/c3e5eb42-2c97-48cd-9565-0ce92512f9af/tmpTI5qDv.tmp
  784. WARNING:toil.leader:W/k/job7syrZa Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.40 with error: network error
  785. WARNING:toil.leader:W/k/job7syrZa Uncaught exception
  786. WARNING:toil.leader:W/k/job7syrZa Traceback (most recent call last):
  787. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  788. WARNING:toil.leader:W/k/job7syrZa job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  789. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  790. WARNING:toil.leader:W/k/job7syrZa super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  791. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  792. WARNING:toil.leader:W/k/job7syrZa returnValues = self._run(jobGraph, fileStore)
  793. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  794. WARNING:toil.leader:W/k/job7syrZa return self.run(fileStore)
  795. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  796. WARNING:toil.leader:W/k/job7syrZa makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  797. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  798. WARNING:toil.leader:W/k/job7syrZa parameters=["cactus_setup"] + args + sequences)
  799. WARNING:toil.leader:W/k/job7syrZa File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  800. WARNING:toil.leader:W/k/job7syrZa raise RuntimeError("Command %s failed with output: %s" % (call, output))
  801. WARNING:toil.leader:W/k/job7syrZa RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.40" port="29336" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpT4fVSU/c3e5eb42-2c97-48cd-9565-0ce92512f9af/tmpqPoIqy.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpT4fVSU/c3e5eb42-2c97-48cd-9565-0ce92512f9af/tmpvscZoB.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpT4fVSU/c3e5eb42-2c97-48cd-9565-0ce92512f9af/tmpYEEHh5.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpT4fVSU/c3e5eb42-2c97-48cd-9565-0ce92512f9af/tmp9Kn4J4.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpT4fVSU/c3e5eb42-2c97-48cd-9565-0ce92512f9af/tmpTI5qDv.tmp'] failed with output:
  802. WARNING:toil.leader:W/k/job7syrZa ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn39.grid.pub.ro
  803. WARNING:toil.leader:W/k/job7syrZa WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' W/k/job7syrZa with ID W/k/job7syrZa to 0
  804. WARNING:toil.leader:Job 'CactusSetupPhase' W/k/job7syrZa with ID W/k/job7syrZa is completely failed
  805. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' 4/i/jobKW0T3L
  806. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' 4/i/jobKW0T3L
  807. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' 4/i/jobKW0T3L with ID 4/i/jobKW0T3L to 0
  808. WARNING:toil.leader:Job 'KtServerService' 4/i/jobKW0T3L with ID 4/i/jobKW0T3L is completely failed
  809. WARNING:toil.leader:Job: N/8/job2cN94i is being restarted as a checkpoint after the total failure of jobs in its subtree.
  810. INFO:toil.leader:Issued job 'StartPrimaryDB' N/8/job2cN94i with job batch system ID: 167 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  811. INFO:toil.leader:Job ended successfully: 'StartPrimaryDB' N/8/job2cN94i
  812. INFO:toil.leader:Issued job 'KtServerService' N/X/jobsLBpFr with job batch system ID: 168 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  813. INFO:toil.leader:Issued job 'CactusSetupPhase' p/n/jobAsp_cK with job batch system ID: 169 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  814. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' N/X/jobsLBpFr
  815. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' N/X/jobsLBpFr
  816. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' N/X/jobsLBpFr with ID N/X/jobsLBpFr to 5
  817. INFO:toil.leader:Issued job 'KtServerService' N/X/jobsLBpFr with job batch system ID: 170 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  818. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' p/n/jobAsp_cK
  819. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' p/n/jobAsp_cK
  820. WARNING:toil.leader:p/n/jobAsp_cK INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  821. WARNING:toil.leader:p/n/jobAsp_cK INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  822. WARNING:toil.leader:p/n/jobAsp_cK WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  823. WARNING:toil.leader:p/n/jobAsp_cK INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  824. WARNING:toil.leader:p/n/jobAsp_cK INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  825. WARNING:toil.leader:p/n/jobAsp_cK INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="32065" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpS7s10s/4bee543a-0196-413d-a723-362e1300aabb/tmpkJEz_U.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpS7s10s/4bee543a-0196-413d-a723-362e1300aabb/tmpYsaxuf.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpS7s10s/4bee543a-0196-413d-a723-362e1300aabb/tmp003DMF.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpS7s10s/4bee543a-0196-413d-a723-362e1300aabb/tmpORUnQx.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpS7s10s/4bee543a-0196-413d-a723-362e1300aabb/tmpMl2kzr.tmp']
  826. WARNING:toil.leader:p/n/jobAsp_cK Set log level to INFO
  827. WARNING:toil.leader:p/n/jobAsp_cK Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  828. WARNING:toil.leader:p/n/jobAsp_cK <kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="32065" />
  829. WARNING:toil.leader:p/n/jobAsp_cK </st_kv_database_conf>
  830. WARNING:toil.leader:p/n/jobAsp_cK
  831. WARNING:toil.leader:p/n/jobAsp_cK Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpS7s10s/4bee543a-0196-413d-a723-362e1300aabb/tmpkJEz_U.tmp
  832. WARNING:toil.leader:p/n/jobAsp_cK Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpS7s10s/4bee543a-0196-413d-a723-362e1300aabb/tmpYsaxuf.tmp
  833. WARNING:toil.leader:p/n/jobAsp_cK Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpS7s10s/4bee543a-0196-413d-a723-362e1300aabb/tmp003DMF.tmp
  834. WARNING:toil.leader:p/n/jobAsp_cK Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpS7s10s/4bee543a-0196-413d-a723-362e1300aabb/tmpORUnQx.tmp
  835. WARNING:toil.leader:p/n/jobAsp_cK Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpS7s10s/4bee543a-0196-413d-a723-362e1300aabb/tmpMl2kzr.tmp
  836. WARNING:toil.leader:p/n/jobAsp_cK Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.35 with error: network error
  837. WARNING:toil.leader:p/n/jobAsp_cK Uncaught exception
  838. WARNING:toil.leader:p/n/jobAsp_cK Traceback (most recent call last):
  839. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  840. WARNING:toil.leader:p/n/jobAsp_cK job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  841. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  842. WARNING:toil.leader:p/n/jobAsp_cK super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  843. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  844. WARNING:toil.leader:p/n/jobAsp_cK returnValues = self._run(jobGraph, fileStore)
  845. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  846. WARNING:toil.leader:p/n/jobAsp_cK return self.run(fileStore)
  847. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  848. WARNING:toil.leader:p/n/jobAsp_cK makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  849. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  850. WARNING:toil.leader:p/n/jobAsp_cK parameters=["cactus_setup"] + args + sequences)
  851. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  852. WARNING:toil.leader:p/n/jobAsp_cK raise RuntimeError("Command %s failed with output: %s" % (call, output))
  853. WARNING:toil.leader:p/n/jobAsp_cK RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="32065" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpS7s10s/4bee543a-0196-413d-a723-362e1300aabb/tmpkJEz_U.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpS7s10s/4bee543a-0196-413d-a723-362e1300aabb/tmpYsaxuf.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpS7s10s/4bee543a-0196-413d-a723-362e1300aabb/tmp003DMF.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpS7s10s/4bee543a-0196-413d-a723-362e1300aabb/tmpORUnQx.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpS7s10s/4bee543a-0196-413d-a723-362e1300aabb/tmpMl2kzr.tmp'] failed with output:
  854. WARNING:toil.leader:p/n/jobAsp_cK ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn34.grid.pub.ro
  855. WARNING:toil.leader:p/n/jobAsp_cK WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' p/n/jobAsp_cK with ID p/n/jobAsp_cK to 5
  856. INFO:toil.leader:Issued job 'CactusSetupPhase' p/n/jobAsp_cK with job batch system ID: 171 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  857. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' N/X/jobsLBpFr
  858. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' N/X/jobsLBpFr
  859. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' N/X/jobsLBpFr with ID N/X/jobsLBpFr to 4
  860. INFO:toil.leader:Issued job 'KtServerService' N/X/jobsLBpFr with job batch system ID: 172 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  861. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' p/n/jobAsp_cK
  862. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' p/n/jobAsp_cK
  863. WARNING:toil.leader:p/n/jobAsp_cK INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  864. WARNING:toil.leader:p/n/jobAsp_cK INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  865. WARNING:toil.leader:p/n/jobAsp_cK WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  866. WARNING:toil.leader:p/n/jobAsp_cK INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  867. WARNING:toil.leader:p/n/jobAsp_cK INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  868. WARNING:toil.leader:p/n/jobAsp_cK INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="32065" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpMIcqi9/c8192e06-b44f-4ff9-ab2b-32d5b1dc78ec/tmp56HOZk.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpMIcqi9/c8192e06-b44f-4ff9-ab2b-32d5b1dc78ec/tmpV0NarA.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpMIcqi9/c8192e06-b44f-4ff9-ab2b-32d5b1dc78ec/tmpGbf9sa.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpMIcqi9/c8192e06-b44f-4ff9-ab2b-32d5b1dc78ec/tmpyCm1yN.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpMIcqi9/c8192e06-b44f-4ff9-ab2b-32d5b1dc78ec/tmpKyzdum.tmp']
  869. WARNING:toil.leader:p/n/jobAsp_cK Set log level to INFO
  870. WARNING:toil.leader:p/n/jobAsp_cK Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  871. WARNING:toil.leader:p/n/jobAsp_cK <kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="32065" />
  872. WARNING:toil.leader:p/n/jobAsp_cK </st_kv_database_conf>
  873. WARNING:toil.leader:p/n/jobAsp_cK
  874. WARNING:toil.leader:p/n/jobAsp_cK Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpMIcqi9/c8192e06-b44f-4ff9-ab2b-32d5b1dc78ec/tmp56HOZk.tmp
  875. WARNING:toil.leader:p/n/jobAsp_cK Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpMIcqi9/c8192e06-b44f-4ff9-ab2b-32d5b1dc78ec/tmpV0NarA.tmp
  876. WARNING:toil.leader:p/n/jobAsp_cK Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpMIcqi9/c8192e06-b44f-4ff9-ab2b-32d5b1dc78ec/tmpGbf9sa.tmp
  877. WARNING:toil.leader:p/n/jobAsp_cK Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpMIcqi9/c8192e06-b44f-4ff9-ab2b-32d5b1dc78ec/tmpyCm1yN.tmp
  878. WARNING:toil.leader:p/n/jobAsp_cK Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpMIcqi9/c8192e06-b44f-4ff9-ab2b-32d5b1dc78ec/tmpKyzdum.tmp
  879. WARNING:toil.leader:p/n/jobAsp_cK Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.35 with error: network error
  880. WARNING:toil.leader:p/n/jobAsp_cK Uncaught exception
  881. WARNING:toil.leader:p/n/jobAsp_cK Traceback (most recent call last):
  882. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  883. WARNING:toil.leader:p/n/jobAsp_cK job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  884. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  885. WARNING:toil.leader:p/n/jobAsp_cK super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  886. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  887. WARNING:toil.leader:p/n/jobAsp_cK returnValues = self._run(jobGraph, fileStore)
  888. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  889. WARNING:toil.leader:p/n/jobAsp_cK return self.run(fileStore)
  890. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  891. WARNING:toil.leader:p/n/jobAsp_cK makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  892. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  893. WARNING:toil.leader:p/n/jobAsp_cK parameters=["cactus_setup"] + args + sequences)
  894. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  895. WARNING:toil.leader:p/n/jobAsp_cK raise RuntimeError("Command %s failed with output: %s" % (call, output))
  896. WARNING:toil.leader:p/n/jobAsp_cK RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="32065" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpMIcqi9/c8192e06-b44f-4ff9-ab2b-32d5b1dc78ec/tmp56HOZk.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpMIcqi9/c8192e06-b44f-4ff9-ab2b-32d5b1dc78ec/tmpV0NarA.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpMIcqi9/c8192e06-b44f-4ff9-ab2b-32d5b1dc78ec/tmpGbf9sa.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpMIcqi9/c8192e06-b44f-4ff9-ab2b-32d5b1dc78ec/tmpyCm1yN.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpMIcqi9/c8192e06-b44f-4ff9-ab2b-32d5b1dc78ec/tmpKyzdum.tmp'] failed with output:
  897. WARNING:toil.leader:p/n/jobAsp_cK ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn39.grid.pub.ro
  898. WARNING:toil.leader:p/n/jobAsp_cK WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' p/n/jobAsp_cK with ID p/n/jobAsp_cK to 4
  899. INFO:toil.leader:Issued job 'CactusSetupPhase' p/n/jobAsp_cK with job batch system ID: 173 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  900. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' N/X/jobsLBpFr
  901. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' N/X/jobsLBpFr
  902. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' N/X/jobsLBpFr with ID N/X/jobsLBpFr to 3
  903. INFO:toil.leader:Issued job 'KtServerService' N/X/jobsLBpFr with job batch system ID: 174 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  904. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' p/n/jobAsp_cK
  905. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' p/n/jobAsp_cK
  906. WARNING:toil.leader:p/n/jobAsp_cK INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  907. WARNING:toil.leader:p/n/jobAsp_cK INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  908. WARNING:toil.leader:p/n/jobAsp_cK WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  909. WARNING:toil.leader:p/n/jobAsp_cK INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  910. WARNING:toil.leader:p/n/jobAsp_cK INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  911. WARNING:toil.leader:p/n/jobAsp_cK INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="22380" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpK2F_ay/5e6103d2-5ba8-49ed-82d5-fc5de042b32b/tmp2MaMFO.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpK2F_ay/5e6103d2-5ba8-49ed-82d5-fc5de042b32b/tmp3iGpob.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpK2F_ay/5e6103d2-5ba8-49ed-82d5-fc5de042b32b/tmpmcCYEy.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpK2F_ay/5e6103d2-5ba8-49ed-82d5-fc5de042b32b/tmpwropPF.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpK2F_ay/5e6103d2-5ba8-49ed-82d5-fc5de042b32b/tmpTTTcHS.tmp']
  912. WARNING:toil.leader:p/n/jobAsp_cK Set log level to INFO
  913. WARNING:toil.leader:p/n/jobAsp_cK Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  914. WARNING:toil.leader:p/n/jobAsp_cK <kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="22380" />
  915. WARNING:toil.leader:p/n/jobAsp_cK </st_kv_database_conf>
  916. WARNING:toil.leader:p/n/jobAsp_cK
  917. WARNING:toil.leader:p/n/jobAsp_cK Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpK2F_ay/5e6103d2-5ba8-49ed-82d5-fc5de042b32b/tmp2MaMFO.tmp
  918. WARNING:toil.leader:p/n/jobAsp_cK Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpK2F_ay/5e6103d2-5ba8-49ed-82d5-fc5de042b32b/tmp3iGpob.tmp
  919. WARNING:toil.leader:p/n/jobAsp_cK Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpK2F_ay/5e6103d2-5ba8-49ed-82d5-fc5de042b32b/tmpmcCYEy.tmp
  920. WARNING:toil.leader:p/n/jobAsp_cK Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpK2F_ay/5e6103d2-5ba8-49ed-82d5-fc5de042b32b/tmpwropPF.tmp
  921. WARNING:toil.leader:p/n/jobAsp_cK Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpK2F_ay/5e6103d2-5ba8-49ed-82d5-fc5de042b32b/tmpTTTcHS.tmp
  922. WARNING:toil.leader:p/n/jobAsp_cK Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.29 with error: network error
  923. WARNING:toil.leader:p/n/jobAsp_cK Uncaught exception
  924. WARNING:toil.leader:p/n/jobAsp_cK Traceback (most recent call last):
  925. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  926. WARNING:toil.leader:p/n/jobAsp_cK job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  927. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  928. WARNING:toil.leader:p/n/jobAsp_cK super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  929. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  930. WARNING:toil.leader:p/n/jobAsp_cK returnValues = self._run(jobGraph, fileStore)
  931. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  932. WARNING:toil.leader:p/n/jobAsp_cK return self.run(fileStore)
  933. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  934. WARNING:toil.leader:p/n/jobAsp_cK makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  935. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  936. WARNING:toil.leader:p/n/jobAsp_cK parameters=["cactus_setup"] + args + sequences)
  937. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  938. WARNING:toil.leader:p/n/jobAsp_cK raise RuntimeError("Command %s failed with output: %s" % (call, output))
  939. WARNING:toil.leader:p/n/jobAsp_cK RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="22380" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpK2F_ay/5e6103d2-5ba8-49ed-82d5-fc5de042b32b/tmp2MaMFO.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpK2F_ay/5e6103d2-5ba8-49ed-82d5-fc5de042b32b/tmp3iGpob.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpK2F_ay/5e6103d2-5ba8-49ed-82d5-fc5de042b32b/tmpmcCYEy.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpK2F_ay/5e6103d2-5ba8-49ed-82d5-fc5de042b32b/tmpwropPF.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpK2F_ay/5e6103d2-5ba8-49ed-82d5-fc5de042b32b/tmpTTTcHS.tmp'] failed with output:
  940. WARNING:toil.leader:p/n/jobAsp_cK ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn39.grid.pub.ro
  941. WARNING:toil.leader:p/n/jobAsp_cK WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' p/n/jobAsp_cK with ID p/n/jobAsp_cK to 3
  942. INFO:toil.leader:Issued job 'CactusSetupPhase' p/n/jobAsp_cK with job batch system ID: 175 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  943. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' N/X/jobsLBpFr
  944. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' N/X/jobsLBpFr
  945. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' N/X/jobsLBpFr with ID N/X/jobsLBpFr to 2
  946. INFO:toil.leader:Issued job 'KtServerService' N/X/jobsLBpFr with job batch system ID: 176 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  947. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' p/n/jobAsp_cK
  948. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' p/n/jobAsp_cK
  949. WARNING:toil.leader:p/n/jobAsp_cK INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  950. WARNING:toil.leader:p/n/jobAsp_cK INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  951. WARNING:toil.leader:p/n/jobAsp_cK WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  952. WARNING:toil.leader:p/n/jobAsp_cK WARNING:toil.fileStore:Detected that job (N/8/job2cN94i/g/tmpBNdXQn-_serialiseJob-stream) prematurely terminated. Fixing the state of the job on disk.
  953. WARNING:toil.leader:p/n/jobAsp_cK INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  954. WARNING:toil.leader:p/n/jobAsp_cK INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  955. WARNING:toil.leader:p/n/jobAsp_cK INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="22938" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpos6gFm/76cee4a6-17fe-4cfb-bb0d-55974026d6ed/tmpO0hjUT.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpos6gFm/76cee4a6-17fe-4cfb-bb0d-55974026d6ed/tmppHU39l.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpos6gFm/76cee4a6-17fe-4cfb-bb0d-55974026d6ed/tmpwA7SN4.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpos6gFm/76cee4a6-17fe-4cfb-bb0d-55974026d6ed/tmpyfsiW1.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpos6gFm/76cee4a6-17fe-4cfb-bb0d-55974026d6ed/tmptigMxY.tmp']
  956. WARNING:toil.leader:p/n/jobAsp_cK Set log level to INFO
  957. WARNING:toil.leader:p/n/jobAsp_cK Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  958. WARNING:toil.leader:p/n/jobAsp_cK <kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="22938" />
  959. WARNING:toil.leader:p/n/jobAsp_cK </st_kv_database_conf>
  960. WARNING:toil.leader:p/n/jobAsp_cK
  961. WARNING:toil.leader:p/n/jobAsp_cK Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpos6gFm/76cee4a6-17fe-4cfb-bb0d-55974026d6ed/tmpO0hjUT.tmp
  962. WARNING:toil.leader:p/n/jobAsp_cK Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpos6gFm/76cee4a6-17fe-4cfb-bb0d-55974026d6ed/tmppHU39l.tmp
  963. WARNING:toil.leader:p/n/jobAsp_cK Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpos6gFm/76cee4a6-17fe-4cfb-bb0d-55974026d6ed/tmpwA7SN4.tmp
  964. WARNING:toil.leader:p/n/jobAsp_cK Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpos6gFm/76cee4a6-17fe-4cfb-bb0d-55974026d6ed/tmpyfsiW1.tmp
  965. WARNING:toil.leader:p/n/jobAsp_cK Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpos6gFm/76cee4a6-17fe-4cfb-bb0d-55974026d6ed/tmptigMxY.tmp
  966. WARNING:toil.leader:p/n/jobAsp_cK Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.35 with error: network error
  967. WARNING:toil.leader:p/n/jobAsp_cK Uncaught exception
  968. WARNING:toil.leader:p/n/jobAsp_cK Traceback (most recent call last):
  969. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  970. WARNING:toil.leader:p/n/jobAsp_cK job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  971. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  972. WARNING:toil.leader:p/n/jobAsp_cK super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  973. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  974. WARNING:toil.leader:p/n/jobAsp_cK returnValues = self._run(jobGraph, fileStore)
  975. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  976. WARNING:toil.leader:p/n/jobAsp_cK return self.run(fileStore)
  977. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  978. WARNING:toil.leader:p/n/jobAsp_cK makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  979. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  980. WARNING:toil.leader:p/n/jobAsp_cK parameters=["cactus_setup"] + args + sequences)
  981. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  982. WARNING:toil.leader:p/n/jobAsp_cK raise RuntimeError("Command %s failed with output: %s" % (call, output))
  983. WARNING:toil.leader:p/n/jobAsp_cK RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="22938" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpos6gFm/76cee4a6-17fe-4cfb-bb0d-55974026d6ed/tmpO0hjUT.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpos6gFm/76cee4a6-17fe-4cfb-bb0d-55974026d6ed/tmppHU39l.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpos6gFm/76cee4a6-17fe-4cfb-bb0d-55974026d6ed/tmpwA7SN4.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpos6gFm/76cee4a6-17fe-4cfb-bb0d-55974026d6ed/tmpyfsiW1.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpos6gFm/76cee4a6-17fe-4cfb-bb0d-55974026d6ed/tmptigMxY.tmp'] failed with output:
  984. WARNING:toil.leader:p/n/jobAsp_cK ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn35.grid.pub.ro
  985. WARNING:toil.leader:p/n/jobAsp_cK WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' p/n/jobAsp_cK with ID p/n/jobAsp_cK to 2
  986. INFO:toil.leader:Issued job 'CactusSetupPhase' p/n/jobAsp_cK with job batch system ID: 177 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  987. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' N/X/jobsLBpFr
  988. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' N/X/jobsLBpFr
  989. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' N/X/jobsLBpFr with ID N/X/jobsLBpFr to 1
  990. INFO:toil.leader:Issued job 'KtServerService' N/X/jobsLBpFr with job batch system ID: 178 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  991. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' p/n/jobAsp_cK
  992. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' p/n/jobAsp_cK
  993. WARNING:toil.leader:p/n/jobAsp_cK INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  994. WARNING:toil.leader:p/n/jobAsp_cK INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  995. WARNING:toil.leader:p/n/jobAsp_cK WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  996. WARNING:toil.leader:p/n/jobAsp_cK INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  997. WARNING:toil.leader:p/n/jobAsp_cK INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  998. WARNING:toil.leader:p/n/jobAsp_cK INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="2100" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpvFu1Wj/6903afbc-0150-4141-8949-709df05797b5/tmpMzUNTZ.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpvFu1Wj/6903afbc-0150-4141-8949-709df05797b5/tmpd8lhaP.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpvFu1Wj/6903afbc-0150-4141-8949-709df05797b5/tmppY85RN.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpvFu1Wj/6903afbc-0150-4141-8949-709df05797b5/tmp5x1ero.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpvFu1Wj/6903afbc-0150-4141-8949-709df05797b5/tmpflrlQr.tmp']
  999. WARNING:toil.leader:p/n/jobAsp_cK Set log level to INFO
  1000. WARNING:toil.leader:p/n/jobAsp_cK Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  1001. WARNING:toil.leader:p/n/jobAsp_cK <kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="2100" />
  1002. WARNING:toil.leader:p/n/jobAsp_cK </st_kv_database_conf>
  1003. WARNING:toil.leader:p/n/jobAsp_cK
  1004. WARNING:toil.leader:p/n/jobAsp_cK Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpvFu1Wj/6903afbc-0150-4141-8949-709df05797b5/tmpMzUNTZ.tmp
  1005. WARNING:toil.leader:p/n/jobAsp_cK Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpvFu1Wj/6903afbc-0150-4141-8949-709df05797b5/tmpd8lhaP.tmp
  1006. WARNING:toil.leader:p/n/jobAsp_cK Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpvFu1Wj/6903afbc-0150-4141-8949-709df05797b5/tmppY85RN.tmp
  1007. WARNING:toil.leader:p/n/jobAsp_cK Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpvFu1Wj/6903afbc-0150-4141-8949-709df05797b5/tmp5x1ero.tmp
  1008. WARNING:toil.leader:p/n/jobAsp_cK Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpvFu1Wj/6903afbc-0150-4141-8949-709df05797b5/tmpflrlQr.tmp
  1009. WARNING:toil.leader:p/n/jobAsp_cK Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.29 with error: network error
  1010. WARNING:toil.leader:p/n/jobAsp_cK Uncaught exception
  1011. WARNING:toil.leader:p/n/jobAsp_cK Traceback (most recent call last):
  1012. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1013. WARNING:toil.leader:p/n/jobAsp_cK job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1014. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1015. WARNING:toil.leader:p/n/jobAsp_cK super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1016. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1017. WARNING:toil.leader:p/n/jobAsp_cK returnValues = self._run(jobGraph, fileStore)
  1018. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1019. WARNING:toil.leader:p/n/jobAsp_cK return self.run(fileStore)
  1020. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  1021. WARNING:toil.leader:p/n/jobAsp_cK makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  1022. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  1023. WARNING:toil.leader:p/n/jobAsp_cK parameters=["cactus_setup"] + args + sequences)
  1024. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1025. WARNING:toil.leader:p/n/jobAsp_cK raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1026. WARNING:toil.leader:p/n/jobAsp_cK RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="2100" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpvFu1Wj/6903afbc-0150-4141-8949-709df05797b5/tmpMzUNTZ.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpvFu1Wj/6903afbc-0150-4141-8949-709df05797b5/tmpd8lhaP.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpvFu1Wj/6903afbc-0150-4141-8949-709df05797b5/tmppY85RN.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpvFu1Wj/6903afbc-0150-4141-8949-709df05797b5/tmp5x1ero.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpvFu1Wj/6903afbc-0150-4141-8949-709df05797b5/tmpflrlQr.tmp'] failed with output:
  1027. WARNING:toil.leader:p/n/jobAsp_cK ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn39.grid.pub.ro
  1028. WARNING:toil.leader:p/n/jobAsp_cK WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' p/n/jobAsp_cK with ID p/n/jobAsp_cK to 1
  1029. INFO:toil.leader:Issued job 'CactusSetupPhase' p/n/jobAsp_cK with job batch system ID: 179 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1030. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' N/X/jobsLBpFr
  1031. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' N/X/jobsLBpFr
  1032. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' N/X/jobsLBpFr with ID N/X/jobsLBpFr to 0
  1033. WARNING:toil.leader:Job 'KtServerService' N/X/jobsLBpFr with ID N/X/jobsLBpFr is completely failed
  1034. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' p/n/jobAsp_cK
  1035. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' p/n/jobAsp_cK
  1036. WARNING:toil.leader:p/n/jobAsp_cK INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1037. WARNING:toil.leader:p/n/jobAsp_cK INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1038. WARNING:toil.leader:p/n/jobAsp_cK WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1039. WARNING:toil.leader:p/n/jobAsp_cK INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  1040. WARNING:toil.leader:p/n/jobAsp_cK INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  1041. WARNING:toil.leader:p/n/jobAsp_cK INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.40" port="15730" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpRDkROu/e60d9a66-f56f-4915-8a7b-04b75dd4aa23/tmp42OLIj.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpRDkROu/e60d9a66-f56f-4915-8a7b-04b75dd4aa23/tmpPIAyB5.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpRDkROu/e60d9a66-f56f-4915-8a7b-04b75dd4aa23/tmpSvnKmd.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpRDkROu/e60d9a66-f56f-4915-8a7b-04b75dd4aa23/tmpOR96zW.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpRDkROu/e60d9a66-f56f-4915-8a7b-04b75dd4aa23/tmpVamHA5.tmp']
  1042. WARNING:toil.leader:p/n/jobAsp_cK Set log level to INFO
  1043. WARNING:toil.leader:p/n/jobAsp_cK Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  1044. WARNING:toil.leader:p/n/jobAsp_cK <kyoto_tycoon database_dir="fakepath" host="172.16.13.40" port="15730" />
  1045. WARNING:toil.leader:p/n/jobAsp_cK </st_kv_database_conf>
  1046. WARNING:toil.leader:p/n/jobAsp_cK
  1047. WARNING:toil.leader:p/n/jobAsp_cK Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpRDkROu/e60d9a66-f56f-4915-8a7b-04b75dd4aa23/tmp42OLIj.tmp
  1048. WARNING:toil.leader:p/n/jobAsp_cK Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpRDkROu/e60d9a66-f56f-4915-8a7b-04b75dd4aa23/tmpPIAyB5.tmp
  1049. WARNING:toil.leader:p/n/jobAsp_cK Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpRDkROu/e60d9a66-f56f-4915-8a7b-04b75dd4aa23/tmpSvnKmd.tmp
  1050. WARNING:toil.leader:p/n/jobAsp_cK Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpRDkROu/e60d9a66-f56f-4915-8a7b-04b75dd4aa23/tmpOR96zW.tmp
  1051. WARNING:toil.leader:p/n/jobAsp_cK Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpRDkROu/e60d9a66-f56f-4915-8a7b-04b75dd4aa23/tmpVamHA5.tmp
  1052. WARNING:toil.leader:p/n/jobAsp_cK Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.40 with error: network error
  1053. WARNING:toil.leader:p/n/jobAsp_cK Uncaught exception
  1054. WARNING:toil.leader:p/n/jobAsp_cK Traceback (most recent call last):
  1055. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1056. WARNING:toil.leader:p/n/jobAsp_cK job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1057. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1058. WARNING:toil.leader:p/n/jobAsp_cK super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1059. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1060. WARNING:toil.leader:p/n/jobAsp_cK returnValues = self._run(jobGraph, fileStore)
  1061. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1062. WARNING:toil.leader:p/n/jobAsp_cK return self.run(fileStore)
  1063. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  1064. WARNING:toil.leader:p/n/jobAsp_cK makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  1065. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  1066. WARNING:toil.leader:p/n/jobAsp_cK parameters=["cactus_setup"] + args + sequences)
  1067. WARNING:toil.leader:p/n/jobAsp_cK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1068. WARNING:toil.leader:p/n/jobAsp_cK raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1069. WARNING:toil.leader:p/n/jobAsp_cK RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.40" port="15730" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpRDkROu/e60d9a66-f56f-4915-8a7b-04b75dd4aa23/tmp42OLIj.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpRDkROu/e60d9a66-f56f-4915-8a7b-04b75dd4aa23/tmpPIAyB5.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpRDkROu/e60d9a66-f56f-4915-8a7b-04b75dd4aa23/tmpSvnKmd.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpRDkROu/e60d9a66-f56f-4915-8a7b-04b75dd4aa23/tmpOR96zW.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpRDkROu/e60d9a66-f56f-4915-8a7b-04b75dd4aa23/tmpVamHA5.tmp'] failed with output:
  1070. WARNING:toil.leader:p/n/jobAsp_cK ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn39.grid.pub.ro
  1071. WARNING:toil.leader:p/n/jobAsp_cK WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' p/n/jobAsp_cK with ID p/n/jobAsp_cK to 0
  1072. WARNING:toil.leader:Job 'CactusSetupPhase' p/n/jobAsp_cK with ID p/n/jobAsp_cK is completely failed
  1073. WARNING:toil.leader:Job: N/8/job2cN94i is being restarted as a checkpoint after the total failure of jobs in its subtree.
  1074. INFO:toil.leader:Issued job 'StartPrimaryDB' N/8/job2cN94i with job batch system ID: 180 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1075. INFO:toil.leader:Job ended successfully: 'StartPrimaryDB' N/8/job2cN94i
  1076. INFO:toil.leader:Issued job 'KtServerService' U/V/jobiIuBle with job batch system ID: 181 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1077. INFO:toil.leader:Issued job 'CactusSetupPhase' D/S/jobfdzqEm with job batch system ID: 182 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1078. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' D/S/jobfdzqEm
  1079. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' D/S/jobfdzqEm
  1080. WARNING:toil.leader:D/S/jobfdzqEm INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1081. WARNING:toil.leader:D/S/jobfdzqEm INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1082. WARNING:toil.leader:D/S/jobfdzqEm WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1083. WARNING:toil.leader:D/S/jobfdzqEm INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  1084. WARNING:toil.leader:D/S/jobfdzqEm INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  1085. WARNING:toil.leader:D/S/jobfdzqEm INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="11145" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpOBJWOW/518fe532-50cf-4026-b9b6-8940068c5d04/tmpzEnNyU.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpOBJWOW/518fe532-50cf-4026-b9b6-8940068c5d04/tmpnV65TG.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpOBJWOW/518fe532-50cf-4026-b9b6-8940068c5d04/tmp_sXZaE.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpOBJWOW/518fe532-50cf-4026-b9b6-8940068c5d04/tmphMMGLZ.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpOBJWOW/518fe532-50cf-4026-b9b6-8940068c5d04/tmpcM_4Ev.tmp']
  1086. WARNING:toil.leader:D/S/jobfdzqEm Set log level to INFO
  1087. WARNING:toil.leader:D/S/jobfdzqEm Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  1088. WARNING:toil.leader:D/S/jobfdzqEm <kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="11145" />
  1089. WARNING:toil.leader:D/S/jobfdzqEm </st_kv_database_conf>
  1090. WARNING:toil.leader:D/S/jobfdzqEm
  1091. WARNING:toil.leader:D/S/jobfdzqEm Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpOBJWOW/518fe532-50cf-4026-b9b6-8940068c5d04/tmpzEnNyU.tmp
  1092. WARNING:toil.leader:D/S/jobfdzqEm Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpOBJWOW/518fe532-50cf-4026-b9b6-8940068c5d04/tmpnV65TG.tmp
  1093. WARNING:toil.leader:D/S/jobfdzqEm Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpOBJWOW/518fe532-50cf-4026-b9b6-8940068c5d04/tmp_sXZaE.tmp
  1094. WARNING:toil.leader:D/S/jobfdzqEm Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpOBJWOW/518fe532-50cf-4026-b9b6-8940068c5d04/tmphMMGLZ.tmp
  1095. WARNING:toil.leader:D/S/jobfdzqEm Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpOBJWOW/518fe532-50cf-4026-b9b6-8940068c5d04/tmpcM_4Ev.tmp
  1096. WARNING:toil.leader:D/S/jobfdzqEm Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.29 with error: network error
  1097. WARNING:toil.leader:D/S/jobfdzqEm Uncaught exception
  1098. WARNING:toil.leader:D/S/jobfdzqEm Traceback (most recent call last):
  1099. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1100. WARNING:toil.leader:D/S/jobfdzqEm job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1101. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1102. WARNING:toil.leader:D/S/jobfdzqEm super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1103. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1104. WARNING:toil.leader:D/S/jobfdzqEm returnValues = self._run(jobGraph, fileStore)
  1105. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1106. WARNING:toil.leader:D/S/jobfdzqEm return self.run(fileStore)
  1107. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  1108. WARNING:toil.leader:D/S/jobfdzqEm makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  1109. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  1110. WARNING:toil.leader:D/S/jobfdzqEm parameters=["cactus_setup"] + args + sequences)
  1111. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1112. WARNING:toil.leader:D/S/jobfdzqEm raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1113. WARNING:toil.leader:D/S/jobfdzqEm RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="11145" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpOBJWOW/518fe532-50cf-4026-b9b6-8940068c5d04/tmpzEnNyU.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpOBJWOW/518fe532-50cf-4026-b9b6-8940068c5d04/tmpnV65TG.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpOBJWOW/518fe532-50cf-4026-b9b6-8940068c5d04/tmp_sXZaE.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpOBJWOW/518fe532-50cf-4026-b9b6-8940068c5d04/tmphMMGLZ.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-b02a3811-2b63-4208-851d-7815af46a62d/tmpOBJWOW/518fe532-50cf-4026-b9b6-8940068c5d04/tmpcM_4Ev.tmp'] failed with output:
  1114. WARNING:toil.leader:D/S/jobfdzqEm ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn39.grid.pub.ro
  1115. WARNING:toil.leader:D/S/jobfdzqEm WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' D/S/jobfdzqEm with ID D/S/jobfdzqEm to 5
  1116. INFO:toil.leader:Issued job 'CactusSetupPhase' D/S/jobfdzqEm with job batch system ID: 183 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1117. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' U/V/jobiIuBle
  1118. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' U/V/jobiIuBle
  1119. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' U/V/jobiIuBle with ID U/V/jobiIuBle to 5
  1120. INFO:toil.leader:Issued job 'KtServerService' U/V/jobiIuBle with job batch system ID: 184 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1121. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' D/S/jobfdzqEm
  1122. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' D/S/jobfdzqEm
  1123. WARNING:toil.leader:D/S/jobfdzqEm INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1124. WARNING:toil.leader:D/S/jobfdzqEm INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1125. WARNING:toil.leader:D/S/jobfdzqEm WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1126. WARNING:toil.leader:D/S/jobfdzqEm INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  1127. WARNING:toil.leader:D/S/jobfdzqEm INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  1128. WARNING:toil.leader:D/S/jobfdzqEm INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="11145" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp2gBumA/2336f012-9b8c-4080-80f8-3a42507da3d0/tmpZf9aSe.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp2gBumA/2336f012-9b8c-4080-80f8-3a42507da3d0/tmpnYRO5M.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp2gBumA/2336f012-9b8c-4080-80f8-3a42507da3d0/tmpd7duy_.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp2gBumA/2336f012-9b8c-4080-80f8-3a42507da3d0/tmpytDCbc.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp2gBumA/2336f012-9b8c-4080-80f8-3a42507da3d0/tmp984jvG.tmp']
  1129. WARNING:toil.leader:D/S/jobfdzqEm Set log level to INFO
  1130. WARNING:toil.leader:D/S/jobfdzqEm Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  1131. WARNING:toil.leader:D/S/jobfdzqEm <kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="11145" />
  1132. WARNING:toil.leader:D/S/jobfdzqEm </st_kv_database_conf>
  1133. WARNING:toil.leader:D/S/jobfdzqEm
  1134. WARNING:toil.leader:D/S/jobfdzqEm Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp2gBumA/2336f012-9b8c-4080-80f8-3a42507da3d0/tmpZf9aSe.tmp
  1135. WARNING:toil.leader:D/S/jobfdzqEm Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp2gBumA/2336f012-9b8c-4080-80f8-3a42507da3d0/tmpnYRO5M.tmp
  1136. WARNING:toil.leader:D/S/jobfdzqEm Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp2gBumA/2336f012-9b8c-4080-80f8-3a42507da3d0/tmpd7duy_.tmp
  1137. WARNING:toil.leader:D/S/jobfdzqEm Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp2gBumA/2336f012-9b8c-4080-80f8-3a42507da3d0/tmpytDCbc.tmp
  1138. WARNING:toil.leader:D/S/jobfdzqEm Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp2gBumA/2336f012-9b8c-4080-80f8-3a42507da3d0/tmp984jvG.tmp
  1139. WARNING:toil.leader:D/S/jobfdzqEm Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.29 with error: network error
  1140. WARNING:toil.leader:D/S/jobfdzqEm Uncaught exception
  1141. WARNING:toil.leader:D/S/jobfdzqEm Traceback (most recent call last):
  1142. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1143. WARNING:toil.leader:D/S/jobfdzqEm job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1144. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1145. WARNING:toil.leader:D/S/jobfdzqEm super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1146. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1147. WARNING:toil.leader:D/S/jobfdzqEm returnValues = self._run(jobGraph, fileStore)
  1148. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1149. WARNING:toil.leader:D/S/jobfdzqEm return self.run(fileStore)
  1150. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  1151. WARNING:toil.leader:D/S/jobfdzqEm makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  1152. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  1153. WARNING:toil.leader:D/S/jobfdzqEm parameters=["cactus_setup"] + args + sequences)
  1154. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1155. WARNING:toil.leader:D/S/jobfdzqEm raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1156. WARNING:toil.leader:D/S/jobfdzqEm RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="11145" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp2gBumA/2336f012-9b8c-4080-80f8-3a42507da3d0/tmpZf9aSe.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp2gBumA/2336f012-9b8c-4080-80f8-3a42507da3d0/tmpnYRO5M.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp2gBumA/2336f012-9b8c-4080-80f8-3a42507da3d0/tmpd7duy_.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp2gBumA/2336f012-9b8c-4080-80f8-3a42507da3d0/tmpytDCbc.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp2gBumA/2336f012-9b8c-4080-80f8-3a42507da3d0/tmp984jvG.tmp'] failed with output:
  1157. WARNING:toil.leader:D/S/jobfdzqEm ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn37.grid.pub.ro
  1158. WARNING:toil.leader:D/S/jobfdzqEm WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' D/S/jobfdzqEm with ID D/S/jobfdzqEm to 4
  1159. INFO:toil.leader:Issued job 'CactusSetupPhase' D/S/jobfdzqEm with job batch system ID: 185 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1160. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' U/V/jobiIuBle
  1161. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' U/V/jobiIuBle
  1162. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' U/V/jobiIuBle with ID U/V/jobiIuBle to 4
  1163. INFO:toil.leader:Issued job 'KtServerService' U/V/jobiIuBle with job batch system ID: 186 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1164. INFO:toil.leader:Reissued any over long jobs
  1165. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' U/V/jobiIuBle
  1166. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' U/V/jobiIuBle
  1167. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' U/V/jobiIuBle with ID U/V/jobiIuBle to 3
  1168. INFO:toil.leader:Issued job 'KtServerService' U/V/jobiIuBle with job batch system ID: 187 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1169. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' D/S/jobfdzqEm
  1170. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' D/S/jobfdzqEm
  1171. WARNING:toil.leader:D/S/jobfdzqEm INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1172. WARNING:toil.leader:D/S/jobfdzqEm INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1173. WARNING:toil.leader:D/S/jobfdzqEm WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1174. WARNING:toil.leader:D/S/jobfdzqEm WARNING:toil.fileStore:Detected that job (N/8/job2cN94i/g/tmpBNdXQn-_serialiseJob-stream) prematurely terminated. Fixing the state of the job on disk.
  1175. WARNING:toil.leader:D/S/jobfdzqEm INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  1176. WARNING:toil.leader:D/S/jobfdzqEm INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  1177. WARNING:toil.leader:D/S/jobfdzqEm INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="5047" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-e0c05519-3899-4a42-9065-c10aa0c432fb/tmp7TRnci/a5884a02-4214-44e1-95be-05f60c602977/tmpmz8oeA.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-e0c05519-3899-4a42-9065-c10aa0c432fb/tmp7TRnci/a5884a02-4214-44e1-95be-05f60c602977/tmpcjhHZU.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-e0c05519-3899-4a42-9065-c10aa0c432fb/tmp7TRnci/a5884a02-4214-44e1-95be-05f60c602977/tmpy4VVNz.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-e0c05519-3899-4a42-9065-c10aa0c432fb/tmp7TRnci/a5884a02-4214-44e1-95be-05f60c602977/tmpElTJgI.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-e0c05519-3899-4a42-9065-c10aa0c432fb/tmp7TRnci/a5884a02-4214-44e1-95be-05f60c602977/tmpnR6cKC.tmp']
  1178. WARNING:toil.leader:D/S/jobfdzqEm Set log level to INFO
  1179. WARNING:toil.leader:D/S/jobfdzqEm Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  1180. WARNING:toil.leader:D/S/jobfdzqEm <kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="5047" />
  1181. WARNING:toil.leader:D/S/jobfdzqEm </st_kv_database_conf>
  1182. WARNING:toil.leader:D/S/jobfdzqEm
  1183. WARNING:toil.leader:D/S/jobfdzqEm Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-e0c05519-3899-4a42-9065-c10aa0c432fb/tmp7TRnci/a5884a02-4214-44e1-95be-05f60c602977/tmpmz8oeA.tmp
  1184. WARNING:toil.leader:D/S/jobfdzqEm Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-e0c05519-3899-4a42-9065-c10aa0c432fb/tmp7TRnci/a5884a02-4214-44e1-95be-05f60c602977/tmpcjhHZU.tmp
  1185. WARNING:toil.leader:D/S/jobfdzqEm Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-e0c05519-3899-4a42-9065-c10aa0c432fb/tmp7TRnci/a5884a02-4214-44e1-95be-05f60c602977/tmpy4VVNz.tmp
  1186. WARNING:toil.leader:D/S/jobfdzqEm Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-e0c05519-3899-4a42-9065-c10aa0c432fb/tmp7TRnci/a5884a02-4214-44e1-95be-05f60c602977/tmpElTJgI.tmp
  1187. WARNING:toil.leader:D/S/jobfdzqEm Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-e0c05519-3899-4a42-9065-c10aa0c432fb/tmp7TRnci/a5884a02-4214-44e1-95be-05f60c602977/tmpnR6cKC.tmp
  1188. WARNING:toil.leader:D/S/jobfdzqEm Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.35 with error: network error
  1189. WARNING:toil.leader:D/S/jobfdzqEm Uncaught exception
  1190. WARNING:toil.leader:D/S/jobfdzqEm Traceback (most recent call last):
  1191. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1192. WARNING:toil.leader:D/S/jobfdzqEm job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1193. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1194. WARNING:toil.leader:D/S/jobfdzqEm super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1195. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1196. WARNING:toil.leader:D/S/jobfdzqEm returnValues = self._run(jobGraph, fileStore)
  1197. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1198. WARNING:toil.leader:D/S/jobfdzqEm return self.run(fileStore)
  1199. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  1200. WARNING:toil.leader:D/S/jobfdzqEm makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  1201. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  1202. WARNING:toil.leader:D/S/jobfdzqEm parameters=["cactus_setup"] + args + sequences)
  1203. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1204. WARNING:toil.leader:D/S/jobfdzqEm raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1205. WARNING:toil.leader:D/S/jobfdzqEm RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="5047" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-e0c05519-3899-4a42-9065-c10aa0c432fb/tmp7TRnci/a5884a02-4214-44e1-95be-05f60c602977/tmpmz8oeA.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-e0c05519-3899-4a42-9065-c10aa0c432fb/tmp7TRnci/a5884a02-4214-44e1-95be-05f60c602977/tmpcjhHZU.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-e0c05519-3899-4a42-9065-c10aa0c432fb/tmp7TRnci/a5884a02-4214-44e1-95be-05f60c602977/tmpy4VVNz.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-e0c05519-3899-4a42-9065-c10aa0c432fb/tmp7TRnci/a5884a02-4214-44e1-95be-05f60c602977/tmpElTJgI.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-e0c05519-3899-4a42-9065-c10aa0c432fb/tmp7TRnci/a5884a02-4214-44e1-95be-05f60c602977/tmpnR6cKC.tmp'] failed with output:
  1206. WARNING:toil.leader:D/S/jobfdzqEm ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn40.grid.pub.ro
  1207. WARNING:toil.leader:D/S/jobfdzqEm WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' D/S/jobfdzqEm with ID D/S/jobfdzqEm to 3
  1208. INFO:toil.leader:Issued job 'CactusSetupPhase' D/S/jobfdzqEm with job batch system ID: 188 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1209. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' U/V/jobiIuBle
  1210. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' U/V/jobiIuBle
  1211. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' U/V/jobiIuBle with ID U/V/jobiIuBle to 2
  1212. INFO:toil.leader:Issued job 'KtServerService' U/V/jobiIuBle with job batch system ID: 189 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1213. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' D/S/jobfdzqEm
  1214. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' D/S/jobfdzqEm
  1215. WARNING:toil.leader:D/S/jobfdzqEm INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1216. WARNING:toil.leader:D/S/jobfdzqEm INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1217. WARNING:toil.leader:D/S/jobfdzqEm WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1218. WARNING:toil.leader:D/S/jobfdzqEm WARNING:toil.fileStore:Detected that job (N/8/job2cN94i/g/tmplNnAXY-_serialiseJob-stream) prematurely terminated. Fixing the state of the job on disk.
  1219. WARNING:toil.leader:D/S/jobfdzqEm INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  1220. WARNING:toil.leader:D/S/jobfdzqEm INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  1221. WARNING:toil.leader:D/S/jobfdzqEm INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.30" port="12500" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpoSAhDg/7218547f-18f7-4f4b-bea2-bb46fba07981/tmpALbXf6.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpoSAhDg/7218547f-18f7-4f4b-bea2-bb46fba07981/tmpH9GXZa.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpoSAhDg/7218547f-18f7-4f4b-bea2-bb46fba07981/tmpLTEXPY.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpoSAhDg/7218547f-18f7-4f4b-bea2-bb46fba07981/tmpMLc40g.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpoSAhDg/7218547f-18f7-4f4b-bea2-bb46fba07981/tmpyNtZPj.tmp']
  1222. WARNING:toil.leader:D/S/jobfdzqEm Set log level to INFO
  1223. WARNING:toil.leader:D/S/jobfdzqEm Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  1224. WARNING:toil.leader:D/S/jobfdzqEm <kyoto_tycoon database_dir="fakepath" host="172.16.13.30" port="12500" />
  1225. WARNING:toil.leader:D/S/jobfdzqEm </st_kv_database_conf>
  1226. WARNING:toil.leader:D/S/jobfdzqEm
  1227. WARNING:toil.leader:D/S/jobfdzqEm Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpoSAhDg/7218547f-18f7-4f4b-bea2-bb46fba07981/tmpALbXf6.tmp
  1228. WARNING:toil.leader:D/S/jobfdzqEm Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpoSAhDg/7218547f-18f7-4f4b-bea2-bb46fba07981/tmpH9GXZa.tmp
  1229. WARNING:toil.leader:D/S/jobfdzqEm Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpoSAhDg/7218547f-18f7-4f4b-bea2-bb46fba07981/tmpLTEXPY.tmp
  1230. WARNING:toil.leader:D/S/jobfdzqEm Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpoSAhDg/7218547f-18f7-4f4b-bea2-bb46fba07981/tmpMLc40g.tmp
  1231. WARNING:toil.leader:D/S/jobfdzqEm Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpoSAhDg/7218547f-18f7-4f4b-bea2-bb46fba07981/tmpyNtZPj.tmp
  1232. WARNING:toil.leader:D/S/jobfdzqEm Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.30 with error: network error
  1233. WARNING:toil.leader:D/S/jobfdzqEm Uncaught exception
  1234. WARNING:toil.leader:D/S/jobfdzqEm Traceback (most recent call last):
  1235. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1236. WARNING:toil.leader:D/S/jobfdzqEm job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1237. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1238. WARNING:toil.leader:D/S/jobfdzqEm super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1239. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1240. WARNING:toil.leader:D/S/jobfdzqEm returnValues = self._run(jobGraph, fileStore)
  1241. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1242. WARNING:toil.leader:D/S/jobfdzqEm return self.run(fileStore)
  1243. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  1244. WARNING:toil.leader:D/S/jobfdzqEm makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  1245. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  1246. WARNING:toil.leader:D/S/jobfdzqEm parameters=["cactus_setup"] + args + sequences)
  1247. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1248. WARNING:toil.leader:D/S/jobfdzqEm raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1249. WARNING:toil.leader:D/S/jobfdzqEm RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.30" port="12500" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpoSAhDg/7218547f-18f7-4f4b-bea2-bb46fba07981/tmpALbXf6.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpoSAhDg/7218547f-18f7-4f4b-bea2-bb46fba07981/tmpH9GXZa.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpoSAhDg/7218547f-18f7-4f4b-bea2-bb46fba07981/tmpLTEXPY.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpoSAhDg/7218547f-18f7-4f4b-bea2-bb46fba07981/tmpMLc40g.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpoSAhDg/7218547f-18f7-4f4b-bea2-bb46fba07981/tmpyNtZPj.tmp'] failed with output:
  1250. WARNING:toil.leader:D/S/jobfdzqEm ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn29.grid.pub.ro
  1251. WARNING:toil.leader:D/S/jobfdzqEm WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' D/S/jobfdzqEm with ID D/S/jobfdzqEm to 2
  1252. INFO:toil.leader:Issued job 'CactusSetupPhase' D/S/jobfdzqEm with job batch system ID: 190 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1253. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' U/V/jobiIuBle
  1254. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' U/V/jobiIuBle
  1255. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' U/V/jobiIuBle with ID U/V/jobiIuBle to 1
  1256. INFO:toil.leader:Issued job 'KtServerService' U/V/jobiIuBle with job batch system ID: 191 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1257. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' D/S/jobfdzqEm
  1258. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' D/S/jobfdzqEm
  1259. WARNING:toil.leader:D/S/jobfdzqEm INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1260. WARNING:toil.leader:D/S/jobfdzqEm INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1261. WARNING:toil.leader:D/S/jobfdzqEm WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1262. WARNING:toil.leader:D/S/jobfdzqEm WARNING:toil.fileStore:Detected that job (N/8/job2cN94i/g/tmplNnAXY-_serialiseJob-stream) prematurely terminated. Fixing the state of the job on disk.
  1263. WARNING:toil.leader:D/S/jobfdzqEm INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  1264. WARNING:toil.leader:D/S/jobfdzqEm INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  1265. WARNING:toil.leader:D/S/jobfdzqEm INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.30" port="12500" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmp3MCHQK/5a6adb54-d54e-4ba7-ae78-7a085907b896/tmplyzwgw.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmp3MCHQK/5a6adb54-d54e-4ba7-ae78-7a085907b896/tmpxErNL4.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmp3MCHQK/5a6adb54-d54e-4ba7-ae78-7a085907b896/tmpTu1XbK.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmp3MCHQK/5a6adb54-d54e-4ba7-ae78-7a085907b896/tmpzKjcRb.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmp3MCHQK/5a6adb54-d54e-4ba7-ae78-7a085907b896/tmpoEkVIR.tmp']
  1266. WARNING:toil.leader:D/S/jobfdzqEm Set log level to INFO
  1267. WARNING:toil.leader:D/S/jobfdzqEm Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  1268. WARNING:toil.leader:D/S/jobfdzqEm <kyoto_tycoon database_dir="fakepath" host="172.16.13.30" port="12500" />
  1269. WARNING:toil.leader:D/S/jobfdzqEm </st_kv_database_conf>
  1270. WARNING:toil.leader:D/S/jobfdzqEm
  1271. WARNING:toil.leader:D/S/jobfdzqEm Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmp3MCHQK/5a6adb54-d54e-4ba7-ae78-7a085907b896/tmplyzwgw.tmp
  1272. WARNING:toil.leader:D/S/jobfdzqEm Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmp3MCHQK/5a6adb54-d54e-4ba7-ae78-7a085907b896/tmpxErNL4.tmp
  1273. WARNING:toil.leader:D/S/jobfdzqEm Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmp3MCHQK/5a6adb54-d54e-4ba7-ae78-7a085907b896/tmpTu1XbK.tmp
  1274. WARNING:toil.leader:D/S/jobfdzqEm Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmp3MCHQK/5a6adb54-d54e-4ba7-ae78-7a085907b896/tmpzKjcRb.tmp
  1275. WARNING:toil.leader:D/S/jobfdzqEm Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmp3MCHQK/5a6adb54-d54e-4ba7-ae78-7a085907b896/tmpoEkVIR.tmp
  1276. WARNING:toil.leader:D/S/jobfdzqEm Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.30 with error: network error
  1277. WARNING:toil.leader:D/S/jobfdzqEm Uncaught exception
  1278. WARNING:toil.leader:D/S/jobfdzqEm Traceback (most recent call last):
  1279. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1280. WARNING:toil.leader:D/S/jobfdzqEm job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1281. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1282. WARNING:toil.leader:D/S/jobfdzqEm super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1283. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1284. WARNING:toil.leader:D/S/jobfdzqEm returnValues = self._run(jobGraph, fileStore)
  1285. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1286. WARNING:toil.leader:D/S/jobfdzqEm return self.run(fileStore)
  1287. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  1288. WARNING:toil.leader:D/S/jobfdzqEm makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  1289. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  1290. WARNING:toil.leader:D/S/jobfdzqEm parameters=["cactus_setup"] + args + sequences)
  1291. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1292. WARNING:toil.leader:D/S/jobfdzqEm raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1293. WARNING:toil.leader:D/S/jobfdzqEm RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.30" port="12500" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmp3MCHQK/5a6adb54-d54e-4ba7-ae78-7a085907b896/tmplyzwgw.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmp3MCHQK/5a6adb54-d54e-4ba7-ae78-7a085907b896/tmpxErNL4.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmp3MCHQK/5a6adb54-d54e-4ba7-ae78-7a085907b896/tmpTu1XbK.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmp3MCHQK/5a6adb54-d54e-4ba7-ae78-7a085907b896/tmpzKjcRb.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmp3MCHQK/5a6adb54-d54e-4ba7-ae78-7a085907b896/tmpoEkVIR.tmp'] failed with output:
  1294. WARNING:toil.leader:D/S/jobfdzqEm ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn35.grid.pub.ro
  1295. WARNING:toil.leader:D/S/jobfdzqEm WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' D/S/jobfdzqEm with ID D/S/jobfdzqEm to 1
  1296. INFO:toil.leader:Issued job 'CactusSetupPhase' D/S/jobfdzqEm with job batch system ID: 192 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1297. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' U/V/jobiIuBle
  1298. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' U/V/jobiIuBle
  1299. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' U/V/jobiIuBle with ID U/V/jobiIuBle to 0
  1300. WARNING:toil.leader:Job 'KtServerService' U/V/jobiIuBle with ID U/V/jobiIuBle is completely failed
  1301. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' D/S/jobfdzqEm
  1302. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' D/S/jobfdzqEm
  1303. WARNING:toil.leader:D/S/jobfdzqEm INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1304. WARNING:toil.leader:D/S/jobfdzqEm INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1305. WARNING:toil.leader:D/S/jobfdzqEm WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1306. WARNING:toil.leader:D/S/jobfdzqEm INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  1307. WARNING:toil.leader:D/S/jobfdzqEm INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  1308. WARNING:toil.leader:D/S/jobfdzqEm INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="31688" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpys3qfX/e58837ce-ff43-4d24-aff0-086f9b126984/tmprnbOS8.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpys3qfX/e58837ce-ff43-4d24-aff0-086f9b126984/tmpmvuwio.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpys3qfX/e58837ce-ff43-4d24-aff0-086f9b126984/tmpEKriJd.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpys3qfX/e58837ce-ff43-4d24-aff0-086f9b126984/tmpzohi_c.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpys3qfX/e58837ce-ff43-4d24-aff0-086f9b126984/tmprOqkGM.tmp']
  1309. WARNING:toil.leader:D/S/jobfdzqEm Set log level to INFO
  1310. WARNING:toil.leader:D/S/jobfdzqEm Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  1311. WARNING:toil.leader:D/S/jobfdzqEm <kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="31688" />
  1312. WARNING:toil.leader:D/S/jobfdzqEm </st_kv_database_conf>
  1313. WARNING:toil.leader:D/S/jobfdzqEm
  1314. WARNING:toil.leader:D/S/jobfdzqEm Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpys3qfX/e58837ce-ff43-4d24-aff0-086f9b126984/tmprnbOS8.tmp
  1315. WARNING:toil.leader:D/S/jobfdzqEm Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpys3qfX/e58837ce-ff43-4d24-aff0-086f9b126984/tmpmvuwio.tmp
  1316. WARNING:toil.leader:D/S/jobfdzqEm Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpys3qfX/e58837ce-ff43-4d24-aff0-086f9b126984/tmpEKriJd.tmp
  1317. WARNING:toil.leader:D/S/jobfdzqEm Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpys3qfX/e58837ce-ff43-4d24-aff0-086f9b126984/tmpzohi_c.tmp
  1318. WARNING:toil.leader:D/S/jobfdzqEm Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpys3qfX/e58837ce-ff43-4d24-aff0-086f9b126984/tmprOqkGM.tmp
  1319. WARNING:toil.leader:D/S/jobfdzqEm Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.29 with error: network error
  1320. WARNING:toil.leader:D/S/jobfdzqEm Uncaught exception
  1321. WARNING:toil.leader:D/S/jobfdzqEm Traceback (most recent call last):
  1322. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1323. WARNING:toil.leader:D/S/jobfdzqEm job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1324. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1325. WARNING:toil.leader:D/S/jobfdzqEm super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1326. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1327. WARNING:toil.leader:D/S/jobfdzqEm returnValues = self._run(jobGraph, fileStore)
  1328. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1329. WARNING:toil.leader:D/S/jobfdzqEm return self.run(fileStore)
  1330. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  1331. WARNING:toil.leader:D/S/jobfdzqEm makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  1332. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  1333. WARNING:toil.leader:D/S/jobfdzqEm parameters=["cactus_setup"] + args + sequences)
  1334. WARNING:toil.leader:D/S/jobfdzqEm File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1335. WARNING:toil.leader:D/S/jobfdzqEm raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1336. WARNING:toil.leader:D/S/jobfdzqEm RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="31688" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpys3qfX/e58837ce-ff43-4d24-aff0-086f9b126984/tmprnbOS8.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpys3qfX/e58837ce-ff43-4d24-aff0-086f9b126984/tmpmvuwio.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpys3qfX/e58837ce-ff43-4d24-aff0-086f9b126984/tmpEKriJd.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpys3qfX/e58837ce-ff43-4d24-aff0-086f9b126984/tmpzohi_c.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpys3qfX/e58837ce-ff43-4d24-aff0-086f9b126984/tmprOqkGM.tmp'] failed with output:
  1337. WARNING:toil.leader:D/S/jobfdzqEm ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn34.grid.pub.ro
  1338. WARNING:toil.leader:D/S/jobfdzqEm WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' D/S/jobfdzqEm with ID D/S/jobfdzqEm to 0
  1339. WARNING:toil.leader:Job 'CactusSetupPhase' D/S/jobfdzqEm with ID D/S/jobfdzqEm is completely failed
  1340. WARNING:toil.leader:Job: N/8/job2cN94i is being restarted as a checkpoint after the total failure of jobs in its subtree.
  1341. INFO:toil.leader:Issued job 'StartPrimaryDB' N/8/job2cN94i with job batch system ID: 193 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1342. INFO:toil.leader:Job ended successfully: 'StartPrimaryDB' N/8/job2cN94i
  1343. INFO:toil.leader:Issued job 'KtServerService' e/K/jobxqb0Y0 with job batch system ID: 194 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1344. INFO:toil.leader:Issued job 'CactusSetupPhase' n/P/jobc3lN0J with job batch system ID: 195 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1345. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' e/K/jobxqb0Y0
  1346. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' e/K/jobxqb0Y0
  1347. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' e/K/jobxqb0Y0 with ID e/K/jobxqb0Y0 to 5
  1348. INFO:toil.leader:Issued job 'KtServerService' e/K/jobxqb0Y0 with job batch system ID: 196 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1349. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' n/P/jobc3lN0J
  1350. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' n/P/jobc3lN0J
  1351. WARNING:toil.leader:n/P/jobc3lN0J INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1352. WARNING:toil.leader:n/P/jobc3lN0J INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1353. WARNING:toil.leader:n/P/jobc3lN0J WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1354. WARNING:toil.leader:n/P/jobc3lN0J INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  1355. WARNING:toil.leader:n/P/jobc3lN0J INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  1356. WARNING:toil.leader:n/P/jobc3lN0J INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.34" port="11956" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpi_ZUzO/d0a1914a-47ef-4fe4-95dd-a86925ca9022/tmppP9JxX.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpi_ZUzO/d0a1914a-47ef-4fe4-95dd-a86925ca9022/tmptYuXfk.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpi_ZUzO/d0a1914a-47ef-4fe4-95dd-a86925ca9022/tmpyyfDcn.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpi_ZUzO/d0a1914a-47ef-4fe4-95dd-a86925ca9022/tmpDKyAJI.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpi_ZUzO/d0a1914a-47ef-4fe4-95dd-a86925ca9022/tmplKzDyh.tmp']
  1357. WARNING:toil.leader:n/P/jobc3lN0J Set log level to INFO
  1358. WARNING:toil.leader:n/P/jobc3lN0J Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  1359. WARNING:toil.leader:n/P/jobc3lN0J <kyoto_tycoon database_dir="fakepath" host="172.16.13.34" port="11956" />
  1360. WARNING:toil.leader:n/P/jobc3lN0J </st_kv_database_conf>
  1361. WARNING:toil.leader:n/P/jobc3lN0J
  1362. WARNING:toil.leader:n/P/jobc3lN0J Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpi_ZUzO/d0a1914a-47ef-4fe4-95dd-a86925ca9022/tmppP9JxX.tmp
  1363. WARNING:toil.leader:n/P/jobc3lN0J Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpi_ZUzO/d0a1914a-47ef-4fe4-95dd-a86925ca9022/tmptYuXfk.tmp
  1364. WARNING:toil.leader:n/P/jobc3lN0J Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpi_ZUzO/d0a1914a-47ef-4fe4-95dd-a86925ca9022/tmpyyfDcn.tmp
  1365. WARNING:toil.leader:n/P/jobc3lN0J Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpi_ZUzO/d0a1914a-47ef-4fe4-95dd-a86925ca9022/tmpDKyAJI.tmp
  1366. WARNING:toil.leader:n/P/jobc3lN0J Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpi_ZUzO/d0a1914a-47ef-4fe4-95dd-a86925ca9022/tmplKzDyh.tmp
  1367. WARNING:toil.leader:n/P/jobc3lN0J Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.34 with error: network error
  1368. WARNING:toil.leader:n/P/jobc3lN0J Uncaught exception
  1369. WARNING:toil.leader:n/P/jobc3lN0J Traceback (most recent call last):
  1370. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1371. WARNING:toil.leader:n/P/jobc3lN0J job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1372. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1373. WARNING:toil.leader:n/P/jobc3lN0J super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1374. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1375. WARNING:toil.leader:n/P/jobc3lN0J returnValues = self._run(jobGraph, fileStore)
  1376. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1377. WARNING:toil.leader:n/P/jobc3lN0J return self.run(fileStore)
  1378. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  1379. WARNING:toil.leader:n/P/jobc3lN0J makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  1380. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  1381. WARNING:toil.leader:n/P/jobc3lN0J parameters=["cactus_setup"] + args + sequences)
  1382. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1383. WARNING:toil.leader:n/P/jobc3lN0J raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1384. WARNING:toil.leader:n/P/jobc3lN0J RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.34" port="11956" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpi_ZUzO/d0a1914a-47ef-4fe4-95dd-a86925ca9022/tmppP9JxX.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpi_ZUzO/d0a1914a-47ef-4fe4-95dd-a86925ca9022/tmptYuXfk.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpi_ZUzO/d0a1914a-47ef-4fe4-95dd-a86925ca9022/tmpyyfDcn.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpi_ZUzO/d0a1914a-47ef-4fe4-95dd-a86925ca9022/tmpDKyAJI.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpi_ZUzO/d0a1914a-47ef-4fe4-95dd-a86925ca9022/tmplKzDyh.tmp'] failed with output:
  1385. WARNING:toil.leader:n/P/jobc3lN0J ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn38.grid.pub.ro
  1386. WARNING:toil.leader:n/P/jobc3lN0J WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' n/P/jobc3lN0J with ID n/P/jobc3lN0J to 5
  1387. INFO:toil.leader:Issued job 'CactusSetupPhase' n/P/jobc3lN0J with job batch system ID: 197 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1388. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' e/K/jobxqb0Y0
  1389. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' e/K/jobxqb0Y0
  1390. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' e/K/jobxqb0Y0 with ID e/K/jobxqb0Y0 to 4
  1391. INFO:toil.leader:Issued job 'KtServerService' e/K/jobxqb0Y0 with job batch system ID: 198 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1392. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' n/P/jobc3lN0J
  1393. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' n/P/jobc3lN0J
  1394. WARNING:toil.leader:n/P/jobc3lN0J INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1395. WARNING:toil.leader:n/P/jobc3lN0J INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1396. WARNING:toil.leader:n/P/jobc3lN0J WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1397. WARNING:toil.leader:n/P/jobc3lN0J INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  1398. WARNING:toil.leader:n/P/jobc3lN0J INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  1399. WARNING:toil.leader:n/P/jobc3lN0J INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.39" port="30151" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-d3c169df-9b53-43e5-8eff-b713994e3764/tmpj43jOo/8b2dfae6-0e35-445b-b96e-58bcda4d4107/tmpNzz9BQ.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-d3c169df-9b53-43e5-8eff-b713994e3764/tmpj43jOo/8b2dfae6-0e35-445b-b96e-58bcda4d4107/tmpLwd5xr.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-d3c169df-9b53-43e5-8eff-b713994e3764/tmpj43jOo/8b2dfae6-0e35-445b-b96e-58bcda4d4107/tmp8HZ0JB.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-d3c169df-9b53-43e5-8eff-b713994e3764/tmpj43jOo/8b2dfae6-0e35-445b-b96e-58bcda4d4107/tmpyRHFIc.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-d3c169df-9b53-43e5-8eff-b713994e3764/tmpj43jOo/8b2dfae6-0e35-445b-b96e-58bcda4d4107/tmpxTD_ZS.tmp']
  1400. WARNING:toil.leader:n/P/jobc3lN0J Set log level to INFO
  1401. WARNING:toil.leader:n/P/jobc3lN0J Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  1402. WARNING:toil.leader:n/P/jobc3lN0J <kyoto_tycoon database_dir="fakepath" host="172.16.13.39" port="30151" />
  1403. WARNING:toil.leader:n/P/jobc3lN0J </st_kv_database_conf>
  1404. WARNING:toil.leader:n/P/jobc3lN0J
  1405. WARNING:toil.leader:n/P/jobc3lN0J Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-d3c169df-9b53-43e5-8eff-b713994e3764/tmpj43jOo/8b2dfae6-0e35-445b-b96e-58bcda4d4107/tmpNzz9BQ.tmp
  1406. WARNING:toil.leader:n/P/jobc3lN0J Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-d3c169df-9b53-43e5-8eff-b713994e3764/tmpj43jOo/8b2dfae6-0e35-445b-b96e-58bcda4d4107/tmpLwd5xr.tmp
  1407. WARNING:toil.leader:n/P/jobc3lN0J Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-d3c169df-9b53-43e5-8eff-b713994e3764/tmpj43jOo/8b2dfae6-0e35-445b-b96e-58bcda4d4107/tmp8HZ0JB.tmp
  1408. WARNING:toil.leader:n/P/jobc3lN0J Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-d3c169df-9b53-43e5-8eff-b713994e3764/tmpj43jOo/8b2dfae6-0e35-445b-b96e-58bcda4d4107/tmpyRHFIc.tmp
  1409. WARNING:toil.leader:n/P/jobc3lN0J Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-d3c169df-9b53-43e5-8eff-b713994e3764/tmpj43jOo/8b2dfae6-0e35-445b-b96e-58bcda4d4107/tmpxTD_ZS.tmp
  1410. WARNING:toil.leader:n/P/jobc3lN0J Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.39 with error: network error
  1411. WARNING:toil.leader:n/P/jobc3lN0J Uncaught exception
  1412. WARNING:toil.leader:n/P/jobc3lN0J Traceback (most recent call last):
  1413. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1414. WARNING:toil.leader:n/P/jobc3lN0J job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1415. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1416. WARNING:toil.leader:n/P/jobc3lN0J super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1417. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1418. WARNING:toil.leader:n/P/jobc3lN0J returnValues = self._run(jobGraph, fileStore)
  1419. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1420. WARNING:toil.leader:n/P/jobc3lN0J return self.run(fileStore)
  1421. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  1422. WARNING:toil.leader:n/P/jobc3lN0J makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  1423. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  1424. WARNING:toil.leader:n/P/jobc3lN0J parameters=["cactus_setup"] + args + sequences)
  1425. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1426. WARNING:toil.leader:n/P/jobc3lN0J raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1427. WARNING:toil.leader:n/P/jobc3lN0J RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.39" port="30151" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-d3c169df-9b53-43e5-8eff-b713994e3764/tmpj43jOo/8b2dfae6-0e35-445b-b96e-58bcda4d4107/tmpNzz9BQ.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-d3c169df-9b53-43e5-8eff-b713994e3764/tmpj43jOo/8b2dfae6-0e35-445b-b96e-58bcda4d4107/tmpLwd5xr.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-d3c169df-9b53-43e5-8eff-b713994e3764/tmpj43jOo/8b2dfae6-0e35-445b-b96e-58bcda4d4107/tmp8HZ0JB.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-d3c169df-9b53-43e5-8eff-b713994e3764/tmpj43jOo/8b2dfae6-0e35-445b-b96e-58bcda4d4107/tmpyRHFIc.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-d3c169df-9b53-43e5-8eff-b713994e3764/tmpj43jOo/8b2dfae6-0e35-445b-b96e-58bcda4d4107/tmpxTD_ZS.tmp'] failed with output:
  1428. WARNING:toil.leader:n/P/jobc3lN0J ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn32.grid.pub.ro
  1429. WARNING:toil.leader:n/P/jobc3lN0J WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' n/P/jobc3lN0J with ID n/P/jobc3lN0J to 4
  1430. INFO:toil.leader:Issued job 'CactusSetupPhase' n/P/jobc3lN0J with job batch system ID: 199 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1431. INFO:toil.leader:Job ended successfully: 'KtServerService' e/K/jobxqb0Y0
  1432. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'KtServerService' e/K/jobxqb0Y0
  1433. WARNING:toil.leader:e/K/jobxqb0Y0 INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1434. WARNING:toil.leader:e/K/jobxqb0Y0 INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1435. WARNING:toil.leader:e/K/jobxqb0Y0 WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1436. WARNING:toil.leader:e/K/jobxqb0Y0 WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1437. WARNING:toil.leader:e/K/jobxqb0Y0 INFO:cactus.shared.common:Running the command ['netstat', '-tuplen']
  1438. WARNING:toil.leader:e/K/jobxqb0Y0 (No info could be read for "-p": geteuid()=98354 but you should be root.)
  1439. WARNING:toil.leader:e/K/jobxqb0Y0 INFO:cactus.shared.common:Running the command ['ktserver', '-port', '14688', '-ls', '-tout', '200000', '-th', '64', '-bgs', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpFGj7mF/2f988771-92a3-42c4-9825-68431a4e398a/tzi_GRE/snapshot', '-bgsc', 'lzo', '-bgsi', '1000000', '-log', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpFGj7mF/2f988771-92a3-42c4-9825-68431a4e398a/tmppsDOzD.tmp', ':#opts=ls#bnum=30m#msiz=50g#ktopts=p']
  1440. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1441. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1442. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1443. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 511
  1444. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1445. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1446. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1447. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 511
  1448. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1449. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1450. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1451. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 512
  1452. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1453. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1454. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1455. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 513
  1456. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1457. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1458. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1459. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 514
  1460. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1461. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1462. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1463. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 515
  1464. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1465. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1466. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1467. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 516
  1468. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1469. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1470. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1471. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 517
  1472. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1473. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1474. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1475. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 518
  1476. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1477. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1478. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1479. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 519
  1480. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1481. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1482. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1483. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 520
  1484. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1485. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1486. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1487. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 521
  1488. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1489. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1490. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1491. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 522
  1492. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1493. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1494. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1495. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 523
  1496. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1497. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1498. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1499. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 524
  1500. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1501. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1502. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1503. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 525
  1504. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1505. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1506. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1507. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 526
  1508. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1509. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1510. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1511. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 527
  1512. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1513. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1514. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1515. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 528
  1516. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1517. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1518. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1519. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 529
  1520. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1521. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1522. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1523. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 530
  1524. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1525. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1526. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1527. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 531
  1528. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1529. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1530. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1531. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 532
  1532. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1533. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1534. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1535. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 533
  1536. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1537. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1538. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1539. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 534
  1540. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1541. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1542. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1543. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 535
  1544. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1545. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1546. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1547. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 536
  1548. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1549. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1550. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1551. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 537
  1552. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1553. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1554. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1555. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 538
  1556. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1557. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1558. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1559. WARNING:toil.leader:e/K/jobxqb0Y0 INFO:toil.lib.bioio:Ktserver running.
  1560. WARNING:toil.leader:e/K/jobxqb0Y0 INFO:toil.lib.bioio:Ktserver running.
  1561. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 539
  1562. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1563. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1564. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1565. WARNING:toil.leader:e/K/jobxqb0Y0 INFO:toil.lib.bioio:Ktserver running.
  1566. WARNING:toil.leader:e/K/jobxqb0Y0 INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '14688', '-host', '172.16.13.38', 'TERMINATE']
  1567. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 541
  1568. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1569. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1570. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1571. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 542
  1572. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1573. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1574. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1575. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 543
  1576. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1577. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1578. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1579. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 544
  1580. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1581. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1582. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1583. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 545
  1584. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1585. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1586. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1587. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 546
  1588. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1589. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1590. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1591. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 547
  1592. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1593. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1594. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1595. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 548
  1596. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1597. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1598. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1599. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 549
  1600. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1601. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1602. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1603. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 550
  1604. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1605. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1606. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1607. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 551
  1608. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1609. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1610. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1611. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 552
  1612. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1613. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1614. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1615. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 553
  1616. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1617. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1618. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1619. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 554
  1620. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1621. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1622. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1623. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 555
  1624. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1625. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1626. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1627. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 556
  1628. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1629. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1630. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1631. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 557
  1632. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1633. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1634. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1635. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 558
  1636. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1637. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1638. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1639. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 559
  1640. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1641. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1642. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1643. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 560
  1644. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1645. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1646. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1647. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 561
  1648. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1649. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1650. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1651. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 562
  1652. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1653. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1654. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1655. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 563
  1656. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1657. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1658. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1659. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 564
  1660. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1661. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1662. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1663. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 565
  1664. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1665. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1666. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1667. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 566
  1668. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1669. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1670. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1671. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 567
  1672. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1673. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1674. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1675. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 568
  1676. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1677. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1678. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1679. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 569
  1680. WARNING:toil.leader:e/K/jobxqb0Y0 Stack limits: Infinity Infinity
  1681. WARNING:toil.leader:e/K/jobxqb0Y0 Process limits: 510853 510853
  1682. WARNING:toil.leader:e/K/jobxqb0Y0 Kernel thread cap: 1021706
  1683. WARNING:toil.leader:e/K/jobxqb0Y0 Current total threads running: 570
  1684. WARNING:toil.leader:e/K/jobxqb0Y0 terminate called after throwing an instance of 'std::runtime_error'
  1685. WARNING:toil.leader:e/K/jobxqb0Y0 what(): pthread_create: EAGAIN: Insufficient resources to create thread, or thread limit reached
  1686. WARNING:toil.leader:e/K/jobxqb0Y0 Process ServerProcess-1:
  1687. WARNING:toil.leader:e/K/jobxqb0Y0 Traceback (most recent call last):
  1688. WARNING:toil.leader:e/K/jobxqb0Y0 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap
  1689. WARNING:toil.leader:e/K/jobxqb0Y0 self.run()
  1690. WARNING:toil.leader:e/K/jobxqb0Y0 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/ktserverControl.py", line 82, in run
  1691. WARNING:toil.leader:e/K/jobxqb0Y0 self.tryRun(*self.args, **self.kwargs)
  1692. WARNING:toil.leader:e/K/jobxqb0Y0 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/ktserverControl.py", line 118, in tryRun
  1693. WARNING:toil.leader:e/K/jobxqb0Y0 raise RuntimeError("KTServer failed. Log: %s" % f.read())
  1694. WARNING:toil.leader:e/K/jobxqb0Y0 RuntimeError: KTServer failed. Log: 2019-03-16T22:06:41.524770+02:00: [SYSTEM]: ================ [START]: pid=27365
  1695. WARNING:toil.leader:e/K/jobxqb0Y0 2019-03-16T22:06:41.524983+02:00: [SYSTEM]: opening a database: path=:#opts=ls#bnum=30m#msiz=50g#ktopts=p
  1696. WARNING:toil.leader:e/K/jobxqb0Y0 2019-03-16T22:06:41.566580+02:00: [SYSTEM]: starting the server: expr=:14688
  1697. WARNING:toil.leader:e/K/jobxqb0Y0 2019-03-16T22:06:41.566704+02:00: [SYSTEM]: server socket opened: expr=:14688 timeout=200000.0
  1698. WARNING:toil.leader:e/K/jobxqb0Y0 2019-03-16T22:06:41.566731+02:00: [SYSTEM]: listening server socket started: fd=4
  1699. WARNING:toil.leader:e/K/jobxqb0Y0
  1700. WARNING:toil.leader:e/K/jobxqb0Y0 INFO:cactus.shared.common:Running the command ['ktremotemgr', 'set', '-port', '14688', '-host', '172.16.13.38', 'TERMINATE', '1']
  1701. WARNING:toil.leader:e/K/jobxqb0Y0 ktremotemgr: DB::open failed: : 6: network error: connection failed
  1702. WARNING:toil.leader:e/K/jobxqb0Y0 Traceback (most recent call last):
  1703. WARNING:toil.leader:e/K/jobxqb0Y0 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1704. WARNING:toil.leader:e/K/jobxqb0Y0 job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1705. WARNING:toil.leader:e/K/jobxqb0Y0 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1706. WARNING:toil.leader:e/K/jobxqb0Y0 returnValues = self._run(jobGraph, fileStore)
  1707. WARNING:toil.leader:e/K/jobxqb0Y0 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1694, in _run
  1708. WARNING:toil.leader:e/K/jobxqb0Y0 returnValues = self.run(fileStore)
  1709. WARNING:toil.leader:e/K/jobxqb0Y0 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1673, in run
  1710. WARNING:toil.leader:e/K/jobxqb0Y0 if not service.check():
  1711. WARNING:toil.leader:e/K/jobxqb0Y0 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/ktserverToil.py", line 55, in check
  1712. WARNING:toil.leader:e/K/jobxqb0Y0 raise RuntimeError(msg)
  1713. WARNING:toil.leader:e/K/jobxqb0Y0 RuntimeError: Traceback (most recent call last):
  1714. WARNING:toil.leader:e/K/jobxqb0Y0 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/ktserverControl.py", line 82, in run
  1715. WARNING:toil.leader:e/K/jobxqb0Y0 self.tryRun(*self.args, **self.kwargs)
  1716. WARNING:toil.leader:e/K/jobxqb0Y0 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/ktserverControl.py", line 118, in tryRun
  1717. WARNING:toil.leader:e/K/jobxqb0Y0 raise RuntimeError("KTServer failed. Log: %s" % f.read())
  1718. WARNING:toil.leader:e/K/jobxqb0Y0 RuntimeError: KTServer failed. Log: 2019-03-16T22:06:41.524770+02:00: [SYSTEM]: ================ [START]: pid=27365
  1719. WARNING:toil.leader:e/K/jobxqb0Y0 2019-03-16T22:06:41.524983+02:00: [SYSTEM]: opening a database: path=:#opts=ls#bnum=30m#msiz=50g#ktopts=p
  1720. WARNING:toil.leader:e/K/jobxqb0Y0 2019-03-16T22:06:41.566580+02:00: [SYSTEM]: starting the server: expr=:14688
  1721. WARNING:toil.leader:e/K/jobxqb0Y0 2019-03-16T22:06:41.566704+02:00: [SYSTEM]: server socket opened: expr=:14688 timeout=200000.0
  1722. WARNING:toil.leader:e/K/jobxqb0Y0 2019-03-16T22:06:41.566731+02:00: [SYSTEM]: listening server socket started: fd=4
  1723. WARNING:toil.leader:e/K/jobxqb0Y0
  1724. WARNING:toil.leader:e/K/jobxqb0Y0
  1725. WARNING:toil.leader:e/K/jobxqb0Y0 ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn38.grid.pub.ro
  1726. WARNING:toil.leader:e/K/jobxqb0Y0 WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' e/K/jobxqb0Y0 with ID e/K/jobxqb0Y0 to 3
  1727. INFO:toil.leader:Issued job 'KtServerService' e/K/jobxqb0Y0 with job batch system ID: 200 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1728. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' n/P/jobc3lN0J
  1729. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' n/P/jobc3lN0J
  1730. WARNING:toil.leader:n/P/jobc3lN0J INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1731. WARNING:toil.leader:n/P/jobc3lN0J INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1732. WARNING:toil.leader:n/P/jobc3lN0J WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1733. WARNING:toil.leader:n/P/jobc3lN0J INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  1734. WARNING:toil.leader:n/P/jobc3lN0J INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  1735. WARNING:toil.leader:n/P/jobc3lN0J INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="3143" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpxbcB7X/01dc540a-e353-4de3-a370-56ea41054078/tmpxGPIqy.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpxbcB7X/01dc540a-e353-4de3-a370-56ea41054078/tmpsZN3Fb.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpxbcB7X/01dc540a-e353-4de3-a370-56ea41054078/tmpdt8UIS.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpxbcB7X/01dc540a-e353-4de3-a370-56ea41054078/tmp8axNQq.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpxbcB7X/01dc540a-e353-4de3-a370-56ea41054078/tmpnypJZo.tmp']
  1736. WARNING:toil.leader:n/P/jobc3lN0J Set log level to INFO
  1737. WARNING:toil.leader:n/P/jobc3lN0J Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  1738. WARNING:toil.leader:n/P/jobc3lN0J <kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="3143" />
  1739. WARNING:toil.leader:n/P/jobc3lN0J </st_kv_database_conf>
  1740. WARNING:toil.leader:n/P/jobc3lN0J
  1741. WARNING:toil.leader:n/P/jobc3lN0J Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpxbcB7X/01dc540a-e353-4de3-a370-56ea41054078/tmpxGPIqy.tmp
  1742. WARNING:toil.leader:n/P/jobc3lN0J Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpxbcB7X/01dc540a-e353-4de3-a370-56ea41054078/tmpsZN3Fb.tmp
  1743. WARNING:toil.leader:n/P/jobc3lN0J Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpxbcB7X/01dc540a-e353-4de3-a370-56ea41054078/tmpdt8UIS.tmp
  1744. WARNING:toil.leader:n/P/jobc3lN0J Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpxbcB7X/01dc540a-e353-4de3-a370-56ea41054078/tmp8axNQq.tmp
  1745. WARNING:toil.leader:n/P/jobc3lN0J Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpxbcB7X/01dc540a-e353-4de3-a370-56ea41054078/tmpnypJZo.tmp
  1746. WARNING:toil.leader:n/P/jobc3lN0J Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.35 with error: network error
  1747. WARNING:toil.leader:n/P/jobc3lN0J Uncaught exception
  1748. WARNING:toil.leader:n/P/jobc3lN0J Traceback (most recent call last):
  1749. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1750. WARNING:toil.leader:n/P/jobc3lN0J job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1751. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1752. WARNING:toil.leader:n/P/jobc3lN0J super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1753. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1754. WARNING:toil.leader:n/P/jobc3lN0J returnValues = self._run(jobGraph, fileStore)
  1755. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1756. WARNING:toil.leader:n/P/jobc3lN0J return self.run(fileStore)
  1757. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  1758. WARNING:toil.leader:n/P/jobc3lN0J makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  1759. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  1760. WARNING:toil.leader:n/P/jobc3lN0J parameters=["cactus_setup"] + args + sequences)
  1761. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1762. WARNING:toil.leader:n/P/jobc3lN0J raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1763. WARNING:toil.leader:n/P/jobc3lN0J RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="3143" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpxbcB7X/01dc540a-e353-4de3-a370-56ea41054078/tmpxGPIqy.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpxbcB7X/01dc540a-e353-4de3-a370-56ea41054078/tmpsZN3Fb.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpxbcB7X/01dc540a-e353-4de3-a370-56ea41054078/tmpdt8UIS.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpxbcB7X/01dc540a-e353-4de3-a370-56ea41054078/tmp8axNQq.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpxbcB7X/01dc540a-e353-4de3-a370-56ea41054078/tmpnypJZo.tmp'] failed with output:
  1764. WARNING:toil.leader:n/P/jobc3lN0J ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn38.grid.pub.ro
  1765. WARNING:toil.leader:n/P/jobc3lN0J WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' n/P/jobc3lN0J with ID n/P/jobc3lN0J to 3
  1766. INFO:toil.leader:Issued job 'CactusSetupPhase' n/P/jobc3lN0J with job batch system ID: 201 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1767. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' e/K/jobxqb0Y0
  1768. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' e/K/jobxqb0Y0
  1769. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' e/K/jobxqb0Y0 with ID e/K/jobxqb0Y0 to 2
  1770. INFO:toil.leader:Issued job 'KtServerService' e/K/jobxqb0Y0 with job batch system ID: 202 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1771. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' n/P/jobc3lN0J
  1772. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' n/P/jobc3lN0J
  1773. WARNING:toil.leader:n/P/jobc3lN0J INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1774. WARNING:toil.leader:n/P/jobc3lN0J INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1775. WARNING:toil.leader:n/P/jobc3lN0J WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1776. WARNING:toil.leader:n/P/jobc3lN0J INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  1777. WARNING:toil.leader:n/P/jobc3lN0J INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  1778. WARNING:toil.leader:n/P/jobc3lN0J INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="3143" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpp9lgnc/4fa85795-ff8c-410c-8d38-051f26b7f3ab/tmpwiY3PT.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpp9lgnc/4fa85795-ff8c-410c-8d38-051f26b7f3ab/tmpggB1w9.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpp9lgnc/4fa85795-ff8c-410c-8d38-051f26b7f3ab/tmpdFuWNs.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpp9lgnc/4fa85795-ff8c-410c-8d38-051f26b7f3ab/tmpzsmRjO.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpp9lgnc/4fa85795-ff8c-410c-8d38-051f26b7f3ab/tmpXCs21_.tmp']
  1779. WARNING:toil.leader:n/P/jobc3lN0J Set log level to INFO
  1780. WARNING:toil.leader:n/P/jobc3lN0J Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  1781. WARNING:toil.leader:n/P/jobc3lN0J <kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="3143" />
  1782. WARNING:toil.leader:n/P/jobc3lN0J </st_kv_database_conf>
  1783. WARNING:toil.leader:n/P/jobc3lN0J
  1784. WARNING:toil.leader:n/P/jobc3lN0J Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpp9lgnc/4fa85795-ff8c-410c-8d38-051f26b7f3ab/tmpwiY3PT.tmp
  1785. WARNING:toil.leader:n/P/jobc3lN0J Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpp9lgnc/4fa85795-ff8c-410c-8d38-051f26b7f3ab/tmpggB1w9.tmp
  1786. WARNING:toil.leader:n/P/jobc3lN0J Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpp9lgnc/4fa85795-ff8c-410c-8d38-051f26b7f3ab/tmpdFuWNs.tmp
  1787. WARNING:toil.leader:n/P/jobc3lN0J Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpp9lgnc/4fa85795-ff8c-410c-8d38-051f26b7f3ab/tmpzsmRjO.tmp
  1788. WARNING:toil.leader:n/P/jobc3lN0J Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpp9lgnc/4fa85795-ff8c-410c-8d38-051f26b7f3ab/tmpXCs21_.tmp
  1789. WARNING:toil.leader:n/P/jobc3lN0J Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.35 with error: network error
  1790. WARNING:toil.leader:n/P/jobc3lN0J Uncaught exception
  1791. WARNING:toil.leader:n/P/jobc3lN0J Traceback (most recent call last):
  1792. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1793. WARNING:toil.leader:n/P/jobc3lN0J job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1794. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1795. WARNING:toil.leader:n/P/jobc3lN0J super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1796. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1797. WARNING:toil.leader:n/P/jobc3lN0J returnValues = self._run(jobGraph, fileStore)
  1798. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1799. WARNING:toil.leader:n/P/jobc3lN0J return self.run(fileStore)
  1800. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  1801. WARNING:toil.leader:n/P/jobc3lN0J makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  1802. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  1803. WARNING:toil.leader:n/P/jobc3lN0J parameters=["cactus_setup"] + args + sequences)
  1804. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1805. WARNING:toil.leader:n/P/jobc3lN0J raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1806. WARNING:toil.leader:n/P/jobc3lN0J RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="3143" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpp9lgnc/4fa85795-ff8c-410c-8d38-051f26b7f3ab/tmpwiY3PT.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpp9lgnc/4fa85795-ff8c-410c-8d38-051f26b7f3ab/tmpggB1w9.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpp9lgnc/4fa85795-ff8c-410c-8d38-051f26b7f3ab/tmpdFuWNs.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpp9lgnc/4fa85795-ff8c-410c-8d38-051f26b7f3ab/tmpzsmRjO.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpp9lgnc/4fa85795-ff8c-410c-8d38-051f26b7f3ab/tmpXCs21_.tmp'] failed with output:
  1807. WARNING:toil.leader:n/P/jobc3lN0J ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn38.grid.pub.ro
  1808. WARNING:toil.leader:n/P/jobc3lN0J WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' n/P/jobc3lN0J with ID n/P/jobc3lN0J to 2
  1809. INFO:toil.leader:Issued job 'CactusSetupPhase' n/P/jobc3lN0J with job batch system ID: 203 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1810. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' e/K/jobxqb0Y0
  1811. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' e/K/jobxqb0Y0
  1812. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' e/K/jobxqb0Y0 with ID e/K/jobxqb0Y0 to 1
  1813. INFO:toil.leader:Issued job 'KtServerService' e/K/jobxqb0Y0 with job batch system ID: 204 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1814. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' n/P/jobc3lN0J
  1815. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' n/P/jobc3lN0J
  1816. WARNING:toil.leader:n/P/jobc3lN0J INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1817. WARNING:toil.leader:n/P/jobc3lN0J INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1818. WARNING:toil.leader:n/P/jobc3lN0J WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1819. WARNING:toil.leader:n/P/jobc3lN0J WARNING:toil.fileStore:Detected that job (N/8/job2cN94i/g/tmp5k3kYp-_serialiseJob-stream) prematurely terminated. Fixing the state of the job on disk.
  1820. WARNING:toil.leader:n/P/jobc3lN0J INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  1821. WARNING:toil.leader:n/P/jobc3lN0J INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  1822. WARNING:toil.leader:n/P/jobc3lN0J INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.38" port="19223" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpdQZXeK/6ce30c50-7562-45e4-a141-f0f0c880101e/tmpwpT1vr.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpdQZXeK/6ce30c50-7562-45e4-a141-f0f0c880101e/tmpVFcfaF.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpdQZXeK/6ce30c50-7562-45e4-a141-f0f0c880101e/tmpV26i3i.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpdQZXeK/6ce30c50-7562-45e4-a141-f0f0c880101e/tmpyngBsc.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpdQZXeK/6ce30c50-7562-45e4-a141-f0f0c880101e/tmp9NXcC_.tmp']
  1823. WARNING:toil.leader:n/P/jobc3lN0J Set log level to INFO
  1824. WARNING:toil.leader:n/P/jobc3lN0J Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  1825. WARNING:toil.leader:n/P/jobc3lN0J <kyoto_tycoon database_dir="fakepath" host="172.16.13.38" port="19223" />
  1826. WARNING:toil.leader:n/P/jobc3lN0J </st_kv_database_conf>
  1827. WARNING:toil.leader:n/P/jobc3lN0J
  1828. WARNING:toil.leader:n/P/jobc3lN0J Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpdQZXeK/6ce30c50-7562-45e4-a141-f0f0c880101e/tmpwpT1vr.tmp
  1829. WARNING:toil.leader:n/P/jobc3lN0J Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpdQZXeK/6ce30c50-7562-45e4-a141-f0f0c880101e/tmpVFcfaF.tmp
  1830. WARNING:toil.leader:n/P/jobc3lN0J Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpdQZXeK/6ce30c50-7562-45e4-a141-f0f0c880101e/tmpV26i3i.tmp
  1831. WARNING:toil.leader:n/P/jobc3lN0J Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpdQZXeK/6ce30c50-7562-45e4-a141-f0f0c880101e/tmpyngBsc.tmp
  1832. WARNING:toil.leader:n/P/jobc3lN0J Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpdQZXeK/6ce30c50-7562-45e4-a141-f0f0c880101e/tmp9NXcC_.tmp
  1833. WARNING:toil.leader:n/P/jobc3lN0J Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.38 with error: network error
  1834. WARNING:toil.leader:n/P/jobc3lN0J Uncaught exception
  1835. WARNING:toil.leader:n/P/jobc3lN0J Traceback (most recent call last):
  1836. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1837. WARNING:toil.leader:n/P/jobc3lN0J job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1838. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1839. WARNING:toil.leader:n/P/jobc3lN0J super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1840. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1841. WARNING:toil.leader:n/P/jobc3lN0J returnValues = self._run(jobGraph, fileStore)
  1842. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1843. WARNING:toil.leader:n/P/jobc3lN0J return self.run(fileStore)
  1844. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  1845. WARNING:toil.leader:n/P/jobc3lN0J makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  1846. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  1847. WARNING:toil.leader:n/P/jobc3lN0J parameters=["cactus_setup"] + args + sequences)
  1848. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1849. WARNING:toil.leader:n/P/jobc3lN0J raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1850. WARNING:toil.leader:n/P/jobc3lN0J RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.38" port="19223" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpdQZXeK/6ce30c50-7562-45e4-a141-f0f0c880101e/tmpwpT1vr.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpdQZXeK/6ce30c50-7562-45e4-a141-f0f0c880101e/tmpVFcfaF.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpdQZXeK/6ce30c50-7562-45e4-a141-f0f0c880101e/tmpV26i3i.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpdQZXeK/6ce30c50-7562-45e4-a141-f0f0c880101e/tmpyngBsc.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpdQZXeK/6ce30c50-7562-45e4-a141-f0f0c880101e/tmp9NXcC_.tmp'] failed with output:
  1851. WARNING:toil.leader:n/P/jobc3lN0J ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn35.grid.pub.ro
  1852. WARNING:toil.leader:n/P/jobc3lN0J WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' n/P/jobc3lN0J with ID n/P/jobc3lN0J to 1
  1853. INFO:toil.leader:Issued job 'CactusSetupPhase' n/P/jobc3lN0J with job batch system ID: 205 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1854. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' e/K/jobxqb0Y0
  1855. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' e/K/jobxqb0Y0
  1856. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' e/K/jobxqb0Y0 with ID e/K/jobxqb0Y0 to 0
  1857. WARNING:toil.leader:Job 'KtServerService' e/K/jobxqb0Y0 with ID e/K/jobxqb0Y0 is completely failed
  1858. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' n/P/jobc3lN0J
  1859. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' n/P/jobc3lN0J
  1860. WARNING:toil.leader:n/P/jobc3lN0J INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1861. WARNING:toil.leader:n/P/jobc3lN0J INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1862. WARNING:toil.leader:n/P/jobc3lN0J WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1863. WARNING:toil.leader:n/P/jobc3lN0J WARNING:toil.fileStore:Detected that job (N/8/job2cN94i/g/tmp5k3kYp-_serialiseJob-stream) prematurely terminated. Fixing the state of the job on disk.
  1864. WARNING:toil.leader:n/P/jobc3lN0J INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  1865. WARNING:toil.leader:n/P/jobc3lN0J INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  1866. WARNING:toil.leader:n/P/jobc3lN0J INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.38" port="30249" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpGAoyEN/85745272-569d-43a6-aad4-09b1565166b5/tmpItAT4k.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpGAoyEN/85745272-569d-43a6-aad4-09b1565166b5/tmpiPBhtW.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpGAoyEN/85745272-569d-43a6-aad4-09b1565166b5/tmppbP119.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpGAoyEN/85745272-569d-43a6-aad4-09b1565166b5/tmpVIyeXE.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpGAoyEN/85745272-569d-43a6-aad4-09b1565166b5/tmpNBQsKS.tmp']
  1867. WARNING:toil.leader:n/P/jobc3lN0J Set log level to INFO
  1868. WARNING:toil.leader:n/P/jobc3lN0J Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  1869. WARNING:toil.leader:n/P/jobc3lN0J <kyoto_tycoon database_dir="fakepath" host="172.16.13.38" port="30249" />
  1870. WARNING:toil.leader:n/P/jobc3lN0J </st_kv_database_conf>
  1871. WARNING:toil.leader:n/P/jobc3lN0J
  1872. WARNING:toil.leader:n/P/jobc3lN0J Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpGAoyEN/85745272-569d-43a6-aad4-09b1565166b5/tmpItAT4k.tmp
  1873. WARNING:toil.leader:n/P/jobc3lN0J Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpGAoyEN/85745272-569d-43a6-aad4-09b1565166b5/tmpiPBhtW.tmp
  1874. WARNING:toil.leader:n/P/jobc3lN0J Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpGAoyEN/85745272-569d-43a6-aad4-09b1565166b5/tmppbP119.tmp
  1875. WARNING:toil.leader:n/P/jobc3lN0J Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpGAoyEN/85745272-569d-43a6-aad4-09b1565166b5/tmpVIyeXE.tmp
  1876. WARNING:toil.leader:n/P/jobc3lN0J Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpGAoyEN/85745272-569d-43a6-aad4-09b1565166b5/tmpNBQsKS.tmp
  1877. WARNING:toil.leader:n/P/jobc3lN0J Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.38 with error: network error
  1878. WARNING:toil.leader:n/P/jobc3lN0J Uncaught exception
  1879. WARNING:toil.leader:n/P/jobc3lN0J Traceback (most recent call last):
  1880. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1881. WARNING:toil.leader:n/P/jobc3lN0J job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1882. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1883. WARNING:toil.leader:n/P/jobc3lN0J super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1884. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1885. WARNING:toil.leader:n/P/jobc3lN0J returnValues = self._run(jobGraph, fileStore)
  1886. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1887. WARNING:toil.leader:n/P/jobc3lN0J return self.run(fileStore)
  1888. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  1889. WARNING:toil.leader:n/P/jobc3lN0J makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  1890. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  1891. WARNING:toil.leader:n/P/jobc3lN0J parameters=["cactus_setup"] + args + sequences)
  1892. WARNING:toil.leader:n/P/jobc3lN0J File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1893. WARNING:toil.leader:n/P/jobc3lN0J raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1894. WARNING:toil.leader:n/P/jobc3lN0J RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.38" port="30249" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpGAoyEN/85745272-569d-43a6-aad4-09b1565166b5/tmpItAT4k.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpGAoyEN/85745272-569d-43a6-aad4-09b1565166b5/tmpiPBhtW.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpGAoyEN/85745272-569d-43a6-aad4-09b1565166b5/tmppbP119.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpGAoyEN/85745272-569d-43a6-aad4-09b1565166b5/tmpVIyeXE.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpGAoyEN/85745272-569d-43a6-aad4-09b1565166b5/tmpNBQsKS.tmp'] failed with output:
  1895. WARNING:toil.leader:n/P/jobc3lN0J ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn34.grid.pub.ro
  1896. WARNING:toil.leader:n/P/jobc3lN0J WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' n/P/jobc3lN0J with ID n/P/jobc3lN0J to 0
  1897. WARNING:toil.leader:Job 'CactusSetupPhase' n/P/jobc3lN0J with ID n/P/jobc3lN0J is completely failed
  1898. WARNING:toil.leader:Job: N/8/job2cN94i is being restarted as a checkpoint after the total failure of jobs in its subtree.
  1899. INFO:toil.leader:Issued job 'StartPrimaryDB' N/8/job2cN94i with job batch system ID: 206 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1900. INFO:toil.leader:Job ended successfully: 'StartPrimaryDB' N/8/job2cN94i
  1901. INFO:toil.leader:Issued job 'KtServerService' 3/R/jobQbZW4w with job batch system ID: 207 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1902. INFO:toil.leader:Issued job 'CactusSetupPhase' Q/B/jobe6uQPN with job batch system ID: 208 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1903. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' 3/R/jobQbZW4w
  1904. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' 3/R/jobQbZW4w
  1905. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' 3/R/jobQbZW4w with ID 3/R/jobQbZW4w to 5
  1906. INFO:toil.leader:Issued job 'KtServerService' 3/R/jobQbZW4w with job batch system ID: 209 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1907. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' Q/B/jobe6uQPN
  1908. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' Q/B/jobe6uQPN
  1909. WARNING:toil.leader:Q/B/jobe6uQPN INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1910. WARNING:toil.leader:Q/B/jobe6uQPN INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1911. WARNING:toil.leader:Q/B/jobe6uQPN WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1912. WARNING:toil.leader:Q/B/jobe6uQPN WARNING:toil.fileStore:Detected that job (N/8/job2cN94i/g/tmp9MRLp0-_serialiseJob-stream) prematurely terminated. Fixing the state of the job on disk.
  1913. WARNING:toil.leader:Q/B/jobe6uQPN INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  1914. WARNING:toil.leader:Q/B/jobe6uQPN INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  1915. WARNING:toil.leader:Q/B/jobe6uQPN INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.38" port="21056" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpQiMVUq/f14c8d00-d239-4777-a430-266676e181b2/tmplqQPqs.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpQiMVUq/f14c8d00-d239-4777-a430-266676e181b2/tmpAO6dpJ.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpQiMVUq/f14c8d00-d239-4777-a430-266676e181b2/tmpyWmEaX.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpQiMVUq/f14c8d00-d239-4777-a430-266676e181b2/tmpdpwodw.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpQiMVUq/f14c8d00-d239-4777-a430-266676e181b2/tmpP2VdBF.tmp']
  1916. WARNING:toil.leader:Q/B/jobe6uQPN Set log level to INFO
  1917. WARNING:toil.leader:Q/B/jobe6uQPN Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  1918. WARNING:toil.leader:Q/B/jobe6uQPN <kyoto_tycoon database_dir="fakepath" host="172.16.13.38" port="21056" />
  1919. WARNING:toil.leader:Q/B/jobe6uQPN </st_kv_database_conf>
  1920. WARNING:toil.leader:Q/B/jobe6uQPN
  1921. WARNING:toil.leader:Q/B/jobe6uQPN Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpQiMVUq/f14c8d00-d239-4777-a430-266676e181b2/tmplqQPqs.tmp
  1922. WARNING:toil.leader:Q/B/jobe6uQPN Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpQiMVUq/f14c8d00-d239-4777-a430-266676e181b2/tmpAO6dpJ.tmp
  1923. WARNING:toil.leader:Q/B/jobe6uQPN Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpQiMVUq/f14c8d00-d239-4777-a430-266676e181b2/tmpyWmEaX.tmp
  1924. WARNING:toil.leader:Q/B/jobe6uQPN Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpQiMVUq/f14c8d00-d239-4777-a430-266676e181b2/tmpdpwodw.tmp
  1925. WARNING:toil.leader:Q/B/jobe6uQPN Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpQiMVUq/f14c8d00-d239-4777-a430-266676e181b2/tmpP2VdBF.tmp
  1926. WARNING:toil.leader:Q/B/jobe6uQPN Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.38 with error: network error
  1927. WARNING:toil.leader:Q/B/jobe6uQPN Uncaught exception
  1928. WARNING:toil.leader:Q/B/jobe6uQPN Traceback (most recent call last):
  1929. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1930. WARNING:toil.leader:Q/B/jobe6uQPN job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1931. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1932. WARNING:toil.leader:Q/B/jobe6uQPN super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1933. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1934. WARNING:toil.leader:Q/B/jobe6uQPN returnValues = self._run(jobGraph, fileStore)
  1935. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1936. WARNING:toil.leader:Q/B/jobe6uQPN return self.run(fileStore)
  1937. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  1938. WARNING:toil.leader:Q/B/jobe6uQPN makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  1939. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  1940. WARNING:toil.leader:Q/B/jobe6uQPN parameters=["cactus_setup"] + args + sequences)
  1941. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1942. WARNING:toil.leader:Q/B/jobe6uQPN raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1943. WARNING:toil.leader:Q/B/jobe6uQPN RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.38" port="21056" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpQiMVUq/f14c8d00-d239-4777-a430-266676e181b2/tmplqQPqs.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpQiMVUq/f14c8d00-d239-4777-a430-266676e181b2/tmpAO6dpJ.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpQiMVUq/f14c8d00-d239-4777-a430-266676e181b2/tmpyWmEaX.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpQiMVUq/f14c8d00-d239-4777-a430-266676e181b2/tmpdpwodw.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpQiMVUq/f14c8d00-d239-4777-a430-266676e181b2/tmpP2VdBF.tmp'] failed with output:
  1944. WARNING:toil.leader:Q/B/jobe6uQPN ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn38.grid.pub.ro
  1945. WARNING:toil.leader:Q/B/jobe6uQPN WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' Q/B/jobe6uQPN with ID Q/B/jobe6uQPN to 5
  1946. INFO:toil.leader:Issued job 'CactusSetupPhase' Q/B/jobe6uQPN with job batch system ID: 210 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1947. INFO:toil.leader:Job ended successfully: 'KtServerService' 3/R/jobQbZW4w
  1948. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'KtServerService' 3/R/jobQbZW4w
  1949. WARNING:toil.leader:3/R/jobQbZW4w INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1950. WARNING:toil.leader:3/R/jobQbZW4w INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1951. WARNING:toil.leader:3/R/jobQbZW4w WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1952. WARNING:toil.leader:3/R/jobQbZW4w WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1953. WARNING:toil.leader:3/R/jobQbZW4w INFO:cactus.shared.common:Running the command ['netstat', '-tuplen']
  1954. WARNING:toil.leader:3/R/jobQbZW4w (No info could be read for "-p": geteuid()=98354 but you should be root.)
  1955. WARNING:toil.leader:3/R/jobQbZW4w INFO:cactus.shared.common:Running the command ['ktserver', '-port', '14112', '-ls', '-tout', '200000', '-th', '64', '-bgs', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpbPqmI9/075174e6-5cf3-4710-81f3-c9c515237f7f/t9QpEA1/snapshot', '-bgsc', 'lzo', '-bgsi', '1000000', '-log', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpbPqmI9/075174e6-5cf3-4710-81f3-c9c515237f7f/tmpVpXRNI.tmp', ':#opts=ls#bnum=30m#msiz=50g#ktopts=p']
  1956. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  1957. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  1958. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  1959. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 513
  1960. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  1961. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  1962. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  1963. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 513
  1964. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  1965. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  1966. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  1967. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 514
  1968. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  1969. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  1970. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  1971. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 515
  1972. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  1973. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  1974. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  1975. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 516
  1976. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  1977. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  1978. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  1979. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 517
  1980. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  1981. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  1982. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  1983. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 518
  1984. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  1985. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  1986. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  1987. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 519
  1988. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  1989. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  1990. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  1991. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 520
  1992. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  1993. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  1994. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  1995. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 521
  1996. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  1997. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  1998. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  1999. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 522
  2000. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2001. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2002. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2003. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 523
  2004. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2005. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2006. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2007. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 524
  2008. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2009. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2010. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2011. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 525
  2012. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2013. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2014. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2015. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 526
  2016. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2017. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2018. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2019. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 527
  2020. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2021. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2022. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2023. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 528
  2024. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2025. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2026. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2027. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 529
  2028. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2029. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2030. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2031. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 530
  2032. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2033. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2034. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2035. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 531
  2036. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2037. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2038. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2039. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 532
  2040. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2041. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2042. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2043. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 533
  2044. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2045. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2046. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2047. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 534
  2048. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2049. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2050. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2051. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 535
  2052. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2053. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2054. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2055. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 536
  2056. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2057. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2058. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2059. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 537
  2060. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2061. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2062. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2063. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 538
  2064. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2065. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2066. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2067. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 539
  2068. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2069. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2070. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2071. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 540
  2072. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2073. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2074. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2075. WARNING:toil.leader:3/R/jobQbZW4w INFO:toil.lib.bioio:Ktserver running.
  2076. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 541
  2077. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2078. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2079. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2080. WARNING:toil.leader:3/R/jobQbZW4w INFO:toil.lib.bioio:Ktserver running.
  2081. WARNING:toil.leader:3/R/jobQbZW4w INFO:toil.lib.bioio:Ktserver running.
  2082. WARNING:toil.leader:3/R/jobQbZW4w INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '14112', '-host', '172.16.13.38', 'TERMINATE']
  2083. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 542
  2084. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2085. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2086. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2087. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 544
  2088. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2089. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2090. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2091. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 545
  2092. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2093. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2094. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2095. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 546
  2096. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2097. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2098. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2099. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 547
  2100. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2101. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2102. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2103. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 548
  2104. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2105. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2106. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2107. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 549
  2108. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2109. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2110. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2111. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 550
  2112. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2113. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2114. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2115. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 551
  2116. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2117. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2118. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2119. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 552
  2120. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2121. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2122. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2123. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 553
  2124. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2125. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2126. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2127. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 554
  2128. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2129. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2130. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2131. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 555
  2132. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2133. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2134. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2135. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 556
  2136. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2137. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2138. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2139. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 557
  2140. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2141. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2142. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2143. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 558
  2144. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2145. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2146. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2147. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 559
  2148. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2149. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2150. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2151. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 560
  2152. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2153. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2154. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2155. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 561
  2156. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2157. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2158. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2159. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 562
  2160. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2161. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2162. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2163. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 563
  2164. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2165. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2166. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2167. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 564
  2168. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2169. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2170. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2171. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 565
  2172. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2173. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2174. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2175. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 566
  2176. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2177. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2178. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2179. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 567
  2180. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2181. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2182. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2183. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 568
  2184. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2185. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2186. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2187. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 569
  2188. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2189. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2190. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2191. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 570
  2192. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2193. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2194. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2195. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 571
  2196. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2197. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 510853 510853
  2198. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1021706
  2199. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 572
  2200. WARNING:toil.leader:3/R/jobQbZW4w terminate called after throwing an instance of 'std::runtime_error'
  2201. WARNING:toil.leader:3/R/jobQbZW4w what(): pthread_create: EAGAIN: Insufficient resources to create thread, or thread limit reached
  2202. WARNING:toil.leader:3/R/jobQbZW4w Process ServerProcess-1:
  2203. WARNING:toil.leader:3/R/jobQbZW4w Traceback (most recent call last):
  2204. WARNING:toil.leader:3/R/jobQbZW4w File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap
  2205. WARNING:toil.leader:3/R/jobQbZW4w self.run()
  2206. WARNING:toil.leader:3/R/jobQbZW4w File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/ktserverControl.py", line 82, in run
  2207. WARNING:toil.leader:3/R/jobQbZW4w self.tryRun(*self.args, **self.kwargs)
  2208. WARNING:toil.leader:3/R/jobQbZW4w File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/ktserverControl.py", line 118, in tryRun
  2209. WARNING:toil.leader:3/R/jobQbZW4w raise RuntimeError("KTServer failed. Log: %s" % f.read())
  2210. WARNING:toil.leader:3/R/jobQbZW4w RuntimeError: KTServer failed. Log: 2019-03-16T22:29:30.315871+02:00: [SYSTEM]: ================ [START]: pid=29427
  2211. WARNING:toil.leader:3/R/jobQbZW4w 2019-03-16T22:29:30.316052+02:00: [SYSTEM]: opening a database: path=:#opts=ls#bnum=30m#msiz=50g#ktopts=p
  2212. WARNING:toil.leader:3/R/jobQbZW4w 2019-03-16T22:29:30.358716+02:00: [SYSTEM]: starting the server: expr=:14112
  2213. WARNING:toil.leader:3/R/jobQbZW4w 2019-03-16T22:29:30.358865+02:00: [SYSTEM]: server socket opened: expr=:14112 timeout=200000.0
  2214. WARNING:toil.leader:3/R/jobQbZW4w 2019-03-16T22:29:30.358896+02:00: [SYSTEM]: listening server socket started: fd=4
  2215. WARNING:toil.leader:3/R/jobQbZW4w
  2216. WARNING:toil.leader:3/R/jobQbZW4w INFO:cactus.shared.common:Running the command ['ktremotemgr', 'set', '-port', '14112', '-host', '172.16.13.38', 'TERMINATE', '1']
  2217. WARNING:toil.leader:3/R/jobQbZW4w ktremotemgr: DB::open failed: : 6: network error: connection failed
  2218. WARNING:toil.leader:3/R/jobQbZW4w Traceback (most recent call last):
  2219. WARNING:toil.leader:3/R/jobQbZW4w File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  2220. WARNING:toil.leader:3/R/jobQbZW4w job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2221. WARNING:toil.leader:3/R/jobQbZW4w File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  2222. WARNING:toil.leader:3/R/jobQbZW4w returnValues = self._run(jobGraph, fileStore)
  2223. WARNING:toil.leader:3/R/jobQbZW4w File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1694, in _run
  2224. WARNING:toil.leader:3/R/jobQbZW4w returnValues = self.run(fileStore)
  2225. WARNING:toil.leader:3/R/jobQbZW4w File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1673, in run
  2226. WARNING:toil.leader:3/R/jobQbZW4w if not service.check():
  2227. WARNING:toil.leader:3/R/jobQbZW4w File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/ktserverToil.py", line 55, in check
  2228. WARNING:toil.leader:3/R/jobQbZW4w raise RuntimeError(msg)
  2229. WARNING:toil.leader:3/R/jobQbZW4w RuntimeError: Traceback (most recent call last):
  2230. WARNING:toil.leader:3/R/jobQbZW4w File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/ktserverControl.py", line 82, in run
  2231. WARNING:toil.leader:3/R/jobQbZW4w self.tryRun(*self.args, **self.kwargs)
  2232. WARNING:toil.leader:3/R/jobQbZW4w File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/ktserverControl.py", line 118, in tryRun
  2233. WARNING:toil.leader:3/R/jobQbZW4w raise RuntimeError("KTServer failed. Log: %s" % f.read())
  2234. WARNING:toil.leader:3/R/jobQbZW4w RuntimeError: KTServer failed. Log: 2019-03-16T22:29:30.315871+02:00: [SYSTEM]: ================ [START]: pid=29427
  2235. WARNING:toil.leader:3/R/jobQbZW4w 2019-03-16T22:29:30.316052+02:00: [SYSTEM]: opening a database: path=:#opts=ls#bnum=30m#msiz=50g#ktopts=p
  2236. WARNING:toil.leader:3/R/jobQbZW4w 2019-03-16T22:29:30.358716+02:00: [SYSTEM]: starting the server: expr=:14112
  2237. WARNING:toil.leader:3/R/jobQbZW4w 2019-03-16T22:29:30.358865+02:00: [SYSTEM]: server socket opened: expr=:14112 timeout=200000.0
  2238. WARNING:toil.leader:3/R/jobQbZW4w 2019-03-16T22:29:30.358896+02:00: [SYSTEM]: listening server socket started: fd=4
  2239. WARNING:toil.leader:3/R/jobQbZW4w
  2240. WARNING:toil.leader:3/R/jobQbZW4w
  2241. WARNING:toil.leader:3/R/jobQbZW4w ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn38.grid.pub.ro
  2242. WARNING:toil.leader:3/R/jobQbZW4w WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' 3/R/jobQbZW4w with ID 3/R/jobQbZW4w to 4
  2243. INFO:toil.leader:Issued job 'KtServerService' 3/R/jobQbZW4w with job batch system ID: 211 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  2244. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' Q/B/jobe6uQPN
  2245. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' Q/B/jobe6uQPN
  2246. WARNING:toil.leader:Q/B/jobe6uQPN INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  2247. WARNING:toil.leader:Q/B/jobe6uQPN INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  2248. WARNING:toil.leader:Q/B/jobe6uQPN WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  2249. WARNING:toil.leader:Q/B/jobe6uQPN INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  2250. WARNING:toil.leader:Q/B/jobe6uQPN INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  2251. WARNING:toil.leader:Q/B/jobe6uQPN INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.38" port="14112" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpYaWuKq/c2ee15f8-c906-416a-859e-f8f905a23475/tmpNAzpXj.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpYaWuKq/c2ee15f8-c906-416a-859e-f8f905a23475/tmpQsOuj9.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpYaWuKq/c2ee15f8-c906-416a-859e-f8f905a23475/tmpjskPad.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpYaWuKq/c2ee15f8-c906-416a-859e-f8f905a23475/tmpv3XiFS.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpYaWuKq/c2ee15f8-c906-416a-859e-f8f905a23475/tmpc6AACt.tmp']
  2252. WARNING:toil.leader:Q/B/jobe6uQPN Set log level to INFO
  2253. WARNING:toil.leader:Q/B/jobe6uQPN Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  2254. WARNING:toil.leader:Q/B/jobe6uQPN <kyoto_tycoon database_dir="fakepath" host="172.16.13.38" port="14112" />
  2255. WARNING:toil.leader:Q/B/jobe6uQPN </st_kv_database_conf>
  2256. WARNING:toil.leader:Q/B/jobe6uQPN
  2257. WARNING:toil.leader:Q/B/jobe6uQPN Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpYaWuKq/c2ee15f8-c906-416a-859e-f8f905a23475/tmpNAzpXj.tmp
  2258. WARNING:toil.leader:Q/B/jobe6uQPN Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpYaWuKq/c2ee15f8-c906-416a-859e-f8f905a23475/tmpQsOuj9.tmp
  2259. WARNING:toil.leader:Q/B/jobe6uQPN Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpYaWuKq/c2ee15f8-c906-416a-859e-f8f905a23475/tmpjskPad.tmp
  2260. WARNING:toil.leader:Q/B/jobe6uQPN Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpYaWuKq/c2ee15f8-c906-416a-859e-f8f905a23475/tmpv3XiFS.tmp
  2261. WARNING:toil.leader:Q/B/jobe6uQPN Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpYaWuKq/c2ee15f8-c906-416a-859e-f8f905a23475/tmpc6AACt.tmp
  2262. WARNING:toil.leader:Q/B/jobe6uQPN Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.38 with error: network error
  2263. WARNING:toil.leader:Q/B/jobe6uQPN Uncaught exception
  2264. WARNING:toil.leader:Q/B/jobe6uQPN Traceback (most recent call last):
  2265. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  2266. WARNING:toil.leader:Q/B/jobe6uQPN job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2267. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  2268. WARNING:toil.leader:Q/B/jobe6uQPN super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2269. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  2270. WARNING:toil.leader:Q/B/jobe6uQPN returnValues = self._run(jobGraph, fileStore)
  2271. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  2272. WARNING:toil.leader:Q/B/jobe6uQPN return self.run(fileStore)
  2273. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  2274. WARNING:toil.leader:Q/B/jobe6uQPN makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  2275. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  2276. WARNING:toil.leader:Q/B/jobe6uQPN parameters=["cactus_setup"] + args + sequences)
  2277. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  2278. WARNING:toil.leader:Q/B/jobe6uQPN raise RuntimeError("Command %s failed with output: %s" % (call, output))
  2279. WARNING:toil.leader:Q/B/jobe6uQPN RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.38" port="14112" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpYaWuKq/c2ee15f8-c906-416a-859e-f8f905a23475/tmpNAzpXj.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpYaWuKq/c2ee15f8-c906-416a-859e-f8f905a23475/tmpQsOuj9.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpYaWuKq/c2ee15f8-c906-416a-859e-f8f905a23475/tmpjskPad.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpYaWuKq/c2ee15f8-c906-416a-859e-f8f905a23475/tmpv3XiFS.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpYaWuKq/c2ee15f8-c906-416a-859e-f8f905a23475/tmpc6AACt.tmp'] failed with output:
  2280. WARNING:toil.leader:Q/B/jobe6uQPN ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn34.grid.pub.ro
  2281. WARNING:toil.leader:Q/B/jobe6uQPN WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' Q/B/jobe6uQPN with ID Q/B/jobe6uQPN to 4
  2282. INFO:toil.leader:Issued job 'CactusSetupPhase' Q/B/jobe6uQPN with job batch system ID: 212 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  2283. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' 3/R/jobQbZW4w
  2284. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' 3/R/jobQbZW4w
  2285. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' 3/R/jobQbZW4w with ID 3/R/jobQbZW4w to 3
  2286. INFO:toil.leader:Issued job 'KtServerService' 3/R/jobQbZW4w with job batch system ID: 213 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  2287. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' Q/B/jobe6uQPN
  2288. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' Q/B/jobe6uQPN
  2289. WARNING:toil.leader:Q/B/jobe6uQPN INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  2290. WARNING:toil.leader:Q/B/jobe6uQPN INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  2291. WARNING:toil.leader:Q/B/jobe6uQPN WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  2292. WARNING:toil.leader:Q/B/jobe6uQPN INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  2293. WARNING:toil.leader:Q/B/jobe6uQPN INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  2294. WARNING:toil.leader:Q/B/jobe6uQPN INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="20346" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpe3NZif/65580754-235e-4c22-a45f-d61f9a4c64f6/tmpWV1jNR.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpe3NZif/65580754-235e-4c22-a45f-d61f9a4c64f6/tmpKYNAM8.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpe3NZif/65580754-235e-4c22-a45f-d61f9a4c64f6/tmpzim97h.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpe3NZif/65580754-235e-4c22-a45f-d61f9a4c64f6/tmpOqIVDU.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpe3NZif/65580754-235e-4c22-a45f-d61f9a4c64f6/tmpkW3O0A.tmp']
  2295. WARNING:toil.leader:Q/B/jobe6uQPN Set log level to INFO
  2296. WARNING:toil.leader:Q/B/jobe6uQPN Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  2297. WARNING:toil.leader:Q/B/jobe6uQPN <kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="20346" />
  2298. WARNING:toil.leader:Q/B/jobe6uQPN </st_kv_database_conf>
  2299. WARNING:toil.leader:Q/B/jobe6uQPN
  2300. WARNING:toil.leader:Q/B/jobe6uQPN Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpe3NZif/65580754-235e-4c22-a45f-d61f9a4c64f6/tmpWV1jNR.tmp
  2301. WARNING:toil.leader:Q/B/jobe6uQPN Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpe3NZif/65580754-235e-4c22-a45f-d61f9a4c64f6/tmpKYNAM8.tmp
  2302. WARNING:toil.leader:Q/B/jobe6uQPN Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpe3NZif/65580754-235e-4c22-a45f-d61f9a4c64f6/tmpzim97h.tmp
  2303. WARNING:toil.leader:Q/B/jobe6uQPN Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpe3NZif/65580754-235e-4c22-a45f-d61f9a4c64f6/tmpOqIVDU.tmp
  2304. WARNING:toil.leader:Q/B/jobe6uQPN Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpe3NZif/65580754-235e-4c22-a45f-d61f9a4c64f6/tmpkW3O0A.tmp
  2305. WARNING:toil.leader:Q/B/jobe6uQPN Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.35 with error: network error
  2306. WARNING:toil.leader:Q/B/jobe6uQPN Uncaught exception
  2307. WARNING:toil.leader:Q/B/jobe6uQPN Traceback (most recent call last):
  2308. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  2309. WARNING:toil.leader:Q/B/jobe6uQPN job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2310. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  2311. WARNING:toil.leader:Q/B/jobe6uQPN super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2312. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  2313. WARNING:toil.leader:Q/B/jobe6uQPN returnValues = self._run(jobGraph, fileStore)
  2314. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  2315. WARNING:toil.leader:Q/B/jobe6uQPN return self.run(fileStore)
  2316. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  2317. WARNING:toil.leader:Q/B/jobe6uQPN makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  2318. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  2319. WARNING:toil.leader:Q/B/jobe6uQPN parameters=["cactus_setup"] + args + sequences)
  2320. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  2321. WARNING:toil.leader:Q/B/jobe6uQPN raise RuntimeError("Command %s failed with output: %s" % (call, output))
  2322. WARNING:toil.leader:Q/B/jobe6uQPN RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="20346" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpe3NZif/65580754-235e-4c22-a45f-d61f9a4c64f6/tmpWV1jNR.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpe3NZif/65580754-235e-4c22-a45f-d61f9a4c64f6/tmpKYNAM8.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpe3NZif/65580754-235e-4c22-a45f-d61f9a4c64f6/tmpzim97h.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpe3NZif/65580754-235e-4c22-a45f-d61f9a4c64f6/tmpOqIVDU.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpe3NZif/65580754-235e-4c22-a45f-d61f9a4c64f6/tmpkW3O0A.tmp'] failed with output:
  2323. WARNING:toil.leader:Q/B/jobe6uQPN ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn34.grid.pub.ro
  2324. WARNING:toil.leader:Q/B/jobe6uQPN WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' Q/B/jobe6uQPN with ID Q/B/jobe6uQPN to 3
  2325. INFO:toil.leader:Issued job 'CactusSetupPhase' Q/B/jobe6uQPN with job batch system ID: 214 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  2326. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' 3/R/jobQbZW4w
  2327. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' 3/R/jobQbZW4w
  2328. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' 3/R/jobQbZW4w with ID 3/R/jobQbZW4w to 2
  2329. INFO:toil.leader:Issued job 'KtServerService' 3/R/jobQbZW4w with job batch system ID: 215 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  2330. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' Q/B/jobe6uQPN
  2331. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' Q/B/jobe6uQPN
  2332. WARNING:toil.leader:Q/B/jobe6uQPN INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  2333. WARNING:toil.leader:Q/B/jobe6uQPN INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  2334. WARNING:toil.leader:Q/B/jobe6uQPN WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  2335. WARNING:toil.leader:Q/B/jobe6uQPN INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  2336. WARNING:toil.leader:Q/B/jobe6uQPN INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  2337. WARNING:toil.leader:Q/B/jobe6uQPN INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="28850" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpg9dI3k/b21a1f43-feb8-40de-8917-57aba5d4d970/tmpvVMby2.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpg9dI3k/b21a1f43-feb8-40de-8917-57aba5d4d970/tmpGZxXMp.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpg9dI3k/b21a1f43-feb8-40de-8917-57aba5d4d970/tmpYSSVuy.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpg9dI3k/b21a1f43-feb8-40de-8917-57aba5d4d970/tmp5_ao9H.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpg9dI3k/b21a1f43-feb8-40de-8917-57aba5d4d970/tmp44pDPL.tmp']
  2338. WARNING:toil.leader:Q/B/jobe6uQPN Set log level to INFO
  2339. WARNING:toil.leader:Q/B/jobe6uQPN Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  2340. WARNING:toil.leader:Q/B/jobe6uQPN <kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="28850" />
  2341. WARNING:toil.leader:Q/B/jobe6uQPN </st_kv_database_conf>
  2342. WARNING:toil.leader:Q/B/jobe6uQPN
  2343. WARNING:toil.leader:Q/B/jobe6uQPN Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpg9dI3k/b21a1f43-feb8-40de-8917-57aba5d4d970/tmpvVMby2.tmp
  2344. WARNING:toil.leader:Q/B/jobe6uQPN Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpg9dI3k/b21a1f43-feb8-40de-8917-57aba5d4d970/tmpGZxXMp.tmp
  2345. WARNING:toil.leader:Q/B/jobe6uQPN Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpg9dI3k/b21a1f43-feb8-40de-8917-57aba5d4d970/tmpYSSVuy.tmp
  2346. WARNING:toil.leader:Q/B/jobe6uQPN Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpg9dI3k/b21a1f43-feb8-40de-8917-57aba5d4d970/tmp5_ao9H.tmp
  2347. WARNING:toil.leader:Q/B/jobe6uQPN Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpg9dI3k/b21a1f43-feb8-40de-8917-57aba5d4d970/tmp44pDPL.tmp
  2348. WARNING:toil.leader:Q/B/jobe6uQPN Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.35 with error: network error
  2349. WARNING:toil.leader:Q/B/jobe6uQPN Uncaught exception
  2350. WARNING:toil.leader:Q/B/jobe6uQPN Traceback (most recent call last):
  2351. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  2352. WARNING:toil.leader:Q/B/jobe6uQPN job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2353. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  2354. WARNING:toil.leader:Q/B/jobe6uQPN super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2355. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  2356. WARNING:toil.leader:Q/B/jobe6uQPN returnValues = self._run(jobGraph, fileStore)
  2357. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  2358. WARNING:toil.leader:Q/B/jobe6uQPN return self.run(fileStore)
  2359. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  2360. WARNING:toil.leader:Q/B/jobe6uQPN makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  2361. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  2362. WARNING:toil.leader:Q/B/jobe6uQPN parameters=["cactus_setup"] + args + sequences)
  2363. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  2364. WARNING:toil.leader:Q/B/jobe6uQPN raise RuntimeError("Command %s failed with output: %s" % (call, output))
  2365. WARNING:toil.leader:Q/B/jobe6uQPN RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="28850" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpg9dI3k/b21a1f43-feb8-40de-8917-57aba5d4d970/tmpvVMby2.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpg9dI3k/b21a1f43-feb8-40de-8917-57aba5d4d970/tmpGZxXMp.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpg9dI3k/b21a1f43-feb8-40de-8917-57aba5d4d970/tmpYSSVuy.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpg9dI3k/b21a1f43-feb8-40de-8917-57aba5d4d970/tmp5_ao9H.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpg9dI3k/b21a1f43-feb8-40de-8917-57aba5d4d970/tmp44pDPL.tmp'] failed with output:
  2366. WARNING:toil.leader:Q/B/jobe6uQPN ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn34.grid.pub.ro
  2367. WARNING:toil.leader:Q/B/jobe6uQPN WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' Q/B/jobe6uQPN with ID Q/B/jobe6uQPN to 2
  2368. INFO:toil.leader:Issued job 'CactusSetupPhase' Q/B/jobe6uQPN with job batch system ID: 216 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  2369. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' 3/R/jobQbZW4w
  2370. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' 3/R/jobQbZW4w
  2371. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' 3/R/jobQbZW4w with ID 3/R/jobQbZW4w to 1
  2372. INFO:toil.leader:Issued job 'KtServerService' 3/R/jobQbZW4w with job batch system ID: 217 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  2373. INFO:toil.leader:Reissued any over long jobs
  2374. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' Q/B/jobe6uQPN
  2375. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' Q/B/jobe6uQPN
  2376. WARNING:toil.leader:Q/B/jobe6uQPN INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  2377. WARNING:toil.leader:Q/B/jobe6uQPN INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  2378. WARNING:toil.leader:Q/B/jobe6uQPN WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  2379. WARNING:toil.leader:Q/B/jobe6uQPN WARNING:toil.fileStore:Detected that job (N/8/job2cN94i/g/tmp9MRLp0-_serialiseJob-stream) prematurely terminated. Fixing the state of the job on disk.
  2380. WARNING:toil.leader:Q/B/jobe6uQPN INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  2381. WARNING:toil.leader:Q/B/jobe6uQPN INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  2382. WARNING:toil.leader:Q/B/jobe6uQPN INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.34" port="24889" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmprfTOd2/621026ec-9ca5-4abc-9d24-fccca865391d/tmpgCMZQf.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmprfTOd2/621026ec-9ca5-4abc-9d24-fccca865391d/tmpi6sTgL.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmprfTOd2/621026ec-9ca5-4abc-9d24-fccca865391d/tmpd7X6b7.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmprfTOd2/621026ec-9ca5-4abc-9d24-fccca865391d/tmpC9mzhM.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmprfTOd2/621026ec-9ca5-4abc-9d24-fccca865391d/tmppFWUpB.tmp']
  2383. WARNING:toil.leader:Q/B/jobe6uQPN Set log level to INFO
  2384. WARNING:toil.leader:Q/B/jobe6uQPN Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  2385. WARNING:toil.leader:Q/B/jobe6uQPN <kyoto_tycoon database_dir="fakepath" host="172.16.13.34" port="24889" />
  2386. WARNING:toil.leader:Q/B/jobe6uQPN </st_kv_database_conf>
  2387. WARNING:toil.leader:Q/B/jobe6uQPN
  2388. WARNING:toil.leader:Q/B/jobe6uQPN Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmprfTOd2/621026ec-9ca5-4abc-9d24-fccca865391d/tmpgCMZQf.tmp
  2389. WARNING:toil.leader:Q/B/jobe6uQPN Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmprfTOd2/621026ec-9ca5-4abc-9d24-fccca865391d/tmpi6sTgL.tmp
  2390. WARNING:toil.leader:Q/B/jobe6uQPN Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmprfTOd2/621026ec-9ca5-4abc-9d24-fccca865391d/tmpd7X6b7.tmp
  2391. WARNING:toil.leader:Q/B/jobe6uQPN Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmprfTOd2/621026ec-9ca5-4abc-9d24-fccca865391d/tmpC9mzhM.tmp
  2392. WARNING:toil.leader:Q/B/jobe6uQPN Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmprfTOd2/621026ec-9ca5-4abc-9d24-fccca865391d/tmppFWUpB.tmp
  2393. WARNING:toil.leader:Q/B/jobe6uQPN Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.34 with error: network error
  2394. WARNING:toil.leader:Q/B/jobe6uQPN Uncaught exception
  2395. WARNING:toil.leader:Q/B/jobe6uQPN Traceback (most recent call last):
  2396. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  2397. WARNING:toil.leader:Q/B/jobe6uQPN job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2398. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  2399. WARNING:toil.leader:Q/B/jobe6uQPN super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2400. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  2401. WARNING:toil.leader:Q/B/jobe6uQPN returnValues = self._run(jobGraph, fileStore)
  2402. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  2403. WARNING:toil.leader:Q/B/jobe6uQPN return self.run(fileStore)
  2404. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  2405. WARNING:toil.leader:Q/B/jobe6uQPN makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  2406. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  2407. WARNING:toil.leader:Q/B/jobe6uQPN parameters=["cactus_setup"] + args + sequences)
  2408. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  2409. WARNING:toil.leader:Q/B/jobe6uQPN raise RuntimeError("Command %s failed with output: %s" % (call, output))
  2410. WARNING:toil.leader:Q/B/jobe6uQPN RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.34" port="24889" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmprfTOd2/621026ec-9ca5-4abc-9d24-fccca865391d/tmpgCMZQf.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmprfTOd2/621026ec-9ca5-4abc-9d24-fccca865391d/tmpi6sTgL.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmprfTOd2/621026ec-9ca5-4abc-9d24-fccca865391d/tmpd7X6b7.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmprfTOd2/621026ec-9ca5-4abc-9d24-fccca865391d/tmpC9mzhM.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmprfTOd2/621026ec-9ca5-4abc-9d24-fccca865391d/tmppFWUpB.tmp'] failed with output:
  2411. WARNING:toil.leader:Q/B/jobe6uQPN ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn35.grid.pub.ro
  2412. WARNING:toil.leader:Q/B/jobe6uQPN WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' Q/B/jobe6uQPN with ID Q/B/jobe6uQPN to 1
  2413. INFO:toil.leader:Issued job 'CactusSetupPhase' Q/B/jobe6uQPN with job batch system ID: 218 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  2414. INFO:toil.leader:Job ended successfully: 'KtServerService' 3/R/jobQbZW4w
  2415. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'KtServerService' 3/R/jobQbZW4w
  2416. WARNING:toil.leader:3/R/jobQbZW4w INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  2417. WARNING:toil.leader:3/R/jobQbZW4w INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  2418. WARNING:toil.leader:3/R/jobQbZW4w WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  2419. WARNING:toil.leader:3/R/jobQbZW4w WARNING:toil.fileStore:Detected that job (N/8/job2cN94i/g/tmp9MRLp0-_serialiseJob-stream) prematurely terminated. Fixing the state of the job on disk.
  2420. WARNING:toil.leader:3/R/jobQbZW4w WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  2421. WARNING:toil.leader:3/R/jobQbZW4w INFO:cactus.shared.common:Running the command ['netstat', '-tuplen']
  2422. WARNING:toil.leader:3/R/jobQbZW4w (No info could be read for "-p": geteuid()=98354 but you should be root.)
  2423. WARNING:toil.leader:3/R/jobQbZW4w INFO:cactus.shared.common:Running the command ['ktserver', '-port', '30415', '-ls', '-tout', '200000', '-th', '64', '-bgs', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpO1FJWj/946cec54-c920-4bc2-91e3-34dbfad074ef/tM3tOdR/snapshot', '-bgsc', 'lzo', '-bgsi', '1000000', '-log', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpO1FJWj/946cec54-c920-4bc2-91e3-34dbfad074ef/tmpv9gyN7.tmp', ':#opts=ls#bnum=30m#msiz=50g#ktopts=p']
  2424. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2425. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2426. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2427. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 558
  2428. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2429. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2430. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2431. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 558
  2432. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2433. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2434. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2435. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 559
  2436. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2437. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2438. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2439. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 560
  2440. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2441. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2442. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2443. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 561
  2444. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2445. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2446. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2447. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 562
  2448. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2449. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2450. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2451. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 563
  2452. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2453. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2454. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2455. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 564
  2456. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2457. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2458. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2459. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 565
  2460. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2461. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2462. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2463. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 566
  2464. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2465. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2466. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2467. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 567
  2468. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2469. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2470. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2471. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 568
  2472. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2473. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2474. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2475. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 569
  2476. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2477. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2478. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2479. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 570
  2480. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2481. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2482. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2483. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 571
  2484. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2485. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2486. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2487. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 572
  2488. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2489. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2490. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2491. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 573
  2492. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2493. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2494. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2495. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 574
  2496. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2497. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2498. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2499. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 575
  2500. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2501. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2502. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2503. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 576
  2504. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2505. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2506. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2507. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 577
  2508. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2509. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2510. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2511. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 578
  2512. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2513. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2514. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2515. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 579
  2516. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2517. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2518. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2519. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 580
  2520. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2521. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2522. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2523. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 581
  2524. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2525. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2526. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2527. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 582
  2528. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2529. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2530. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2531. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 583
  2532. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2533. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2534. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2535. WARNING:toil.leader:3/R/jobQbZW4w INFO:toil.lib.bioio:Ktserver running.
  2536. WARNING:toil.leader:3/R/jobQbZW4w INFO:toil.lib.bioio:Ktserver running.
  2537. WARNING:toil.leader:3/R/jobQbZW4w INFO:toil.lib.bioio:Ktserver running.
  2538. WARNING:toil.leader:3/R/jobQbZW4w INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '30415', '-host', '172.16.13.34', 'TERMINATE']
  2539. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 584
  2540. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2541. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2542. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2543. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 586
  2544. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2545. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2546. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2547. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 587
  2548. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2549. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2550. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2551. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 588
  2552. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2553. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2554. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2555. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 589
  2556. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2557. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2558. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2559. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 590
  2560. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2561. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2562. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2563. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 591
  2564. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2565. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2566. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2567. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 592
  2568. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2569. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2570. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2571. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 593
  2572. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2573. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2574. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2575. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 594
  2576. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2577. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2578. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2579. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 595
  2580. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2581. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2582. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2583. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 596
  2584. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2585. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2586. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2587. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 597
  2588. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2589. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2590. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2591. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 598
  2592. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2593. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2594. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2595. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 599
  2596. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2597. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2598. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2599. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 600
  2600. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2601. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2602. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2603. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 601
  2604. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2605. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2606. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2607. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 602
  2608. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2609. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2610. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2611. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 603
  2612. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2613. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2614. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2615. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 604
  2616. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2617. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2618. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2619. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 605
  2620. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2621. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2622. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2623. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 606
  2624. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2625. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2626. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2627. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 607
  2628. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2629. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2630. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2631. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 608
  2632. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2633. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2634. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2635. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 609
  2636. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2637. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2638. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2639. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 610
  2640. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2641. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2642. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2643. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 611
  2644. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2645. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2646. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2647. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 612
  2648. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2649. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2650. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2651. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 613
  2652. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2653. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2654. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2655. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 614
  2656. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2657. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2658. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2659. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 615
  2660. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2661. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2662. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2663. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 616
  2664. WARNING:toil.leader:3/R/jobQbZW4w Stack limits: Infinity Infinity
  2665. WARNING:toil.leader:3/R/jobQbZW4w Process limits: 511096 511096
  2666. WARNING:toil.leader:3/R/jobQbZW4w Kernel thread cap: 1022192
  2667. WARNING:toil.leader:3/R/jobQbZW4w Current total threads running: 617
  2668. WARNING:toil.leader:3/R/jobQbZW4w terminate called after throwing an instance of 'std::runtime_error'
  2669. WARNING:toil.leader:3/R/jobQbZW4w what(): pthread_create: EAGAIN: Insufficient resources to create thread, or thread limit reached
  2670. WARNING:toil.leader:3/R/jobQbZW4w Process ServerProcess-1:
  2671. WARNING:toil.leader:3/R/jobQbZW4w Traceback (most recent call last):
  2672. WARNING:toil.leader:3/R/jobQbZW4w File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap
  2673. WARNING:toil.leader:3/R/jobQbZW4w self.run()
  2674. WARNING:toil.leader:3/R/jobQbZW4w File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/ktserverControl.py", line 82, in run
  2675. WARNING:toil.leader:3/R/jobQbZW4w self.tryRun(*self.args, **self.kwargs)
  2676. WARNING:toil.leader:3/R/jobQbZW4w File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/ktserverControl.py", line 118, in tryRun
  2677. WARNING:toil.leader:3/R/jobQbZW4w raise RuntimeError("KTServer failed. Log: %s" % f.read())
  2678. WARNING:toil.leader:3/R/jobQbZW4w RuntimeError: KTServer failed. Log: 2019-03-16T22:41:59.786614+02:00: [SYSTEM]: ================ [START]: pid=22381
  2679. WARNING:toil.leader:3/R/jobQbZW4w 2019-03-16T22:41:59.786803+02:00: [SYSTEM]: opening a database: path=:#opts=ls#bnum=30m#msiz=50g#ktopts=p
  2680. WARNING:toil.leader:3/R/jobQbZW4w 2019-03-16T22:41:59.834009+02:00: [SYSTEM]: starting the server: expr=:30415
  2681. WARNING:toil.leader:3/R/jobQbZW4w 2019-03-16T22:41:59.834166+02:00: [SYSTEM]: server socket opened: expr=:30415 timeout=200000.0
  2682. WARNING:toil.leader:3/R/jobQbZW4w 2019-03-16T22:41:59.834194+02:00: [SYSTEM]: listening server socket started: fd=4
  2683. WARNING:toil.leader:3/R/jobQbZW4w
  2684. WARNING:toil.leader:3/R/jobQbZW4w INFO:cactus.shared.common:Running the command ['ktremotemgr', 'set', '-port', '30415', '-host', '172.16.13.34', 'TERMINATE', '1']
  2685. WARNING:toil.leader:3/R/jobQbZW4w ktremotemgr: DB::open failed: : 6: network error: connection failed
  2686. WARNING:toil.leader:3/R/jobQbZW4w Traceback (most recent call last):
  2687. WARNING:toil.leader:3/R/jobQbZW4w File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  2688. WARNING:toil.leader:3/R/jobQbZW4w job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2689. WARNING:toil.leader:3/R/jobQbZW4w File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  2690. WARNING:toil.leader:3/R/jobQbZW4w returnValues = self._run(jobGraph, fileStore)
  2691. WARNING:toil.leader:3/R/jobQbZW4w File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1694, in _run
  2692. WARNING:toil.leader:3/R/jobQbZW4w returnValues = self.run(fileStore)
  2693. WARNING:toil.leader:3/R/jobQbZW4w File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1673, in run
  2694. WARNING:toil.leader:3/R/jobQbZW4w if not service.check():
  2695. WARNING:toil.leader:3/R/jobQbZW4w File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/ktserverToil.py", line 55, in check
  2696. WARNING:toil.leader:3/R/jobQbZW4w raise RuntimeError(msg)
  2697. WARNING:toil.leader:3/R/jobQbZW4w RuntimeError: Traceback (most recent call last):
  2698. WARNING:toil.leader:3/R/jobQbZW4w File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/ktserverControl.py", line 82, in run
  2699. WARNING:toil.leader:3/R/jobQbZW4w self.tryRun(*self.args, **self.kwargs)
  2700. WARNING:toil.leader:3/R/jobQbZW4w File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/ktserverControl.py", line 118, in tryRun
  2701. WARNING:toil.leader:3/R/jobQbZW4w raise RuntimeError("KTServer failed. Log: %s" % f.read())
  2702. WARNING:toil.leader:3/R/jobQbZW4w RuntimeError: KTServer failed. Log: 2019-03-16T22:41:59.786614+02:00: [SYSTEM]: ================ [START]: pid=22381
  2703. WARNING:toil.leader:3/R/jobQbZW4w 2019-03-16T22:41:59.786803+02:00: [SYSTEM]: opening a database: path=:#opts=ls#bnum=30m#msiz=50g#ktopts=p
  2704. WARNING:toil.leader:3/R/jobQbZW4w 2019-03-16T22:41:59.834009+02:00: [SYSTEM]: starting the server: expr=:30415
  2705. WARNING:toil.leader:3/R/jobQbZW4w 2019-03-16T22:41:59.834166+02:00: [SYSTEM]: server socket opened: expr=:30415 timeout=200000.0
  2706. WARNING:toil.leader:3/R/jobQbZW4w 2019-03-16T22:41:59.834194+02:00: [SYSTEM]: listening server socket started: fd=4
  2707. WARNING:toil.leader:3/R/jobQbZW4w
  2708. WARNING:toil.leader:3/R/jobQbZW4w
  2709. WARNING:toil.leader:3/R/jobQbZW4w ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn34.grid.pub.ro
  2710. WARNING:toil.leader:3/R/jobQbZW4w WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' 3/R/jobQbZW4w with ID 3/R/jobQbZW4w to 0
  2711. WARNING:toil.leader:Job 'KtServerService' 3/R/jobQbZW4w with ID 3/R/jobQbZW4w is completely failed
  2712. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' Q/B/jobe6uQPN
  2713. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' Q/B/jobe6uQPN
  2714. WARNING:toil.leader:Q/B/jobe6uQPN INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  2715. WARNING:toil.leader:Q/B/jobe6uQPN INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  2716. WARNING:toil.leader:Q/B/jobe6uQPN WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  2717. WARNING:toil.leader:Q/B/jobe6uQPN INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  2718. WARNING:toil.leader:Q/B/jobe6uQPN INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  2719. WARNING:toil.leader:Q/B/jobe6uQPN INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.34" port="30415" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpF4y0h2/8a70ffef-23ce-4a6c-bd72-f433dc1c49b1/tmpJSx4UY.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpF4y0h2/8a70ffef-23ce-4a6c-bd72-f433dc1c49b1/tmpjOifs1.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpF4y0h2/8a70ffef-23ce-4a6c-bd72-f433dc1c49b1/tmpRXmdnG.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpF4y0h2/8a70ffef-23ce-4a6c-bd72-f433dc1c49b1/tmpeDF2ji.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpF4y0h2/8a70ffef-23ce-4a6c-bd72-f433dc1c49b1/tmph0NmVg.tmp']
  2720. WARNING:toil.leader:Q/B/jobe6uQPN Set log level to INFO
  2721. WARNING:toil.leader:Q/B/jobe6uQPN Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  2722. WARNING:toil.leader:Q/B/jobe6uQPN <kyoto_tycoon database_dir="fakepath" host="172.16.13.34" port="30415" />
  2723. WARNING:toil.leader:Q/B/jobe6uQPN </st_kv_database_conf>
  2724. WARNING:toil.leader:Q/B/jobe6uQPN
  2725. WARNING:toil.leader:Q/B/jobe6uQPN Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpF4y0h2/8a70ffef-23ce-4a6c-bd72-f433dc1c49b1/tmpJSx4UY.tmp
  2726. WARNING:toil.leader:Q/B/jobe6uQPN Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpF4y0h2/8a70ffef-23ce-4a6c-bd72-f433dc1c49b1/tmpjOifs1.tmp
  2727. WARNING:toil.leader:Q/B/jobe6uQPN Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpF4y0h2/8a70ffef-23ce-4a6c-bd72-f433dc1c49b1/tmpRXmdnG.tmp
  2728. WARNING:toil.leader:Q/B/jobe6uQPN Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpF4y0h2/8a70ffef-23ce-4a6c-bd72-f433dc1c49b1/tmpeDF2ji.tmp
  2729. WARNING:toil.leader:Q/B/jobe6uQPN Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpF4y0h2/8a70ffef-23ce-4a6c-bd72-f433dc1c49b1/tmph0NmVg.tmp
  2730. WARNING:toil.leader:Q/B/jobe6uQPN Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.34 with error: network error
  2731. WARNING:toil.leader:Q/B/jobe6uQPN Uncaught exception
  2732. WARNING:toil.leader:Q/B/jobe6uQPN Traceback (most recent call last):
  2733. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  2734. WARNING:toil.leader:Q/B/jobe6uQPN job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2735. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  2736. WARNING:toil.leader:Q/B/jobe6uQPN super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2737. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  2738. WARNING:toil.leader:Q/B/jobe6uQPN returnValues = self._run(jobGraph, fileStore)
  2739. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  2740. WARNING:toil.leader:Q/B/jobe6uQPN return self.run(fileStore)
  2741. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  2742. WARNING:toil.leader:Q/B/jobe6uQPN makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  2743. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  2744. WARNING:toil.leader:Q/B/jobe6uQPN parameters=["cactus_setup"] + args + sequences)
  2745. WARNING:toil.leader:Q/B/jobe6uQPN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  2746. WARNING:toil.leader:Q/B/jobe6uQPN raise RuntimeError("Command %s failed with output: %s" % (call, output))
  2747. WARNING:toil.leader:Q/B/jobe6uQPN RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.34" port="30415" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpF4y0h2/8a70ffef-23ce-4a6c-bd72-f433dc1c49b1/tmpJSx4UY.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpF4y0h2/8a70ffef-23ce-4a6c-bd72-f433dc1c49b1/tmpjOifs1.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpF4y0h2/8a70ffef-23ce-4a6c-bd72-f433dc1c49b1/tmpRXmdnG.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpF4y0h2/8a70ffef-23ce-4a6c-bd72-f433dc1c49b1/tmpeDF2ji.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-906375ca-4612-4b9c-aace-ad64276eee7f/tmpF4y0h2/8a70ffef-23ce-4a6c-bd72-f433dc1c49b1/tmph0NmVg.tmp'] failed with output:
  2748. WARNING:toil.leader:Q/B/jobe6uQPN ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn34.grid.pub.ro
  2749. WARNING:toil.leader:Q/B/jobe6uQPN WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' Q/B/jobe6uQPN with ID Q/B/jobe6uQPN to 0
  2750. WARNING:toil.leader:Job 'CactusSetupPhase' Q/B/jobe6uQPN with ID Q/B/jobe6uQPN is completely failed
  2751. WARNING:toil.leader:Job: N/8/job2cN94i is being restarted as a checkpoint after the total failure of jobs in its subtree.
  2752. INFO:toil.leader:Issued job 'StartPrimaryDB' N/8/job2cN94i with job batch system ID: 219 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  2753. INFO:toil.leader:Job ended successfully: 'StartPrimaryDB' N/8/job2cN94i
  2754. INFO:toil.leader:Issued job 'KtServerService' 1/r/jobM3ECWT with job batch system ID: 220 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  2755. INFO:toil.leader:Issued job 'CactusSetupPhase' h/G/jobNV0VjP with job batch system ID: 221 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  2756. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' 1/r/jobM3ECWT
  2757. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' 1/r/jobM3ECWT
  2758. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' 1/r/jobM3ECWT with ID 1/r/jobM3ECWT to 5
  2759. INFO:toil.leader:Issued job 'KtServerService' 1/r/jobM3ECWT with job batch system ID: 222 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  2760. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' h/G/jobNV0VjP
  2761. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' h/G/jobNV0VjP
  2762. WARNING:toil.leader:h/G/jobNV0VjP INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  2763. WARNING:toil.leader:h/G/jobNV0VjP INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  2764. WARNING:toil.leader:h/G/jobNV0VjP WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  2765. WARNING:toil.leader:h/G/jobNV0VjP INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  2766. WARNING:toil.leader:h/G/jobNV0VjP INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  2767. WARNING:toil.leader:h/G/jobNV0VjP INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.34" port="7396" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmp31FEhL/f80c544d-1ca0-4d29-ac29-2605907b6974/tmpAp5bNF.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmp31FEhL/f80c544d-1ca0-4d29-ac29-2605907b6974/tmpl6xf_X.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmp31FEhL/f80c544d-1ca0-4d29-ac29-2605907b6974/tmpCM791z.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmp31FEhL/f80c544d-1ca0-4d29-ac29-2605907b6974/tmpeqDxZC.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmp31FEhL/f80c544d-1ca0-4d29-ac29-2605907b6974/tmpU7EI8_.tmp']
  2768. WARNING:toil.leader:h/G/jobNV0VjP Set log level to INFO
  2769. WARNING:toil.leader:h/G/jobNV0VjP Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  2770. WARNING:toil.leader:h/G/jobNV0VjP <kyoto_tycoon database_dir="fakepath" host="172.16.13.34" port="7396" />
  2771. WARNING:toil.leader:h/G/jobNV0VjP </st_kv_database_conf>
  2772. WARNING:toil.leader:h/G/jobNV0VjP
  2773. WARNING:toil.leader:h/G/jobNV0VjP Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmp31FEhL/f80c544d-1ca0-4d29-ac29-2605907b6974/tmpAp5bNF.tmp
  2774. WARNING:toil.leader:h/G/jobNV0VjP Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmp31FEhL/f80c544d-1ca0-4d29-ac29-2605907b6974/tmpl6xf_X.tmp
  2775. WARNING:toil.leader:h/G/jobNV0VjP Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmp31FEhL/f80c544d-1ca0-4d29-ac29-2605907b6974/tmpCM791z.tmp
  2776. WARNING:toil.leader:h/G/jobNV0VjP Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmp31FEhL/f80c544d-1ca0-4d29-ac29-2605907b6974/tmpeqDxZC.tmp
  2777. WARNING:toil.leader:h/G/jobNV0VjP Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmp31FEhL/f80c544d-1ca0-4d29-ac29-2605907b6974/tmpU7EI8_.tmp
  2778. WARNING:toil.leader:h/G/jobNV0VjP Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.34 with error: network error
  2779. WARNING:toil.leader:h/G/jobNV0VjP Uncaught exception
  2780. WARNING:toil.leader:h/G/jobNV0VjP Traceback (most recent call last):
  2781. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  2782. WARNING:toil.leader:h/G/jobNV0VjP job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2783. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  2784. WARNING:toil.leader:h/G/jobNV0VjP super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2785. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  2786. WARNING:toil.leader:h/G/jobNV0VjP returnValues = self._run(jobGraph, fileStore)
  2787. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  2788. WARNING:toil.leader:h/G/jobNV0VjP return self.run(fileStore)
  2789. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  2790. WARNING:toil.leader:h/G/jobNV0VjP makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  2791. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  2792. WARNING:toil.leader:h/G/jobNV0VjP parameters=["cactus_setup"] + args + sequences)
  2793. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  2794. WARNING:toil.leader:h/G/jobNV0VjP raise RuntimeError("Command %s failed with output: %s" % (call, output))
  2795. WARNING:toil.leader:h/G/jobNV0VjP RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.34" port="7396" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmp31FEhL/f80c544d-1ca0-4d29-ac29-2605907b6974/tmpAp5bNF.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmp31FEhL/f80c544d-1ca0-4d29-ac29-2605907b6974/tmpl6xf_X.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmp31FEhL/f80c544d-1ca0-4d29-ac29-2605907b6974/tmpCM791z.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmp31FEhL/f80c544d-1ca0-4d29-ac29-2605907b6974/tmpeqDxZC.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmp31FEhL/f80c544d-1ca0-4d29-ac29-2605907b6974/tmpU7EI8_.tmp'] failed with output:
  2796. WARNING:toil.leader:h/G/jobNV0VjP ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn30.grid.pub.ro
  2797. WARNING:toil.leader:h/G/jobNV0VjP WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' h/G/jobNV0VjP with ID h/G/jobNV0VjP to 5
  2798. INFO:toil.leader:Issued job 'CactusSetupPhase' h/G/jobNV0VjP with job batch system ID: 223 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  2799. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' 1/r/jobM3ECWT
  2800. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' 1/r/jobM3ECWT
  2801. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' 1/r/jobM3ECWT with ID 1/r/jobM3ECWT to 4
  2802. INFO:toil.leader:Issued job 'KtServerService' 1/r/jobM3ECWT with job batch system ID: 224 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  2803. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' 1/r/jobM3ECWT
  2804. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' 1/r/jobM3ECWT
  2805. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' 1/r/jobM3ECWT with ID 1/r/jobM3ECWT to 3
  2806. INFO:toil.leader:Issued job 'KtServerService' 1/r/jobM3ECWT with job batch system ID: 225 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  2807. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' h/G/jobNV0VjP
  2808. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' h/G/jobNV0VjP
  2809. WARNING:toil.leader:h/G/jobNV0VjP INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  2810. WARNING:toil.leader:h/G/jobNV0VjP INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  2811. WARNING:toil.leader:h/G/jobNV0VjP WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  2812. WARNING:toil.leader:h/G/jobNV0VjP WARNING:toil.fileStore:Detected that job (N/8/job2cN94i/g/tmp5SHicD-_serialiseJob-stream) prematurely terminated. Fixing the state of the job on disk.
  2813. WARNING:toil.leader:h/G/jobNV0VjP INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  2814. WARNING:toil.leader:h/G/jobNV0VjP INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  2815. WARNING:toil.leader:h/G/jobNV0VjP INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.30" port="11973" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpmUdMtI/9f4d0fe3-4824-4451-9a66-40ad570c0745/tmpV_bAx1.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpmUdMtI/9f4d0fe3-4824-4451-9a66-40ad570c0745/tmp7rJqnd.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpmUdMtI/9f4d0fe3-4824-4451-9a66-40ad570c0745/tmpgftLy1.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpmUdMtI/9f4d0fe3-4824-4451-9a66-40ad570c0745/tmpXkOJ8w.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpmUdMtI/9f4d0fe3-4824-4451-9a66-40ad570c0745/tmpm5JwMF.tmp']
  2816. WARNING:toil.leader:h/G/jobNV0VjP Set log level to INFO
  2817. WARNING:toil.leader:h/G/jobNV0VjP Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  2818. WARNING:toil.leader:h/G/jobNV0VjP <kyoto_tycoon database_dir="fakepath" host="172.16.13.30" port="11973" />
  2819. WARNING:toil.leader:h/G/jobNV0VjP </st_kv_database_conf>
  2820. WARNING:toil.leader:h/G/jobNV0VjP
  2821. WARNING:toil.leader:h/G/jobNV0VjP Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpmUdMtI/9f4d0fe3-4824-4451-9a66-40ad570c0745/tmpV_bAx1.tmp
  2822. WARNING:toil.leader:h/G/jobNV0VjP Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpmUdMtI/9f4d0fe3-4824-4451-9a66-40ad570c0745/tmp7rJqnd.tmp
  2823. WARNING:toil.leader:h/G/jobNV0VjP Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpmUdMtI/9f4d0fe3-4824-4451-9a66-40ad570c0745/tmpgftLy1.tmp
  2824. WARNING:toil.leader:h/G/jobNV0VjP Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpmUdMtI/9f4d0fe3-4824-4451-9a66-40ad570c0745/tmpXkOJ8w.tmp
  2825. WARNING:toil.leader:h/G/jobNV0VjP Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpmUdMtI/9f4d0fe3-4824-4451-9a66-40ad570c0745/tmpm5JwMF.tmp
  2826. WARNING:toil.leader:h/G/jobNV0VjP Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.30 with error: network error
  2827. WARNING:toil.leader:h/G/jobNV0VjP Uncaught exception
  2828. WARNING:toil.leader:h/G/jobNV0VjP Traceback (most recent call last):
  2829. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  2830. WARNING:toil.leader:h/G/jobNV0VjP job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2831. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  2832. WARNING:toil.leader:h/G/jobNV0VjP super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2833. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  2834. WARNING:toil.leader:h/G/jobNV0VjP returnValues = self._run(jobGraph, fileStore)
  2835. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  2836. WARNING:toil.leader:h/G/jobNV0VjP return self.run(fileStore)
  2837. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  2838. WARNING:toil.leader:h/G/jobNV0VjP makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  2839. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  2840. WARNING:toil.leader:h/G/jobNV0VjP parameters=["cactus_setup"] + args + sequences)
  2841. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  2842. WARNING:toil.leader:h/G/jobNV0VjP raise RuntimeError("Command %s failed with output: %s" % (call, output))
  2843. WARNING:toil.leader:h/G/jobNV0VjP RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.30" port="11973" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpmUdMtI/9f4d0fe3-4824-4451-9a66-40ad570c0745/tmpV_bAx1.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpmUdMtI/9f4d0fe3-4824-4451-9a66-40ad570c0745/tmp7rJqnd.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpmUdMtI/9f4d0fe3-4824-4451-9a66-40ad570c0745/tmpgftLy1.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpmUdMtI/9f4d0fe3-4824-4451-9a66-40ad570c0745/tmpXkOJ8w.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpmUdMtI/9f4d0fe3-4824-4451-9a66-40ad570c0745/tmpm5JwMF.tmp'] failed with output:
  2844. WARNING:toil.leader:h/G/jobNV0VjP ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn41.grid.pub.ro
  2845. WARNING:toil.leader:h/G/jobNV0VjP WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' h/G/jobNV0VjP with ID h/G/jobNV0VjP to 4
  2846. INFO:toil.leader:Issued job 'CactusSetupPhase' h/G/jobNV0VjP with job batch system ID: 226 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  2847. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' 1/r/jobM3ECWT
  2848. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' 1/r/jobM3ECWT
  2849. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' 1/r/jobM3ECWT with ID 1/r/jobM3ECWT to 2
  2850. INFO:toil.leader:Issued job 'KtServerService' 1/r/jobM3ECWT with job batch system ID: 227 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  2851. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' h/G/jobNV0VjP
  2852. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' h/G/jobNV0VjP
  2853. WARNING:toil.leader:h/G/jobNV0VjP INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  2854. WARNING:toil.leader:h/G/jobNV0VjP INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  2855. WARNING:toil.leader:h/G/jobNV0VjP WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  2856. WARNING:toil.leader:h/G/jobNV0VjP INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  2857. WARNING:toil.leader:h/G/jobNV0VjP INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  2858. WARNING:toil.leader:h/G/jobNV0VjP INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.30" port="11973" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpUC7Mty/57b2b5df-3f44-4227-94b2-8451568631a7/tmp1dQyNQ.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpUC7Mty/57b2b5df-3f44-4227-94b2-8451568631a7/tmpNNFQhk.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpUC7Mty/57b2b5df-3f44-4227-94b2-8451568631a7/tmpsPAMNX.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpUC7Mty/57b2b5df-3f44-4227-94b2-8451568631a7/tmpulYfMp.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpUC7Mty/57b2b5df-3f44-4227-94b2-8451568631a7/tmpKNlG0J.tmp']
  2859. WARNING:toil.leader:h/G/jobNV0VjP Set log level to INFO
  2860. WARNING:toil.leader:h/G/jobNV0VjP Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  2861. WARNING:toil.leader:h/G/jobNV0VjP <kyoto_tycoon database_dir="fakepath" host="172.16.13.30" port="11973" />
  2862. WARNING:toil.leader:h/G/jobNV0VjP </st_kv_database_conf>
  2863. WARNING:toil.leader:h/G/jobNV0VjP
  2864. WARNING:toil.leader:h/G/jobNV0VjP Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpUC7Mty/57b2b5df-3f44-4227-94b2-8451568631a7/tmp1dQyNQ.tmp
  2865. WARNING:toil.leader:h/G/jobNV0VjP Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpUC7Mty/57b2b5df-3f44-4227-94b2-8451568631a7/tmpNNFQhk.tmp
  2866. WARNING:toil.leader:h/G/jobNV0VjP Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpUC7Mty/57b2b5df-3f44-4227-94b2-8451568631a7/tmpsPAMNX.tmp
  2867. WARNING:toil.leader:h/G/jobNV0VjP Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpUC7Mty/57b2b5df-3f44-4227-94b2-8451568631a7/tmpulYfMp.tmp
  2868. WARNING:toil.leader:h/G/jobNV0VjP Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpUC7Mty/57b2b5df-3f44-4227-94b2-8451568631a7/tmpKNlG0J.tmp
  2869. WARNING:toil.leader:h/G/jobNV0VjP Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.30 with error: network error
  2870. WARNING:toil.leader:h/G/jobNV0VjP Uncaught exception
  2871. WARNING:toil.leader:h/G/jobNV0VjP Traceback (most recent call last):
  2872. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  2873. WARNING:toil.leader:h/G/jobNV0VjP job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2874. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  2875. WARNING:toil.leader:h/G/jobNV0VjP super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2876. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  2877. WARNING:toil.leader:h/G/jobNV0VjP returnValues = self._run(jobGraph, fileStore)
  2878. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  2879. WARNING:toil.leader:h/G/jobNV0VjP return self.run(fileStore)
  2880. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  2881. WARNING:toil.leader:h/G/jobNV0VjP makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  2882. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  2883. WARNING:toil.leader:h/G/jobNV0VjP parameters=["cactus_setup"] + args + sequences)
  2884. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  2885. WARNING:toil.leader:h/G/jobNV0VjP raise RuntimeError("Command %s failed with output: %s" % (call, output))
  2886. WARNING:toil.leader:h/G/jobNV0VjP RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.30" port="11973" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpUC7Mty/57b2b5df-3f44-4227-94b2-8451568631a7/tmp1dQyNQ.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpUC7Mty/57b2b5df-3f44-4227-94b2-8451568631a7/tmpNNFQhk.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpUC7Mty/57b2b5df-3f44-4227-94b2-8451568631a7/tmpsPAMNX.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpUC7Mty/57b2b5df-3f44-4227-94b2-8451568631a7/tmpulYfMp.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmpUC7Mty/57b2b5df-3f44-4227-94b2-8451568631a7/tmpKNlG0J.tmp'] failed with output:
  2887. WARNING:toil.leader:h/G/jobNV0VjP ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn35.grid.pub.ro
  2888. WARNING:toil.leader:h/G/jobNV0VjP WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' h/G/jobNV0VjP with ID h/G/jobNV0VjP to 3
  2889. INFO:toil.leader:Issued job 'CactusSetupPhase' h/G/jobNV0VjP with job batch system ID: 228 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  2890. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' 1/r/jobM3ECWT
  2891. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' 1/r/jobM3ECWT
  2892. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' 1/r/jobM3ECWT with ID 1/r/jobM3ECWT to 1
  2893. INFO:toil.leader:Issued job 'KtServerService' 1/r/jobM3ECWT with job batch system ID: 229 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  2894. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' h/G/jobNV0VjP
  2895. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' h/G/jobNV0VjP
  2896. WARNING:toil.leader:h/G/jobNV0VjP INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  2897. WARNING:toil.leader:h/G/jobNV0VjP INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  2898. WARNING:toil.leader:h/G/jobNV0VjP WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  2899. WARNING:toil.leader:h/G/jobNV0VjP INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  2900. WARNING:toil.leader:h/G/jobNV0VjP INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  2901. WARNING:toil.leader:h/G/jobNV0VjP INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="27393" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpjC6eIg/83d4c2b7-7b49-4259-8bc7-75fe64459e88/tmpW0GLKA.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpjC6eIg/83d4c2b7-7b49-4259-8bc7-75fe64459e88/tmpa9vYBF.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpjC6eIg/83d4c2b7-7b49-4259-8bc7-75fe64459e88/tmpqrFnLS.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpjC6eIg/83d4c2b7-7b49-4259-8bc7-75fe64459e88/tmpcDICnw.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpjC6eIg/83d4c2b7-7b49-4259-8bc7-75fe64459e88/tmpFxTUjv.tmp']
  2902. WARNING:toil.leader:h/G/jobNV0VjP Set log level to INFO
  2903. WARNING:toil.leader:h/G/jobNV0VjP Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  2904. WARNING:toil.leader:h/G/jobNV0VjP <kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="27393" />
  2905. WARNING:toil.leader:h/G/jobNV0VjP </st_kv_database_conf>
  2906. WARNING:toil.leader:h/G/jobNV0VjP
  2907. WARNING:toil.leader:h/G/jobNV0VjP Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpjC6eIg/83d4c2b7-7b49-4259-8bc7-75fe64459e88/tmpW0GLKA.tmp
  2908. WARNING:toil.leader:h/G/jobNV0VjP Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpjC6eIg/83d4c2b7-7b49-4259-8bc7-75fe64459e88/tmpa9vYBF.tmp
  2909. WARNING:toil.leader:h/G/jobNV0VjP Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpjC6eIg/83d4c2b7-7b49-4259-8bc7-75fe64459e88/tmpqrFnLS.tmp
  2910. WARNING:toil.leader:h/G/jobNV0VjP Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpjC6eIg/83d4c2b7-7b49-4259-8bc7-75fe64459e88/tmpcDICnw.tmp
  2911. WARNING:toil.leader:h/G/jobNV0VjP Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpjC6eIg/83d4c2b7-7b49-4259-8bc7-75fe64459e88/tmpFxTUjv.tmp
  2912. WARNING:toil.leader:h/G/jobNV0VjP Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.41 with error: network error
  2913. WARNING:toil.leader:h/G/jobNV0VjP Uncaught exception
  2914. WARNING:toil.leader:h/G/jobNV0VjP Traceback (most recent call last):
  2915. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  2916. WARNING:toil.leader:h/G/jobNV0VjP job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2917. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  2918. WARNING:toil.leader:h/G/jobNV0VjP super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2919. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  2920. WARNING:toil.leader:h/G/jobNV0VjP returnValues = self._run(jobGraph, fileStore)
  2921. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  2922. WARNING:toil.leader:h/G/jobNV0VjP return self.run(fileStore)
  2923. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  2924. WARNING:toil.leader:h/G/jobNV0VjP makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  2925. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  2926. WARNING:toil.leader:h/G/jobNV0VjP parameters=["cactus_setup"] + args + sequences)
  2927. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  2928. WARNING:toil.leader:h/G/jobNV0VjP raise RuntimeError("Command %s failed with output: %s" % (call, output))
  2929. WARNING:toil.leader:h/G/jobNV0VjP RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="27393" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpjC6eIg/83d4c2b7-7b49-4259-8bc7-75fe64459e88/tmpW0GLKA.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpjC6eIg/83d4c2b7-7b49-4259-8bc7-75fe64459e88/tmpa9vYBF.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpjC6eIg/83d4c2b7-7b49-4259-8bc7-75fe64459e88/tmpqrFnLS.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpjC6eIg/83d4c2b7-7b49-4259-8bc7-75fe64459e88/tmpcDICnw.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpjC6eIg/83d4c2b7-7b49-4259-8bc7-75fe64459e88/tmpFxTUjv.tmp'] failed with output:
  2930. WARNING:toil.leader:h/G/jobNV0VjP ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn38.grid.pub.ro
  2931. WARNING:toil.leader:h/G/jobNV0VjP WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' h/G/jobNV0VjP with ID h/G/jobNV0VjP to 2
  2932. INFO:toil.leader:Issued job 'CactusSetupPhase' h/G/jobNV0VjP with job batch system ID: 230 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  2933. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' 1/r/jobM3ECWT
  2934. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' 1/r/jobM3ECWT
  2935. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' 1/r/jobM3ECWT with ID 1/r/jobM3ECWT to 0
  2936. WARNING:toil.leader:Job 'KtServerService' 1/r/jobM3ECWT with ID 1/r/jobM3ECWT is completely failed
  2937. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' h/G/jobNV0VjP
  2938. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' h/G/jobNV0VjP
  2939. WARNING:toil.leader:h/G/jobNV0VjP INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  2940. WARNING:toil.leader:h/G/jobNV0VjP INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  2941. WARNING:toil.leader:h/G/jobNV0VjP WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  2942. WARNING:toil.leader:h/G/jobNV0VjP INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  2943. WARNING:toil.leader:h/G/jobNV0VjP INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  2944. WARNING:toil.leader:h/G/jobNV0VjP INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.38" port="8662" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmp3YBSbi/7c7403eb-20a3-4eb0-99fd-2cdc1232ac97/tmpbNeBYK.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmp3YBSbi/7c7403eb-20a3-4eb0-99fd-2cdc1232ac97/tmpTu4qqc.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmp3YBSbi/7c7403eb-20a3-4eb0-99fd-2cdc1232ac97/tmpDdTNJy.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmp3YBSbi/7c7403eb-20a3-4eb0-99fd-2cdc1232ac97/tmpD3mGF1.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmp3YBSbi/7c7403eb-20a3-4eb0-99fd-2cdc1232ac97/tmpi0xJkT.tmp']
  2945. WARNING:toil.leader:h/G/jobNV0VjP Set log level to INFO
  2946. WARNING:toil.leader:h/G/jobNV0VjP Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  2947. WARNING:toil.leader:h/G/jobNV0VjP <kyoto_tycoon database_dir="fakepath" host="172.16.13.38" port="8662" />
  2948. WARNING:toil.leader:h/G/jobNV0VjP </st_kv_database_conf>
  2949. WARNING:toil.leader:h/G/jobNV0VjP
  2950. WARNING:toil.leader:h/G/jobNV0VjP Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmp3YBSbi/7c7403eb-20a3-4eb0-99fd-2cdc1232ac97/tmpbNeBYK.tmp
  2951. WARNING:toil.leader:h/G/jobNV0VjP Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmp3YBSbi/7c7403eb-20a3-4eb0-99fd-2cdc1232ac97/tmpTu4qqc.tmp
  2952. WARNING:toil.leader:h/G/jobNV0VjP Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmp3YBSbi/7c7403eb-20a3-4eb0-99fd-2cdc1232ac97/tmpDdTNJy.tmp
  2953. WARNING:toil.leader:h/G/jobNV0VjP Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmp3YBSbi/7c7403eb-20a3-4eb0-99fd-2cdc1232ac97/tmpD3mGF1.tmp
  2954. WARNING:toil.leader:h/G/jobNV0VjP Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmp3YBSbi/7c7403eb-20a3-4eb0-99fd-2cdc1232ac97/tmpi0xJkT.tmp
  2955. WARNING:toil.leader:h/G/jobNV0VjP Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.38 with error: network error
  2956. WARNING:toil.leader:h/G/jobNV0VjP Uncaught exception
  2957. WARNING:toil.leader:h/G/jobNV0VjP Traceback (most recent call last):
  2958. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  2959. WARNING:toil.leader:h/G/jobNV0VjP job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2960. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  2961. WARNING:toil.leader:h/G/jobNV0VjP super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2962. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  2963. WARNING:toil.leader:h/G/jobNV0VjP returnValues = self._run(jobGraph, fileStore)
  2964. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  2965. WARNING:toil.leader:h/G/jobNV0VjP return self.run(fileStore)
  2966. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  2967. WARNING:toil.leader:h/G/jobNV0VjP makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  2968. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  2969. WARNING:toil.leader:h/G/jobNV0VjP parameters=["cactus_setup"] + args + sequences)
  2970. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  2971. WARNING:toil.leader:h/G/jobNV0VjP raise RuntimeError("Command %s failed with output: %s" % (call, output))
  2972. WARNING:toil.leader:h/G/jobNV0VjP RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.38" port="8662" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmp3YBSbi/7c7403eb-20a3-4eb0-99fd-2cdc1232ac97/tmpbNeBYK.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmp3YBSbi/7c7403eb-20a3-4eb0-99fd-2cdc1232ac97/tmpTu4qqc.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmp3YBSbi/7c7403eb-20a3-4eb0-99fd-2cdc1232ac97/tmpDdTNJy.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmp3YBSbi/7c7403eb-20a3-4eb0-99fd-2cdc1232ac97/tmpD3mGF1.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-88711676-5948-47d0-acd3-569974301115/tmp3YBSbi/7c7403eb-20a3-4eb0-99fd-2cdc1232ac97/tmpi0xJkT.tmp'] failed with output:
  2973. WARNING:toil.leader:h/G/jobNV0VjP ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn35.grid.pub.ro
  2974. WARNING:toil.leader:h/G/jobNV0VjP WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' h/G/jobNV0VjP with ID h/G/jobNV0VjP to 1
  2975. INFO:toil.leader:Issued job 'CactusSetupPhase' h/G/jobNV0VjP with job batch system ID: 231 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  2976. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' h/G/jobNV0VjP
  2977. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' h/G/jobNV0VjP
  2978. WARNING:toil.leader:h/G/jobNV0VjP INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  2979. WARNING:toil.leader:h/G/jobNV0VjP INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  2980. WARNING:toil.leader:h/G/jobNV0VjP WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  2981. WARNING:toil.leader:h/G/jobNV0VjP WARNING:toil.fileStore:Detected that job (N/8/job2cN94i/g/tmpSr72ee-_serialiseJob-stream) prematurely terminated. Fixing the state of the job on disk.
  2982. WARNING:toil.leader:h/G/jobNV0VjP INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  2983. WARNING:toil.leader:h/G/jobNV0VjP INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  2984. WARNING:toil.leader:h/G/jobNV0VjP INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.38" port="8662" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpREoveY/fb4f0e39-5389-41bb-b827-d47cf1432dcc/tmp5uNzHe.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpREoveY/fb4f0e39-5389-41bb-b827-d47cf1432dcc/tmpOnvnlh.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpREoveY/fb4f0e39-5389-41bb-b827-d47cf1432dcc/tmpaLPAUw.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpREoveY/fb4f0e39-5389-41bb-b827-d47cf1432dcc/tmpv0982n.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpREoveY/fb4f0e39-5389-41bb-b827-d47cf1432dcc/tmphhm0bo.tmp']
  2985. WARNING:toil.leader:h/G/jobNV0VjP Set log level to INFO
  2986. WARNING:toil.leader:h/G/jobNV0VjP Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  2987. WARNING:toil.leader:h/G/jobNV0VjP <kyoto_tycoon database_dir="fakepath" host="172.16.13.38" port="8662" />
  2988. WARNING:toil.leader:h/G/jobNV0VjP </st_kv_database_conf>
  2989. WARNING:toil.leader:h/G/jobNV0VjP
  2990. WARNING:toil.leader:h/G/jobNV0VjP Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpREoveY/fb4f0e39-5389-41bb-b827-d47cf1432dcc/tmp5uNzHe.tmp
  2991. WARNING:toil.leader:h/G/jobNV0VjP Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpREoveY/fb4f0e39-5389-41bb-b827-d47cf1432dcc/tmpOnvnlh.tmp
  2992. WARNING:toil.leader:h/G/jobNV0VjP Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpREoveY/fb4f0e39-5389-41bb-b827-d47cf1432dcc/tmpaLPAUw.tmp
  2993. WARNING:toil.leader:h/G/jobNV0VjP Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpREoveY/fb4f0e39-5389-41bb-b827-d47cf1432dcc/tmpv0982n.tmp
  2994. WARNING:toil.leader:h/G/jobNV0VjP Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpREoveY/fb4f0e39-5389-41bb-b827-d47cf1432dcc/tmphhm0bo.tmp
  2995. WARNING:toil.leader:h/G/jobNV0VjP Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.38 with error: network error
  2996. WARNING:toil.leader:h/G/jobNV0VjP Uncaught exception
  2997. WARNING:toil.leader:h/G/jobNV0VjP Traceback (most recent call last):
  2998. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  2999. WARNING:toil.leader:h/G/jobNV0VjP job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  3000. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  3001. WARNING:toil.leader:h/G/jobNV0VjP super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  3002. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  3003. WARNING:toil.leader:h/G/jobNV0VjP returnValues = self._run(jobGraph, fileStore)
  3004. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  3005. WARNING:toil.leader:h/G/jobNV0VjP return self.run(fileStore)
  3006. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  3007. WARNING:toil.leader:h/G/jobNV0VjP makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  3008. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  3009. WARNING:toil.leader:h/G/jobNV0VjP parameters=["cactus_setup"] + args + sequences)
  3010. WARNING:toil.leader:h/G/jobNV0VjP File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  3011. WARNING:toil.leader:h/G/jobNV0VjP raise RuntimeError("Command %s failed with output: %s" % (call, output))
  3012. WARNING:toil.leader:h/G/jobNV0VjP RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.38" port="8662" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpREoveY/fb4f0e39-5389-41bb-b827-d47cf1432dcc/tmp5uNzHe.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpREoveY/fb4f0e39-5389-41bb-b827-d47cf1432dcc/tmpOnvnlh.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpREoveY/fb4f0e39-5389-41bb-b827-d47cf1432dcc/tmpaLPAUw.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpREoveY/fb4f0e39-5389-41bb-b827-d47cf1432dcc/tmpv0982n.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-4a1d8798-1fbf-46ce-a59d-dbcf0d1a641b-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpREoveY/fb4f0e39-5389-41bb-b827-d47cf1432dcc/tmphhm0bo.tmp'] failed with output:
  3013. WARNING:toil.leader:h/G/jobNV0VjP ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn41.grid.pub.ro
  3014. WARNING:toil.leader:h/G/jobNV0VjP WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' h/G/jobNV0VjP with ID h/G/jobNV0VjP to 0
  3015. WARNING:toil.leader:Job 'CactusSetupPhase' h/G/jobNV0VjP with ID h/G/jobNV0VjP is completely failed
  3016. INFO:toil.leader:Finished toil run with 10 failed jobs.
  3017. INFO:toil.leader:Failed jobs at end of the run: 'CactusSetupCheckpoint' m/g/jobFH953E 'ProgressiveDown' R/B/jobHI0bbU 'StartPrimaryDB' N/8/job2cN94i 'KtServerService' 1/r/jobM3ECWT 'ProgressiveUp' H/4/job4fjXBN 'RunCactusPreprocessorThenProgressiveDown' n/R/jobJ7SRhl 'CactusSetupPhase' h/G/jobNV0VjP 'ProgressiveNext' n/a/jobcxAFzs 'RunCactusPreprocessorThenProgressiveDown2' w/J/jobQeKsUS 'CactusTrimmingBlastPhase' N/t/jobC7X2Fy
  3018. Traceback (most recent call last):
  3019. File "/export/home/ncit/external/a.mizeranschi/toil_conda/bin/cactus", line 11, in <module>
  3020. sys.exit(main())
  3021. File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/progressive/cactus_progressive.py", line 520, in main
  3022. halID = toil.start(RunCactusPreprocessorThenProgressiveDown(options, project, memory=configWrapper.getDefaultMemory()))
  3023. File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/common.py", line 784, in start
  3024. return self._runMainLoop(rootJobGraph)
  3025. File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/common.py", line 1059, in _runMainLoop
  3026. jobCache=self._jobCache).run()
  3027. File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/leader.py", line 237, in run
  3028. raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore)
  3029. toil.leader.FailedJobsException
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!