API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Mar 18th, 2019
78
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 377.30 KB | None | 0 0
raw download clone embed print report
  1. INFO:cactus.progressive.projectWrapper:Using config from path /export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/cactus_progressive_config.xml.
  2. INFO:toil.lib.bioio:xmlRoot = <multi_cactus inputSequences="simMouse.chr6 simRat.chr6 simHuman.chr6 simDog.chr6 simCow.chr6">
  3. <tree>(simMouse_chr6:0.084509,simRat_chr6:0.091589,simHuman_chr6:0.415992,simDog_chr6:0.488495,simCow_chr6:0.514545)mr;</tree>
  4. <cactus experiment_path="/scratch/tmp/1071822.1.campus-haswell.q/tmplrA7Zx/progressiveAlignment/mr/mr_experiment.xml" name="mr" />
  5. </multi_cactus>
  6. WARNING:toil.batchSystems.singleMachine:Limiting maxMemory to physically available memory (134158856192).
  7. WARNING:toil.batchSystems.singleMachine:Limiting maxDisk to physically available disk (2339446456320).
  8. WARNING:toil.common:Batch system does not support auto-deployment. The user script ModuleDescriptor(dirPath='/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages', name='cactus.progressive.cactus_progressive', fromVirtualEnv=False) will have to be present at the same location on every worker.
  9. INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  10. INFO:toil.leader:Issued job 'RunCactusPreprocessorThenProgressiveDown' E/U/jobU_k7ZC with job batch system ID: 0 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  11. INFO:toil.statsAndLogging:Got message from job at time 03-18-2019 12:43:01: Using the following configuration:
  12. <cactusWorkflowConfig distanceToAddToRootAlignment="0.1">
  13. <constants defaultCpu="1" defaultMemory="3500000000" defaultOverlargeCpu="1" defaultOverlargeMemory="3500000000">
  14.  
  15. <divergences five="0.35" four="0.25" one="0.1" three="0.2" two="0.15" useDefault="0" />
  16. </constants>
  17. <preprocessor check="1" checkAssemblyHub="1" memory="2000000000" preprocessJob="checkUniqueHeaders" />
  18. <preprocessor chunkSize="3000000" lastzOpts="--step=3 --ambiguous=iupac,100,100 --ungapped --queryhsplimit=keep,nowarn:1500" memory="2000000000" minPeriod="50" preprocessJob="lastzRepeatMask" proportionToSample="0.2" unmask="0" />
  19.  
  20. <trimBlast doTrimStrategy="1" keepParalogs="0" trimFlanking="10" trimMinSize="100" trimOutgroupDepth="1" trimOutgroupFlanking="2000" trimThreshold="1.0" trimWindowSize="1" />
  21. <ktserver memory="3500000000" />
  22. <setup makeEventHeadersAlphaNumeric="0" />
  23.  
  24. <caf alignmentFilter="filterSecondariesByMultipleSpecies" alpha="0.001" annealingRounds="128" blockTrim="5" chunkSize="25000000" compressFiles="1" deannealingRounds="2 8" doPhylogeny="0" filterByIdentity="0" identityRatio="3" lastzDisk="mediumDisk" lastzMemory="2000000000" maxAdjacencyComponentSizeRatio="50" maxAlignmentsPerSite="5" maxRecoverableChainLength="500000" maxRecoverableChainsIterations="5" maximumMedianSequenceLengthBetweenLinkedEnds="1000" minLengthForChromosome="1000000" minimumBlockDegree="2" minimumBlockDegreeToCheckSupport="10" minimumBlockHomologySupport="0.05" minimumDistance="0.01" minimumIngroupDegree="1" minimumMapQValue="0.0" minimumNumberOfSpecies="1" minimumOutgroupDegree="0" minimumSequenceLengthForBlast="30" minimumTreeCoverage="0.0" numTreeBuildingThreads="2" overlapSize="10000" phylogenyBreakpointScalingFactor="1.0" phylogenyCostPerDupPerBase="0.00" phylogenyCostPerLossPerBase="0.02" phylogenyDistanceCorrectionMethod="jukesCantor" phylogenyDoSplitsWithSupportHigherThanThisAllAtOnce="0.44" phylogenyHomologyUnitType="chain" phylogenyKeepSingleDegreeBlocks="0" phylogenyMaxBaseDistance="100" phylogenyMaxBlockDistance="50" phylogenyNumTrees="30" phylogenyRootingMethod="bestRecon" phylogenyScoringMethod="reconCost" phylogenySkipSingleCopyBlocks="1" phylogenyTreeBuildingMethod="guidedNeighborJoining,splitDecomposition" proportionOfUnalignedBasesForNewChromosome="0.8" realign="1" realignArguments="--gapGamma 0.0 --matchGamma 0.9 --diagonalExpansion 4 --splitMatrixBiggerThanThis 10 --constraintDiagonalTrim 0 --alignAmbiguityCharacters --splitIndelsLongerThanThis 99" removeRecoverableChains="unequalNumberOfIngroupCopies" runMapQFiltering="1" trim="0 0">
  25. <divergence argName="lastzArguments" default="--step=1 --ambiguous=iupac,100,100 --ydrop=3000" five="--step=2 --ambiguous=iupac,100,100 --ydrop=3000" four="--step=3 --ambiguous=iupac,100,100 --ydrop=3000" one="--step=2 --ambiguous=iupac,100,100 --ydrop=3000 --notransition" three="--step=4 --ambiguous=iupac,100,100 --ydrop=3000" two="--step=5 --ambiguous=iupac,100,100 --ydrop=3000" />
  26. <CactusCafRecursion maxFlowerGroupSize="100000000" />
  27. <CactusCafWrapper maxFlowerGroupSize="25000000" minFlowerSize="1" />
  28. <CactusCafWrapperLarge2 overlargeMemory="5000000000" />
  29. </caf>
  30.  
  31. <bar alignAmbiguityCharacters="1" anchorMatrixBiggerThanThis="500" bandingLimit="1000000" constraintDiagonalTrim="14" diagonalExpansion="20" gapGamma="0.0" largeEndSize="5000" matchGamma="0.2" minimumBlockDegree="2" minimumCoverageToRescue="0.5" minimumIngroupDegree="1" minimumNumberOfSpecies="1" minimumOutgroupDegree="0" minimumSizeToRescue="100" pruneOutStubAlignments="1" repeatMaskMatrixBiggerThanThis="500" rescue="0" runBar="1" spanningTrees="5" splitMatrixBiggerThanThis="3000" useBanding="1" useProgressiveMerging="1" veryLargeEndSize="2000000">
  32. <CactusBarRecursion maxFlowerGroupSize="100000000" />
  33. <CactusBarWrapper maxFlowerGroupSize="2000000" memory="2000000000" />
  34. <CactusBarWrapperLarge maxFlowerGroupSize="2000000" />
  35. <CactusBarEndAlignerWrapper memory="2000000000" />
  36. </bar>
  37. <normal iterations="0">
  38. <CactusNormalRecursion maxFlowerGroupSize="100000000" maxFlowerWrapperGroupSize="10000000" />
  39. <CactusNormalWrapper />
  40. </normal>
  41. <avg buildAvgs="0">
  42. <CactusAVGRecursion maxFlowerGroupSize="100000000" maxFlowerWrapperGroupSize="10000000" />
  43. <CactusAVGWrapper />
  44. </avg>
  45. <reference buildReference="1" ignoreUnalignedGaps="1" makeScaffolds="1" matchingAlgorithm="blossom5" maxWalkForCalculatingZ="100000" minNumberOfSequencesToSupportAdjacency="1" numberOfNs="10" permutations="10" phi="1.0" reference="mr" theta="0.0001" useSimulatedAnnealing="1" wiggle="0.9999">
  46. <CactusReferenceRecursion maxFlowerGroupSize="100000000" maxFlowerWrapperGroupSize="2000000" />
  47.  
  48. <CactusReferenceWrapper />
  49.  
  50. <CactusSetReferenceCoordinatesUpWrapper />
  51.  
  52. <CactusSetReferenceCoordinatesDownRecursion maxFlowerGroupSize="100000000" maxFlowerWrapperGroupSize="2000000" />
  53.  
  54. <CactusSetReferenceCoordinatesDownWrapper />
  55. </reference>
  56. <check runCheck="0">
  57. <CactusCheckRecursion maxFlowerGroupSize="100000000" maxFlowerWrapperGroupSize="2000000" />
  58. <CactusCheckWrapper />
  59. </check>
  60. <hal buildFasta="1" buildHal="1">
  61. <CactusHalGeneratorRecursion maxFlowerGroupSize="2000000" />
  62. <CactusHalGeneratorUpWrapper />
  63. </hal>
  64.  
  65. <multi_cactus>
  66. <outgroup ancestor_quality_fraction="0.75" max_num_outgroups="3" strategy="greedyPreference" threshold="0" />
  67.  
  68. <decomposition default_internal_node_prefix="Anc" max_parallel_subtrees="50000" self_alignment="false" subtree_size="2" />
  69.  
  70. </multi_cactus>
  71.  
  72. <ProgressiveNext />
  73.  
  74. <ProgressiveUp />
  75.  
  76. <ProgressiveDown />
  77.  
  78. <ProgressiveOut />
  79.  
  80. <RunCactusPreprocessorThenProgressiveDown />
  81.  
  82. <RunCactusPreprocessorThenProgressiveDown2 />
  83.  
  84. <exportHal disk="2000000000" />
  85. </cactusWorkflowConfig>
  86. INFO:toil.statsAndLogging:Got message from job at time 03-18-2019 12:43:01: Leaf names = ['simMouse_chr6', 'simRat_chr6', 'simHuman_chr6', 'simDog_chr6', 'simCow_chr6']
  87. INFO:toil.leader:Job ended successfully: 'RunCactusPreprocessorThenProgressiveDown' E/U/jobU_k7ZC
  88. INFO:toil.leader:Issued job 'logAssemblyStats' n/I/jobgHF13G with job batch system ID: 1 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  89. INFO:toil.leader:Issued job 'logAssemblyStats' 9/d/jobLmSpID with job batch system ID: 2 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  90. INFO:toil.leader:Issued job 'logAssemblyStats' R/l/jobtntRV0 with job batch system ID: 3 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  91. INFO:toil.leader:Issued job 'logAssemblyStats' g/W/jobaWQu7g with job batch system ID: 4 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  92. INFO:toil.leader:Issued job 'logAssemblyStats' W/g/jobCdZDTm with job batch system ID: 5 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  93. INFO:toil.leader:Issued job 'CactusPreprocessor' I/g/jobNNIdRR with job batch system ID: 6 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  94. INFO:toil.statsAndLogging:Got message from job at time 03-18-2019 12:45:04: Before preprocessing, got assembly stats for genome simCow_chr6: Input-sample: /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpkOl0Wq/0e860c6d-72df-4448-a544-1bad76657d34/tmp7_0uPU.tmp Total-sequences: 1 Total-length: 602619 Proportion-repeat-masked: 0.054570 ProportionNs: 0.000000 Total-Ns: 0 N50: 602619 Median-sequence-length: 602619 Max-sequence-length: 602619 Min-sequence-length: 602619
  95.  
  96. INFO:toil.statsAndLogging:Got message from job at time 03-18-2019 12:45:05: Before preprocessing, got assembly stats for genome simHuman_chr6: Input-sample: /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpNFd6X8/3b065712-9dcf-4580-a4b9-750459f499f6/tmp4CIkGZ.tmp Total-sequences: 1 Total-length: 601863 Proportion-repeat-masked: 0.074751 ProportionNs: 0.000000 Total-Ns: 0 N50: 601863 Median-sequence-length: 601863 Max-sequence-length: 601863 Min-sequence-length: 601863
  97.  
  98. INFO:toil.statsAndLogging:Got message from job at time 03-18-2019 12:45:05: Before preprocessing, got assembly stats for genome simMouse_chr6: Input-sample: /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-b02a3811-2b63-4208-851d-7815af46a62d/tmpeCFVJ7/43d36972-ba65-4a40-9db2-6c9355d50ce6/tmpUflXC_.tmp Total-sequences: 1 Total-length: 636262 Proportion-repeat-masked: 0.056477 ProportionNs: 0.000000 Total-Ns: 0 N50: 636262 Median-sequence-length: 636262 Max-sequence-length: 636262 Min-sequence-length: 636262
  99.  
  100. INFO:toil.statsAndLogging:Got message from job at time 03-18-2019 12:45:06: Before preprocessing, got assembly stats for genome simDog_chr6: Input-sample: /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmp0J6MpZ/f7bd4132-4fa7-4fa8-a3b1-d7c6aea7a503/tmpqa07FB.tmp Total-sequences: 1 Total-length: 593897 Proportion-repeat-masked: 0.045009 ProportionNs: 0.000000 Total-Ns: 0 N50: 593897 Median-sequence-length: 593897 Max-sequence-length: 593897 Min-sequence-length: 593897
  101.  
  102. INFO:toil.statsAndLogging:Got message from job at time 03-18-2019 12:45:06: Before preprocessing, got assembly stats for genome simRat_chr6: Input-sample: /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmp7bpXxu/018bedd3-7ac1-41e7-853d-0baea960b739/tmp7sToye.tmp Total-sequences: 1 Total-length: 647215 Proportion-repeat-masked: 0.075276 ProportionNs: 0.000000 Total-Ns: 0 N50: 647215 Median-sequence-length: 647215 Max-sequence-length: 647215 Min-sequence-length: 647215
  103.  
  104. INFO:toil.leader:Job ended successfully: 'logAssemblyStats' W/g/jobCdZDTm
  105. INFO:toil.leader:Job ended successfully: 'CactusPreprocessor' I/g/jobNNIdRR
  106. INFO:toil.leader:Issued job 'CactusPreprocessor2' W/R/jobQuzQ7a with job batch system ID: 7 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  107. INFO:toil.leader:Issued job 'CactusPreprocessor2' q/z/job6Nlkw9 with job batch system ID: 8 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  108. INFO:toil.leader:Issued job 'CactusPreprocessor2' f/A/jobaZeZ7G with job batch system ID: 9 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  109. INFO:toil.leader:Issued job 'CactusPreprocessor2' k/1/jobveU_JE with job batch system ID: 10 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  110. INFO:toil.leader:Issued job 'CactusPreprocessor2' t/D/jobBGSltl with job batch system ID: 11 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  111. INFO:toil.leader:Job ended successfully: 'logAssemblyStats' n/I/jobgHF13G
  112. INFO:toil.leader:Job ended successfully: 'logAssemblyStats' 9/d/jobLmSpID
  113. INFO:toil.leader:Job ended successfully: 'logAssemblyStats' R/l/jobtntRV0
  114. INFO:toil.leader:Job ended successfully: 'logAssemblyStats' g/W/jobaWQu7g
  115. INFO:toil.leader:Job ended successfully: 'CactusPreprocessor2' k/1/jobveU_JE
  116. INFO:toil.leader:Issued job 'BatchPreprocessor' F/i/jobsHRoQg with job batch system ID: 12 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  117. INFO:toil.leader:Job ended successfully: 'CactusPreprocessor2' t/D/jobBGSltl
  118. INFO:toil.leader:Issued job 'BatchPreprocessor' k/t/jobEdSDw_ with job batch system ID: 13 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  119. INFO:toil.leader:Job ended successfully: 'CactusPreprocessor2' W/R/jobQuzQ7a
  120. INFO:toil.leader:Issued job 'BatchPreprocessor' Q/l/job248xQT with job batch system ID: 14 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  121. INFO:toil.leader:Job ended successfully: 'CactusPreprocessor2' q/z/job6Nlkw9
  122. INFO:toil.leader:Issued job 'BatchPreprocessor' Q/V/jobIy_6Xu with job batch system ID: 15 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  123. INFO:toil.leader:Job ended successfully: 'CactusPreprocessor2' f/A/jobaZeZ7G
  124. INFO:toil.leader:Issued job 'BatchPreprocessor' 4/o/jobTmK4q2 with job batch system ID: 16 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  125. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' k/t/jobEdSDw_
  126. INFO:toil.leader:Issued job 'PreprocessSequence' C/C/jobn4KmlL with job batch system ID: 17 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  127. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' Q/l/job248xQT
  128. INFO:toil.leader:Issued job 'PreprocessSequence' c/h/jobEXfgEC with job batch system ID: 18 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  129. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' Q/V/jobIy_6Xu
  130. INFO:toil.leader:Issued job 'PreprocessSequence' A/R/job95XpMq with job batch system ID: 19 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  131. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' 4/o/jobTmK4q2
  132. INFO:toil.leader:Issued job 'PreprocessSequence' s/y/jobUOdm7P with job batch system ID: 20 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  133. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' F/i/jobsHRoQg
  134. INFO:toil.leader:Issued job 'PreprocessSequence' L/h/job7hMNDW with job batch system ID: 21 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  135. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' C/C/jobn4KmlL
  136. INFO:toil.leader:Issued job 'CheckUniqueHeaders' 7/s/jobOXSvjb with job batch system ID: 22 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  137. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' c/h/jobEXfgEC
  138. INFO:toil.leader:Issued job 'CheckUniqueHeaders' I/8/job3OIV06 with job batch system ID: 23 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  139. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' A/R/job95XpMq
  140. INFO:toil.leader:Issued job 'CheckUniqueHeaders' u/2/jobVOYMEi with job batch system ID: 24 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  141. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' s/y/jobUOdm7P
  142. INFO:toil.leader:Issued job 'CheckUniqueHeaders' X/U/jobMBj8ad with job batch system ID: 25 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  143. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' L/h/job7hMNDW
  144. INFO:toil.leader:Issued job 'CheckUniqueHeaders' s/f/jobXh1arN with job batch system ID: 26 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  145. INFO:toil.leader:Job ended successfully: 'CheckUniqueHeaders' I/8/job3OIV06
  146. INFO:toil.leader:Issued job 'PreprocessSequence' c/h/jobEXfgEC with job batch system ID: 27 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  147. INFO:toil.leader:Job ended successfully: 'CheckUniqueHeaders' 7/s/jobOXSvjb
  148. INFO:toil.leader:Issued job 'PreprocessSequence' C/C/jobn4KmlL with job batch system ID: 28 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  149. INFO:toil.leader:Job ended successfully: 'CheckUniqueHeaders' u/2/jobVOYMEi
  150. INFO:toil.leader:Issued job 'PreprocessSequence' A/R/job95XpMq with job batch system ID: 29 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  151. INFO:toil.leader:Job ended successfully: 'CheckUniqueHeaders' X/U/jobMBj8ad
  152. INFO:toil.leader:Issued job 'PreprocessSequence' s/y/jobUOdm7P with job batch system ID: 30 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  153. INFO:toil.leader:Job ended successfully: 'CheckUniqueHeaders' s/f/jobXh1arN
  154. INFO:toil.leader:Issued job 'PreprocessSequence' L/h/job7hMNDW with job batch system ID: 31 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  155. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' L/h/job7hMNDW
  156. INFO:toil.leader:Issued job 'BatchPreprocessor' 2/G/jobRjshcC with job batch system ID: 32 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  157. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' c/h/jobEXfgEC
  158. INFO:toil.leader:Issued job 'BatchPreprocessor' I/f/jobBAmKJE with job batch system ID: 33 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  159. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' C/C/jobn4KmlL
  160. INFO:toil.leader:Issued job 'BatchPreprocessor' J/R/jobCNYZaa with job batch system ID: 34 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  161. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' A/R/job95XpMq
  162. INFO:toil.leader:Issued job 'BatchPreprocessor' 6/x/jobKFdrbC with job batch system ID: 35 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  163. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' s/y/jobUOdm7P
  164. INFO:toil.leader:Issued job 'BatchPreprocessor' o/c/jobFGgNut with job batch system ID: 36 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  165. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' 2/G/jobRjshcC
  166. INFO:toil.leader:Issued job 'PreprocessSequence' K/C/jobIVGDp6 with job batch system ID: 37 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  167. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' J/R/jobCNYZaa
  168. INFO:toil.leader:Issued job 'PreprocessSequence' A/o/jobpia8ch with job batch system ID: 38 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  169. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' 6/x/jobKFdrbC
  170. INFO:toil.leader:Issued job 'PreprocessSequence' O/u/jobZgFZOc with job batch system ID: 39 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  171. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' o/c/jobFGgNut
  172. INFO:toil.leader:Issued job 'PreprocessSequence' 6/p/jobN1PZ59 with job batch system ID: 40 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  173. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' K/C/jobIVGDp6
  174. INFO:toil.leader:Issued job 'LastzRepeatMaskJob' M/O/jobxnijHw with job batch system ID: 41 and cores: 1, disk: 100.0 M, and memory: 4.0 G
  175. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' I/f/jobBAmKJE
  176. INFO:toil.leader:Issued job 'PreprocessSequence' 6/Q/job1q2sUC with job batch system ID: 42 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  177. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' A/o/jobpia8ch
  178. INFO:toil.leader:Issued job 'LastzRepeatMaskJob' v/v/jobXcazsY with job batch system ID: 43 and cores: 1, disk: 100.0 M, and memory: 4.0 G
  179. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' O/u/jobZgFZOc
  180. INFO:toil.leader:Issued job 'LastzRepeatMaskJob' T/t/jobwo2LI_ with job batch system ID: 44 and cores: 1, disk: 100.0 M, and memory: 4.0 G
  181. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' 6/p/jobN1PZ59
  182. INFO:toil.leader:Issued job 'LastzRepeatMaskJob' s/l/jobN2uGh5 with job batch system ID: 45 and cores: 1, disk: 100.0 M, and memory: 4.0 G
  183. INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' M/O/jobxnijHw
  184. INFO:toil.leader:Issued job 'MergeChunks' L/u/jobxHeSOX with job batch system ID: 46 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  185. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' 6/Q/job1q2sUC
  186. INFO:toil.leader:Issued job 'LastzRepeatMaskJob' V/f/jobu0X4St with job batch system ID: 47 and cores: 1, disk: 100.0 M, and memory: 4.0 G
  187. INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' v/v/jobXcazsY
  188. INFO:toil.leader:Issued job 'MergeChunks' Z/J/jobWZPxgG with job batch system ID: 48 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  189. INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' T/t/jobwo2LI_
  190. INFO:toil.leader:Issued job 'MergeChunks' U/v/jobZn988B with job batch system ID: 49 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  191. INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' s/l/jobN2uGh5
  192. INFO:toil.leader:Issued job 'MergeChunks' o/h/jobB7TD1t with job batch system ID: 50 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  193. INFO:toil.leader:Job ended successfully: 'MergeChunks' L/u/jobxHeSOX
  194. INFO:toil.leader:Issued job 'MergeChunks2' F/L/jobb42xAa with job batch system ID: 51 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  195. INFO:toil.leader:Job ended successfully: 'LastzRepeatMaskJob' V/f/jobu0X4St
  196. INFO:toil.leader:Issued job 'MergeChunks' B/8/jobt7eUYX with job batch system ID: 52 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  197. INFO:toil.leader:Job ended successfully: 'MergeChunks' o/h/jobB7TD1t
  198. INFO:toil.leader:Issued job 'MergeChunks2' t/C/job8hDtn7 with job batch system ID: 53 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  199. INFO:toil.leader:Job ended successfully: 'MergeChunks2' F/L/jobb42xAa
  200. INFO:toil.leader:Issued job 'MergeChunks' L/u/jobxHeSOX with job batch system ID: 54 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  201. INFO:toil.leader:Job ended successfully: 'MergeChunks' B/8/jobt7eUYX
  202. INFO:toil.leader:Issued job 'MergeChunks2' l/i/jobu_bEjV with job batch system ID: 55 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  203. INFO:toil.leader:Job ended successfully: 'MergeChunks' Z/J/jobWZPxgG
  204. INFO:toil.leader:Issued job 'MergeChunks2' 4/I/jobhe_e4I with job batch system ID: 56 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  205. INFO:toil.leader:Job ended successfully: 'MergeChunks' U/v/jobZn988B
  206. INFO:toil.leader:Issued job 'MergeChunks2' K/6/jobUHd_7s with job batch system ID: 57 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  207. INFO:toil.leader:Job ended successfully: 'MergeChunks2' t/C/job8hDtn7
  208. INFO:toil.leader:Issued job 'MergeChunks' o/h/jobB7TD1t with job batch system ID: 58 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  209. INFO:toil.leader:Job ended successfully: 'MergeChunks' L/u/jobxHeSOX
  210. INFO:toil.leader:Issued job 'PreprocessSequence' K/C/jobIVGDp6 with job batch system ID: 59 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  211. INFO:toil.leader:Job ended successfully: 'MergeChunks2' l/i/jobu_bEjV
  212. INFO:toil.leader:Issued job 'MergeChunks' B/8/jobt7eUYX with job batch system ID: 60 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  213. INFO:toil.leader:Job ended successfully: 'MergeChunks2' 4/I/jobhe_e4I
  214. INFO:toil.leader:Issued job 'MergeChunks' Z/J/jobWZPxgG with job batch system ID: 61 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  215. INFO:toil.leader:Job ended successfully: 'MergeChunks2' K/6/jobUHd_7s
  216. INFO:toil.leader:Issued job 'MergeChunks' U/v/jobZn988B with job batch system ID: 62 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  217. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' K/C/jobIVGDp6
  218. INFO:toil.leader:Issued job 'BatchPreprocessor' 2/G/jobRjshcC with job batch system ID: 63 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  219. INFO:toil.leader:Job ended successfully: 'MergeChunks' B/8/jobt7eUYX
  220. INFO:toil.leader:Issued job 'PreprocessSequence' 6/Q/job1q2sUC with job batch system ID: 64 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  221. INFO:toil.leader:Job ended successfully: 'MergeChunks' Z/J/jobWZPxgG
  222. INFO:toil.leader:Issued job 'PreprocessSequence' A/o/jobpia8ch with job batch system ID: 65 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  223. INFO:toil.leader:Job ended successfully: 'MergeChunks' U/v/jobZn988B
  224. INFO:toil.leader:Issued job 'PreprocessSequence' O/u/jobZgFZOc with job batch system ID: 66 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  225. INFO:toil.leader:Job ended successfully: 'MergeChunks' o/h/jobB7TD1t
  226. INFO:toil.leader:Issued job 'PreprocessSequence' 6/p/jobN1PZ59 with job batch system ID: 67 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  227. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' 2/G/jobRjshcC
  228. INFO:toil.leader:Issued job 'BatchPreprocessor' F/i/jobsHRoQg with job batch system ID: 68 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  229. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' 6/Q/job1q2sUC
  230. INFO:toil.leader:Issued job 'BatchPreprocessor' I/f/jobBAmKJE with job batch system ID: 69 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  231. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' A/o/jobpia8ch
  232. INFO:toil.leader:Issued job 'BatchPreprocessor' J/R/jobCNYZaa with job batch system ID: 70 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  233. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' O/u/jobZgFZOc
  234. INFO:toil.leader:Issued job 'BatchPreprocessor' 6/x/jobKFdrbC with job batch system ID: 71 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  235. INFO:toil.leader:Job ended successfully: 'PreprocessSequence' 6/p/jobN1PZ59
  236. INFO:toil.leader:Issued job 'BatchPreprocessor' o/c/jobFGgNut with job batch system ID: 72 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  237. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' o/c/jobFGgNut
  238. INFO:toil.leader:Issued job 'BatchPreprocessor' 4/o/jobTmK4q2 with job batch system ID: 73 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  239. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' F/i/jobsHRoQg
  240. INFO:toil.leader:Issued job 'CactusPreprocessor2' k/1/jobveU_JE with job batch system ID: 74 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  241. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' I/f/jobBAmKJE
  242. INFO:toil.leader:Issued job 'BatchPreprocessor' Q/l/job248xQT with job batch system ID: 75 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  243. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' J/R/jobCNYZaa
  244. INFO:toil.leader:Issued job 'BatchPreprocessor' k/t/jobEdSDw_ with job batch system ID: 76 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  245. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' 6/x/jobKFdrbC
  246. INFO:toil.leader:Issued job 'BatchPreprocessor' Q/V/jobIy_6Xu with job batch system ID: 77 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  247. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' Q/l/job248xQT
  248. INFO:toil.leader:Issued job 'CactusPreprocessor2' W/R/jobQuzQ7a with job batch system ID: 78 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  249. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' k/t/jobEdSDw_
  250. INFO:toil.leader:Issued job 'CactusPreprocessor2' t/D/jobBGSltl with job batch system ID: 79 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  251. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' Q/V/jobIy_6Xu
  252. INFO:toil.leader:Issued job 'CactusPreprocessor2' q/z/job6Nlkw9 with job batch system ID: 80 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  253. INFO:toil.leader:Job ended successfully: 'BatchPreprocessor' 4/o/jobTmK4q2
  254. INFO:toil.leader:Issued job 'CactusPreprocessor2' f/A/jobaZeZ7G with job batch system ID: 81 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  255. INFO:toil.leader:Job ended successfully: 'CactusPreprocessor2' k/1/jobveU_JE
  256. INFO:toil.leader:Job ended successfully: 'CactusPreprocessor2' W/R/jobQuzQ7a
  257. INFO:toil.leader:Job ended successfully: 'CactusPreprocessor2' t/D/jobBGSltl
  258. INFO:toil.leader:Job ended successfully: 'CactusPreprocessor2' q/z/job6Nlkw9
  259. INFO:toil.leader:Job ended successfully: 'CactusPreprocessor2' f/A/jobaZeZ7G
  260. INFO:toil.leader:Issued job 'CactusPreprocessor' I/g/jobNNIdRR with job batch system ID: 82 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  261. INFO:toil.leader:Job ended successfully: 'CactusPreprocessor' I/g/jobNNIdRR
  262. INFO:toil.leader:Issued job 'RunCactusPreprocessorThenProgressiveDown2' d/c/jobyROBMF with job batch system ID: 83 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  263. INFO:toil.leader:Job ended successfully: 'RunCactusPreprocessorThenProgressiveDown2' d/c/jobyROBMF
  264. INFO:toil.leader:Issued job 'logAssemblyStats' 7/8/jobiSYXWo with job batch system ID: 84 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  265. INFO:toil.leader:Issued job 'logAssemblyStats' D/f/job1GU_Yw with job batch system ID: 85 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  266. INFO:toil.leader:Issued job 'logAssemblyStats' S/0/jobsU7qn1 with job batch system ID: 86 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  267. INFO:toil.leader:Issued job 'logAssemblyStats' 9/1/jobMvLJQy with job batch system ID: 87 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  268. INFO:toil.leader:Issued job 'logAssemblyStats' n/N/jobmBLwjD with job batch system ID: 88 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  269. INFO:toil.leader:Issued job 'ProgressiveDown' w/1/jobSc89ZQ with job batch system ID: 89 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  270. INFO:toil.statsAndLogging:Got message from job at time 03-18-2019 13:18:44: There are 0 dependent projects
  271. INFO:toil.statsAndLogging:Got message from job at time 03-18-2019 13:18:46: After preprocessing, got assembly stats for genome simMouse_chr6: Input-sample: /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpyfc98C/9e42ad13-8b50-489b-a61d-8f2e60298792/tmpkSXYZh.tmp Total-sequences: 1 Total-length: 636262 Proportion-repeat-masked: 0.056477 ProportionNs: 0.000000 Total-Ns: 0 N50: 636262 Median-sequence-length: 636262 Max-sequence-length: 636262 Min-sequence-length: 636262
  272.  
  273. INFO:toil.statsAndLogging:Got message from job at time 03-18-2019 13:18:47: After preprocessing, got assembly stats for genome simCow_chr6: Input-sample: /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpkRKZCI/83cd4cdc-26ff-4006-90c2-bdca02f7190c/tmpj97eRa.tmp Total-sequences: 1 Total-length: 602619 Proportion-repeat-masked: 0.054570 ProportionNs: 0.000000 Total-Ns: 0 N50: 602619 Median-sequence-length: 602619 Max-sequence-length: 602619 Min-sequence-length: 602619
  274.  
  275. INFO:toil.statsAndLogging:Got message from job at time 03-18-2019 13:18:47: After preprocessing, got assembly stats for genome simDog_chr6: Input-sample: /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpq3xmFI/dbdb849c-202c-4359-88ef-d5bd7ba280d1/tmpxh5a3j.tmp Total-sequences: 1 Total-length: 593897 Proportion-repeat-masked: 0.045009 ProportionNs: 0.000000 Total-Ns: 0 N50: 593897 Median-sequence-length: 593897 Max-sequence-length: 593897 Min-sequence-length: 593897
  276.  
  277. INFO:toil.statsAndLogging:Got message from job at time 03-18-2019 13:18:47: After preprocessing, got assembly stats for genome simRat_chr6: Input-sample: /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-b02a3811-2b63-4208-851d-7815af46a62d/tmpmsLXX_/51ce0e6e-35eb-42b0-a238-87152cbd6f68/tmp9rriSf.tmp Total-sequences: 1 Total-length: 647215 Proportion-repeat-masked: 0.075276 ProportionNs: 0.000000 Total-Ns: 0 N50: 647215 Median-sequence-length: 647215 Max-sequence-length: 647215 Min-sequence-length: 647215
  278.  
  279. INFO:toil.statsAndLogging:Got message from job at time 03-18-2019 13:18:50: After preprocessing, got assembly stats for genome simHuman_chr6: Input-sample: /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpEh58IV/9213fab0-91c0-4778-9141-102e25d2b6cf/tmpvIhaPU.tmp Total-sequences: 1 Total-length: 601863 Proportion-repeat-masked: 0.074751 ProportionNs: 0.000000 Total-Ns: 0 N50: 601863 Median-sequence-length: 601863 Max-sequence-length: 601863 Min-sequence-length: 601863
  280.  
  281. INFO:toil.leader:Job ended successfully: 'logAssemblyStats' n/N/jobmBLwjD
  282. INFO:toil.leader:Job ended successfully: 'ProgressiveDown' w/1/jobSc89ZQ
  283. INFO:toil.leader:Issued job 'ProgressiveNext' H/f/jobOf_7TQ with job batch system ID: 90 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  284. INFO:toil.leader:Job ended successfully: 'logAssemblyStats' 7/8/jobiSYXWo
  285. INFO:toil.leader:Job ended successfully: 'logAssemblyStats' D/f/job1GU_Yw
  286. INFO:toil.leader:Job ended successfully: 'logAssemblyStats' S/0/jobsU7qn1
  287. INFO:toil.leader:Job ended successfully: 'logAssemblyStats' 9/1/jobMvLJQy
  288. INFO:toil.statsAndLogging:Got message from job at time 03-18-2019 13:20:49: Project has 0 dependencies
  289. INFO:toil.leader:Job ended successfully: 'ProgressiveNext' H/f/jobOf_7TQ
  290. INFO:toil.leader:Issued job 'ProgressiveUp' O/t/jobBTKSzR with job batch system ID: 91 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  291. INFO:toil.leader:Job ended successfully: 'ProgressiveUp' O/t/jobBTKSzR
  292. INFO:toil.leader:Issued job 'CactusTrimmingBlastPhase' 6/c/job8rdfrw with job batch system ID: 92 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  293. INFO:toil.statsAndLogging:Got message from job at time 03-18-2019 13:23:48: Running blast using the trimming strategy
  294. INFO:toil.statsAndLogging:Got message from job at time 03-18-2019 13:23:48: Ingroup sequences: [('simMouse_chr6', '6/c/job8rdfrw/g/tmpE4vV6H-run-tmpNSP5iE.tmp'), ('simRat_chr6', '6/c/job8rdfrw/g/tmpWnmPXA-run-tmpedIuR0.tmp')]
  295. INFO:toil.statsAndLogging:Got message from job at time 03-18-2019 13:23:48: Outgroup sequences: [('simHuman_chr6', '6/c/job8rdfrw/g/tmpPUPDdM-run-tmpdQrrpe.tmp'), ('simDog_chr6', '6/c/job8rdfrw/g/tmpiw_eNY-run-tmptEJ_iV.tmp'), ('simCow_chr6', '6/c/job8rdfrw/g/tmpv3zZTk-run-tmpdvd0DX.tmp')]
  296. INFO:toil.statsAndLogging:Got message from job at time 03-18-2019 13:23:48: Running mapQ uniquifying with parameters, minimumMapQValue: 0.0, maxAlignmentsPerSite 5, alpha: 0.001
  297. INFO:toil.leader:Job ended successfully: 'CactusTrimmingBlastPhase' 6/c/job8rdfrw
  298. INFO:toil.leader:Issued job 'EncapsulatedJob' D/G/jobAaKlOK with job batch system ID: 93 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  299. INFO:toil.leader:Job ended successfully: 'EncapsulatedJob' D/G/jobAaKlOK
  300. INFO:toil.leader:Issued job 'BlastIngroupsAndOutgroups' 1/P/jobIdB1xQ with job batch system ID: 94 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  301. INFO:toil.statsAndLogging:Got message from job at time 03-18-2019 13:27:01: Blasting ingroups vs outgroups. Ingroup genomes: simMouse_chr6, simRat_chr6, outgroup genomes: simHuman_chr6, simDog_chr6, simCow_chr6
  302. INFO:toil.leader:Job ended successfully: 'BlastIngroupsAndOutgroups' 1/P/jobIdB1xQ
  303. INFO:toil.leader:Issued job 'BlastSequencesAllAgainstAll' y/G/jobtd23h5 with job batch system ID: 95 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  304. INFO:toil.leader:Issued job 'BlastFirstOutgroup' N/J/jobkKP7HD with job batch system ID: 96 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  305. INFO:toil.leader:Job ended successfully: 'BlastFirstOutgroup' N/J/jobkKP7HD
  306. INFO:toil.leader:Issued job 'BlastSequencesAgainstEachOther' Y/4/jobRhpg0d with job batch system ID: 97 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  307. INFO:toil.leader:Job ended successfully: 'BlastSequencesAllAgainstAll' y/G/jobtd23h5
  308. INFO:toil.leader:Issued job 'MakeSelfBlasts' l/p/jobH3cKG_ with job batch system ID: 98 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  309. INFO:toil.leader:Issued job 'MakeOffDiagonalBlasts' 8/a/joblSgcs6 with job batch system ID: 99 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  310. INFO:toil.leader:Job ended successfully: 'BlastSequencesAgainstEachOther' Y/4/jobRhpg0d
  311. INFO:toil.leader:Issued job 'RunBlast' i/i/jobeQniUc with job batch system ID: 100 and cores: 1, disk: 100.0 M, and memory: 100.0 M
  312. INFO:toil.leader:Job ended successfully: 'MakeSelfBlasts' l/p/jobH3cKG_
  313. INFO:toil.leader:Issued job 'RunSelfBlast' P/l/job3wKUaN with job batch system ID: 101 and cores: 1, disk: 100.0 M, and memory: 100.0 M
  314. INFO:toil.leader:Job ended successfully: 'MakeOffDiagonalBlasts' 8/a/joblSgcs6
  315. INFO:toil.leader:Issued job 'CollateBlasts' x/w/jobTzxuRe with job batch system ID: 102 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  316. WARNING:toil.leader:Job failed with exit value 1: 'RunBlast' i/i/jobeQniUc
  317. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'RunBlast' i/i/jobeQniUc
  318. WARNING:toil.leader:i/i/jobeQniUc INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  319. WARNING:toil.leader:i/i/jobeQniUc INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  320. WARNING:toil.leader:i/i/jobeQniUc WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  321. WARNING:toil.leader:i/i/jobeQniUc INFO:cactus.shared.common:Running the command ['cPecanLastz', '--format=cigar', '--notrivial', '--step=1', '--ambiguous=iupac,100,100', '--ydrop=3000', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpLDKJU8/f5b7a302-0440-473a-866e-52760f61d709/tmptcYNQH.tmp[multiple][nameparse=darkspace]', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpLDKJU8/f5b7a302-0440-473a-866e-52760f61d709/tmpDUz6ZM.tmp[nameparse=darkspace]']
  322. WARNING:toil.leader:i/i/jobeQniUc FAILURE: call to malloc failed to allocate 72,242,864 bytes, for new_position_table
  323. WARNING:toil.leader:i/i/jobeQniUc Traceback (most recent call last):
  324. WARNING:toil.leader:i/i/jobeQniUc File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  325. WARNING:toil.leader:i/i/jobeQniUc job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  326. WARNING:toil.leader:i/i/jobeQniUc File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  327. WARNING:toil.leader:i/i/jobeQniUc super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  328. WARNING:toil.leader:i/i/jobeQniUc File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  329. WARNING:toil.leader:i/i/jobeQniUc returnValues = self._run(jobGraph, fileStore)
  330. WARNING:toil.leader:i/i/jobeQniUc File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  331. WARNING:toil.leader:i/i/jobeQniUc return self.run(fileStore)
  332. WARNING:toil.leader:i/i/jobeQniUc File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/blast/blast.py", line 443, in run
  333. WARNING:toil.leader:i/i/jobeQniUc runLastz(seqFile1, seqFile2, blastResultsFile, lastzArguments = self.blastOptions.lastzArguments)
  334. WARNING:toil.leader:i/i/jobeQniUc File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 758, in runLastz
  335. WARNING:toil.leader:i/i/jobeQniUc soft_timeout=5400)
  336. WARNING:toil.leader:i/i/jobeQniUc File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  337. WARNING:toil.leader:i/i/jobeQniUc raise RuntimeError("Command %s failed with output: %s" % (call, output))
  338. WARNING:toil.leader:i/i/jobeQniUc RuntimeError: Command ['cPecanLastz', '--format=cigar', '--notrivial', '--step=1', '--ambiguous=iupac,100,100', '--ydrop=3000', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpLDKJU8/f5b7a302-0440-473a-866e-52760f61d709/tmptcYNQH.tmp[multiple][nameparse=darkspace]', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpLDKJU8/f5b7a302-0440-473a-866e-52760f61d709/tmpDUz6ZM.tmp[nameparse=darkspace]'] failed with output: None
  339. WARNING:toil.leader:i/i/jobeQniUc ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn38.grid.pub.ro
  340. WARNING:toil.leader:i/i/jobeQniUc WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'RunBlast' i/i/jobeQniUc with ID i/i/jobeQniUc to 5
  341. WARNING:toil.leader:i/i/jobeQniUc WARNING:toil.jobGraph:We have increased the default memory of the failed job 'RunBlast' i/i/jobeQniUc to 2147483648 bytes
  342. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'RunBlast' i/i/jobeQniUc with ID i/i/jobeQniUc to 4
  343. INFO:toil.leader:Issued job 'RunBlast' i/i/jobeQniUc with job batch system ID: 103 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  344. WARNING:toil.leader:Job failed with exit value 1: 'RunSelfBlast' P/l/job3wKUaN
  345. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'RunSelfBlast' P/l/job3wKUaN
  346. WARNING:toil.leader:P/l/job3wKUaN INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  347. WARNING:toil.leader:P/l/job3wKUaN INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  348. WARNING:toil.leader:P/l/job3wKUaN WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  349. WARNING:toil.leader:P/l/job3wKUaN INFO:cactus.shared.common:Running the command ['cPecanLastz', '--format=cigar', '--notrivial', '--step=1', '--ambiguous=iupac,100,100', '--ydrop=3000', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-906375ca-4612-4b9c-aace-ad64276eee7f/tmp2Nvegg/a376bc48-ab13-4c29-b05c-134f536dacf8/tmptG_qao.tmp[multiple][nameparse=darkspace]', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-906375ca-4612-4b9c-aace-ad64276eee7f/tmp2Nvegg/a376bc48-ab13-4c29-b05c-134f536dacf8/tmptG_qao.tmp[nameparse=darkspace]']
  350. WARNING:toil.leader:P/l/job3wKUaN FAILURE: call to malloc failed to allocate 72,242,864 bytes, for new_position_table
  351. WARNING:toil.leader:P/l/job3wKUaN Traceback (most recent call last):
  352. WARNING:toil.leader:P/l/job3wKUaN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  353. WARNING:toil.leader:P/l/job3wKUaN job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  354. WARNING:toil.leader:P/l/job3wKUaN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  355. WARNING:toil.leader:P/l/job3wKUaN super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  356. WARNING:toil.leader:P/l/job3wKUaN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  357. WARNING:toil.leader:P/l/job3wKUaN returnValues = self._run(jobGraph, fileStore)
  358. WARNING:toil.leader:P/l/job3wKUaN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  359. WARNING:toil.leader:P/l/job3wKUaN return self.run(fileStore)
  360. WARNING:toil.leader:P/l/job3wKUaN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/blast/blast.py", line 402, in run
  361. WARNING:toil.leader:P/l/job3wKUaN runSelfLastz(seqFile, blastResultsFile, lastzArguments=self.blastOptions.lastzArguments)
  362. WARNING:toil.leader:P/l/job3wKUaN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 769, in runSelfLastz
  363. WARNING:toil.leader:P/l/job3wKUaN soft_timeout=5400)
  364. WARNING:toil.leader:P/l/job3wKUaN File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  365. WARNING:toil.leader:P/l/job3wKUaN raise RuntimeError("Command %s failed with output: %s" % (call, output))
  366. WARNING:toil.leader:P/l/job3wKUaN RuntimeError: Command ['cPecanLastz', '--format=cigar', '--notrivial', '--step=1', '--ambiguous=iupac,100,100', '--ydrop=3000', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-906375ca-4612-4b9c-aace-ad64276eee7f/tmp2Nvegg/a376bc48-ab13-4c29-b05c-134f536dacf8/tmptG_qao.tmp[multiple][nameparse=darkspace]', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-906375ca-4612-4b9c-aace-ad64276eee7f/tmp2Nvegg/a376bc48-ab13-4c29-b05c-134f536dacf8/tmptG_qao.tmp[nameparse=darkspace]'] failed with output: None
  367. WARNING:toil.leader:P/l/job3wKUaN ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn34.grid.pub.ro
  368. WARNING:toil.leader:P/l/job3wKUaN WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'RunSelfBlast' P/l/job3wKUaN with ID P/l/job3wKUaN to 5
  369. WARNING:toil.leader:P/l/job3wKUaN WARNING:toil.jobGraph:We have increased the default memory of the failed job 'RunSelfBlast' P/l/job3wKUaN to 2147483648 bytes
  370. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'RunSelfBlast' P/l/job3wKUaN with ID P/l/job3wKUaN to 4
  371. INFO:toil.leader:Issued job 'RunSelfBlast' P/l/job3wKUaN with job batch system ID: 104 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  372. INFO:toil.leader:Job ended successfully: 'CollateBlasts' x/w/jobTzxuRe
  373. INFO:toil.leader:Issued job 'CollateBlasts2' V/3/jobtIzCkE with job batch system ID: 105 and cores: 1, disk: 0.0 , and memory: 2.0 G
  374. WARNING:toil.statsAndLogging:Got message from job at time 03-18-2019 13:33:11: Job used more disk than requested. Consider modifying the user script to avoid the chance of failure due to incorrectly requested resources. Job x/w/jobTzxuRe/g/tmpBayj9z-_serialiseJob-stream used 0.00% (4.0 KB [4096B] used, 0.0 B [0B] requested) at the end of its run.
  375. INFO:toil.leader:Job ended successfully: 'RunBlast' i/i/jobeQniUc
  376. INFO:toil.leader:Issued job 'CollateBlasts' 7/k/jobbMyjKA with job batch system ID: 106 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  377. INFO:toil.leader:Job ended successfully: 'RunSelfBlast' P/l/job3wKUaN
  378. INFO:toil.leader:Issued job 'CollateBlasts' I/p/job0tojhf with job batch system ID: 107 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  379. INFO:toil.leader:Job ended successfully: 'CollateBlasts2' V/3/jobtIzCkE
  380. INFO:toil.leader:Issued job 'CollateBlasts' x/w/jobTzxuRe with job batch system ID: 108 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  381. INFO:toil.leader:Job ended successfully: 'CollateBlasts' x/w/jobTzxuRe
  382. INFO:toil.leader:Issued job 'MakeOffDiagonalBlasts' 8/a/joblSgcs6 with job batch system ID: 109 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  383. INFO:toil.leader:Job ended successfully: 'CollateBlasts' 7/k/jobbMyjKA
  384. INFO:toil.leader:Issued job 'CollateBlasts2' Z/f/job3vf6zp with job batch system ID: 110 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  385. INFO:toil.leader:Job ended successfully: 'CollateBlasts' I/p/job0tojhf
  386. INFO:toil.leader:Issued job 'CollateBlasts2' B/0/jobZsBZBM with job batch system ID: 111 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  387. INFO:toil.leader:Job ended successfully: 'CollateBlasts2' B/0/jobZsBZBM
  388. INFO:toil.leader:Issued job 'CollateBlasts' I/p/job0tojhf with job batch system ID: 112 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  389. INFO:toil.leader:Job ended successfully: 'MakeOffDiagonalBlasts' 8/a/joblSgcs6
  390. INFO:toil.leader:Job ended successfully: 'CollateBlasts2' Z/f/job3vf6zp
  391. INFO:toil.leader:Issued job 'CollateBlasts' 7/k/jobbMyjKA with job batch system ID: 113 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  392. INFO:toil.leader:Job ended successfully: 'CollateBlasts' I/p/job0tojhf
  393. INFO:toil.leader:Issued job 'MakeSelfBlasts' l/p/jobH3cKG_ with job batch system ID: 114 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  394. INFO:toil.leader:Job ended successfully: 'CollateBlasts' 7/k/jobbMyjKA
  395. INFO:toil.leader:Issued job 'BlastSequencesAgainstEachOther' Y/4/jobRhpg0d with job batch system ID: 115 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  396. INFO:toil.leader:Job ended successfully: 'BlastSequencesAgainstEachOther' Y/4/jobRhpg0d
  397. INFO:toil.leader:Issued job 'TrimAndRecurseOnOutgroups' 1/G/jobdVAEBo with job batch system ID: 116 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  398. INFO:toil.leader:Job ended successfully: 'MakeSelfBlasts' l/p/jobH3cKG_
  399. INFO:toil.leader:Issued job 'CollateBlasts' 5/D/job_rRpe7 with job batch system ID: 117 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  400. INFO:toil.statsAndLogging:Got message from job at time 03-18-2019 13:41:47: Coverage on simMouse_chr6 from outgroup #1, simHuman_chr6: 42.9126680518% (current ingroup length 636262, untrimmed length 636262). Outgroup trimmed to 593077 bp from 601863
  401. INFO:toil.statsAndLogging:Got message from job at time 03-18-2019 13:41:47: Coverage on simRat_chr6 from outgroup #1, simHuman_chr6: 41.0293333745% (current ingroup length 647215, untrimmed length 647215). Outgroup trimmed to 593077 bp from 601863
  402. INFO:toil.statsAndLogging:Got message from job at time 03-18-2019 13:41:47: Cumulative coverage of 1 outgroups on ingroup simMouse_chr6: 42.9126680518
  403. INFO:toil.statsAndLogging:Got message from job at time 03-18-2019 13:41:47: Cumulative coverage of 1 outgroups on ingroup simRat_chr6: 41.0293333745
  404. INFO:toil.leader:Reissued any over long jobs
  405. INFO:toil.leader:Job ended successfully: 'CollateBlasts' 5/D/job_rRpe7
  406. INFO:toil.leader:Issued job 'CollateBlasts2' z/I/jobGgGGrO with job batch system ID: 118 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  407. INFO:toil.leader:Job ended successfully: 'TrimAndRecurseOnOutgroups' 1/G/jobdVAEBo
  408. INFO:toil.leader:Issued job 'BlastFirstOutgroup' m/L/jobYbzYEB with job batch system ID: 119 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  409. INFO:toil.leader:Job ended successfully: 'CollateBlasts2' z/I/jobGgGGrO
  410. INFO:toil.leader:Issued job 'CollateBlasts' 5/D/job_rRpe7 with job batch system ID: 120 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  411. INFO:toil.leader:Job ended successfully: 'CollateBlasts' 5/D/job_rRpe7
  412. INFO:toil.leader:Issued job 'BlastSequencesAllAgainstAll' y/G/jobtd23h5 with job batch system ID: 121 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  413. INFO:toil.leader:Job ended successfully: 'BlastFirstOutgroup' m/L/jobYbzYEB
  414. INFO:toil.leader:Issued job 'BlastSequencesAgainstEachOther' x/S/jobBC1_1J with job batch system ID: 122 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  415. INFO:toil.leader:Job ended successfully: 'BlastSequencesAllAgainstAll' y/G/jobtd23h5
  416. INFO:toil.leader:Job ended successfully: 'BlastSequencesAgainstEachOther' x/S/jobBC1_1J
  417. INFO:toil.leader:Issued job 'RunBlast' L/k/jobSZ3vdK with job batch system ID: 123 and cores: 1, disk: 100.0 M, and memory: 100.0 M
  418. WARNING:toil.leader:Job failed with exit value 1: 'RunBlast' L/k/jobSZ3vdK
  419. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'RunBlast' L/k/jobSZ3vdK
  420. WARNING:toil.leader:L/k/jobSZ3vdK INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  421. WARNING:toil.leader:L/k/jobSZ3vdK INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  422. WARNING:toil.leader:L/k/jobSZ3vdK WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  423. WARNING:toil.leader:L/k/jobSZ3vdK INFO:cactus.shared.common:Running the command ['cPecanLastz', '--format=cigar', '--notrivial', '--step=1', '--ambiguous=iupac,100,100', '--ydrop=3000', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpcQlY4F/599ba8bb-166b-471d-8067-58911f50fea6/tmps0mU25.tmp[multiple][nameparse=darkspace]', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpcQlY4F/599ba8bb-166b-471d-8067-58911f50fea6/tmpiSImLT.tmp[nameparse=darkspace]']
  424. WARNING:toil.leader:L/k/jobSZ3vdK FAILURE: call to malloc failed to allocate 69,733,056 bytes, for new_position_table
  425. WARNING:toil.leader:L/k/jobSZ3vdK Traceback (most recent call last):
  426. WARNING:toil.leader:L/k/jobSZ3vdK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  427. WARNING:toil.leader:L/k/jobSZ3vdK job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  428. WARNING:toil.leader:L/k/jobSZ3vdK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  429. WARNING:toil.leader:L/k/jobSZ3vdK super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  430. WARNING:toil.leader:L/k/jobSZ3vdK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  431. WARNING:toil.leader:L/k/jobSZ3vdK returnValues = self._run(jobGraph, fileStore)
  432. WARNING:toil.leader:L/k/jobSZ3vdK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  433. WARNING:toil.leader:L/k/jobSZ3vdK return self.run(fileStore)
  434. WARNING:toil.leader:L/k/jobSZ3vdK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/blast/blast.py", line 443, in run
  435. WARNING:toil.leader:L/k/jobSZ3vdK runLastz(seqFile1, seqFile2, blastResultsFile, lastzArguments = self.blastOptions.lastzArguments)
  436. WARNING:toil.leader:L/k/jobSZ3vdK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 758, in runLastz
  437. WARNING:toil.leader:L/k/jobSZ3vdK soft_timeout=5400)
  438. WARNING:toil.leader:L/k/jobSZ3vdK File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  439. WARNING:toil.leader:L/k/jobSZ3vdK raise RuntimeError("Command %s failed with output: %s" % (call, output))
  440. WARNING:toil.leader:L/k/jobSZ3vdK RuntimeError: Command ['cPecanLastz', '--format=cigar', '--notrivial', '--step=1', '--ambiguous=iupac,100,100', '--ydrop=3000', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpcQlY4F/599ba8bb-166b-471d-8067-58911f50fea6/tmps0mU25.tmp[multiple][nameparse=darkspace]', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpcQlY4F/599ba8bb-166b-471d-8067-58911f50fea6/tmpiSImLT.tmp[nameparse=darkspace]'] failed with output: None
  441. WARNING:toil.leader:L/k/jobSZ3vdK ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn37.grid.pub.ro
  442. WARNING:toil.leader:L/k/jobSZ3vdK WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'RunBlast' L/k/jobSZ3vdK with ID L/k/jobSZ3vdK to 5
  443. WARNING:toil.leader:L/k/jobSZ3vdK WARNING:toil.jobGraph:We have increased the default memory of the failed job 'RunBlast' L/k/jobSZ3vdK to 2147483648 bytes
  444. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'RunBlast' L/k/jobSZ3vdK with ID L/k/jobSZ3vdK to 4
  445. INFO:toil.leader:Issued job 'RunBlast' L/k/jobSZ3vdK with job batch system ID: 124 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  446. INFO:toil.leader:Job ended successfully: 'RunBlast' L/k/jobSZ3vdK
  447. INFO:toil.leader:Issued job 'CollateBlasts' V/e/jobWzY8cT with job batch system ID: 125 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  448. INFO:toil.leader:Job ended successfully: 'CollateBlasts' V/e/jobWzY8cT
  449. INFO:toil.leader:Issued job 'CollateBlasts2' e/7/jobkYxSzb with job batch system ID: 126 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  450. INFO:toil.leader:Job ended successfully: 'CollateBlasts2' e/7/jobkYxSzb
  451. INFO:toil.leader:Issued job 'CollateBlasts' V/e/jobWzY8cT with job batch system ID: 127 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  452. INFO:toil.leader:Job ended successfully: 'CollateBlasts' V/e/jobWzY8cT
  453. INFO:toil.leader:Issued job 'BlastSequencesAgainstEachOther' x/S/jobBC1_1J with job batch system ID: 128 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  454. INFO:toil.leader:Job ended successfully: 'BlastSequencesAgainstEachOther' x/S/jobBC1_1J
  455. INFO:toil.leader:Issued job 'TrimAndRecurseOnOutgroups' D/B/jobRf42tL with job batch system ID: 129 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  456. INFO:toil.statsAndLogging:Got message from job at time 03-18-2019 13:59:02: Coverage on simMouse_chr6 from outgroup #2, simDog_chr6: 5.02493867797% (current ingroup length 317178, untrimmed length 636262). Outgroup trimmed to 269178 bp from 593897
  457. INFO:toil.statsAndLogging:Got message from job at time 03-18-2019 13:59:02: Coverage on simRat_chr6 from outgroup #2, simDog_chr6: 4.32875040364% (current ingroup length 337557, untrimmed length 647215). Outgroup trimmed to 269178 bp from 593897
  458. INFO:toil.statsAndLogging:Got message from job at time 03-18-2019 13:59:02: Cumulative coverage of 2 outgroups on ingroup simMouse_chr6: 45.3963933097
  459. INFO:toil.statsAndLogging:Got message from job at time 03-18-2019 13:59:02: Cumulative coverage of 2 outgroups on ingroup simRat_chr6: 43.2647574608
  460. INFO:toil.leader:Job ended successfully: 'TrimAndRecurseOnOutgroups' D/B/jobRf42tL
  461. INFO:toil.leader:Issued job 'BlastFirstOutgroup' 2/E/jobCoF2tq with job batch system ID: 130 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  462. INFO:toil.leader:Job ended successfully: 'BlastFirstOutgroup' 2/E/jobCoF2tq
  463. INFO:toil.leader:Issued job 'BlastSequencesAgainstEachOther' 5/n/jobHbBr7k with job batch system ID: 131 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  464. INFO:toil.leader:Job ended successfully: 'BlastSequencesAgainstEachOther' 5/n/jobHbBr7k
  465. INFO:toil.leader:Issued job 'RunBlast' S/n/jobOrlMT5 with job batch system ID: 132 and cores: 1, disk: 100.0 M, and memory: 100.0 M
  466. WARNING:toil.leader:Job failed with exit value 1: 'RunBlast' S/n/jobOrlMT5
  467. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'RunBlast' S/n/jobOrlMT5
  468. WARNING:toil.leader:S/n/jobOrlMT5 INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  469. WARNING:toil.leader:S/n/jobOrlMT5 INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  470. WARNING:toil.leader:S/n/jobOrlMT5 WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  471. WARNING:toil.leader:S/n/jobOrlMT5 INFO:cactus.shared.common:Running the command ['cPecanLastz', '--format=cigar', '--notrivial', '--step=1', '--ambiguous=iupac,100,100', '--ydrop=3000', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpS66Sfl/d536d584-7f05-47a3-91fc-71337ca7c7e3/tmpGYLNSg.tmp[multiple][nameparse=darkspace]', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpS66Sfl/d536d584-7f05-47a3-91fc-71337ca7c7e3/tmp83d07p.tmp[nameparse=darkspace]']
  472. WARNING:toil.leader:S/n/jobOrlMT5 FAILURE: call to malloc failed to allocate 69,592,544 bytes, for new_position_table
  473. WARNING:toil.leader:S/n/jobOrlMT5 Traceback (most recent call last):
  474. WARNING:toil.leader:S/n/jobOrlMT5 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  475. WARNING:toil.leader:S/n/jobOrlMT5 job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  476. WARNING:toil.leader:S/n/jobOrlMT5 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  477. WARNING:toil.leader:S/n/jobOrlMT5 super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  478. WARNING:toil.leader:S/n/jobOrlMT5 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  479. WARNING:toil.leader:S/n/jobOrlMT5 returnValues = self._run(jobGraph, fileStore)
  480. WARNING:toil.leader:S/n/jobOrlMT5 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  481. WARNING:toil.leader:S/n/jobOrlMT5 return self.run(fileStore)
  482. WARNING:toil.leader:S/n/jobOrlMT5 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/blast/blast.py", line 443, in run
  483. WARNING:toil.leader:S/n/jobOrlMT5 runLastz(seqFile1, seqFile2, blastResultsFile, lastzArguments = self.blastOptions.lastzArguments)
  484. WARNING:toil.leader:S/n/jobOrlMT5 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 758, in runLastz
  485. WARNING:toil.leader:S/n/jobOrlMT5 soft_timeout=5400)
  486. WARNING:toil.leader:S/n/jobOrlMT5 File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  487. WARNING:toil.leader:S/n/jobOrlMT5 raise RuntimeError("Command %s failed with output: %s" % (call, output))
  488. WARNING:toil.leader:S/n/jobOrlMT5 RuntimeError: Command ['cPecanLastz', '--format=cigar', '--notrivial', '--step=1', '--ambiguous=iupac,100,100', '--ydrop=3000', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpS66Sfl/d536d584-7f05-47a3-91fc-71337ca7c7e3/tmpGYLNSg.tmp[multiple][nameparse=darkspace]', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpS66Sfl/d536d584-7f05-47a3-91fc-71337ca7c7e3/tmp83d07p.tmp[nameparse=darkspace]'] failed with output: None
  489. WARNING:toil.leader:S/n/jobOrlMT5 ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn38.grid.pub.ro
  490. WARNING:toil.leader:S/n/jobOrlMT5 WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'RunBlast' S/n/jobOrlMT5 with ID S/n/jobOrlMT5 to 5
  491. WARNING:toil.leader:S/n/jobOrlMT5 WARNING:toil.jobGraph:We have increased the default memory of the failed job 'RunBlast' S/n/jobOrlMT5 to 2147483648 bytes
  492. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'RunBlast' S/n/jobOrlMT5 with ID S/n/jobOrlMT5 to 4
  493. INFO:toil.leader:Issued job 'RunBlast' S/n/jobOrlMT5 with job batch system ID: 133 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  494. INFO:toil.leader:Job ended successfully: 'RunBlast' S/n/jobOrlMT5
  495. INFO:toil.leader:Issued job 'CollateBlasts' o/I/jobhy7sVY with job batch system ID: 134 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  496. INFO:toil.leader:Job ended successfully: 'CollateBlasts' o/I/jobhy7sVY
  497. INFO:toil.leader:Issued job 'CollateBlasts2' l/v/job9F5xjl with job batch system ID: 135 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  498. INFO:toil.leader:Job ended successfully: 'CollateBlasts2' l/v/job9F5xjl
  499. INFO:toil.leader:Issued job 'CollateBlasts' o/I/jobhy7sVY with job batch system ID: 136 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  500. INFO:toil.leader:Job ended successfully: 'CollateBlasts' o/I/jobhy7sVY
  501. INFO:toil.leader:Issued job 'BlastSequencesAgainstEachOther' 5/n/jobHbBr7k with job batch system ID: 137 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  502. INFO:toil.leader:Job ended successfully: 'BlastSequencesAgainstEachOther' 5/n/jobHbBr7k
  503. INFO:toil.leader:Issued job 'TrimAndRecurseOnOutgroups' L/z/job7Sk5Gu with job batch system ID: 138 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  504. INFO:toil.statsAndLogging:Got message from job at time 03-18-2019 14:14:43: Coverage on simMouse_chr6 from outgroup #3, simCow_chr6: 1.45565380493% (current ingroup length 298560, untrimmed length 636262). Outgroup trimmed to 108335 bp from 602619
  505. INFO:toil.statsAndLogging:Got message from job at time 03-18-2019 14:14:43: Coverage on simRat_chr6 from outgroup #3, simCow_chr6: 1.14062280934% (current ingroup length 320965, untrimmed length 647215). Outgroup trimmed to 108335 bp from 602619
  506. INFO:toil.statsAndLogging:Got message from job at time 03-18-2019 14:14:43: Cumulative coverage of 3 outgroups on ingroup simMouse_chr6: 46.0489546759
  507. INFO:toil.statsAndLogging:Got message from job at time 03-18-2019 14:14:43: Cumulative coverage of 3 outgroups on ingroup simRat_chr6: 43.8301028252
  508. INFO:toil.leader:Job ended successfully: 'TrimAndRecurseOnOutgroups' L/z/job7Sk5Gu
  509. INFO:toil.leader:Issued job 'BlastFirstOutgroup' 2/E/jobCoF2tq with job batch system ID: 139 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  510. INFO:toil.leader:Job ended successfully: 'BlastFirstOutgroup' 2/E/jobCoF2tq
  511. INFO:toil.leader:Issued job 'TrimAndRecurseOnOutgroups' D/B/jobRf42tL with job batch system ID: 140 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  512. INFO:toil.leader:Job ended successfully: 'TrimAndRecurseOnOutgroups' D/B/jobRf42tL
  513. INFO:toil.leader:Issued job 'BlastFirstOutgroup' m/L/jobYbzYEB with job batch system ID: 141 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  514. INFO:toil.leader:Job ended successfully: 'BlastFirstOutgroup' m/L/jobYbzYEB
  515. INFO:toil.leader:Issued job 'TrimAndRecurseOnOutgroups' 1/G/jobdVAEBo with job batch system ID: 142 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  516. INFO:toil.leader:Job ended successfully: 'TrimAndRecurseOnOutgroups' 1/G/jobdVAEBo
  517. INFO:toil.leader:Issued job 'BlastFirstOutgroup' N/J/jobkKP7HD with job batch system ID: 143 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  518. INFO:toil.leader:Job ended successfully: 'BlastFirstOutgroup' N/J/jobkKP7HD
  519. INFO:toil.leader:Issued job 'CollateBlasts' c/P/jobuK0f3G with job batch system ID: 144 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  520. INFO:toil.leader:Job ended successfully: 'CollateBlasts' c/P/jobuK0f3G
  521. INFO:toil.leader:Issued job 'CollateBlasts2' z/I/jobTJu6dW with job batch system ID: 145 and cores: 1, disk: 100.0 M, and memory: 2.0 G
  522. INFO:toil.leader:Job ended successfully: 'CollateBlasts2' z/I/jobTJu6dW
  523. INFO:toil.leader:Issued job 'CollateBlasts' c/P/jobuK0f3G with job batch system ID: 146 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  524. INFO:toil.leader:Job ended successfully: 'CollateBlasts' c/P/jobuK0f3G
  525. INFO:toil.leader:Issued job 'BlastIngroupsAndOutgroups' 1/P/jobIdB1xQ with job batch system ID: 147 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  526. INFO:toil.leader:Job ended successfully: 'BlastIngroupsAndOutgroups' 1/P/jobIdB1xQ
  527. INFO:toil.leader:Issued job 'Job' 5/m/jobkhQTMv with job batch system ID: 148 and cores: 0, disk: 1.0 M, and memory: 32.0 M
  528. WARNING:toil.leader:Job failed with exit value 1: 'Job' 5/m/jobkhQTMv
  529. WARNING:toil.leader:No log file is present, despite job failing: 'Job' 5/m/jobkhQTMv
  530. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'Job' 5/m/jobkhQTMv with ID 5/m/jobkhQTMv to 5
  531. WARNING:toil.jobGraph:We have increased the default memory of the failed job 'Job' 5/m/jobkhQTMv to 2147483648 bytes
  532. INFO:toil.leader:Issued job 'Job' 5/m/jobkhQTMv with job batch system ID: 149 and cores: 0, disk: 1.0 M, and memory: 2.0 G
  533. INFO:toil.leader:Job ended successfully: 'Job' 5/m/jobkhQTMv
  534. INFO:toil.leader:Issued job 'mappingQualityRescoring' I/T/jobh7hfYo with job batch system ID: 150 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  535. INFO:toil.statsAndLogging:Got message from job at time 03-18-2019 14:34:39: Input cigar file has 7362 lines
  536. INFO:toil.statsAndLogging:Got message from job at time 03-18-2019 14:34:39: Filtered, non-overlapping primary cigar file has 11273 lines
  537. INFO:toil.statsAndLogging:Got message from job at time 03-18-2019 14:34:39: Filtered, non-overlapping secondary cigar file has 31711 lines
  538. INFO:toil.leader:Job ended successfully: 'mappingQualityRescoring' I/T/jobh7hfYo
  539. INFO:toil.leader:Issued job 'Job' 5/m/jobkhQTMv with job batch system ID: 151 and cores: 0, disk: 1.0 M, and memory: 2.0 G
  540. INFO:toil.leader:Job ended successfully: 'Job' 5/m/jobkhQTMv
  541. INFO:toil.leader:Issued job 'EncapsulatedJob' D/G/jobAaKlOK with job batch system ID: 152 and cores: 1, disk: 2.0 G, and memory: 2.0 G
  542. INFO:toil.leader:Job ended successfully: 'EncapsulatedJob' D/G/jobAaKlOK
  543. INFO:toil.leader:Issued job 'CactusSetupCheckpoint' P/9/jobVgO4XI with job batch system ID: 153 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  544. INFO:toil.leader:Job ended successfully: 'CactusSetupCheckpoint' P/9/jobVgO4XI
  545. INFO:toil.leader:Issued job 'StartPrimaryDB' a/n/jobKwRkH2 with job batch system ID: 154 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  546. INFO:toil.leader:Reissued any over long jobs
  547. INFO:toil.leader:Job ended successfully: 'StartPrimaryDB' a/n/jobKwRkH2
  548. INFO:toil.leader:Issued job 'KtServerService' w/R/job74nnk2 with job batch system ID: 155 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  549. INFO:toil.leader:Issued job 'CactusSetupPhase' a/K/job40aWLl with job batch system ID: 156 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  550. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' w/R/job74nnk2
  551. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' w/R/job74nnk2
  552. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' w/R/job74nnk2 with ID w/R/job74nnk2 to 5
  553. INFO:toil.leader:Issued job 'KtServerService' w/R/job74nnk2 with job batch system ID: 157 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  554. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' a/K/job40aWLl
  555. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' a/K/job40aWLl
  556. WARNING:toil.leader:a/K/job40aWLl INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  557. WARNING:toil.leader:a/K/job40aWLl INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  558. WARNING:toil.leader:a/K/job40aWLl WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  559. WARNING:toil.leader:a/K/job40aWLl WARNING:toil.fileStore:Detected that job (a/n/jobKwRkH2/g/tmpZ5X_AY-_serialiseJob-stream) prematurely terminated. Fixing the state of the job on disk.
  560. WARNING:toil.leader:a/K/job40aWLl INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  561. WARNING:toil.leader:a/K/job40aWLl INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  562. WARNING:toil.leader:a/K/job40aWLl INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="24170" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmp4oGtfX/93ab2165-9120-4629-9867-2f63ecc43055/tmpK6IlI5.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmp4oGtfX/93ab2165-9120-4629-9867-2f63ecc43055/tmpZqyDL6.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmp4oGtfX/93ab2165-9120-4629-9867-2f63ecc43055/tmpuVqUM6.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmp4oGtfX/93ab2165-9120-4629-9867-2f63ecc43055/tmpVB20wM.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmp4oGtfX/93ab2165-9120-4629-9867-2f63ecc43055/tmpFDpNxo.tmp']
  563. WARNING:toil.leader:a/K/job40aWLl Set log level to INFO
  564. WARNING:toil.leader:a/K/job40aWLl Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  565. WARNING:toil.leader:a/K/job40aWLl <kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="24170" />
  566. WARNING:toil.leader:a/K/job40aWLl </st_kv_database_conf>
  567. WARNING:toil.leader:a/K/job40aWLl
  568. WARNING:toil.leader:a/K/job40aWLl Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmp4oGtfX/93ab2165-9120-4629-9867-2f63ecc43055/tmpK6IlI5.tmp
  569. WARNING:toil.leader:a/K/job40aWLl Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmp4oGtfX/93ab2165-9120-4629-9867-2f63ecc43055/tmpZqyDL6.tmp
  570. WARNING:toil.leader:a/K/job40aWLl Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmp4oGtfX/93ab2165-9120-4629-9867-2f63ecc43055/tmpuVqUM6.tmp
  571. WARNING:toil.leader:a/K/job40aWLl Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmp4oGtfX/93ab2165-9120-4629-9867-2f63ecc43055/tmpVB20wM.tmp
  572. WARNING:toil.leader:a/K/job40aWLl Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmp4oGtfX/93ab2165-9120-4629-9867-2f63ecc43055/tmpFDpNxo.tmp
  573. WARNING:toil.leader:a/K/job40aWLl Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.29 with error: network error
  574. WARNING:toil.leader:a/K/job40aWLl Uncaught exception
  575. WARNING:toil.leader:a/K/job40aWLl Traceback (most recent call last):
  576. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  577. WARNING:toil.leader:a/K/job40aWLl job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  578. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  579. WARNING:toil.leader:a/K/job40aWLl super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  580. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  581. WARNING:toil.leader:a/K/job40aWLl returnValues = self._run(jobGraph, fileStore)
  582. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  583. WARNING:toil.leader:a/K/job40aWLl return self.run(fileStore)
  584. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  585. WARNING:toil.leader:a/K/job40aWLl makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  586. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  587. WARNING:toil.leader:a/K/job40aWLl parameters=["cactus_setup"] + args + sequences)
  588. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  589. WARNING:toil.leader:a/K/job40aWLl raise RuntimeError("Command %s failed with output: %s" % (call, output))
  590. WARNING:toil.leader:a/K/job40aWLl RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="24170" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmp4oGtfX/93ab2165-9120-4629-9867-2f63ecc43055/tmpK6IlI5.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmp4oGtfX/93ab2165-9120-4629-9867-2f63ecc43055/tmpZqyDL6.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmp4oGtfX/93ab2165-9120-4629-9867-2f63ecc43055/tmpuVqUM6.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmp4oGtfX/93ab2165-9120-4629-9867-2f63ecc43055/tmpVB20wM.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmp4oGtfX/93ab2165-9120-4629-9867-2f63ecc43055/tmpFDpNxo.tmp'] failed with output:
  591. WARNING:toil.leader:a/K/job40aWLl ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn29.grid.pub.ro
  592. WARNING:toil.leader:a/K/job40aWLl WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' a/K/job40aWLl with ID a/K/job40aWLl to 5
  593. INFO:toil.leader:Issued job 'CactusSetupPhase' a/K/job40aWLl with job batch system ID: 158 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  594. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' w/R/job74nnk2
  595. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' w/R/job74nnk2
  596. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' w/R/job74nnk2 with ID w/R/job74nnk2 to 4
  597. INFO:toil.leader:Issued job 'KtServerService' w/R/job74nnk2 with job batch system ID: 159 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  598. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' a/K/job40aWLl
  599. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' a/K/job40aWLl
  600. WARNING:toil.leader:a/K/job40aWLl INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  601. WARNING:toil.leader:a/K/job40aWLl INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  602. WARNING:toil.leader:a/K/job40aWLl WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  603. WARNING:toil.leader:a/K/job40aWLl INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  604. WARNING:toil.leader:a/K/job40aWLl INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  605. WARNING:toil.leader:a/K/job40aWLl INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="13235" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpY1IZ2g/185468b7-9035-4776-a38a-7958c32a5fef/tmp3AjZD5.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpY1IZ2g/185468b7-9035-4776-a38a-7958c32a5fef/tmpmAPN95.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpY1IZ2g/185468b7-9035-4776-a38a-7958c32a5fef/tmpvjftI4.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpY1IZ2g/185468b7-9035-4776-a38a-7958c32a5fef/tmpL1JdkO.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpY1IZ2g/185468b7-9035-4776-a38a-7958c32a5fef/tmpBQqqOt.tmp']
  606. WARNING:toil.leader:a/K/job40aWLl Set log level to INFO
  607. WARNING:toil.leader:a/K/job40aWLl Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  608. WARNING:toil.leader:a/K/job40aWLl <kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="13235" />
  609. WARNING:toil.leader:a/K/job40aWLl </st_kv_database_conf>
  610. WARNING:toil.leader:a/K/job40aWLl
  611. WARNING:toil.leader:a/K/job40aWLl Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpY1IZ2g/185468b7-9035-4776-a38a-7958c32a5fef/tmp3AjZD5.tmp
  612. WARNING:toil.leader:a/K/job40aWLl Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpY1IZ2g/185468b7-9035-4776-a38a-7958c32a5fef/tmpmAPN95.tmp
  613. WARNING:toil.leader:a/K/job40aWLl Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpY1IZ2g/185468b7-9035-4776-a38a-7958c32a5fef/tmpvjftI4.tmp
  614. WARNING:toil.leader:a/K/job40aWLl Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpY1IZ2g/185468b7-9035-4776-a38a-7958c32a5fef/tmpL1JdkO.tmp
  615. WARNING:toil.leader:a/K/job40aWLl Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpY1IZ2g/185468b7-9035-4776-a38a-7958c32a5fef/tmpBQqqOt.tmp
  616. WARNING:toil.leader:a/K/job40aWLl Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.41 with error: network error
  617. WARNING:toil.leader:a/K/job40aWLl Uncaught exception
  618. WARNING:toil.leader:a/K/job40aWLl Traceback (most recent call last):
  619. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  620. WARNING:toil.leader:a/K/job40aWLl job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  621. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  622. WARNING:toil.leader:a/K/job40aWLl super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  623. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  624. WARNING:toil.leader:a/K/job40aWLl returnValues = self._run(jobGraph, fileStore)
  625. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  626. WARNING:toil.leader:a/K/job40aWLl return self.run(fileStore)
  627. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  628. WARNING:toil.leader:a/K/job40aWLl makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  629. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  630. WARNING:toil.leader:a/K/job40aWLl parameters=["cactus_setup"] + args + sequences)
  631. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  632. WARNING:toil.leader:a/K/job40aWLl raise RuntimeError("Command %s failed with output: %s" % (call, output))
  633. WARNING:toil.leader:a/K/job40aWLl RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="13235" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpY1IZ2g/185468b7-9035-4776-a38a-7958c32a5fef/tmp3AjZD5.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpY1IZ2g/185468b7-9035-4776-a38a-7958c32a5fef/tmpmAPN95.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpY1IZ2g/185468b7-9035-4776-a38a-7958c32a5fef/tmpvjftI4.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpY1IZ2g/185468b7-9035-4776-a38a-7958c32a5fef/tmpL1JdkO.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpY1IZ2g/185468b7-9035-4776-a38a-7958c32a5fef/tmpBQqqOt.tmp'] failed with output:
  634. WARNING:toil.leader:a/K/job40aWLl ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn35.grid.pub.ro
  635. WARNING:toil.leader:a/K/job40aWLl WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' a/K/job40aWLl with ID a/K/job40aWLl to 4
  636. INFO:toil.leader:Issued job 'CactusSetupPhase' a/K/job40aWLl with job batch system ID: 160 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  637. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' a/K/job40aWLl
  638. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' a/K/job40aWLl
  639. WARNING:toil.leader:a/K/job40aWLl INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  640. WARNING:toil.leader:a/K/job40aWLl INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  641. WARNING:toil.leader:a/K/job40aWLl WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  642. WARNING:toil.leader:a/K/job40aWLl INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  643. WARNING:toil.leader:a/K/job40aWLl INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  644. WARNING:toil.leader:a/K/job40aWLl INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="13235" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpU5rIJ1/e5a04bde-360f-446b-8746-3e83d159e137/tmptthcPB.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpU5rIJ1/e5a04bde-360f-446b-8746-3e83d159e137/tmppQNHap.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpU5rIJ1/e5a04bde-360f-446b-8746-3e83d159e137/tmpp6OOc1.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpU5rIJ1/e5a04bde-360f-446b-8746-3e83d159e137/tmpNxj3PQ.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpU5rIJ1/e5a04bde-360f-446b-8746-3e83d159e137/tmpW2aBXw.tmp']
  645. WARNING:toil.leader:a/K/job40aWLl Set log level to INFO
  646. WARNING:toil.leader:a/K/job40aWLl Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  647. WARNING:toil.leader:a/K/job40aWLl <kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="13235" />
  648. WARNING:toil.leader:a/K/job40aWLl </st_kv_database_conf>
  649. WARNING:toil.leader:a/K/job40aWLl
  650. WARNING:toil.leader:a/K/job40aWLl Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpU5rIJ1/e5a04bde-360f-446b-8746-3e83d159e137/tmptthcPB.tmp
  651. WARNING:toil.leader:a/K/job40aWLl Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpU5rIJ1/e5a04bde-360f-446b-8746-3e83d159e137/tmppQNHap.tmp
  652. WARNING:toil.leader:a/K/job40aWLl Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpU5rIJ1/e5a04bde-360f-446b-8746-3e83d159e137/tmpp6OOc1.tmp
  653. WARNING:toil.leader:a/K/job40aWLl Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpU5rIJ1/e5a04bde-360f-446b-8746-3e83d159e137/tmpNxj3PQ.tmp
  654. WARNING:toil.leader:a/K/job40aWLl Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpU5rIJ1/e5a04bde-360f-446b-8746-3e83d159e137/tmpW2aBXw.tmp
  655. WARNING:toil.leader:a/K/job40aWLl Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.41 with error: network error
  656. WARNING:toil.leader:a/K/job40aWLl Uncaught exception
  657. WARNING:toil.leader:a/K/job40aWLl Traceback (most recent call last):
  658. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  659. WARNING:toil.leader:a/K/job40aWLl job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  660. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  661. WARNING:toil.leader:a/K/job40aWLl super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  662. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  663. WARNING:toil.leader:a/K/job40aWLl returnValues = self._run(jobGraph, fileStore)
  664. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  665. WARNING:toil.leader:a/K/job40aWLl return self.run(fileStore)
  666. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  667. WARNING:toil.leader:a/K/job40aWLl makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  668. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  669. WARNING:toil.leader:a/K/job40aWLl parameters=["cactus_setup"] + args + sequences)
  670. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  671. WARNING:toil.leader:a/K/job40aWLl raise RuntimeError("Command %s failed with output: %s" % (call, output))
  672. WARNING:toil.leader:a/K/job40aWLl RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="13235" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpU5rIJ1/e5a04bde-360f-446b-8746-3e83d159e137/tmptthcPB.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpU5rIJ1/e5a04bde-360f-446b-8746-3e83d159e137/tmppQNHap.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpU5rIJ1/e5a04bde-360f-446b-8746-3e83d159e137/tmpp6OOc1.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpU5rIJ1/e5a04bde-360f-446b-8746-3e83d159e137/tmpNxj3PQ.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpU5rIJ1/e5a04bde-360f-446b-8746-3e83d159e137/tmpW2aBXw.tmp'] failed with output:
  673. WARNING:toil.leader:a/K/job40aWLl ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn40.grid.pub.ro
  674. WARNING:toil.leader:a/K/job40aWLl WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' a/K/job40aWLl with ID a/K/job40aWLl to 3
  675. INFO:toil.leader:Issued job 'CactusSetupPhase' a/K/job40aWLl with job batch system ID: 161 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  676. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' w/R/job74nnk2
  677. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' w/R/job74nnk2
  678. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' w/R/job74nnk2 with ID w/R/job74nnk2 to 3
  679. INFO:toil.leader:Issued job 'KtServerService' w/R/job74nnk2 with job batch system ID: 162 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  680. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' a/K/job40aWLl
  681. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' a/K/job40aWLl
  682. WARNING:toil.leader:a/K/job40aWLl INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  683. WARNING:toil.leader:a/K/job40aWLl INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  684. WARNING:toil.leader:a/K/job40aWLl WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  685. WARNING:toil.leader:a/K/job40aWLl INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  686. WARNING:toil.leader:a/K/job40aWLl INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  687. WARNING:toil.leader:a/K/job40aWLl INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.34" port="23228" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp5NBBLs/1a1c6c74-9d15-4115-a08d-0cb14c1c0698/tmpfJ_EKH.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp5NBBLs/1a1c6c74-9d15-4115-a08d-0cb14c1c0698/tmpQTZ37k.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp5NBBLs/1a1c6c74-9d15-4115-a08d-0cb14c1c0698/tmpzvSvqF.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp5NBBLs/1a1c6c74-9d15-4115-a08d-0cb14c1c0698/tmp7u13MZ.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp5NBBLs/1a1c6c74-9d15-4115-a08d-0cb14c1c0698/tmpP6oUt9.tmp']
  688. WARNING:toil.leader:a/K/job40aWLl Set log level to INFO
  689. WARNING:toil.leader:a/K/job40aWLl Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  690. WARNING:toil.leader:a/K/job40aWLl <kyoto_tycoon database_dir="fakepath" host="172.16.13.34" port="23228" />
  691. WARNING:toil.leader:a/K/job40aWLl </st_kv_database_conf>
  692. WARNING:toil.leader:a/K/job40aWLl
  693. WARNING:toil.leader:a/K/job40aWLl Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp5NBBLs/1a1c6c74-9d15-4115-a08d-0cb14c1c0698/tmpfJ_EKH.tmp
  694. WARNING:toil.leader:a/K/job40aWLl Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp5NBBLs/1a1c6c74-9d15-4115-a08d-0cb14c1c0698/tmpQTZ37k.tmp
  695. WARNING:toil.leader:a/K/job40aWLl Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp5NBBLs/1a1c6c74-9d15-4115-a08d-0cb14c1c0698/tmpzvSvqF.tmp
  696. WARNING:toil.leader:a/K/job40aWLl Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp5NBBLs/1a1c6c74-9d15-4115-a08d-0cb14c1c0698/tmp7u13MZ.tmp
  697. WARNING:toil.leader:a/K/job40aWLl Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp5NBBLs/1a1c6c74-9d15-4115-a08d-0cb14c1c0698/tmpP6oUt9.tmp
  698. WARNING:toil.leader:a/K/job40aWLl Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.34 with error: network error
  699. WARNING:toil.leader:a/K/job40aWLl Uncaught exception
  700. WARNING:toil.leader:a/K/job40aWLl Traceback (most recent call last):
  701. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  702. WARNING:toil.leader:a/K/job40aWLl job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  703. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  704. WARNING:toil.leader:a/K/job40aWLl super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  705. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  706. WARNING:toil.leader:a/K/job40aWLl returnValues = self._run(jobGraph, fileStore)
  707. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  708. WARNING:toil.leader:a/K/job40aWLl return self.run(fileStore)
  709. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  710. WARNING:toil.leader:a/K/job40aWLl makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  711. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  712. WARNING:toil.leader:a/K/job40aWLl parameters=["cactus_setup"] + args + sequences)
  713. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  714. WARNING:toil.leader:a/K/job40aWLl raise RuntimeError("Command %s failed with output: %s" % (call, output))
  715. WARNING:toil.leader:a/K/job40aWLl RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.34" port="23228" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp5NBBLs/1a1c6c74-9d15-4115-a08d-0cb14c1c0698/tmpfJ_EKH.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp5NBBLs/1a1c6c74-9d15-4115-a08d-0cb14c1c0698/tmpQTZ37k.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp5NBBLs/1a1c6c74-9d15-4115-a08d-0cb14c1c0698/tmpzvSvqF.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp5NBBLs/1a1c6c74-9d15-4115-a08d-0cb14c1c0698/tmp7u13MZ.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp5NBBLs/1a1c6c74-9d15-4115-a08d-0cb14c1c0698/tmpP6oUt9.tmp'] failed with output:
  716. WARNING:toil.leader:a/K/job40aWLl ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn37.grid.pub.ro
  717. WARNING:toil.leader:a/K/job40aWLl WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' a/K/job40aWLl with ID a/K/job40aWLl to 2
  718. INFO:toil.leader:Issued job 'CactusSetupPhase' a/K/job40aWLl with job batch system ID: 163 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  719. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' w/R/job74nnk2
  720. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' w/R/job74nnk2
  721. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' w/R/job74nnk2 with ID w/R/job74nnk2 to 2
  722. INFO:toil.leader:Issued job 'KtServerService' w/R/job74nnk2 with job batch system ID: 164 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  723. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' a/K/job40aWLl
  724. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' a/K/job40aWLl
  725. WARNING:toil.leader:a/K/job40aWLl INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  726. WARNING:toil.leader:a/K/job40aWLl INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  727. WARNING:toil.leader:a/K/job40aWLl WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  728. WARNING:toil.leader:a/K/job40aWLl INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  729. WARNING:toil.leader:a/K/job40aWLl INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  730. WARNING:toil.leader:a/K/job40aWLl INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="10141" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpVvDBwK/0bda02b7-3a11-4d05-a91a-fa85b1e1d5f5/tmpJPw3oD.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpVvDBwK/0bda02b7-3a11-4d05-a91a-fa85b1e1d5f5/tmp3uflfn.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpVvDBwK/0bda02b7-3a11-4d05-a91a-fa85b1e1d5f5/tmpq9RW2l.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpVvDBwK/0bda02b7-3a11-4d05-a91a-fa85b1e1d5f5/tmp0MkhyN.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpVvDBwK/0bda02b7-3a11-4d05-a91a-fa85b1e1d5f5/tmpgd7Kpn.tmp']
  731. WARNING:toil.leader:a/K/job40aWLl Set log level to INFO
  732. WARNING:toil.leader:a/K/job40aWLl Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  733. WARNING:toil.leader:a/K/job40aWLl <kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="10141" />
  734. WARNING:toil.leader:a/K/job40aWLl </st_kv_database_conf>
  735. WARNING:toil.leader:a/K/job40aWLl
  736. WARNING:toil.leader:a/K/job40aWLl Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpVvDBwK/0bda02b7-3a11-4d05-a91a-fa85b1e1d5f5/tmpJPw3oD.tmp
  737. WARNING:toil.leader:a/K/job40aWLl Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpVvDBwK/0bda02b7-3a11-4d05-a91a-fa85b1e1d5f5/tmp3uflfn.tmp
  738. WARNING:toil.leader:a/K/job40aWLl Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpVvDBwK/0bda02b7-3a11-4d05-a91a-fa85b1e1d5f5/tmpq9RW2l.tmp
  739. WARNING:toil.leader:a/K/job40aWLl Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpVvDBwK/0bda02b7-3a11-4d05-a91a-fa85b1e1d5f5/tmp0MkhyN.tmp
  740. WARNING:toil.leader:a/K/job40aWLl Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpVvDBwK/0bda02b7-3a11-4d05-a91a-fa85b1e1d5f5/tmpgd7Kpn.tmp
  741. WARNING:toil.leader:a/K/job40aWLl Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.41 with error: network error
  742. WARNING:toil.leader:a/K/job40aWLl Uncaught exception
  743. WARNING:toil.leader:a/K/job40aWLl Traceback (most recent call last):
  744. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  745. WARNING:toil.leader:a/K/job40aWLl job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  746. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  747. WARNING:toil.leader:a/K/job40aWLl super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  748. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  749. WARNING:toil.leader:a/K/job40aWLl returnValues = self._run(jobGraph, fileStore)
  750. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  751. WARNING:toil.leader:a/K/job40aWLl return self.run(fileStore)
  752. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  753. WARNING:toil.leader:a/K/job40aWLl makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  754. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  755. WARNING:toil.leader:a/K/job40aWLl parameters=["cactus_setup"] + args + sequences)
  756. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  757. WARNING:toil.leader:a/K/job40aWLl raise RuntimeError("Command %s failed with output: %s" % (call, output))
  758. WARNING:toil.leader:a/K/job40aWLl RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="10141" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpVvDBwK/0bda02b7-3a11-4d05-a91a-fa85b1e1d5f5/tmpJPw3oD.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpVvDBwK/0bda02b7-3a11-4d05-a91a-fa85b1e1d5f5/tmp3uflfn.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpVvDBwK/0bda02b7-3a11-4d05-a91a-fa85b1e1d5f5/tmpq9RW2l.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpVvDBwK/0bda02b7-3a11-4d05-a91a-fa85b1e1d5f5/tmp0MkhyN.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpVvDBwK/0bda02b7-3a11-4d05-a91a-fa85b1e1d5f5/tmpgd7Kpn.tmp'] failed with output:
  759. WARNING:toil.leader:a/K/job40aWLl ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn38.grid.pub.ro
  760. WARNING:toil.leader:a/K/job40aWLl WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' a/K/job40aWLl with ID a/K/job40aWLl to 1
  761. INFO:toil.leader:Issued job 'CactusSetupPhase' a/K/job40aWLl with job batch system ID: 165 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  762. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' w/R/job74nnk2
  763. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' w/R/job74nnk2
  764. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' w/R/job74nnk2 with ID w/R/job74nnk2 to 1
  765. INFO:toil.leader:Issued job 'KtServerService' w/R/job74nnk2 with job batch system ID: 166 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  766. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' a/K/job40aWLl
  767. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' a/K/job40aWLl
  768. WARNING:toil.leader:a/K/job40aWLl INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  769. WARNING:toil.leader:a/K/job40aWLl INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  770. WARNING:toil.leader:a/K/job40aWLl WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  771. WARNING:toil.leader:a/K/job40aWLl INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  772. WARNING:toil.leader:a/K/job40aWLl INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  773. WARNING:toil.leader:a/K/job40aWLl INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="14197" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpXwxtNw/3cc55379-2082-4f7e-9e29-09fcda4317ea/tmpmaSeGb.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpXwxtNw/3cc55379-2082-4f7e-9e29-09fcda4317ea/tmpRPzpfj.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpXwxtNw/3cc55379-2082-4f7e-9e29-09fcda4317ea/tmpWEhHyS.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpXwxtNw/3cc55379-2082-4f7e-9e29-09fcda4317ea/tmprCogAD.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpXwxtNw/3cc55379-2082-4f7e-9e29-09fcda4317ea/tmph0Q5bM.tmp']
  774. WARNING:toil.leader:a/K/job40aWLl Set log level to INFO
  775. WARNING:toil.leader:a/K/job40aWLl Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  776. WARNING:toil.leader:a/K/job40aWLl <kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="14197" />
  777. WARNING:toil.leader:a/K/job40aWLl </st_kv_database_conf>
  778. WARNING:toil.leader:a/K/job40aWLl
  779. WARNING:toil.leader:a/K/job40aWLl Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpXwxtNw/3cc55379-2082-4f7e-9e29-09fcda4317ea/tmpmaSeGb.tmp
  780. WARNING:toil.leader:a/K/job40aWLl Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpXwxtNw/3cc55379-2082-4f7e-9e29-09fcda4317ea/tmpRPzpfj.tmp
  781. WARNING:toil.leader:a/K/job40aWLl Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpXwxtNw/3cc55379-2082-4f7e-9e29-09fcda4317ea/tmpWEhHyS.tmp
  782. WARNING:toil.leader:a/K/job40aWLl Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpXwxtNw/3cc55379-2082-4f7e-9e29-09fcda4317ea/tmprCogAD.tmp
  783. WARNING:toil.leader:a/K/job40aWLl Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpXwxtNw/3cc55379-2082-4f7e-9e29-09fcda4317ea/tmph0Q5bM.tmp
  784. WARNING:toil.leader:a/K/job40aWLl Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.29 with error: network error
  785. WARNING:toil.leader:a/K/job40aWLl Uncaught exception
  786. WARNING:toil.leader:a/K/job40aWLl Traceback (most recent call last):
  787. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  788. WARNING:toil.leader:a/K/job40aWLl job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  789. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  790. WARNING:toil.leader:a/K/job40aWLl super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  791. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  792. WARNING:toil.leader:a/K/job40aWLl returnValues = self._run(jobGraph, fileStore)
  793. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  794. WARNING:toil.leader:a/K/job40aWLl return self.run(fileStore)
  795. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  796. WARNING:toil.leader:a/K/job40aWLl makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  797. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  798. WARNING:toil.leader:a/K/job40aWLl parameters=["cactus_setup"] + args + sequences)
  799. WARNING:toil.leader:a/K/job40aWLl File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  800. WARNING:toil.leader:a/K/job40aWLl raise RuntimeError("Command %s failed with output: %s" % (call, output))
  801. WARNING:toil.leader:a/K/job40aWLl RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="14197" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpXwxtNw/3cc55379-2082-4f7e-9e29-09fcda4317ea/tmpmaSeGb.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpXwxtNw/3cc55379-2082-4f7e-9e29-09fcda4317ea/tmpRPzpfj.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpXwxtNw/3cc55379-2082-4f7e-9e29-09fcda4317ea/tmpWEhHyS.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpXwxtNw/3cc55379-2082-4f7e-9e29-09fcda4317ea/tmprCogAD.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpXwxtNw/3cc55379-2082-4f7e-9e29-09fcda4317ea/tmph0Q5bM.tmp'] failed with output:
  802. WARNING:toil.leader:a/K/job40aWLl ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn35.grid.pub.ro
  803. WARNING:toil.leader:a/K/job40aWLl WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' a/K/job40aWLl with ID a/K/job40aWLl to 0
  804. WARNING:toil.leader:Job 'CactusSetupPhase' a/K/job40aWLl with ID a/K/job40aWLl is completely failed
  805. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' w/R/job74nnk2
  806. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' w/R/job74nnk2
  807. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' w/R/job74nnk2 with ID w/R/job74nnk2 to 0
  808. WARNING:toil.leader:Job 'KtServerService' w/R/job74nnk2 with ID w/R/job74nnk2 is completely failed
  809. WARNING:toil.leader:Job: a/n/jobKwRkH2 is being restarted as a checkpoint after the total failure of jobs in its subtree.
  810. INFO:toil.leader:Issued job 'StartPrimaryDB' a/n/jobKwRkH2 with job batch system ID: 167 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  811. INFO:toil.leader:Job ended successfully: 'StartPrimaryDB' a/n/jobKwRkH2
  812. INFO:toil.leader:Issued job 'KtServerService' S/N/jobPEeSof with job batch system ID: 168 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  813. INFO:toil.leader:Issued job 'CactusSetupPhase' f/J/jobbzmqrZ with job batch system ID: 169 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  814. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' S/N/jobPEeSof
  815. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' S/N/jobPEeSof
  816. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' S/N/jobPEeSof with ID S/N/jobPEeSof to 5
  817. INFO:toil.leader:Issued job 'KtServerService' S/N/jobPEeSof with job batch system ID: 170 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  818. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' f/J/jobbzmqrZ
  819. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' f/J/jobbzmqrZ
  820. WARNING:toil.leader:f/J/jobbzmqrZ INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  821. WARNING:toil.leader:f/J/jobbzmqrZ INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  822. WARNING:toil.leader:f/J/jobbzmqrZ WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  823. WARNING:toil.leader:f/J/jobbzmqrZ INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  824. WARNING:toil.leader:f/J/jobbzmqrZ INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  825. WARNING:toil.leader:f/J/jobbzmqrZ INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.38" port="18646" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpjj3LSX/a16c4fc0-f023-4c97-8682-fc82e9bffa4e/tmpD3d5Qx.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpjj3LSX/a16c4fc0-f023-4c97-8682-fc82e9bffa4e/tmpmhinES.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpjj3LSX/a16c4fc0-f023-4c97-8682-fc82e9bffa4e/tmpBSYETR.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpjj3LSX/a16c4fc0-f023-4c97-8682-fc82e9bffa4e/tmpwjcjVK.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpjj3LSX/a16c4fc0-f023-4c97-8682-fc82e9bffa4e/tmpcfpcU3.tmp']
  826. WARNING:toil.leader:f/J/jobbzmqrZ Set log level to INFO
  827. WARNING:toil.leader:f/J/jobbzmqrZ Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  828. WARNING:toil.leader:f/J/jobbzmqrZ <kyoto_tycoon database_dir="fakepath" host="172.16.13.38" port="18646" />
  829. WARNING:toil.leader:f/J/jobbzmqrZ </st_kv_database_conf>
  830. WARNING:toil.leader:f/J/jobbzmqrZ
  831. WARNING:toil.leader:f/J/jobbzmqrZ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpjj3LSX/a16c4fc0-f023-4c97-8682-fc82e9bffa4e/tmpD3d5Qx.tmp
  832. WARNING:toil.leader:f/J/jobbzmqrZ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpjj3LSX/a16c4fc0-f023-4c97-8682-fc82e9bffa4e/tmpmhinES.tmp
  833. WARNING:toil.leader:f/J/jobbzmqrZ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpjj3LSX/a16c4fc0-f023-4c97-8682-fc82e9bffa4e/tmpBSYETR.tmp
  834. WARNING:toil.leader:f/J/jobbzmqrZ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpjj3LSX/a16c4fc0-f023-4c97-8682-fc82e9bffa4e/tmpwjcjVK.tmp
  835. WARNING:toil.leader:f/J/jobbzmqrZ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpjj3LSX/a16c4fc0-f023-4c97-8682-fc82e9bffa4e/tmpcfpcU3.tmp
  836. WARNING:toil.leader:f/J/jobbzmqrZ Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.38 with error: network error
  837. WARNING:toil.leader:f/J/jobbzmqrZ Uncaught exception
  838. WARNING:toil.leader:f/J/jobbzmqrZ Traceback (most recent call last):
  839. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  840. WARNING:toil.leader:f/J/jobbzmqrZ job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  841. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  842. WARNING:toil.leader:f/J/jobbzmqrZ super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  843. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  844. WARNING:toil.leader:f/J/jobbzmqrZ returnValues = self._run(jobGraph, fileStore)
  845. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  846. WARNING:toil.leader:f/J/jobbzmqrZ return self.run(fileStore)
  847. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  848. WARNING:toil.leader:f/J/jobbzmqrZ makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  849. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  850. WARNING:toil.leader:f/J/jobbzmqrZ parameters=["cactus_setup"] + args + sequences)
  851. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  852. WARNING:toil.leader:f/J/jobbzmqrZ raise RuntimeError("Command %s failed with output: %s" % (call, output))
  853. WARNING:toil.leader:f/J/jobbzmqrZ RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.38" port="18646" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpjj3LSX/a16c4fc0-f023-4c97-8682-fc82e9bffa4e/tmpD3d5Qx.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpjj3LSX/a16c4fc0-f023-4c97-8682-fc82e9bffa4e/tmpmhinES.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpjj3LSX/a16c4fc0-f023-4c97-8682-fc82e9bffa4e/tmpBSYETR.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpjj3LSX/a16c4fc0-f023-4c97-8682-fc82e9bffa4e/tmpwjcjVK.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpjj3LSX/a16c4fc0-f023-4c97-8682-fc82e9bffa4e/tmpcfpcU3.tmp'] failed with output:
  854. WARNING:toil.leader:f/J/jobbzmqrZ ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn35.grid.pub.ro
  855. WARNING:toil.leader:f/J/jobbzmqrZ WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' f/J/jobbzmqrZ with ID f/J/jobbzmqrZ to 5
  856. INFO:toil.leader:Issued job 'CactusSetupPhase' f/J/jobbzmqrZ with job batch system ID: 171 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  857. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' S/N/jobPEeSof
  858. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' S/N/jobPEeSof
  859. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' S/N/jobPEeSof with ID S/N/jobPEeSof to 4
  860. INFO:toil.leader:Issued job 'KtServerService' S/N/jobPEeSof with job batch system ID: 172 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  861. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' S/N/jobPEeSof
  862. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' S/N/jobPEeSof
  863. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' S/N/jobPEeSof with ID S/N/jobPEeSof to 3
  864. INFO:toil.leader:Issued job 'KtServerService' S/N/jobPEeSof with job batch system ID: 173 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  865. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' f/J/jobbzmqrZ
  866. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' f/J/jobbzmqrZ
  867. WARNING:toil.leader:f/J/jobbzmqrZ INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  868. WARNING:toil.leader:f/J/jobbzmqrZ INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  869. WARNING:toil.leader:f/J/jobbzmqrZ WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  870. WARNING:toil.leader:f/J/jobbzmqrZ WARNING:toil.fileStore:Detected that job (a/n/jobKwRkH2/g/tmpZ5X_AY-_serialiseJob-stream) prematurely terminated. Fixing the state of the job on disk.
  871. WARNING:toil.leader:f/J/jobbzmqrZ INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  872. WARNING:toil.leader:f/J/jobbzmqrZ INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  873. WARNING:toil.leader:f/J/jobbzmqrZ INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.37" port="5402" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpNneY0O/3abf20b1-bd4f-473e-a226-cac1f9bdaa48/tmpRyEtbR.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpNneY0O/3abf20b1-bd4f-473e-a226-cac1f9bdaa48/tmp08nwsY.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpNneY0O/3abf20b1-bd4f-473e-a226-cac1f9bdaa48/tmpCxTmG0.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpNneY0O/3abf20b1-bd4f-473e-a226-cac1f9bdaa48/tmpR7TMhL.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpNneY0O/3abf20b1-bd4f-473e-a226-cac1f9bdaa48/tmp6SDolC.tmp']
  874. WARNING:toil.leader:f/J/jobbzmqrZ Set log level to INFO
  875. WARNING:toil.leader:f/J/jobbzmqrZ Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  876. WARNING:toil.leader:f/J/jobbzmqrZ <kyoto_tycoon database_dir="fakepath" host="172.16.13.37" port="5402" />
  877. WARNING:toil.leader:f/J/jobbzmqrZ </st_kv_database_conf>
  878. WARNING:toil.leader:f/J/jobbzmqrZ
  879. WARNING:toil.leader:f/J/jobbzmqrZ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpNneY0O/3abf20b1-bd4f-473e-a226-cac1f9bdaa48/tmpRyEtbR.tmp
  880. WARNING:toil.leader:f/J/jobbzmqrZ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpNneY0O/3abf20b1-bd4f-473e-a226-cac1f9bdaa48/tmp08nwsY.tmp
  881. WARNING:toil.leader:f/J/jobbzmqrZ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpNneY0O/3abf20b1-bd4f-473e-a226-cac1f9bdaa48/tmpCxTmG0.tmp
  882. WARNING:toil.leader:f/J/jobbzmqrZ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpNneY0O/3abf20b1-bd4f-473e-a226-cac1f9bdaa48/tmpR7TMhL.tmp
  883. WARNING:toil.leader:f/J/jobbzmqrZ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpNneY0O/3abf20b1-bd4f-473e-a226-cac1f9bdaa48/tmp6SDolC.tmp
  884. WARNING:toil.leader:f/J/jobbzmqrZ Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.37 with error: network error
  885. WARNING:toil.leader:f/J/jobbzmqrZ Uncaught exception
  886. WARNING:toil.leader:f/J/jobbzmqrZ Traceback (most recent call last):
  887. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  888. WARNING:toil.leader:f/J/jobbzmqrZ job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  889. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  890. WARNING:toil.leader:f/J/jobbzmqrZ super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  891. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  892. WARNING:toil.leader:f/J/jobbzmqrZ returnValues = self._run(jobGraph, fileStore)
  893. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  894. WARNING:toil.leader:f/J/jobbzmqrZ return self.run(fileStore)
  895. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  896. WARNING:toil.leader:f/J/jobbzmqrZ makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  897. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  898. WARNING:toil.leader:f/J/jobbzmqrZ parameters=["cactus_setup"] + args + sequences)
  899. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  900. WARNING:toil.leader:f/J/jobbzmqrZ raise RuntimeError("Command %s failed with output: %s" % (call, output))
  901. WARNING:toil.leader:f/J/jobbzmqrZ RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.37" port="5402" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpNneY0O/3abf20b1-bd4f-473e-a226-cac1f9bdaa48/tmpRyEtbR.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpNneY0O/3abf20b1-bd4f-473e-a226-cac1f9bdaa48/tmp08nwsY.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpNneY0O/3abf20b1-bd4f-473e-a226-cac1f9bdaa48/tmpCxTmG0.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpNneY0O/3abf20b1-bd4f-473e-a226-cac1f9bdaa48/tmpR7TMhL.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpNneY0O/3abf20b1-bd4f-473e-a226-cac1f9bdaa48/tmp6SDolC.tmp'] failed with output:
  902. WARNING:toil.leader:f/J/jobbzmqrZ ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn29.grid.pub.ro
  903. WARNING:toil.leader:f/J/jobbzmqrZ WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' f/J/jobbzmqrZ with ID f/J/jobbzmqrZ to 4
  904. INFO:toil.leader:Issued job 'CactusSetupPhase' f/J/jobbzmqrZ with job batch system ID: 174 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  905. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' S/N/jobPEeSof
  906. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' S/N/jobPEeSof
  907. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' S/N/jobPEeSof with ID S/N/jobPEeSof to 2
  908. INFO:toil.leader:Issued job 'KtServerService' S/N/jobPEeSof with job batch system ID: 175 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  909. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' f/J/jobbzmqrZ
  910. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' f/J/jobbzmqrZ
  911. WARNING:toil.leader:f/J/jobbzmqrZ INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  912. WARNING:toil.leader:f/J/jobbzmqrZ INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  913. WARNING:toil.leader:f/J/jobbzmqrZ WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  914. WARNING:toil.leader:f/J/jobbzmqrZ WARNING:toil.fileStore:Detected that job (a/n/jobKwRkH2/g/tmpqp43fG-_serialiseJob-stream) prematurely terminated. Fixing the state of the job on disk.
  915. WARNING:toil.leader:f/J/jobbzmqrZ INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  916. WARNING:toil.leader:f/J/jobbzmqrZ INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  917. WARNING:toil.leader:f/J/jobbzmqrZ INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="15268" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmphElk2g/ab2e5dd3-5c8a-4620-902b-da2bd102ae55/tmpZNnF5j.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmphElk2g/ab2e5dd3-5c8a-4620-902b-da2bd102ae55/tmp9Tpwhw.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmphElk2g/ab2e5dd3-5c8a-4620-902b-da2bd102ae55/tmpVBeSuN.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmphElk2g/ab2e5dd3-5c8a-4620-902b-da2bd102ae55/tmp6IlzR8.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmphElk2g/ab2e5dd3-5c8a-4620-902b-da2bd102ae55/tmp2J3gvc.tmp']
  918. WARNING:toil.leader:f/J/jobbzmqrZ Set log level to INFO
  919. WARNING:toil.leader:f/J/jobbzmqrZ Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  920. WARNING:toil.leader:f/J/jobbzmqrZ <kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="15268" />
  921. WARNING:toil.leader:f/J/jobbzmqrZ </st_kv_database_conf>
  922. WARNING:toil.leader:f/J/jobbzmqrZ
  923. WARNING:toil.leader:f/J/jobbzmqrZ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmphElk2g/ab2e5dd3-5c8a-4620-902b-da2bd102ae55/tmpZNnF5j.tmp
  924. WARNING:toil.leader:f/J/jobbzmqrZ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmphElk2g/ab2e5dd3-5c8a-4620-902b-da2bd102ae55/tmp9Tpwhw.tmp
  925. WARNING:toil.leader:f/J/jobbzmqrZ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmphElk2g/ab2e5dd3-5c8a-4620-902b-da2bd102ae55/tmpVBeSuN.tmp
  926. WARNING:toil.leader:f/J/jobbzmqrZ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmphElk2g/ab2e5dd3-5c8a-4620-902b-da2bd102ae55/tmp6IlzR8.tmp
  927. WARNING:toil.leader:f/J/jobbzmqrZ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmphElk2g/ab2e5dd3-5c8a-4620-902b-da2bd102ae55/tmp2J3gvc.tmp
  928. WARNING:toil.leader:f/J/jobbzmqrZ Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.41 with error: network error
  929. WARNING:toil.leader:f/J/jobbzmqrZ Uncaught exception
  930. WARNING:toil.leader:f/J/jobbzmqrZ Traceback (most recent call last):
  931. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  932. WARNING:toil.leader:f/J/jobbzmqrZ job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  933. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  934. WARNING:toil.leader:f/J/jobbzmqrZ super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  935. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  936. WARNING:toil.leader:f/J/jobbzmqrZ returnValues = self._run(jobGraph, fileStore)
  937. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  938. WARNING:toil.leader:f/J/jobbzmqrZ return self.run(fileStore)
  939. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  940. WARNING:toil.leader:f/J/jobbzmqrZ makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  941. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  942. WARNING:toil.leader:f/J/jobbzmqrZ parameters=["cactus_setup"] + args + sequences)
  943. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  944. WARNING:toil.leader:f/J/jobbzmqrZ raise RuntimeError("Command %s failed with output: %s" % (call, output))
  945. WARNING:toil.leader:f/J/jobbzmqrZ RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="15268" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmphElk2g/ab2e5dd3-5c8a-4620-902b-da2bd102ae55/tmpZNnF5j.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmphElk2g/ab2e5dd3-5c8a-4620-902b-da2bd102ae55/tmp9Tpwhw.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmphElk2g/ab2e5dd3-5c8a-4620-902b-da2bd102ae55/tmpVBeSuN.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmphElk2g/ab2e5dd3-5c8a-4620-902b-da2bd102ae55/tmp6IlzR8.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmphElk2g/ab2e5dd3-5c8a-4620-902b-da2bd102ae55/tmp2J3gvc.tmp'] failed with output:
  946. WARNING:toil.leader:f/J/jobbzmqrZ ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn37.grid.pub.ro
  947. WARNING:toil.leader:f/J/jobbzmqrZ WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' f/J/jobbzmqrZ with ID f/J/jobbzmqrZ to 3
  948. INFO:toil.leader:Issued job 'CactusSetupPhase' f/J/jobbzmqrZ with job batch system ID: 176 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  949. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' f/J/jobbzmqrZ
  950. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' f/J/jobbzmqrZ
  951. WARNING:toil.leader:f/J/jobbzmqrZ INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  952. WARNING:toil.leader:f/J/jobbzmqrZ INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  953. WARNING:toil.leader:f/J/jobbzmqrZ WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  954. WARNING:toil.leader:f/J/jobbzmqrZ INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  955. WARNING:toil.leader:f/J/jobbzmqrZ INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  956. WARNING:toil.leader:f/J/jobbzmqrZ INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="15268" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp9EqBjq/a3f46a57-b9fb-4df3-92d6-c2b33e68a437/tmpJrYK9S.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp9EqBjq/a3f46a57-b9fb-4df3-92d6-c2b33e68a437/tmp9TqPqB.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp9EqBjq/a3f46a57-b9fb-4df3-92d6-c2b33e68a437/tmpmy9aky.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp9EqBjq/a3f46a57-b9fb-4df3-92d6-c2b33e68a437/tmp5c5L7s.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp9EqBjq/a3f46a57-b9fb-4df3-92d6-c2b33e68a437/tmpxxAZ_2.tmp']
  957. WARNING:toil.leader:f/J/jobbzmqrZ Set log level to INFO
  958. WARNING:toil.leader:f/J/jobbzmqrZ Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  959. WARNING:toil.leader:f/J/jobbzmqrZ <kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="15268" />
  960. WARNING:toil.leader:f/J/jobbzmqrZ </st_kv_database_conf>
  961. WARNING:toil.leader:f/J/jobbzmqrZ
  962. WARNING:toil.leader:f/J/jobbzmqrZ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp9EqBjq/a3f46a57-b9fb-4df3-92d6-c2b33e68a437/tmpJrYK9S.tmp
  963. WARNING:toil.leader:f/J/jobbzmqrZ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp9EqBjq/a3f46a57-b9fb-4df3-92d6-c2b33e68a437/tmp9TqPqB.tmp
  964. WARNING:toil.leader:f/J/jobbzmqrZ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp9EqBjq/a3f46a57-b9fb-4df3-92d6-c2b33e68a437/tmpmy9aky.tmp
  965. WARNING:toil.leader:f/J/jobbzmqrZ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp9EqBjq/a3f46a57-b9fb-4df3-92d6-c2b33e68a437/tmp5c5L7s.tmp
  966. WARNING:toil.leader:f/J/jobbzmqrZ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp9EqBjq/a3f46a57-b9fb-4df3-92d6-c2b33e68a437/tmpxxAZ_2.tmp
  967. WARNING:toil.leader:f/J/jobbzmqrZ Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.41 with error: network error
  968. WARNING:toil.leader:f/J/jobbzmqrZ Uncaught exception
  969. WARNING:toil.leader:f/J/jobbzmqrZ Traceback (most recent call last):
  970. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  971. WARNING:toil.leader:f/J/jobbzmqrZ job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  972. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  973. WARNING:toil.leader:f/J/jobbzmqrZ super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  974. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  975. WARNING:toil.leader:f/J/jobbzmqrZ returnValues = self._run(jobGraph, fileStore)
  976. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  977. WARNING:toil.leader:f/J/jobbzmqrZ return self.run(fileStore)
  978. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  979. WARNING:toil.leader:f/J/jobbzmqrZ makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  980. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  981. WARNING:toil.leader:f/J/jobbzmqrZ parameters=["cactus_setup"] + args + sequences)
  982. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  983. WARNING:toil.leader:f/J/jobbzmqrZ raise RuntimeError("Command %s failed with output: %s" % (call, output))
  984. WARNING:toil.leader:f/J/jobbzmqrZ RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="15268" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp9EqBjq/a3f46a57-b9fb-4df3-92d6-c2b33e68a437/tmpJrYK9S.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp9EqBjq/a3f46a57-b9fb-4df3-92d6-c2b33e68a437/tmp9TqPqB.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp9EqBjq/a3f46a57-b9fb-4df3-92d6-c2b33e68a437/tmpmy9aky.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp9EqBjq/a3f46a57-b9fb-4df3-92d6-c2b33e68a437/tmp5c5L7s.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmp9EqBjq/a3f46a57-b9fb-4df3-92d6-c2b33e68a437/tmpxxAZ_2.tmp'] failed with output:
  985. WARNING:toil.leader:f/J/jobbzmqrZ ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn37.grid.pub.ro
  986. WARNING:toil.leader:f/J/jobbzmqrZ WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' f/J/jobbzmqrZ with ID f/J/jobbzmqrZ to 2
  987. INFO:toil.leader:Issued job 'CactusSetupPhase' f/J/jobbzmqrZ with job batch system ID: 177 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  988. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' S/N/jobPEeSof
  989. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' S/N/jobPEeSof
  990. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' S/N/jobPEeSof with ID S/N/jobPEeSof to 1
  991. INFO:toil.leader:Issued job 'KtServerService' S/N/jobPEeSof with job batch system ID: 178 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  992. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' f/J/jobbzmqrZ
  993. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' f/J/jobbzmqrZ
  994. WARNING:toil.leader:f/J/jobbzmqrZ INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  995. WARNING:toil.leader:f/J/jobbzmqrZ INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  996. WARNING:toil.leader:f/J/jobbzmqrZ WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  997. WARNING:toil.leader:f/J/jobbzmqrZ INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  998. WARNING:toil.leader:f/J/jobbzmqrZ INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  999. WARNING:toil.leader:f/J/jobbzmqrZ INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.40" port="30137" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpZtzw51/0db6b510-5af9-4f9a-8c6e-4071020b4f67/tmp_oio49.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpZtzw51/0db6b510-5af9-4f9a-8c6e-4071020b4f67/tmp7YYMrl.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpZtzw51/0db6b510-5af9-4f9a-8c6e-4071020b4f67/tmpH1WXWB.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpZtzw51/0db6b510-5af9-4f9a-8c6e-4071020b4f67/tmpcAkJak.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpZtzw51/0db6b510-5af9-4f9a-8c6e-4071020b4f67/tmp_3A4lB.tmp']
  1000. WARNING:toil.leader:f/J/jobbzmqrZ Set log level to INFO
  1001. WARNING:toil.leader:f/J/jobbzmqrZ Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  1002. WARNING:toil.leader:f/J/jobbzmqrZ <kyoto_tycoon database_dir="fakepath" host="172.16.13.40" port="30137" />
  1003. WARNING:toil.leader:f/J/jobbzmqrZ </st_kv_database_conf>
  1004. WARNING:toil.leader:f/J/jobbzmqrZ
  1005. WARNING:toil.leader:f/J/jobbzmqrZ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpZtzw51/0db6b510-5af9-4f9a-8c6e-4071020b4f67/tmp_oio49.tmp
  1006. WARNING:toil.leader:f/J/jobbzmqrZ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpZtzw51/0db6b510-5af9-4f9a-8c6e-4071020b4f67/tmp7YYMrl.tmp
  1007. WARNING:toil.leader:f/J/jobbzmqrZ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpZtzw51/0db6b510-5af9-4f9a-8c6e-4071020b4f67/tmpH1WXWB.tmp
  1008. WARNING:toil.leader:f/J/jobbzmqrZ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpZtzw51/0db6b510-5af9-4f9a-8c6e-4071020b4f67/tmpcAkJak.tmp
  1009. WARNING:toil.leader:f/J/jobbzmqrZ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpZtzw51/0db6b510-5af9-4f9a-8c6e-4071020b4f67/tmp_3A4lB.tmp
  1010. WARNING:toil.leader:f/J/jobbzmqrZ Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.40 with error: network error
  1011. WARNING:toil.leader:f/J/jobbzmqrZ Uncaught exception
  1012. WARNING:toil.leader:f/J/jobbzmqrZ Traceback (most recent call last):
  1013. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1014. WARNING:toil.leader:f/J/jobbzmqrZ job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1015. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1016. WARNING:toil.leader:f/J/jobbzmqrZ super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1017. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1018. WARNING:toil.leader:f/J/jobbzmqrZ returnValues = self._run(jobGraph, fileStore)
  1019. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1020. WARNING:toil.leader:f/J/jobbzmqrZ return self.run(fileStore)
  1021. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  1022. WARNING:toil.leader:f/J/jobbzmqrZ makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  1023. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  1024. WARNING:toil.leader:f/J/jobbzmqrZ parameters=["cactus_setup"] + args + sequences)
  1025. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1026. WARNING:toil.leader:f/J/jobbzmqrZ raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1027. WARNING:toil.leader:f/J/jobbzmqrZ RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.40" port="30137" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpZtzw51/0db6b510-5af9-4f9a-8c6e-4071020b4f67/tmp_oio49.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpZtzw51/0db6b510-5af9-4f9a-8c6e-4071020b4f67/tmp7YYMrl.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpZtzw51/0db6b510-5af9-4f9a-8c6e-4071020b4f67/tmpH1WXWB.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpZtzw51/0db6b510-5af9-4f9a-8c6e-4071020b4f67/tmpcAkJak.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpZtzw51/0db6b510-5af9-4f9a-8c6e-4071020b4f67/tmp_3A4lB.tmp'] failed with output:
  1028. WARNING:toil.leader:f/J/jobbzmqrZ ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn37.grid.pub.ro
  1029. WARNING:toil.leader:f/J/jobbzmqrZ WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' f/J/jobbzmqrZ with ID f/J/jobbzmqrZ to 1
  1030. INFO:toil.leader:Issued job 'CactusSetupPhase' f/J/jobbzmqrZ with job batch system ID: 179 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1031. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' S/N/jobPEeSof
  1032. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' S/N/jobPEeSof
  1033. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' S/N/jobPEeSof with ID S/N/jobPEeSof to 0
  1034. WARNING:toil.leader:Job 'KtServerService' S/N/jobPEeSof with ID S/N/jobPEeSof is completely failed
  1035. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' f/J/jobbzmqrZ
  1036. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' f/J/jobbzmqrZ
  1037. WARNING:toil.leader:f/J/jobbzmqrZ INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1038. WARNING:toil.leader:f/J/jobbzmqrZ INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1039. WARNING:toil.leader:f/J/jobbzmqrZ WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1040. WARNING:toil.leader:f/J/jobbzmqrZ INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  1041. WARNING:toil.leader:f/J/jobbzmqrZ INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  1042. WARNING:toil.leader:f/J/jobbzmqrZ INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.30" port="2728" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmp4t2sYd/4a968585-52b1-46ca-9325-7b6eb4f87011/tmp5LPR8E.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmp4t2sYd/4a968585-52b1-46ca-9325-7b6eb4f87011/tmpjOVZ99.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmp4t2sYd/4a968585-52b1-46ca-9325-7b6eb4f87011/tmparKP0g.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmp4t2sYd/4a968585-52b1-46ca-9325-7b6eb4f87011/tmpawIKri.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmp4t2sYd/4a968585-52b1-46ca-9325-7b6eb4f87011/tmpE1XVWc.tmp']
  1043. WARNING:toil.leader:f/J/jobbzmqrZ Set log level to INFO
  1044. WARNING:toil.leader:f/J/jobbzmqrZ Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  1045. WARNING:toil.leader:f/J/jobbzmqrZ <kyoto_tycoon database_dir="fakepath" host="172.16.13.30" port="2728" />
  1046. WARNING:toil.leader:f/J/jobbzmqrZ </st_kv_database_conf>
  1047. WARNING:toil.leader:f/J/jobbzmqrZ
  1048. WARNING:toil.leader:f/J/jobbzmqrZ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmp4t2sYd/4a968585-52b1-46ca-9325-7b6eb4f87011/tmp5LPR8E.tmp
  1049. WARNING:toil.leader:f/J/jobbzmqrZ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmp4t2sYd/4a968585-52b1-46ca-9325-7b6eb4f87011/tmpjOVZ99.tmp
  1050. WARNING:toil.leader:f/J/jobbzmqrZ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmp4t2sYd/4a968585-52b1-46ca-9325-7b6eb4f87011/tmparKP0g.tmp
  1051. WARNING:toil.leader:f/J/jobbzmqrZ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmp4t2sYd/4a968585-52b1-46ca-9325-7b6eb4f87011/tmpawIKri.tmp
  1052. WARNING:toil.leader:f/J/jobbzmqrZ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmp4t2sYd/4a968585-52b1-46ca-9325-7b6eb4f87011/tmpE1XVWc.tmp
  1053. WARNING:toil.leader:f/J/jobbzmqrZ Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.30 with error: network error
  1054. WARNING:toil.leader:f/J/jobbzmqrZ Uncaught exception
  1055. WARNING:toil.leader:f/J/jobbzmqrZ Traceback (most recent call last):
  1056. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1057. WARNING:toil.leader:f/J/jobbzmqrZ job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1058. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1059. WARNING:toil.leader:f/J/jobbzmqrZ super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1060. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1061. WARNING:toil.leader:f/J/jobbzmqrZ returnValues = self._run(jobGraph, fileStore)
  1062. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1063. WARNING:toil.leader:f/J/jobbzmqrZ return self.run(fileStore)
  1064. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  1065. WARNING:toil.leader:f/J/jobbzmqrZ makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  1066. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  1067. WARNING:toil.leader:f/J/jobbzmqrZ parameters=["cactus_setup"] + args + sequences)
  1068. WARNING:toil.leader:f/J/jobbzmqrZ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1069. WARNING:toil.leader:f/J/jobbzmqrZ raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1070. WARNING:toil.leader:f/J/jobbzmqrZ RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.30" port="2728" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmp4t2sYd/4a968585-52b1-46ca-9325-7b6eb4f87011/tmp5LPR8E.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmp4t2sYd/4a968585-52b1-46ca-9325-7b6eb4f87011/tmpjOVZ99.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmp4t2sYd/4a968585-52b1-46ca-9325-7b6eb4f87011/tmparKP0g.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmp4t2sYd/4a968585-52b1-46ca-9325-7b6eb4f87011/tmpawIKri.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmp4t2sYd/4a968585-52b1-46ca-9325-7b6eb4f87011/tmpE1XVWc.tmp'] failed with output:
  1071. WARNING:toil.leader:f/J/jobbzmqrZ ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn35.grid.pub.ro
  1072. WARNING:toil.leader:f/J/jobbzmqrZ WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' f/J/jobbzmqrZ with ID f/J/jobbzmqrZ to 0
  1073. WARNING:toil.leader:Job 'CactusSetupPhase' f/J/jobbzmqrZ with ID f/J/jobbzmqrZ is completely failed
  1074. WARNING:toil.leader:Job: a/n/jobKwRkH2 is being restarted as a checkpoint after the total failure of jobs in its subtree.
  1075. INFO:toil.leader:Issued job 'StartPrimaryDB' a/n/jobKwRkH2 with job batch system ID: 180 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1076. INFO:toil.leader:Job ended successfully: 'StartPrimaryDB' a/n/jobKwRkH2
  1077. INFO:toil.leader:Issued job 'KtServerService' x/v/jobBOV8ZO with job batch system ID: 181 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1078. INFO:toil.leader:Issued job 'CactusSetupPhase' k/K/jobgviNIp with job batch system ID: 182 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1079. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' k/K/jobgviNIp
  1080. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' k/K/jobgviNIp
  1081. WARNING:toil.leader:k/K/jobgviNIp INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1082. WARNING:toil.leader:k/K/jobgviNIp INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1083. WARNING:toil.leader:k/K/jobgviNIp WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1084. WARNING:toil.leader:k/K/jobgviNIp INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  1085. WARNING:toil.leader:k/K/jobgviNIp INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  1086. WARNING:toil.leader:k/K/jobgviNIp INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="18867" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpW0tzWU/496f3539-3c57-4056-acd0-a5ac184f4a89/tmpEoIMus.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpW0tzWU/496f3539-3c57-4056-acd0-a5ac184f4a89/tmpZqmni6.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpW0tzWU/496f3539-3c57-4056-acd0-a5ac184f4a89/tmpkd8GxQ.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpW0tzWU/496f3539-3c57-4056-acd0-a5ac184f4a89/tmpDhf0C0.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpW0tzWU/496f3539-3c57-4056-acd0-a5ac184f4a89/tmp7bCbmd.tmp']
  1087. WARNING:toil.leader:k/K/jobgviNIp Set log level to INFO
  1088. WARNING:toil.leader:k/K/jobgviNIp Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  1089. WARNING:toil.leader:k/K/jobgviNIp <kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="18867" />
  1090. WARNING:toil.leader:k/K/jobgviNIp </st_kv_database_conf>
  1091. WARNING:toil.leader:k/K/jobgviNIp
  1092. WARNING:toil.leader:k/K/jobgviNIp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpW0tzWU/496f3539-3c57-4056-acd0-a5ac184f4a89/tmpEoIMus.tmp
  1093. WARNING:toil.leader:k/K/jobgviNIp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpW0tzWU/496f3539-3c57-4056-acd0-a5ac184f4a89/tmpZqmni6.tmp
  1094. WARNING:toil.leader:k/K/jobgviNIp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpW0tzWU/496f3539-3c57-4056-acd0-a5ac184f4a89/tmpkd8GxQ.tmp
  1095. WARNING:toil.leader:k/K/jobgviNIp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpW0tzWU/496f3539-3c57-4056-acd0-a5ac184f4a89/tmpDhf0C0.tmp
  1096. WARNING:toil.leader:k/K/jobgviNIp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpW0tzWU/496f3539-3c57-4056-acd0-a5ac184f4a89/tmp7bCbmd.tmp
  1097. WARNING:toil.leader:k/K/jobgviNIp Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.29 with error: network error
  1098. WARNING:toil.leader:k/K/jobgviNIp Uncaught exception
  1099. WARNING:toil.leader:k/K/jobgviNIp Traceback (most recent call last):
  1100. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1101. WARNING:toil.leader:k/K/jobgviNIp job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1102. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1103. WARNING:toil.leader:k/K/jobgviNIp super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1104. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1105. WARNING:toil.leader:k/K/jobgviNIp returnValues = self._run(jobGraph, fileStore)
  1106. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1107. WARNING:toil.leader:k/K/jobgviNIp return self.run(fileStore)
  1108. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  1109. WARNING:toil.leader:k/K/jobgviNIp makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  1110. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  1111. WARNING:toil.leader:k/K/jobgviNIp parameters=["cactus_setup"] + args + sequences)
  1112. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1113. WARNING:toil.leader:k/K/jobgviNIp raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1114. WARNING:toil.leader:k/K/jobgviNIp RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="18867" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpW0tzWU/496f3539-3c57-4056-acd0-a5ac184f4a89/tmpEoIMus.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpW0tzWU/496f3539-3c57-4056-acd0-a5ac184f4a89/tmpZqmni6.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpW0tzWU/496f3539-3c57-4056-acd0-a5ac184f4a89/tmpkd8GxQ.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpW0tzWU/496f3539-3c57-4056-acd0-a5ac184f4a89/tmpDhf0C0.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpW0tzWU/496f3539-3c57-4056-acd0-a5ac184f4a89/tmp7bCbmd.tmp'] failed with output:
  1115. WARNING:toil.leader:k/K/jobgviNIp ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn37.grid.pub.ro
  1116. WARNING:toil.leader:k/K/jobgviNIp WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' k/K/jobgviNIp with ID k/K/jobgviNIp to 5
  1117. INFO:toil.leader:Issued job 'CactusSetupPhase' k/K/jobgviNIp with job batch system ID: 183 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1118. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' x/v/jobBOV8ZO
  1119. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' x/v/jobBOV8ZO
  1120. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' x/v/jobBOV8ZO with ID x/v/jobBOV8ZO to 5
  1121. INFO:toil.leader:Issued job 'KtServerService' x/v/jobBOV8ZO with job batch system ID: 184 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1122. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' k/K/jobgviNIp
  1123. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' k/K/jobgviNIp
  1124. WARNING:toil.leader:k/K/jobgviNIp INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1125. WARNING:toil.leader:k/K/jobgviNIp INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1126. WARNING:toil.leader:k/K/jobgviNIp WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1127. WARNING:toil.leader:k/K/jobgviNIp INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  1128. WARNING:toil.leader:k/K/jobgviNIp INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  1129. WARNING:toil.leader:k/K/jobgviNIp INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="18867" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpn_1SNa/543b1cc5-f310-4bd6-82eb-03ed451b0777/tmptHiQ4p.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpn_1SNa/543b1cc5-f310-4bd6-82eb-03ed451b0777/tmpgosSfz.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpn_1SNa/543b1cc5-f310-4bd6-82eb-03ed451b0777/tmpwB01or.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpn_1SNa/543b1cc5-f310-4bd6-82eb-03ed451b0777/tmpIvohDC.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpn_1SNa/543b1cc5-f310-4bd6-82eb-03ed451b0777/tmp8MFLV3.tmp']
  1130. WARNING:toil.leader:k/K/jobgviNIp Set log level to INFO
  1131. WARNING:toil.leader:k/K/jobgviNIp Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  1132. WARNING:toil.leader:k/K/jobgviNIp <kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="18867" />
  1133. WARNING:toil.leader:k/K/jobgviNIp </st_kv_database_conf>
  1134. WARNING:toil.leader:k/K/jobgviNIp
  1135. WARNING:toil.leader:k/K/jobgviNIp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpn_1SNa/543b1cc5-f310-4bd6-82eb-03ed451b0777/tmptHiQ4p.tmp
  1136. WARNING:toil.leader:k/K/jobgviNIp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpn_1SNa/543b1cc5-f310-4bd6-82eb-03ed451b0777/tmpgosSfz.tmp
  1137. WARNING:toil.leader:k/K/jobgviNIp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpn_1SNa/543b1cc5-f310-4bd6-82eb-03ed451b0777/tmpwB01or.tmp
  1138. WARNING:toil.leader:k/K/jobgviNIp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpn_1SNa/543b1cc5-f310-4bd6-82eb-03ed451b0777/tmpIvohDC.tmp
  1139. WARNING:toil.leader:k/K/jobgviNIp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpn_1SNa/543b1cc5-f310-4bd6-82eb-03ed451b0777/tmp8MFLV3.tmp
  1140. WARNING:toil.leader:k/K/jobgviNIp Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.29 with error: network error
  1141. WARNING:toil.leader:k/K/jobgviNIp Uncaught exception
  1142. WARNING:toil.leader:k/K/jobgviNIp Traceback (most recent call last):
  1143. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1144. WARNING:toil.leader:k/K/jobgviNIp job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1145. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1146. WARNING:toil.leader:k/K/jobgviNIp super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1147. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1148. WARNING:toil.leader:k/K/jobgviNIp returnValues = self._run(jobGraph, fileStore)
  1149. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1150. WARNING:toil.leader:k/K/jobgviNIp return self.run(fileStore)
  1151. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  1152. WARNING:toil.leader:k/K/jobgviNIp makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  1153. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  1154. WARNING:toil.leader:k/K/jobgviNIp parameters=["cactus_setup"] + args + sequences)
  1155. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1156. WARNING:toil.leader:k/K/jobgviNIp raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1157. WARNING:toil.leader:k/K/jobgviNIp RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="18867" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpn_1SNa/543b1cc5-f310-4bd6-82eb-03ed451b0777/tmptHiQ4p.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpn_1SNa/543b1cc5-f310-4bd6-82eb-03ed451b0777/tmpgosSfz.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpn_1SNa/543b1cc5-f310-4bd6-82eb-03ed451b0777/tmpwB01or.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpn_1SNa/543b1cc5-f310-4bd6-82eb-03ed451b0777/tmpIvohDC.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmpn_1SNa/543b1cc5-f310-4bd6-82eb-03ed451b0777/tmp8MFLV3.tmp'] failed with output:
  1158. WARNING:toil.leader:k/K/jobgviNIp ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn35.grid.pub.ro
  1159. WARNING:toil.leader:k/K/jobgviNIp WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' k/K/jobgviNIp with ID k/K/jobgviNIp to 4
  1160. INFO:toil.leader:Issued job 'CactusSetupPhase' k/K/jobgviNIp with job batch system ID: 185 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1161. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' x/v/jobBOV8ZO
  1162. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' x/v/jobBOV8ZO
  1163. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' x/v/jobBOV8ZO with ID x/v/jobBOV8ZO to 4
  1164. INFO:toil.leader:Issued job 'KtServerService' x/v/jobBOV8ZO with job batch system ID: 186 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1165. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' x/v/jobBOV8ZO
  1166. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' x/v/jobBOV8ZO
  1167. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' x/v/jobBOV8ZO with ID x/v/jobBOV8ZO to 3
  1168. INFO:toil.leader:Issued job 'KtServerService' x/v/jobBOV8ZO with job batch system ID: 187 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1169. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' k/K/jobgviNIp
  1170. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' k/K/jobgviNIp
  1171. WARNING:toil.leader:k/K/jobgviNIp INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1172. WARNING:toil.leader:k/K/jobgviNIp INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1173. WARNING:toil.leader:k/K/jobgviNIp WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1174. WARNING:toil.leader:k/K/jobgviNIp WARNING:toil.fileStore:Detected that job (a/n/jobKwRkH2/g/tmpqp43fG-_serialiseJob-stream) prematurely terminated. Fixing the state of the job on disk.
  1175. WARNING:toil.leader:k/K/jobgviNIp INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  1176. WARNING:toil.leader:k/K/jobgviNIp INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  1177. WARNING:toil.leader:k/K/jobgviNIp INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.37" port="11718" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpN218df/3c103ec0-563b-404d-a41c-aa5b4bc46a49/tmpHPByPs.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpN218df/3c103ec0-563b-404d-a41c-aa5b4bc46a49/tmpw_t5dj.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpN218df/3c103ec0-563b-404d-a41c-aa5b4bc46a49/tmpmDUUmg.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpN218df/3c103ec0-563b-404d-a41c-aa5b4bc46a49/tmp9UFVha.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpN218df/3c103ec0-563b-404d-a41c-aa5b4bc46a49/tmpv5iPL_.tmp']
  1178. WARNING:toil.leader:k/K/jobgviNIp Set log level to INFO
  1179. WARNING:toil.leader:k/K/jobgviNIp Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  1180. WARNING:toil.leader:k/K/jobgviNIp <kyoto_tycoon database_dir="fakepath" host="172.16.13.37" port="11718" />
  1181. WARNING:toil.leader:k/K/jobgviNIp </st_kv_database_conf>
  1182. WARNING:toil.leader:k/K/jobgviNIp
  1183. WARNING:toil.leader:k/K/jobgviNIp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpN218df/3c103ec0-563b-404d-a41c-aa5b4bc46a49/tmpHPByPs.tmp
  1184. WARNING:toil.leader:k/K/jobgviNIp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpN218df/3c103ec0-563b-404d-a41c-aa5b4bc46a49/tmpw_t5dj.tmp
  1185. WARNING:toil.leader:k/K/jobgviNIp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpN218df/3c103ec0-563b-404d-a41c-aa5b4bc46a49/tmpmDUUmg.tmp
  1186. WARNING:toil.leader:k/K/jobgviNIp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpN218df/3c103ec0-563b-404d-a41c-aa5b4bc46a49/tmp9UFVha.tmp
  1187. WARNING:toil.leader:k/K/jobgviNIp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpN218df/3c103ec0-563b-404d-a41c-aa5b4bc46a49/tmpv5iPL_.tmp
  1188. WARNING:toil.leader:k/K/jobgviNIp Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.37 with error: network error
  1189. WARNING:toil.leader:k/K/jobgviNIp Uncaught exception
  1190. WARNING:toil.leader:k/K/jobgviNIp Traceback (most recent call last):
  1191. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1192. WARNING:toil.leader:k/K/jobgviNIp job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1193. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1194. WARNING:toil.leader:k/K/jobgviNIp super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1195. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1196. WARNING:toil.leader:k/K/jobgviNIp returnValues = self._run(jobGraph, fileStore)
  1197. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1198. WARNING:toil.leader:k/K/jobgviNIp return self.run(fileStore)
  1199. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  1200. WARNING:toil.leader:k/K/jobgviNIp makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  1201. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  1202. WARNING:toil.leader:k/K/jobgviNIp parameters=["cactus_setup"] + args + sequences)
  1203. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1204. WARNING:toil.leader:k/K/jobgviNIp raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1205. WARNING:toil.leader:k/K/jobgviNIp RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.37" port="11718" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpN218df/3c103ec0-563b-404d-a41c-aa5b4bc46a49/tmpHPByPs.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpN218df/3c103ec0-563b-404d-a41c-aa5b4bc46a49/tmpw_t5dj.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpN218df/3c103ec0-563b-404d-a41c-aa5b4bc46a49/tmpmDUUmg.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpN218df/3c103ec0-563b-404d-a41c-aa5b4bc46a49/tmp9UFVha.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpN218df/3c103ec0-563b-404d-a41c-aa5b4bc46a49/tmpv5iPL_.tmp'] failed with output:
  1206. WARNING:toil.leader:k/K/jobgviNIp ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn38.grid.pub.ro
  1207. WARNING:toil.leader:k/K/jobgviNIp WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' k/K/jobgviNIp with ID k/K/jobgviNIp to 3
  1208. INFO:toil.leader:Issued job 'CactusSetupPhase' k/K/jobgviNIp with job batch system ID: 188 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1209. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' k/K/jobgviNIp
  1210. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' k/K/jobgviNIp
  1211. WARNING:toil.leader:k/K/jobgviNIp INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1212. WARNING:toil.leader:k/K/jobgviNIp INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1213. WARNING:toil.leader:k/K/jobgviNIp WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1214. WARNING:toil.leader:k/K/jobgviNIp WARNING:toil.fileStore:Detected that job (a/n/jobKwRkH2/g/tmpYTbc6n-_serialiseJob-stream) prematurely terminated. Fixing the state of the job on disk.
  1215. WARNING:toil.leader:k/K/jobgviNIp INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  1216. WARNING:toil.leader:k/K/jobgviNIp INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  1217. WARNING:toil.leader:k/K/jobgviNIp INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="6905" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpnQeMdQ/f1ba48a6-02b3-48e1-ac5a-eb1029f1f714/tmpBHWvcT.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpnQeMdQ/f1ba48a6-02b3-48e1-ac5a-eb1029f1f714/tmpBnNuKc.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpnQeMdQ/f1ba48a6-02b3-48e1-ac5a-eb1029f1f714/tmpCvzqUb.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpnQeMdQ/f1ba48a6-02b3-48e1-ac5a-eb1029f1f714/tmpA2iGk_.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpnQeMdQ/f1ba48a6-02b3-48e1-ac5a-eb1029f1f714/tmpX80Wxg.tmp']
  1218. WARNING:toil.leader:k/K/jobgviNIp Set log level to INFO
  1219. WARNING:toil.leader:k/K/jobgviNIp Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  1220. WARNING:toil.leader:k/K/jobgviNIp <kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="6905" />
  1221. WARNING:toil.leader:k/K/jobgviNIp </st_kv_database_conf>
  1222. WARNING:toil.leader:k/K/jobgviNIp
  1223. WARNING:toil.leader:k/K/jobgviNIp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpnQeMdQ/f1ba48a6-02b3-48e1-ac5a-eb1029f1f714/tmpBHWvcT.tmp
  1224. WARNING:toil.leader:k/K/jobgviNIp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpnQeMdQ/f1ba48a6-02b3-48e1-ac5a-eb1029f1f714/tmpBnNuKc.tmp
  1225. WARNING:toil.leader:k/K/jobgviNIp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpnQeMdQ/f1ba48a6-02b3-48e1-ac5a-eb1029f1f714/tmpCvzqUb.tmp
  1226. WARNING:toil.leader:k/K/jobgviNIp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpnQeMdQ/f1ba48a6-02b3-48e1-ac5a-eb1029f1f714/tmpA2iGk_.tmp
  1227. WARNING:toil.leader:k/K/jobgviNIp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpnQeMdQ/f1ba48a6-02b3-48e1-ac5a-eb1029f1f714/tmpX80Wxg.tmp
  1228. WARNING:toil.leader:k/K/jobgviNIp Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.41 with error: network error
  1229. WARNING:toil.leader:k/K/jobgviNIp Uncaught exception
  1230. WARNING:toil.leader:k/K/jobgviNIp Traceback (most recent call last):
  1231. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1232. WARNING:toil.leader:k/K/jobgviNIp job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1233. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1234. WARNING:toil.leader:k/K/jobgviNIp super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1235. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1236. WARNING:toil.leader:k/K/jobgviNIp returnValues = self._run(jobGraph, fileStore)
  1237. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1238. WARNING:toil.leader:k/K/jobgviNIp return self.run(fileStore)
  1239. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  1240. WARNING:toil.leader:k/K/jobgviNIp makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  1241. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  1242. WARNING:toil.leader:k/K/jobgviNIp parameters=["cactus_setup"] + args + sequences)
  1243. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1244. WARNING:toil.leader:k/K/jobgviNIp raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1245. WARNING:toil.leader:k/K/jobgviNIp RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="6905" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpnQeMdQ/f1ba48a6-02b3-48e1-ac5a-eb1029f1f714/tmpBHWvcT.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpnQeMdQ/f1ba48a6-02b3-48e1-ac5a-eb1029f1f714/tmpBnNuKc.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpnQeMdQ/f1ba48a6-02b3-48e1-ac5a-eb1029f1f714/tmpCvzqUb.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpnQeMdQ/f1ba48a6-02b3-48e1-ac5a-eb1029f1f714/tmpA2iGk_.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpnQeMdQ/f1ba48a6-02b3-48e1-ac5a-eb1029f1f714/tmpX80Wxg.tmp'] failed with output:
  1246. WARNING:toil.leader:k/K/jobgviNIp ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn29.grid.pub.ro
  1247. WARNING:toil.leader:k/K/jobgviNIp WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' k/K/jobgviNIp with ID k/K/jobgviNIp to 2
  1248. INFO:toil.leader:Issued job 'CactusSetupPhase' k/K/jobgviNIp with job batch system ID: 189 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1249. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' x/v/jobBOV8ZO
  1250. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' x/v/jobBOV8ZO
  1251. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' x/v/jobBOV8ZO with ID x/v/jobBOV8ZO to 2
  1252. INFO:toil.leader:Issued job 'KtServerService' x/v/jobBOV8ZO with job batch system ID: 190 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1253. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' k/K/jobgviNIp
  1254. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' k/K/jobgviNIp
  1255. WARNING:toil.leader:k/K/jobgviNIp INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1256. WARNING:toil.leader:k/K/jobgviNIp INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1257. WARNING:toil.leader:k/K/jobgviNIp WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1258. WARNING:toil.leader:k/K/jobgviNIp INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  1259. WARNING:toil.leader:k/K/jobgviNIp INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  1260. WARNING:toil.leader:k/K/jobgviNIp INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.34" port="30332" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmp3tCSNh/8a2f7e88-1321-4b4a-86ff-0da46dbbe720/tmpffO6Vi.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmp3tCSNh/8a2f7e88-1321-4b4a-86ff-0da46dbbe720/tmpJf8x7E.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmp3tCSNh/8a2f7e88-1321-4b4a-86ff-0da46dbbe720/tmpUgrCDY.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmp3tCSNh/8a2f7e88-1321-4b4a-86ff-0da46dbbe720/tmpPtH_Kr.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmp3tCSNh/8a2f7e88-1321-4b4a-86ff-0da46dbbe720/tmpw6lDyO.tmp']
  1261. WARNING:toil.leader:k/K/jobgviNIp Set log level to INFO
  1262. WARNING:toil.leader:k/K/jobgviNIp Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  1263. WARNING:toil.leader:k/K/jobgviNIp <kyoto_tycoon database_dir="fakepath" host="172.16.13.34" port="30332" />
  1264. WARNING:toil.leader:k/K/jobgviNIp </st_kv_database_conf>
  1265. WARNING:toil.leader:k/K/jobgviNIp
  1266. WARNING:toil.leader:k/K/jobgviNIp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmp3tCSNh/8a2f7e88-1321-4b4a-86ff-0da46dbbe720/tmpffO6Vi.tmp
  1267. WARNING:toil.leader:k/K/jobgviNIp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmp3tCSNh/8a2f7e88-1321-4b4a-86ff-0da46dbbe720/tmpJf8x7E.tmp
  1268. WARNING:toil.leader:k/K/jobgviNIp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmp3tCSNh/8a2f7e88-1321-4b4a-86ff-0da46dbbe720/tmpUgrCDY.tmp
  1269. WARNING:toil.leader:k/K/jobgviNIp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmp3tCSNh/8a2f7e88-1321-4b4a-86ff-0da46dbbe720/tmpPtH_Kr.tmp
  1270. WARNING:toil.leader:k/K/jobgviNIp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmp3tCSNh/8a2f7e88-1321-4b4a-86ff-0da46dbbe720/tmpw6lDyO.tmp
  1271. WARNING:toil.leader:k/K/jobgviNIp Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.34 with error: network error
  1272. WARNING:toil.leader:k/K/jobgviNIp Uncaught exception
  1273. WARNING:toil.leader:k/K/jobgviNIp Traceback (most recent call last):
  1274. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1275. WARNING:toil.leader:k/K/jobgviNIp job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1276. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1277. WARNING:toil.leader:k/K/jobgviNIp super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1278. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1279. WARNING:toil.leader:k/K/jobgviNIp returnValues = self._run(jobGraph, fileStore)
  1280. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1281. WARNING:toil.leader:k/K/jobgviNIp return self.run(fileStore)
  1282. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  1283. WARNING:toil.leader:k/K/jobgviNIp makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  1284. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  1285. WARNING:toil.leader:k/K/jobgviNIp parameters=["cactus_setup"] + args + sequences)
  1286. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1287. WARNING:toil.leader:k/K/jobgviNIp raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1288. WARNING:toil.leader:k/K/jobgviNIp RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.34" port="30332" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmp3tCSNh/8a2f7e88-1321-4b4a-86ff-0da46dbbe720/tmpffO6Vi.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmp3tCSNh/8a2f7e88-1321-4b4a-86ff-0da46dbbe720/tmpJf8x7E.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmp3tCSNh/8a2f7e88-1321-4b4a-86ff-0da46dbbe720/tmpUgrCDY.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmp3tCSNh/8a2f7e88-1321-4b4a-86ff-0da46dbbe720/tmpPtH_Kr.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-88711676-5948-47d0-acd3-569974301115/tmp3tCSNh/8a2f7e88-1321-4b4a-86ff-0da46dbbe720/tmpw6lDyO.tmp'] failed with output:
  1289. WARNING:toil.leader:k/K/jobgviNIp ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn35.grid.pub.ro
  1290. WARNING:toil.leader:k/K/jobgviNIp WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' k/K/jobgviNIp with ID k/K/jobgviNIp to 1
  1291. INFO:toil.leader:Issued job 'CactusSetupPhase' k/K/jobgviNIp with job batch system ID: 191 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1292. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' x/v/jobBOV8ZO
  1293. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' x/v/jobBOV8ZO
  1294. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' x/v/jobBOV8ZO with ID x/v/jobBOV8ZO to 1
  1295. INFO:toil.leader:Issued job 'KtServerService' x/v/jobBOV8ZO with job batch system ID: 192 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1296. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' x/v/jobBOV8ZO
  1297. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' x/v/jobBOV8ZO
  1298. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' x/v/jobBOV8ZO with ID x/v/jobBOV8ZO to 0
  1299. WARNING:toil.leader:Job 'KtServerService' x/v/jobBOV8ZO with ID x/v/jobBOV8ZO is completely failed
  1300. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' k/K/jobgviNIp
  1301. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' k/K/jobgviNIp
  1302. WARNING:toil.leader:k/K/jobgviNIp INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1303. WARNING:toil.leader:k/K/jobgviNIp INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1304. WARNING:toil.leader:k/K/jobgviNIp WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1305. WARNING:toil.leader:k/K/jobgviNIp WARNING:toil.fileStore:Detected that job (a/n/jobKwRkH2/g/tmpqp43fG-_serialiseJob-stream) prematurely terminated. Fixing the state of the job on disk.
  1306. WARNING:toil.leader:k/K/jobgviNIp INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  1307. WARNING:toil.leader:k/K/jobgviNIp INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  1308. WARNING:toil.leader:k/K/jobgviNIp INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="19791" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpwDjxzk/19932c56-36f8-474c-9d00-291d34467d48/tmptzLbag.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpwDjxzk/19932c56-36f8-474c-9d00-291d34467d48/tmpBPZZrR.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpwDjxzk/19932c56-36f8-474c-9d00-291d34467d48/tmpgIHZck.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpwDjxzk/19932c56-36f8-474c-9d00-291d34467d48/tmpdQiynP.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpwDjxzk/19932c56-36f8-474c-9d00-291d34467d48/tmphLqsEi.tmp']
  1309. WARNING:toil.leader:k/K/jobgviNIp Set log level to INFO
  1310. WARNING:toil.leader:k/K/jobgviNIp Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  1311. WARNING:toil.leader:k/K/jobgviNIp <kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="19791" />
  1312. WARNING:toil.leader:k/K/jobgviNIp </st_kv_database_conf>
  1313. WARNING:toil.leader:k/K/jobgviNIp
  1314. WARNING:toil.leader:k/K/jobgviNIp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpwDjxzk/19932c56-36f8-474c-9d00-291d34467d48/tmptzLbag.tmp
  1315. WARNING:toil.leader:k/K/jobgviNIp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpwDjxzk/19932c56-36f8-474c-9d00-291d34467d48/tmpBPZZrR.tmp
  1316. WARNING:toil.leader:k/K/jobgviNIp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpwDjxzk/19932c56-36f8-474c-9d00-291d34467d48/tmpgIHZck.tmp
  1317. WARNING:toil.leader:k/K/jobgviNIp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpwDjxzk/19932c56-36f8-474c-9d00-291d34467d48/tmpdQiynP.tmp
  1318. WARNING:toil.leader:k/K/jobgviNIp Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpwDjxzk/19932c56-36f8-474c-9d00-291d34467d48/tmphLqsEi.tmp
  1319. WARNING:toil.leader:k/K/jobgviNIp Set up the flower disk
  1320. WARNING:toil.leader:k/K/jobgviNIp Constructed the flower
  1321. WARNING:toil.leader:k/K/jobgviNIp Going to build the event tree with newick string: ((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;
  1322. WARNING:toil.leader:k/K/jobgviNIp Parsed the tree
  1323. WARNING:toil.leader:k/K/jobgviNIp can't allocate 190 bytes of memory
  1324. WARNING:toil.leader:k/K/jobgviNIp Traceback (most recent call last):
  1325. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1326. WARNING:toil.leader:k/K/jobgviNIp job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1327. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1328. WARNING:toil.leader:k/K/jobgviNIp super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1329. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1330. WARNING:toil.leader:k/K/jobgviNIp returnValues = self._run(jobGraph, fileStore)
  1331. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1332. WARNING:toil.leader:k/K/jobgviNIp return self.run(fileStore)
  1333. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  1334. WARNING:toil.leader:k/K/jobgviNIp makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  1335. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  1336. WARNING:toil.leader:k/K/jobgviNIp parameters=["cactus_setup"] + args + sequences)
  1337. WARNING:toil.leader:k/K/jobgviNIp File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1338. WARNING:toil.leader:k/K/jobgviNIp raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1339. WARNING:toil.leader:k/K/jobgviNIp RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="19791" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpwDjxzk/19932c56-36f8-474c-9d00-291d34467d48/tmptzLbag.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpwDjxzk/19932c56-36f8-474c-9d00-291d34467d48/tmpBPZZrR.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpwDjxzk/19932c56-36f8-474c-9d00-291d34467d48/tmpgIHZck.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpwDjxzk/19932c56-36f8-474c-9d00-291d34467d48/tmpdQiynP.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpwDjxzk/19932c56-36f8-474c-9d00-291d34467d48/tmphLqsEi.tmp'] failed with output:
  1340. WARNING:toil.leader:k/K/jobgviNIp ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn40.grid.pub.ro
  1341. WARNING:toil.leader:k/K/jobgviNIp WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' k/K/jobgviNIp with ID k/K/jobgviNIp to 0
  1342. WARNING:toil.leader:Job 'CactusSetupPhase' k/K/jobgviNIp with ID k/K/jobgviNIp is completely failed
  1343. WARNING:toil.leader:Job: a/n/jobKwRkH2 is being restarted as a checkpoint after the total failure of jobs in its subtree.
  1344. INFO:toil.leader:Issued job 'StartPrimaryDB' a/n/jobKwRkH2 with job batch system ID: 193 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1345. INFO:toil.leader:Job ended successfully: 'StartPrimaryDB' a/n/jobKwRkH2
  1346. INFO:toil.leader:Issued job 'KtServerService' c/1/jobBsZm2L with job batch system ID: 194 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1347. INFO:toil.leader:Issued job 'CactusSetupPhase' V/k/jobtmtU9v with job batch system ID: 195 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1348. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' V/k/jobtmtU9v
  1349. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' V/k/jobtmtU9v
  1350. WARNING:toil.leader:V/k/jobtmtU9v INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1351. WARNING:toil.leader:V/k/jobtmtU9v INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1352. WARNING:toil.leader:V/k/jobtmtU9v WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1353. WARNING:toil.leader:V/k/jobtmtU9v WARNING:toil.fileStore:Detected that job (a/n/jobKwRkH2/g/tmpYTbc6n-_serialiseJob-stream) prematurely terminated. Fixing the state of the job on disk.
  1354. WARNING:toil.leader:V/k/jobtmtU9v INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  1355. WARNING:toil.leader:V/k/jobtmtU9v INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  1356. WARNING:toil.leader:V/k/jobtmtU9v INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.30" port="11878" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-906375ca-4612-4b9c-aace-ad64276eee7f/tmpbYebvM/161672a2-5854-44f8-b8fb-685fb775f59d/tmpn_MOE_.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-906375ca-4612-4b9c-aace-ad64276eee7f/tmpbYebvM/161672a2-5854-44f8-b8fb-685fb775f59d/tmpcha5gB.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-906375ca-4612-4b9c-aace-ad64276eee7f/tmpbYebvM/161672a2-5854-44f8-b8fb-685fb775f59d/tmpRmfjxg.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-906375ca-4612-4b9c-aace-ad64276eee7f/tmpbYebvM/161672a2-5854-44f8-b8fb-685fb775f59d/tmprXhgzJ.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-906375ca-4612-4b9c-aace-ad64276eee7f/tmpbYebvM/161672a2-5854-44f8-b8fb-685fb775f59d/tmpUqneWL.tmp']
  1357. WARNING:toil.leader:V/k/jobtmtU9v Set log level to INFO
  1358. WARNING:toil.leader:V/k/jobtmtU9v Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  1359. WARNING:toil.leader:V/k/jobtmtU9v <kyoto_tycoon database_dir="fakepath" host="172.16.13.30" port="11878" />
  1360. WARNING:toil.leader:V/k/jobtmtU9v </st_kv_database_conf>
  1361. WARNING:toil.leader:V/k/jobtmtU9v
  1362. WARNING:toil.leader:V/k/jobtmtU9v Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-906375ca-4612-4b9c-aace-ad64276eee7f/tmpbYebvM/161672a2-5854-44f8-b8fb-685fb775f59d/tmpn_MOE_.tmp
  1363. WARNING:toil.leader:V/k/jobtmtU9v Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-906375ca-4612-4b9c-aace-ad64276eee7f/tmpbYebvM/161672a2-5854-44f8-b8fb-685fb775f59d/tmpcha5gB.tmp
  1364. WARNING:toil.leader:V/k/jobtmtU9v Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-906375ca-4612-4b9c-aace-ad64276eee7f/tmpbYebvM/161672a2-5854-44f8-b8fb-685fb775f59d/tmpRmfjxg.tmp
  1365. WARNING:toil.leader:V/k/jobtmtU9v Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-906375ca-4612-4b9c-aace-ad64276eee7f/tmpbYebvM/161672a2-5854-44f8-b8fb-685fb775f59d/tmprXhgzJ.tmp
  1366. WARNING:toil.leader:V/k/jobtmtU9v Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-906375ca-4612-4b9c-aace-ad64276eee7f/tmpbYebvM/161672a2-5854-44f8-b8fb-685fb775f59d/tmpUqneWL.tmp
  1367. WARNING:toil.leader:V/k/jobtmtU9v Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.30 with error: network error
  1368. WARNING:toil.leader:V/k/jobtmtU9v Uncaught exception
  1369. WARNING:toil.leader:V/k/jobtmtU9v Traceback (most recent call last):
  1370. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1371. WARNING:toil.leader:V/k/jobtmtU9v job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1372. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1373. WARNING:toil.leader:V/k/jobtmtU9v super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1374. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1375. WARNING:toil.leader:V/k/jobtmtU9v returnValues = self._run(jobGraph, fileStore)
  1376. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1377. WARNING:toil.leader:V/k/jobtmtU9v return self.run(fileStore)
  1378. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  1379. WARNING:toil.leader:V/k/jobtmtU9v makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  1380. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  1381. WARNING:toil.leader:V/k/jobtmtU9v parameters=["cactus_setup"] + args + sequences)
  1382. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1383. WARNING:toil.leader:V/k/jobtmtU9v raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1384. WARNING:toil.leader:V/k/jobtmtU9v RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.30" port="11878" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-906375ca-4612-4b9c-aace-ad64276eee7f/tmpbYebvM/161672a2-5854-44f8-b8fb-685fb775f59d/tmpn_MOE_.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-906375ca-4612-4b9c-aace-ad64276eee7f/tmpbYebvM/161672a2-5854-44f8-b8fb-685fb775f59d/tmpcha5gB.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-906375ca-4612-4b9c-aace-ad64276eee7f/tmpbYebvM/161672a2-5854-44f8-b8fb-685fb775f59d/tmpRmfjxg.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-906375ca-4612-4b9c-aace-ad64276eee7f/tmpbYebvM/161672a2-5854-44f8-b8fb-685fb775f59d/tmprXhgzJ.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-906375ca-4612-4b9c-aace-ad64276eee7f/tmpbYebvM/161672a2-5854-44f8-b8fb-685fb775f59d/tmpUqneWL.tmp'] failed with output:
  1385. WARNING:toil.leader:V/k/jobtmtU9v ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn34.grid.pub.ro
  1386. WARNING:toil.leader:V/k/jobtmtU9v WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' V/k/jobtmtU9v with ID V/k/jobtmtU9v to 5
  1387. INFO:toil.leader:Issued job 'CactusSetupPhase' V/k/jobtmtU9v with job batch system ID: 196 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1388. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' c/1/jobBsZm2L
  1389. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' c/1/jobBsZm2L
  1390. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' c/1/jobBsZm2L with ID c/1/jobBsZm2L to 5
  1391. INFO:toil.leader:Issued job 'KtServerService' c/1/jobBsZm2L with job batch system ID: 197 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1392. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' V/k/jobtmtU9v
  1393. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' V/k/jobtmtU9v
  1394. WARNING:toil.leader:V/k/jobtmtU9v INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1395. WARNING:toil.leader:V/k/jobtmtU9v INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1396. WARNING:toil.leader:V/k/jobtmtU9v WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1397. WARNING:toil.leader:V/k/jobtmtU9v INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  1398. WARNING:toil.leader:V/k/jobtmtU9v INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  1399. WARNING:toil.leader:V/k/jobtmtU9v INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.30" port="11878" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpR6t_mA/7e78d8a5-6742-4310-a193-e1c2d5f5f3f5/tmpIb6Xyx.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpR6t_mA/7e78d8a5-6742-4310-a193-e1c2d5f5f3f5/tmphMQB0C.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpR6t_mA/7e78d8a5-6742-4310-a193-e1c2d5f5f3f5/tmp8SPiGX.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpR6t_mA/7e78d8a5-6742-4310-a193-e1c2d5f5f3f5/tmpLC3jju.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpR6t_mA/7e78d8a5-6742-4310-a193-e1c2d5f5f3f5/tmpN0RpLH.tmp']
  1400. WARNING:toil.leader:V/k/jobtmtU9v Set log level to INFO
  1401. WARNING:toil.leader:V/k/jobtmtU9v Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  1402. WARNING:toil.leader:V/k/jobtmtU9v <kyoto_tycoon database_dir="fakepath" host="172.16.13.30" port="11878" />
  1403. WARNING:toil.leader:V/k/jobtmtU9v </st_kv_database_conf>
  1404. WARNING:toil.leader:V/k/jobtmtU9v
  1405. WARNING:toil.leader:V/k/jobtmtU9v Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpR6t_mA/7e78d8a5-6742-4310-a193-e1c2d5f5f3f5/tmpIb6Xyx.tmp
  1406. WARNING:toil.leader:V/k/jobtmtU9v Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpR6t_mA/7e78d8a5-6742-4310-a193-e1c2d5f5f3f5/tmphMQB0C.tmp
  1407. WARNING:toil.leader:V/k/jobtmtU9v Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpR6t_mA/7e78d8a5-6742-4310-a193-e1c2d5f5f3f5/tmp8SPiGX.tmp
  1408. WARNING:toil.leader:V/k/jobtmtU9v Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpR6t_mA/7e78d8a5-6742-4310-a193-e1c2d5f5f3f5/tmpLC3jju.tmp
  1409. WARNING:toil.leader:V/k/jobtmtU9v Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpR6t_mA/7e78d8a5-6742-4310-a193-e1c2d5f5f3f5/tmpN0RpLH.tmp
  1410. WARNING:toil.leader:V/k/jobtmtU9v Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.30 with error: network error
  1411. WARNING:toil.leader:V/k/jobtmtU9v Uncaught exception
  1412. WARNING:toil.leader:V/k/jobtmtU9v Traceback (most recent call last):
  1413. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1414. WARNING:toil.leader:V/k/jobtmtU9v job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1415. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1416. WARNING:toil.leader:V/k/jobtmtU9v super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1417. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1418. WARNING:toil.leader:V/k/jobtmtU9v returnValues = self._run(jobGraph, fileStore)
  1419. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1420. WARNING:toil.leader:V/k/jobtmtU9v return self.run(fileStore)
  1421. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  1422. WARNING:toil.leader:V/k/jobtmtU9v makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  1423. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  1424. WARNING:toil.leader:V/k/jobtmtU9v parameters=["cactus_setup"] + args + sequences)
  1425. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1426. WARNING:toil.leader:V/k/jobtmtU9v raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1427. WARNING:toil.leader:V/k/jobtmtU9v RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.30" port="11878" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpR6t_mA/7e78d8a5-6742-4310-a193-e1c2d5f5f3f5/tmpIb6Xyx.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpR6t_mA/7e78d8a5-6742-4310-a193-e1c2d5f5f3f5/tmphMQB0C.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpR6t_mA/7e78d8a5-6742-4310-a193-e1c2d5f5f3f5/tmp8SPiGX.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpR6t_mA/7e78d8a5-6742-4310-a193-e1c2d5f5f3f5/tmpLC3jju.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-2bc2432e-0437-4e28-8030-69bdeb17d728/tmpR6t_mA/7e78d8a5-6742-4310-a193-e1c2d5f5f3f5/tmpN0RpLH.tmp'] failed with output:
  1428. WARNING:toil.leader:V/k/jobtmtU9v ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn38.grid.pub.ro
  1429. WARNING:toil.leader:V/k/jobtmtU9v WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' V/k/jobtmtU9v with ID V/k/jobtmtU9v to 4
  1430. INFO:toil.leader:Issued job 'CactusSetupPhase' V/k/jobtmtU9v with job batch system ID: 198 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1431. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' c/1/jobBsZm2L
  1432. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' c/1/jobBsZm2L
  1433. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' c/1/jobBsZm2L with ID c/1/jobBsZm2L to 4
  1434. INFO:toil.leader:Issued job 'KtServerService' c/1/jobBsZm2L with job batch system ID: 199 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1435. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' V/k/jobtmtU9v
  1436. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' V/k/jobtmtU9v
  1437. WARNING:toil.leader:V/k/jobtmtU9v INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1438. WARNING:toil.leader:V/k/jobtmtU9v INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1439. WARNING:toil.leader:V/k/jobtmtU9v WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1440. WARNING:toil.leader:V/k/jobtmtU9v INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  1441. WARNING:toil.leader:V/k/jobtmtU9v INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  1442. WARNING:toil.leader:V/k/jobtmtU9v INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.30" port="11878" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmplpJ1kD/873075bd-5377-425a-8113-3ac501cdad91/tmpoA1ZTO.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmplpJ1kD/873075bd-5377-425a-8113-3ac501cdad91/tmp1WTnQX.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmplpJ1kD/873075bd-5377-425a-8113-3ac501cdad91/tmpGs_CaL.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmplpJ1kD/873075bd-5377-425a-8113-3ac501cdad91/tmpnmESmj.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmplpJ1kD/873075bd-5377-425a-8113-3ac501cdad91/tmpYabwvU.tmp']
  1443. WARNING:toil.leader:V/k/jobtmtU9v Set log level to INFO
  1444. WARNING:toil.leader:V/k/jobtmtU9v Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  1445. WARNING:toil.leader:V/k/jobtmtU9v <kyoto_tycoon database_dir="fakepath" host="172.16.13.30" port="11878" />
  1446. WARNING:toil.leader:V/k/jobtmtU9v </st_kv_database_conf>
  1447. WARNING:toil.leader:V/k/jobtmtU9v
  1448. WARNING:toil.leader:V/k/jobtmtU9v Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmplpJ1kD/873075bd-5377-425a-8113-3ac501cdad91/tmpoA1ZTO.tmp
  1449. WARNING:toil.leader:V/k/jobtmtU9v Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmplpJ1kD/873075bd-5377-425a-8113-3ac501cdad91/tmp1WTnQX.tmp
  1450. WARNING:toil.leader:V/k/jobtmtU9v Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmplpJ1kD/873075bd-5377-425a-8113-3ac501cdad91/tmpGs_CaL.tmp
  1451. WARNING:toil.leader:V/k/jobtmtU9v Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmplpJ1kD/873075bd-5377-425a-8113-3ac501cdad91/tmpnmESmj.tmp
  1452. WARNING:toil.leader:V/k/jobtmtU9v Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmplpJ1kD/873075bd-5377-425a-8113-3ac501cdad91/tmpYabwvU.tmp
  1453. WARNING:toil.leader:V/k/jobtmtU9v Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.30 with error: network error
  1454. WARNING:toil.leader:V/k/jobtmtU9v Uncaught exception
  1455. WARNING:toil.leader:V/k/jobtmtU9v Traceback (most recent call last):
  1456. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1457. WARNING:toil.leader:V/k/jobtmtU9v job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1458. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1459. WARNING:toil.leader:V/k/jobtmtU9v super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1460. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1461. WARNING:toil.leader:V/k/jobtmtU9v returnValues = self._run(jobGraph, fileStore)
  1462. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1463. WARNING:toil.leader:V/k/jobtmtU9v return self.run(fileStore)
  1464. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  1465. WARNING:toil.leader:V/k/jobtmtU9v makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  1466. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  1467. WARNING:toil.leader:V/k/jobtmtU9v parameters=["cactus_setup"] + args + sequences)
  1468. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1469. WARNING:toil.leader:V/k/jobtmtU9v raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1470. WARNING:toil.leader:V/k/jobtmtU9v RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.30" port="11878" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmplpJ1kD/873075bd-5377-425a-8113-3ac501cdad91/tmpoA1ZTO.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmplpJ1kD/873075bd-5377-425a-8113-3ac501cdad91/tmp1WTnQX.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmplpJ1kD/873075bd-5377-425a-8113-3ac501cdad91/tmpGs_CaL.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmplpJ1kD/873075bd-5377-425a-8113-3ac501cdad91/tmpnmESmj.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmplpJ1kD/873075bd-5377-425a-8113-3ac501cdad91/tmpYabwvU.tmp'] failed with output:
  1471. WARNING:toil.leader:V/k/jobtmtU9v ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn29.grid.pub.ro
  1472. WARNING:toil.leader:V/k/jobtmtU9v WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' V/k/jobtmtU9v with ID V/k/jobtmtU9v to 3
  1473. INFO:toil.leader:Issued job 'CactusSetupPhase' V/k/jobtmtU9v with job batch system ID: 200 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1474. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' c/1/jobBsZm2L
  1475. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' c/1/jobBsZm2L
  1476. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' c/1/jobBsZm2L with ID c/1/jobBsZm2L to 3
  1477. INFO:toil.leader:Issued job 'KtServerService' c/1/jobBsZm2L with job batch system ID: 201 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1478. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' V/k/jobtmtU9v
  1479. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' V/k/jobtmtU9v
  1480. WARNING:toil.leader:V/k/jobtmtU9v INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1481. WARNING:toil.leader:V/k/jobtmtU9v INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1482. WARNING:toil.leader:V/k/jobtmtU9v WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1483. WARNING:toil.leader:V/k/jobtmtU9v WARNING:toil.fileStore:Detected that job (a/n/jobKwRkH2/g/tmpYTbc6n-_serialiseJob-stream) prematurely terminated. Fixing the state of the job on disk.
  1484. WARNING:toil.leader:V/k/jobtmtU9v INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  1485. WARNING:toil.leader:V/k/jobtmtU9v INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  1486. WARNING:toil.leader:V/k/jobtmtU9v INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.30" port="11878" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpfKv5f0/93905f36-6f60-499a-8d03-b6d9198765ea/tmpK4eOda.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpfKv5f0/93905f36-6f60-499a-8d03-b6d9198765ea/tmpBgkq1N.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpfKv5f0/93905f36-6f60-499a-8d03-b6d9198765ea/tmpQiXCmj.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpfKv5f0/93905f36-6f60-499a-8d03-b6d9198765ea/tmp08uSSx.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpfKv5f0/93905f36-6f60-499a-8d03-b6d9198765ea/tmpl14n5N.tmp']
  1487. WARNING:toil.leader:V/k/jobtmtU9v Set log level to INFO
  1488. WARNING:toil.leader:V/k/jobtmtU9v Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  1489. WARNING:toil.leader:V/k/jobtmtU9v <kyoto_tycoon database_dir="fakepath" host="172.16.13.30" port="11878" />
  1490. WARNING:toil.leader:V/k/jobtmtU9v </st_kv_database_conf>
  1491. WARNING:toil.leader:V/k/jobtmtU9v
  1492. WARNING:toil.leader:V/k/jobtmtU9v Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpfKv5f0/93905f36-6f60-499a-8d03-b6d9198765ea/tmpK4eOda.tmp
  1493. WARNING:toil.leader:V/k/jobtmtU9v Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpfKv5f0/93905f36-6f60-499a-8d03-b6d9198765ea/tmpBgkq1N.tmp
  1494. WARNING:toil.leader:V/k/jobtmtU9v Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpfKv5f0/93905f36-6f60-499a-8d03-b6d9198765ea/tmpQiXCmj.tmp
  1495. WARNING:toil.leader:V/k/jobtmtU9v Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpfKv5f0/93905f36-6f60-499a-8d03-b6d9198765ea/tmp08uSSx.tmp
  1496. WARNING:toil.leader:V/k/jobtmtU9v Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpfKv5f0/93905f36-6f60-499a-8d03-b6d9198765ea/tmpl14n5N.tmp
  1497. WARNING:toil.leader:V/k/jobtmtU9v Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.30 with error: network error
  1498. WARNING:toil.leader:V/k/jobtmtU9v Uncaught exception
  1499. WARNING:toil.leader:V/k/jobtmtU9v Traceback (most recent call last):
  1500. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1501. WARNING:toil.leader:V/k/jobtmtU9v job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1502. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1503. WARNING:toil.leader:V/k/jobtmtU9v super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1504. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1505. WARNING:toil.leader:V/k/jobtmtU9v returnValues = self._run(jobGraph, fileStore)
  1506. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1507. WARNING:toil.leader:V/k/jobtmtU9v return self.run(fileStore)
  1508. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  1509. WARNING:toil.leader:V/k/jobtmtU9v makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  1510. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  1511. WARNING:toil.leader:V/k/jobtmtU9v parameters=["cactus_setup"] + args + sequences)
  1512. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1513. WARNING:toil.leader:V/k/jobtmtU9v raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1514. WARNING:toil.leader:V/k/jobtmtU9v RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.30" port="11878" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpfKv5f0/93905f36-6f60-499a-8d03-b6d9198765ea/tmpK4eOda.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpfKv5f0/93905f36-6f60-499a-8d03-b6d9198765ea/tmpBgkq1N.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpfKv5f0/93905f36-6f60-499a-8d03-b6d9198765ea/tmpQiXCmj.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpfKv5f0/93905f36-6f60-499a-8d03-b6d9198765ea/tmp08uSSx.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpfKv5f0/93905f36-6f60-499a-8d03-b6d9198765ea/tmpl14n5N.tmp'] failed with output:
  1515. WARNING:toil.leader:V/k/jobtmtU9v ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn37.grid.pub.ro
  1516. WARNING:toil.leader:V/k/jobtmtU9v WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' V/k/jobtmtU9v with ID V/k/jobtmtU9v to 2
  1517. INFO:toil.leader:Issued job 'CactusSetupPhase' V/k/jobtmtU9v with job batch system ID: 202 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1518. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' c/1/jobBsZm2L
  1519. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' c/1/jobBsZm2L
  1520. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' c/1/jobBsZm2L with ID c/1/jobBsZm2L to 2
  1521. INFO:toil.leader:Issued job 'KtServerService' c/1/jobBsZm2L with job batch system ID: 203 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1522. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' V/k/jobtmtU9v
  1523. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' V/k/jobtmtU9v
  1524. WARNING:toil.leader:V/k/jobtmtU9v INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1525. WARNING:toil.leader:V/k/jobtmtU9v INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1526. WARNING:toil.leader:V/k/jobtmtU9v WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1527. WARNING:toil.leader:V/k/jobtmtU9v WARNING:toil.fileStore:Detected that job (a/n/jobKwRkH2/g/tmpZ5X_AY-_serialiseJob-stream) prematurely terminated. Fixing the state of the job on disk.
  1528. WARNING:toil.leader:V/k/jobtmtU9v INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  1529. WARNING:toil.leader:V/k/jobtmtU9v INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  1530. WARNING:toil.leader:V/k/jobtmtU9v INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="7115" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-b02a3811-2b63-4208-851d-7815af46a62d/tmp2fuoyQ/4e43ddb8-20b8-47eb-88c2-c787df382bcf/tmpf_3l2y.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-b02a3811-2b63-4208-851d-7815af46a62d/tmp2fuoyQ/4e43ddb8-20b8-47eb-88c2-c787df382bcf/tmp9nPQ48.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-b02a3811-2b63-4208-851d-7815af46a62d/tmp2fuoyQ/4e43ddb8-20b8-47eb-88c2-c787df382bcf/tmpaCqMKU.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-b02a3811-2b63-4208-851d-7815af46a62d/tmp2fuoyQ/4e43ddb8-20b8-47eb-88c2-c787df382bcf/tmpO2hzIH.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-b02a3811-2b63-4208-851d-7815af46a62d/tmp2fuoyQ/4e43ddb8-20b8-47eb-88c2-c787df382bcf/tmpKA1_Kc.tmp']
  1531. WARNING:toil.leader:V/k/jobtmtU9v Set log level to INFO
  1532. WARNING:toil.leader:V/k/jobtmtU9v Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  1533. WARNING:toil.leader:V/k/jobtmtU9v <kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="7115" />
  1534. WARNING:toil.leader:V/k/jobtmtU9v </st_kv_database_conf>
  1535. WARNING:toil.leader:V/k/jobtmtU9v
  1536. WARNING:toil.leader:V/k/jobtmtU9v Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-b02a3811-2b63-4208-851d-7815af46a62d/tmp2fuoyQ/4e43ddb8-20b8-47eb-88c2-c787df382bcf/tmpf_3l2y.tmp
  1537. WARNING:toil.leader:V/k/jobtmtU9v Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-b02a3811-2b63-4208-851d-7815af46a62d/tmp2fuoyQ/4e43ddb8-20b8-47eb-88c2-c787df382bcf/tmp9nPQ48.tmp
  1538. WARNING:toil.leader:V/k/jobtmtU9v Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-b02a3811-2b63-4208-851d-7815af46a62d/tmp2fuoyQ/4e43ddb8-20b8-47eb-88c2-c787df382bcf/tmpaCqMKU.tmp
  1539. WARNING:toil.leader:V/k/jobtmtU9v Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-b02a3811-2b63-4208-851d-7815af46a62d/tmp2fuoyQ/4e43ddb8-20b8-47eb-88c2-c787df382bcf/tmpO2hzIH.tmp
  1540. WARNING:toil.leader:V/k/jobtmtU9v Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-b02a3811-2b63-4208-851d-7815af46a62d/tmp2fuoyQ/4e43ddb8-20b8-47eb-88c2-c787df382bcf/tmpKA1_Kc.tmp
  1541. WARNING:toil.leader:V/k/jobtmtU9v Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.41 with error: network error
  1542. WARNING:toil.leader:V/k/jobtmtU9v Uncaught exception
  1543. WARNING:toil.leader:V/k/jobtmtU9v Traceback (most recent call last):
  1544. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1545. WARNING:toil.leader:V/k/jobtmtU9v job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1546. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1547. WARNING:toil.leader:V/k/jobtmtU9v super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1548. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1549. WARNING:toil.leader:V/k/jobtmtU9v returnValues = self._run(jobGraph, fileStore)
  1550. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1551. WARNING:toil.leader:V/k/jobtmtU9v return self.run(fileStore)
  1552. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  1553. WARNING:toil.leader:V/k/jobtmtU9v makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  1554. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  1555. WARNING:toil.leader:V/k/jobtmtU9v parameters=["cactus_setup"] + args + sequences)
  1556. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1557. WARNING:toil.leader:V/k/jobtmtU9v raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1558. WARNING:toil.leader:V/k/jobtmtU9v RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="7115" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-b02a3811-2b63-4208-851d-7815af46a62d/tmp2fuoyQ/4e43ddb8-20b8-47eb-88c2-c787df382bcf/tmpf_3l2y.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-b02a3811-2b63-4208-851d-7815af46a62d/tmp2fuoyQ/4e43ddb8-20b8-47eb-88c2-c787df382bcf/tmp9nPQ48.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-b02a3811-2b63-4208-851d-7815af46a62d/tmp2fuoyQ/4e43ddb8-20b8-47eb-88c2-c787df382bcf/tmpaCqMKU.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-b02a3811-2b63-4208-851d-7815af46a62d/tmp2fuoyQ/4e43ddb8-20b8-47eb-88c2-c787df382bcf/tmpO2hzIH.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-b02a3811-2b63-4208-851d-7815af46a62d/tmp2fuoyQ/4e43ddb8-20b8-47eb-88c2-c787df382bcf/tmpKA1_Kc.tmp'] failed with output:
  1559. WARNING:toil.leader:V/k/jobtmtU9v ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn39.grid.pub.ro
  1560. WARNING:toil.leader:V/k/jobtmtU9v WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' V/k/jobtmtU9v with ID V/k/jobtmtU9v to 1
  1561. INFO:toil.leader:Issued job 'CactusSetupPhase' V/k/jobtmtU9v with job batch system ID: 204 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1562. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' c/1/jobBsZm2L
  1563. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' c/1/jobBsZm2L
  1564. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' c/1/jobBsZm2L with ID c/1/jobBsZm2L to 1
  1565. INFO:toil.leader:Issued job 'KtServerService' c/1/jobBsZm2L with job batch system ID: 205 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1566. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' V/k/jobtmtU9v
  1567. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' V/k/jobtmtU9v
  1568. WARNING:toil.leader:V/k/jobtmtU9v INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1569. WARNING:toil.leader:V/k/jobtmtU9v INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1570. WARNING:toil.leader:V/k/jobtmtU9v WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1571. WARNING:toil.leader:V/k/jobtmtU9v WARNING:toil.fileStore:Detected that job (a/n/jobKwRkH2/g/tmpLhkUCC-_serialiseJob-stream) prematurely terminated. Fixing the state of the job on disk.
  1572. WARNING:toil.leader:V/k/jobtmtU9v INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  1573. WARNING:toil.leader:V/k/jobtmtU9v INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  1574. WARNING:toil.leader:V/k/jobtmtU9v INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.40" port="4514" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmpnxWqOR/fad3d93d-6359-43f9-ab99-4244a70cd8d9/tmp0sTnMc.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmpnxWqOR/fad3d93d-6359-43f9-ab99-4244a70cd8d9/tmp7Xo2K1.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmpnxWqOR/fad3d93d-6359-43f9-ab99-4244a70cd8d9/tmpeG9cU8.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmpnxWqOR/fad3d93d-6359-43f9-ab99-4244a70cd8d9/tmp78Ozm1.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmpnxWqOR/fad3d93d-6359-43f9-ab99-4244a70cd8d9/tmp03LkyE.tmp']
  1575. WARNING:toil.leader:V/k/jobtmtU9v Set log level to INFO
  1576. WARNING:toil.leader:V/k/jobtmtU9v Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  1577. WARNING:toil.leader:V/k/jobtmtU9v <kyoto_tycoon database_dir="fakepath" host="172.16.13.40" port="4514" />
  1578. WARNING:toil.leader:V/k/jobtmtU9v </st_kv_database_conf>
  1579. WARNING:toil.leader:V/k/jobtmtU9v
  1580. WARNING:toil.leader:V/k/jobtmtU9v Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmpnxWqOR/fad3d93d-6359-43f9-ab99-4244a70cd8d9/tmp0sTnMc.tmp
  1581. WARNING:toil.leader:V/k/jobtmtU9v Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmpnxWqOR/fad3d93d-6359-43f9-ab99-4244a70cd8d9/tmp7Xo2K1.tmp
  1582. WARNING:toil.leader:V/k/jobtmtU9v Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmpnxWqOR/fad3d93d-6359-43f9-ab99-4244a70cd8d9/tmpeG9cU8.tmp
  1583. WARNING:toil.leader:V/k/jobtmtU9v Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmpnxWqOR/fad3d93d-6359-43f9-ab99-4244a70cd8d9/tmp78Ozm1.tmp
  1584. WARNING:toil.leader:V/k/jobtmtU9v Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmpnxWqOR/fad3d93d-6359-43f9-ab99-4244a70cd8d9/tmp03LkyE.tmp
  1585. WARNING:toil.leader:V/k/jobtmtU9v Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.40 with error: network error
  1586. WARNING:toil.leader:V/k/jobtmtU9v Uncaught exception
  1587. WARNING:toil.leader:V/k/jobtmtU9v Traceback (most recent call last):
  1588. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1589. WARNING:toil.leader:V/k/jobtmtU9v job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1590. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1591. WARNING:toil.leader:V/k/jobtmtU9v super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1592. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1593. WARNING:toil.leader:V/k/jobtmtU9v returnValues = self._run(jobGraph, fileStore)
  1594. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1595. WARNING:toil.leader:V/k/jobtmtU9v return self.run(fileStore)
  1596. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  1597. WARNING:toil.leader:V/k/jobtmtU9v makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  1598. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  1599. WARNING:toil.leader:V/k/jobtmtU9v parameters=["cactus_setup"] + args + sequences)
  1600. WARNING:toil.leader:V/k/jobtmtU9v File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1601. WARNING:toil.leader:V/k/jobtmtU9v raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1602. WARNING:toil.leader:V/k/jobtmtU9v RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.40" port="4514" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmpnxWqOR/fad3d93d-6359-43f9-ab99-4244a70cd8d9/tmp0sTnMc.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmpnxWqOR/fad3d93d-6359-43f9-ab99-4244a70cd8d9/tmp7Xo2K1.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmpnxWqOR/fad3d93d-6359-43f9-ab99-4244a70cd8d9/tmpeG9cU8.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmpnxWqOR/fad3d93d-6359-43f9-ab99-4244a70cd8d9/tmp78Ozm1.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmpnxWqOR/fad3d93d-6359-43f9-ab99-4244a70cd8d9/tmp03LkyE.tmp'] failed with output:
  1603. WARNING:toil.leader:V/k/jobtmtU9v ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn30.grid.pub.ro
  1604. WARNING:toil.leader:V/k/jobtmtU9v WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' V/k/jobtmtU9v with ID V/k/jobtmtU9v to 0
  1605. WARNING:toil.leader:Job 'CactusSetupPhase' V/k/jobtmtU9v with ID V/k/jobtmtU9v is completely failed
  1606. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' c/1/jobBsZm2L
  1607. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' c/1/jobBsZm2L
  1608. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' c/1/jobBsZm2L with ID c/1/jobBsZm2L to 0
  1609. WARNING:toil.leader:Job 'KtServerService' c/1/jobBsZm2L with ID c/1/jobBsZm2L is completely failed
  1610. WARNING:toil.leader:Job: a/n/jobKwRkH2 is being restarted as a checkpoint after the total failure of jobs in its subtree.
  1611. INFO:toil.leader:Issued job 'StartPrimaryDB' a/n/jobKwRkH2 with job batch system ID: 206 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1612. INFO:toil.leader:Job ended successfully: 'StartPrimaryDB' a/n/jobKwRkH2
  1613. INFO:toil.leader:Issued job 'KtServerService' U/f/jobtXLYls with job batch system ID: 207 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1614. INFO:toil.leader:Issued job 'CactusSetupPhase' 5/K/job4XwpkJ with job batch system ID: 208 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1615. INFO:toil.leader:Reissued any over long jobs
  1616. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' U/f/jobtXLYls
  1617. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' U/f/jobtXLYls
  1618. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' U/f/jobtXLYls with ID U/f/jobtXLYls to 5
  1619. INFO:toil.leader:Issued job 'KtServerService' U/f/jobtXLYls with job batch system ID: 209 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1620. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' 5/K/job4XwpkJ
  1621. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' 5/K/job4XwpkJ
  1622. WARNING:toil.leader:5/K/job4XwpkJ INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1623. WARNING:toil.leader:5/K/job4XwpkJ INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1624. WARNING:toil.leader:5/K/job4XwpkJ WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1625. WARNING:toil.leader:5/K/job4XwpkJ WARNING:toil.fileStore:Detected that job (a/n/jobKwRkH2/g/tmpwA1eA_-_serialiseJob-stream) prematurely terminated. Fixing the state of the job on disk.
  1626. WARNING:toil.leader:5/K/job4XwpkJ INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  1627. WARNING:toil.leader:5/K/job4XwpkJ INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  1628. WARNING:toil.leader:5/K/job4XwpkJ INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="27088" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpl1t8y7/a84fbabf-0c51-473a-9093-62e559b3e47f/tmpQS0WjS.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpl1t8y7/a84fbabf-0c51-473a-9093-62e559b3e47f/tmpHG6Yri.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpl1t8y7/a84fbabf-0c51-473a-9093-62e559b3e47f/tmpet0R4y.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpl1t8y7/a84fbabf-0c51-473a-9093-62e559b3e47f/tmpkUMSgv.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpl1t8y7/a84fbabf-0c51-473a-9093-62e559b3e47f/tmpx0zR7W.tmp']
  1629. WARNING:toil.leader:5/K/job4XwpkJ Set log level to INFO
  1630. WARNING:toil.leader:5/K/job4XwpkJ Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  1631. WARNING:toil.leader:5/K/job4XwpkJ <kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="27088" />
  1632. WARNING:toil.leader:5/K/job4XwpkJ </st_kv_database_conf>
  1633. WARNING:toil.leader:5/K/job4XwpkJ
  1634. WARNING:toil.leader:5/K/job4XwpkJ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpl1t8y7/a84fbabf-0c51-473a-9093-62e559b3e47f/tmpQS0WjS.tmp
  1635. WARNING:toil.leader:5/K/job4XwpkJ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpl1t8y7/a84fbabf-0c51-473a-9093-62e559b3e47f/tmpHG6Yri.tmp
  1636. WARNING:toil.leader:5/K/job4XwpkJ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpl1t8y7/a84fbabf-0c51-473a-9093-62e559b3e47f/tmpet0R4y.tmp
  1637. WARNING:toil.leader:5/K/job4XwpkJ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpl1t8y7/a84fbabf-0c51-473a-9093-62e559b3e47f/tmpkUMSgv.tmp
  1638. WARNING:toil.leader:5/K/job4XwpkJ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpl1t8y7/a84fbabf-0c51-473a-9093-62e559b3e47f/tmpx0zR7W.tmp
  1639. WARNING:toil.leader:5/K/job4XwpkJ Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.41 with error: network error
  1640. WARNING:toil.leader:5/K/job4XwpkJ Uncaught exception
  1641. WARNING:toil.leader:5/K/job4XwpkJ Traceback (most recent call last):
  1642. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1643. WARNING:toil.leader:5/K/job4XwpkJ job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1644. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1645. WARNING:toil.leader:5/K/job4XwpkJ super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1646. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1647. WARNING:toil.leader:5/K/job4XwpkJ returnValues = self._run(jobGraph, fileStore)
  1648. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1649. WARNING:toil.leader:5/K/job4XwpkJ return self.run(fileStore)
  1650. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  1651. WARNING:toil.leader:5/K/job4XwpkJ makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  1652. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  1653. WARNING:toil.leader:5/K/job4XwpkJ parameters=["cactus_setup"] + args + sequences)
  1654. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1655. WARNING:toil.leader:5/K/job4XwpkJ raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1656. WARNING:toil.leader:5/K/job4XwpkJ RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.41" port="27088" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpl1t8y7/a84fbabf-0c51-473a-9093-62e559b3e47f/tmpQS0WjS.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpl1t8y7/a84fbabf-0c51-473a-9093-62e559b3e47f/tmpHG6Yri.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpl1t8y7/a84fbabf-0c51-473a-9093-62e559b3e47f/tmpet0R4y.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpl1t8y7/a84fbabf-0c51-473a-9093-62e559b3e47f/tmpkUMSgv.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-ee32135c-bc45-4f9b-bd5f-12666414cf0b/tmpl1t8y7/a84fbabf-0c51-473a-9093-62e559b3e47f/tmpx0zR7W.tmp'] failed with output:
  1657. WARNING:toil.leader:5/K/job4XwpkJ ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn41.grid.pub.ro
  1658. WARNING:toil.leader:5/K/job4XwpkJ WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' 5/K/job4XwpkJ with ID 5/K/job4XwpkJ to 5
  1659. INFO:toil.leader:Issued job 'CactusSetupPhase' 5/K/job4XwpkJ with job batch system ID: 210 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1660. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' U/f/jobtXLYls
  1661. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' U/f/jobtXLYls
  1662. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' U/f/jobtXLYls with ID U/f/jobtXLYls to 4
  1663. INFO:toil.leader:Issued job 'KtServerService' U/f/jobtXLYls with job batch system ID: 211 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1664. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' 5/K/job4XwpkJ
  1665. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' 5/K/job4XwpkJ
  1666. WARNING:toil.leader:5/K/job4XwpkJ INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1667. WARNING:toil.leader:5/K/job4XwpkJ INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1668. WARNING:toil.leader:5/K/job4XwpkJ WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1669. WARNING:toil.leader:5/K/job4XwpkJ INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  1670. WARNING:toil.leader:5/K/job4XwpkJ INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  1671. WARNING:toil.leader:5/K/job4XwpkJ INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="11015" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmp1HZkP8/6b9ec5b0-2e88-484f-b3d2-01f435758663/tmpkYfhu7.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmp1HZkP8/6b9ec5b0-2e88-484f-b3d2-01f435758663/tmp2KeZWo.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmp1HZkP8/6b9ec5b0-2e88-484f-b3d2-01f435758663/tmpHGrnLa.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmp1HZkP8/6b9ec5b0-2e88-484f-b3d2-01f435758663/tmph2TuFl.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmp1HZkP8/6b9ec5b0-2e88-484f-b3d2-01f435758663/tmp44uFky.tmp']
  1672. WARNING:toil.leader:5/K/job4XwpkJ Set log level to INFO
  1673. WARNING:toil.leader:5/K/job4XwpkJ Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  1674. WARNING:toil.leader:5/K/job4XwpkJ <kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="11015" />
  1675. WARNING:toil.leader:5/K/job4XwpkJ </st_kv_database_conf>
  1676. WARNING:toil.leader:5/K/job4XwpkJ
  1677. WARNING:toil.leader:5/K/job4XwpkJ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmp1HZkP8/6b9ec5b0-2e88-484f-b3d2-01f435758663/tmpkYfhu7.tmp
  1678. WARNING:toil.leader:5/K/job4XwpkJ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmp1HZkP8/6b9ec5b0-2e88-484f-b3d2-01f435758663/tmp2KeZWo.tmp
  1679. WARNING:toil.leader:5/K/job4XwpkJ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmp1HZkP8/6b9ec5b0-2e88-484f-b3d2-01f435758663/tmpHGrnLa.tmp
  1680. WARNING:toil.leader:5/K/job4XwpkJ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmp1HZkP8/6b9ec5b0-2e88-484f-b3d2-01f435758663/tmph2TuFl.tmp
  1681. WARNING:toil.leader:5/K/job4XwpkJ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmp1HZkP8/6b9ec5b0-2e88-484f-b3d2-01f435758663/tmp44uFky.tmp
  1682. WARNING:toil.leader:5/K/job4XwpkJ Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.35 with error: network error
  1683. WARNING:toil.leader:5/K/job4XwpkJ Uncaught exception
  1684. WARNING:toil.leader:5/K/job4XwpkJ Traceback (most recent call last):
  1685. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1686. WARNING:toil.leader:5/K/job4XwpkJ job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1687. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1688. WARNING:toil.leader:5/K/job4XwpkJ super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1689. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1690. WARNING:toil.leader:5/K/job4XwpkJ returnValues = self._run(jobGraph, fileStore)
  1691. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1692. WARNING:toil.leader:5/K/job4XwpkJ return self.run(fileStore)
  1693. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  1694. WARNING:toil.leader:5/K/job4XwpkJ makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  1695. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  1696. WARNING:toil.leader:5/K/job4XwpkJ parameters=["cactus_setup"] + args + sequences)
  1697. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1698. WARNING:toil.leader:5/K/job4XwpkJ raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1699. WARNING:toil.leader:5/K/job4XwpkJ RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="11015" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmp1HZkP8/6b9ec5b0-2e88-484f-b3d2-01f435758663/tmpkYfhu7.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmp1HZkP8/6b9ec5b0-2e88-484f-b3d2-01f435758663/tmp2KeZWo.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmp1HZkP8/6b9ec5b0-2e88-484f-b3d2-01f435758663/tmpHGrnLa.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmp1HZkP8/6b9ec5b0-2e88-484f-b3d2-01f435758663/tmph2TuFl.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmp1HZkP8/6b9ec5b0-2e88-484f-b3d2-01f435758663/tmp44uFky.tmp'] failed with output:
  1700. WARNING:toil.leader:5/K/job4XwpkJ ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn29.grid.pub.ro
  1701. WARNING:toil.leader:5/K/job4XwpkJ WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' 5/K/job4XwpkJ with ID 5/K/job4XwpkJ to 4
  1702. INFO:toil.leader:Issued job 'CactusSetupPhase' 5/K/job4XwpkJ with job batch system ID: 212 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1703. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' U/f/jobtXLYls
  1704. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' U/f/jobtXLYls
  1705. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' U/f/jobtXLYls with ID U/f/jobtXLYls to 3
  1706. INFO:toil.leader:Issued job 'KtServerService' U/f/jobtXLYls with job batch system ID: 213 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1707. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' 5/K/job4XwpkJ
  1708. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' 5/K/job4XwpkJ
  1709. WARNING:toil.leader:5/K/job4XwpkJ INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1710. WARNING:toil.leader:5/K/job4XwpkJ INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1711. WARNING:toil.leader:5/K/job4XwpkJ WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1712. WARNING:toil.leader:5/K/job4XwpkJ WARNING:toil.fileStore:Detected that job (a/n/jobKwRkH2/g/tmpLhkUCC-_serialiseJob-stream) prematurely terminated. Fixing the state of the job on disk.
  1713. WARNING:toil.leader:5/K/job4XwpkJ INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  1714. WARNING:toil.leader:5/K/job4XwpkJ INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  1715. WARNING:toil.leader:5/K/job4XwpkJ INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="11015" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpdE4tAd/cc64af61-4160-4d50-a6b5-633f3f00b052/tmpnEQeSR.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpdE4tAd/cc64af61-4160-4d50-a6b5-633f3f00b052/tmp2tMp2P.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpdE4tAd/cc64af61-4160-4d50-a6b5-633f3f00b052/tmpWKPaSQ.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpdE4tAd/cc64af61-4160-4d50-a6b5-633f3f00b052/tmptv4_ux.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpdE4tAd/cc64af61-4160-4d50-a6b5-633f3f00b052/tmp2JADvD.tmp']
  1716. WARNING:toil.leader:5/K/job4XwpkJ Set log level to INFO
  1717. WARNING:toil.leader:5/K/job4XwpkJ Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  1718. WARNING:toil.leader:5/K/job4XwpkJ <kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="11015" />
  1719. WARNING:toil.leader:5/K/job4XwpkJ </st_kv_database_conf>
  1720. WARNING:toil.leader:5/K/job4XwpkJ
  1721. WARNING:toil.leader:5/K/job4XwpkJ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpdE4tAd/cc64af61-4160-4d50-a6b5-633f3f00b052/tmpnEQeSR.tmp
  1722. WARNING:toil.leader:5/K/job4XwpkJ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpdE4tAd/cc64af61-4160-4d50-a6b5-633f3f00b052/tmp2tMp2P.tmp
  1723. WARNING:toil.leader:5/K/job4XwpkJ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpdE4tAd/cc64af61-4160-4d50-a6b5-633f3f00b052/tmpWKPaSQ.tmp
  1724. WARNING:toil.leader:5/K/job4XwpkJ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpdE4tAd/cc64af61-4160-4d50-a6b5-633f3f00b052/tmptv4_ux.tmp
  1725. WARNING:toil.leader:5/K/job4XwpkJ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpdE4tAd/cc64af61-4160-4d50-a6b5-633f3f00b052/tmp2JADvD.tmp
  1726. WARNING:toil.leader:5/K/job4XwpkJ Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.35 with error: network error
  1727. WARNING:toil.leader:5/K/job4XwpkJ Uncaught exception
  1728. WARNING:toil.leader:5/K/job4XwpkJ Traceback (most recent call last):
  1729. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1730. WARNING:toil.leader:5/K/job4XwpkJ job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1731. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1732. WARNING:toil.leader:5/K/job4XwpkJ super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1733. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1734. WARNING:toil.leader:5/K/job4XwpkJ returnValues = self._run(jobGraph, fileStore)
  1735. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1736. WARNING:toil.leader:5/K/job4XwpkJ return self.run(fileStore)
  1737. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  1738. WARNING:toil.leader:5/K/job4XwpkJ makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  1739. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  1740. WARNING:toil.leader:5/K/job4XwpkJ parameters=["cactus_setup"] + args + sequences)
  1741. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1742. WARNING:toil.leader:5/K/job4XwpkJ raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1743. WARNING:toil.leader:5/K/job4XwpkJ RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.35" port="11015" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpdE4tAd/cc64af61-4160-4d50-a6b5-633f3f00b052/tmpnEQeSR.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpdE4tAd/cc64af61-4160-4d50-a6b5-633f3f00b052/tmp2tMp2P.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpdE4tAd/cc64af61-4160-4d50-a6b5-633f3f00b052/tmpWKPaSQ.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpdE4tAd/cc64af61-4160-4d50-a6b5-633f3f00b052/tmptv4_ux.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpdE4tAd/cc64af61-4160-4d50-a6b5-633f3f00b052/tmp2JADvD.tmp'] failed with output:
  1744. WARNING:toil.leader:5/K/job4XwpkJ ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn40.grid.pub.ro
  1745. WARNING:toil.leader:5/K/job4XwpkJ WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' 5/K/job4XwpkJ with ID 5/K/job4XwpkJ to 3
  1746. INFO:toil.leader:Issued job 'CactusSetupPhase' 5/K/job4XwpkJ with job batch system ID: 214 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1747. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' U/f/jobtXLYls
  1748. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' U/f/jobtXLYls
  1749. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' U/f/jobtXLYls with ID U/f/jobtXLYls to 2
  1750. INFO:toil.leader:Issued job 'KtServerService' U/f/jobtXLYls with job batch system ID: 215 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1751. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' 5/K/job4XwpkJ
  1752. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' 5/K/job4XwpkJ
  1753. WARNING:toil.leader:5/K/job4XwpkJ INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1754. WARNING:toil.leader:5/K/job4XwpkJ INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1755. WARNING:toil.leader:5/K/job4XwpkJ WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1756. WARNING:toil.leader:5/K/job4XwpkJ INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  1757. WARNING:toil.leader:5/K/job4XwpkJ INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  1758. WARNING:toil.leader:5/K/job4XwpkJ INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.34" port="31063" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmpJ5spBY/216546c1-9520-46e6-9982-f467dca64295/tmp4JgiBR.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmpJ5spBY/216546c1-9520-46e6-9982-f467dca64295/tmpK_TKqc.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmpJ5spBY/216546c1-9520-46e6-9982-f467dca64295/tmptB8_Od.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmpJ5spBY/216546c1-9520-46e6-9982-f467dca64295/tmpiKcbvG.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmpJ5spBY/216546c1-9520-46e6-9982-f467dca64295/tmp55IN42.tmp']
  1759. WARNING:toil.leader:5/K/job4XwpkJ Set log level to INFO
  1760. WARNING:toil.leader:5/K/job4XwpkJ Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  1761. WARNING:toil.leader:5/K/job4XwpkJ <kyoto_tycoon database_dir="fakepath" host="172.16.13.34" port="31063" />
  1762. WARNING:toil.leader:5/K/job4XwpkJ </st_kv_database_conf>
  1763. WARNING:toil.leader:5/K/job4XwpkJ
  1764. WARNING:toil.leader:5/K/job4XwpkJ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmpJ5spBY/216546c1-9520-46e6-9982-f467dca64295/tmp4JgiBR.tmp
  1765. WARNING:toil.leader:5/K/job4XwpkJ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmpJ5spBY/216546c1-9520-46e6-9982-f467dca64295/tmpK_TKqc.tmp
  1766. WARNING:toil.leader:5/K/job4XwpkJ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmpJ5spBY/216546c1-9520-46e6-9982-f467dca64295/tmptB8_Od.tmp
  1767. WARNING:toil.leader:5/K/job4XwpkJ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmpJ5spBY/216546c1-9520-46e6-9982-f467dca64295/tmpiKcbvG.tmp
  1768. WARNING:toil.leader:5/K/job4XwpkJ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmpJ5spBY/216546c1-9520-46e6-9982-f467dca64295/tmp55IN42.tmp
  1769. WARNING:toil.leader:5/K/job4XwpkJ Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.34 with error: network error
  1770. WARNING:toil.leader:5/K/job4XwpkJ Uncaught exception
  1771. WARNING:toil.leader:5/K/job4XwpkJ Traceback (most recent call last):
  1772. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1773. WARNING:toil.leader:5/K/job4XwpkJ job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1774. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1775. WARNING:toil.leader:5/K/job4XwpkJ super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1776. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1777. WARNING:toil.leader:5/K/job4XwpkJ returnValues = self._run(jobGraph, fileStore)
  1778. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1779. WARNING:toil.leader:5/K/job4XwpkJ return self.run(fileStore)
  1780. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  1781. WARNING:toil.leader:5/K/job4XwpkJ makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  1782. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  1783. WARNING:toil.leader:5/K/job4XwpkJ parameters=["cactus_setup"] + args + sequences)
  1784. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1785. WARNING:toil.leader:5/K/job4XwpkJ raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1786. WARNING:toil.leader:5/K/job4XwpkJ RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.34" port="31063" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmpJ5spBY/216546c1-9520-46e6-9982-f467dca64295/tmp4JgiBR.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmpJ5spBY/216546c1-9520-46e6-9982-f467dca64295/tmpK_TKqc.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmpJ5spBY/216546c1-9520-46e6-9982-f467dca64295/tmptB8_Od.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmpJ5spBY/216546c1-9520-46e6-9982-f467dca64295/tmpiKcbvG.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-922cc1a4-c8a6-4178-9bce-9d5e6fde2fc2/tmpJ5spBY/216546c1-9520-46e6-9982-f467dca64295/tmp55IN42.tmp'] failed with output:
  1787. WARNING:toil.leader:5/K/job4XwpkJ ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn30.grid.pub.ro
  1788. WARNING:toil.leader:5/K/job4XwpkJ WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' 5/K/job4XwpkJ with ID 5/K/job4XwpkJ to 2
  1789. INFO:toil.leader:Issued job 'CactusSetupPhase' 5/K/job4XwpkJ with job batch system ID: 216 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1790. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' U/f/jobtXLYls
  1791. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' U/f/jobtXLYls
  1792. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' U/f/jobtXLYls with ID U/f/jobtXLYls to 1
  1793. INFO:toil.leader:Issued job 'KtServerService' U/f/jobtXLYls with job batch system ID: 217 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1794. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' 5/K/job4XwpkJ
  1795. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' 5/K/job4XwpkJ
  1796. WARNING:toil.leader:5/K/job4XwpkJ INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1797. WARNING:toil.leader:5/K/job4XwpkJ INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1798. WARNING:toil.leader:5/K/job4XwpkJ WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1799. WARNING:toil.leader:5/K/job4XwpkJ INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  1800. WARNING:toil.leader:5/K/job4XwpkJ INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  1801. WARNING:toil.leader:5/K/job4XwpkJ INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.39" port="4317" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpTw1nk8/331e0ffb-a80d-421a-aedc-2416c3a82062/tmpbVrNXM.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpTw1nk8/331e0ffb-a80d-421a-aedc-2416c3a82062/tmpr8ZjYG.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpTw1nk8/331e0ffb-a80d-421a-aedc-2416c3a82062/tmpbB8HFa.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpTw1nk8/331e0ffb-a80d-421a-aedc-2416c3a82062/tmp7VBp1S.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpTw1nk8/331e0ffb-a80d-421a-aedc-2416c3a82062/tmpk6ndWY.tmp']
  1802. WARNING:toil.leader:5/K/job4XwpkJ Set log level to INFO
  1803. WARNING:toil.leader:5/K/job4XwpkJ Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  1804. WARNING:toil.leader:5/K/job4XwpkJ <kyoto_tycoon database_dir="fakepath" host="172.16.13.39" port="4317" />
  1805. WARNING:toil.leader:5/K/job4XwpkJ </st_kv_database_conf>
  1806. WARNING:toil.leader:5/K/job4XwpkJ
  1807. WARNING:toil.leader:5/K/job4XwpkJ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpTw1nk8/331e0ffb-a80d-421a-aedc-2416c3a82062/tmpbVrNXM.tmp
  1808. WARNING:toil.leader:5/K/job4XwpkJ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpTw1nk8/331e0ffb-a80d-421a-aedc-2416c3a82062/tmpr8ZjYG.tmp
  1809. WARNING:toil.leader:5/K/job4XwpkJ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpTw1nk8/331e0ffb-a80d-421a-aedc-2416c3a82062/tmpbB8HFa.tmp
  1810. WARNING:toil.leader:5/K/job4XwpkJ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpTw1nk8/331e0ffb-a80d-421a-aedc-2416c3a82062/tmp7VBp1S.tmp
  1811. WARNING:toil.leader:5/K/job4XwpkJ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpTw1nk8/331e0ffb-a80d-421a-aedc-2416c3a82062/tmpk6ndWY.tmp
  1812. WARNING:toil.leader:5/K/job4XwpkJ Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.39 with error: network error
  1813. WARNING:toil.leader:5/K/job4XwpkJ Uncaught exception
  1814. WARNING:toil.leader:5/K/job4XwpkJ Traceback (most recent call last):
  1815. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1816. WARNING:toil.leader:5/K/job4XwpkJ job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1817. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1818. WARNING:toil.leader:5/K/job4XwpkJ super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1819. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1820. WARNING:toil.leader:5/K/job4XwpkJ returnValues = self._run(jobGraph, fileStore)
  1821. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1822. WARNING:toil.leader:5/K/job4XwpkJ return self.run(fileStore)
  1823. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  1824. WARNING:toil.leader:5/K/job4XwpkJ makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  1825. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  1826. WARNING:toil.leader:5/K/job4XwpkJ parameters=["cactus_setup"] + args + sequences)
  1827. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1828. WARNING:toil.leader:5/K/job4XwpkJ raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1829. WARNING:toil.leader:5/K/job4XwpkJ RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.39" port="4317" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpTw1nk8/331e0ffb-a80d-421a-aedc-2416c3a82062/tmpbVrNXM.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpTw1nk8/331e0ffb-a80d-421a-aedc-2416c3a82062/tmpr8ZjYG.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpTw1nk8/331e0ffb-a80d-421a-aedc-2416c3a82062/tmpbB8HFa.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpTw1nk8/331e0ffb-a80d-421a-aedc-2416c3a82062/tmp7VBp1S.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmpTw1nk8/331e0ffb-a80d-421a-aedc-2416c3a82062/tmpk6ndWY.tmp'] failed with output:
  1830. WARNING:toil.leader:5/K/job4XwpkJ ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn40.grid.pub.ro
  1831. WARNING:toil.leader:5/K/job4XwpkJ WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' 5/K/job4XwpkJ with ID 5/K/job4XwpkJ to 1
  1832. INFO:toil.leader:Issued job 'CactusSetupPhase' 5/K/job4XwpkJ with job batch system ID: 218 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1833. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' U/f/jobtXLYls
  1834. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' U/f/jobtXLYls
  1835. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' U/f/jobtXLYls with ID U/f/jobtXLYls to 0
  1836. WARNING:toil.leader:Job 'KtServerService' U/f/jobtXLYls with ID U/f/jobtXLYls is completely failed
  1837. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' 5/K/job4XwpkJ
  1838. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' 5/K/job4XwpkJ
  1839. WARNING:toil.leader:5/K/job4XwpkJ INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1840. WARNING:toil.leader:5/K/job4XwpkJ INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1841. WARNING:toil.leader:5/K/job4XwpkJ WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1842. WARNING:toil.leader:5/K/job4XwpkJ WARNING:toil.fileStore:Detected that job (a/n/jobKwRkH2/g/tmpwA1eA_-_serialiseJob-stream) prematurely terminated. Fixing the state of the job on disk.
  1843. WARNING:toil.leader:5/K/job4XwpkJ INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  1844. WARNING:toil.leader:5/K/job4XwpkJ INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  1845. WARNING:toil.leader:5/K/job4XwpkJ INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.37" port="13710" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpfzwyfX/efc0ffc7-7eaa-4081-a844-6b5685bcc416/tmpTH2IwD.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpfzwyfX/efc0ffc7-7eaa-4081-a844-6b5685bcc416/tmpAqTdBw.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpfzwyfX/efc0ffc7-7eaa-4081-a844-6b5685bcc416/tmpN8v7MP.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpfzwyfX/efc0ffc7-7eaa-4081-a844-6b5685bcc416/tmpCX5ulI.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpfzwyfX/efc0ffc7-7eaa-4081-a844-6b5685bcc416/tmpFbxkLm.tmp']
  1846. WARNING:toil.leader:5/K/job4XwpkJ Set log level to INFO
  1847. WARNING:toil.leader:5/K/job4XwpkJ Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  1848. WARNING:toil.leader:5/K/job4XwpkJ <kyoto_tycoon database_dir="fakepath" host="172.16.13.37" port="13710" />
  1849. WARNING:toil.leader:5/K/job4XwpkJ </st_kv_database_conf>
  1850. WARNING:toil.leader:5/K/job4XwpkJ
  1851. WARNING:toil.leader:5/K/job4XwpkJ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpfzwyfX/efc0ffc7-7eaa-4081-a844-6b5685bcc416/tmpTH2IwD.tmp
  1852. WARNING:toil.leader:5/K/job4XwpkJ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpfzwyfX/efc0ffc7-7eaa-4081-a844-6b5685bcc416/tmpAqTdBw.tmp
  1853. WARNING:toil.leader:5/K/job4XwpkJ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpfzwyfX/efc0ffc7-7eaa-4081-a844-6b5685bcc416/tmpN8v7MP.tmp
  1854. WARNING:toil.leader:5/K/job4XwpkJ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpfzwyfX/efc0ffc7-7eaa-4081-a844-6b5685bcc416/tmpCX5ulI.tmp
  1855. WARNING:toil.leader:5/K/job4XwpkJ Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpfzwyfX/efc0ffc7-7eaa-4081-a844-6b5685bcc416/tmpFbxkLm.tmp
  1856. WARNING:toil.leader:5/K/job4XwpkJ Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.37 with error: network error
  1857. WARNING:toil.leader:5/K/job4XwpkJ Uncaught exception
  1858. WARNING:toil.leader:5/K/job4XwpkJ Traceback (most recent call last):
  1859. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1860. WARNING:toil.leader:5/K/job4XwpkJ job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1861. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1862. WARNING:toil.leader:5/K/job4XwpkJ super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1863. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1864. WARNING:toil.leader:5/K/job4XwpkJ returnValues = self._run(jobGraph, fileStore)
  1865. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1866. WARNING:toil.leader:5/K/job4XwpkJ return self.run(fileStore)
  1867. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  1868. WARNING:toil.leader:5/K/job4XwpkJ makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  1869. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  1870. WARNING:toil.leader:5/K/job4XwpkJ parameters=["cactus_setup"] + args + sequences)
  1871. WARNING:toil.leader:5/K/job4XwpkJ File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1872. WARNING:toil.leader:5/K/job4XwpkJ raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1873. WARNING:toil.leader:5/K/job4XwpkJ RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.37" port="13710" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpfzwyfX/efc0ffc7-7eaa-4081-a844-6b5685bcc416/tmpTH2IwD.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpfzwyfX/efc0ffc7-7eaa-4081-a844-6b5685bcc416/tmpAqTdBw.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpfzwyfX/efc0ffc7-7eaa-4081-a844-6b5685bcc416/tmpN8v7MP.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpfzwyfX/efc0ffc7-7eaa-4081-a844-6b5685bcc416/tmpCX5ulI.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpfzwyfX/efc0ffc7-7eaa-4081-a844-6b5685bcc416/tmpFbxkLm.tmp'] failed with output:
  1874. WARNING:toil.leader:5/K/job4XwpkJ ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn37.grid.pub.ro
  1875. WARNING:toil.leader:5/K/job4XwpkJ WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' 5/K/job4XwpkJ with ID 5/K/job4XwpkJ to 0
  1876. WARNING:toil.leader:Job 'CactusSetupPhase' 5/K/job4XwpkJ with ID 5/K/job4XwpkJ is completely failed
  1877. WARNING:toil.leader:Job: a/n/jobKwRkH2 is being restarted as a checkpoint after the total failure of jobs in its subtree.
  1878. INFO:toil.leader:Issued job 'StartPrimaryDB' a/n/jobKwRkH2 with job batch system ID: 219 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1879. WARNING:toil.leader:Job: a/n/jobKwRkH2 is being restarted as a checkpoint after the total failure of jobs in its subtree.
  1880. INFO:toil.leader:Issued job 'StartPrimaryDB' a/n/jobKwRkH2 with job batch system ID: 220 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1881. INFO:toil.leader:Job ended successfully: 'StartPrimaryDB' a/n/jobKwRkH2
  1882. INFO:toil.leader:Issued job 'KtServerService' t/N/job7paZYg with job batch system ID: 221 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1883. INFO:toil.leader:Job ended successfully: 'StartPrimaryDB' a/n/jobKwRkH2
  1884. INFO:toil.leader:Issued job 'CactusSetupPhase' W/W/jobNeEy8O with job batch system ID: 222 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1885. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' t/N/job7paZYg
  1886. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' t/N/job7paZYg
  1887. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' t/N/job7paZYg with ID t/N/job7paZYg to 5
  1888. INFO:toil.leader:Issued job 'KtServerService' t/N/job7paZYg with job batch system ID: 223 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1889. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' W/W/jobNeEy8O
  1890. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' W/W/jobNeEy8O
  1891. WARNING:toil.leader:W/W/jobNeEy8O INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1892. WARNING:toil.leader:W/W/jobNeEy8O INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1893. WARNING:toil.leader:W/W/jobNeEy8O WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1894. WARNING:toil.leader:W/W/jobNeEy8O WARNING:toil.fileStore:Detected that job (a/n/jobKwRkH2/g/tmpkoE42E-_serialiseJob-stream) prematurely terminated. Fixing the state of the job on disk.
  1895. WARNING:toil.leader:W/W/jobNeEy8O INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  1896. WARNING:toil.leader:W/W/jobNeEy8O INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  1897. WARNING:toil.leader:W/W/jobNeEy8O INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.40" port="27754" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmp3Vx_bH/f641a8fb-7dab-4192-999e-56ed0bd2ff3f/tmp9rPwcN.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmp3Vx_bH/f641a8fb-7dab-4192-999e-56ed0bd2ff3f/tmp6g4Dhk.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmp3Vx_bH/f641a8fb-7dab-4192-999e-56ed0bd2ff3f/tmpKWnjh0.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmp3Vx_bH/f641a8fb-7dab-4192-999e-56ed0bd2ff3f/tmp4J7ZRh.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmp3Vx_bH/f641a8fb-7dab-4192-999e-56ed0bd2ff3f/tmpQGO55k.tmp']
  1898. WARNING:toil.leader:W/W/jobNeEy8O Set log level to INFO
  1899. WARNING:toil.leader:W/W/jobNeEy8O Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  1900. WARNING:toil.leader:W/W/jobNeEy8O <kyoto_tycoon database_dir="fakepath" host="172.16.13.40" port="27754" />
  1901. WARNING:toil.leader:W/W/jobNeEy8O </st_kv_database_conf>
  1902. WARNING:toil.leader:W/W/jobNeEy8O
  1903. WARNING:toil.leader:W/W/jobNeEy8O Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmp3Vx_bH/f641a8fb-7dab-4192-999e-56ed0bd2ff3f/tmp9rPwcN.tmp
  1904. WARNING:toil.leader:W/W/jobNeEy8O Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmp3Vx_bH/f641a8fb-7dab-4192-999e-56ed0bd2ff3f/tmp6g4Dhk.tmp
  1905. WARNING:toil.leader:W/W/jobNeEy8O Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmp3Vx_bH/f641a8fb-7dab-4192-999e-56ed0bd2ff3f/tmpKWnjh0.tmp
  1906. WARNING:toil.leader:W/W/jobNeEy8O Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmp3Vx_bH/f641a8fb-7dab-4192-999e-56ed0bd2ff3f/tmp4J7ZRh.tmp
  1907. WARNING:toil.leader:W/W/jobNeEy8O Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmp3Vx_bH/f641a8fb-7dab-4192-999e-56ed0bd2ff3f/tmpQGO55k.tmp
  1908. WARNING:toil.leader:W/W/jobNeEy8O Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.40 with error: network error
  1909. WARNING:toil.leader:W/W/jobNeEy8O Uncaught exception
  1910. WARNING:toil.leader:W/W/jobNeEy8O Traceback (most recent call last):
  1911. WARNING:toil.leader:W/W/jobNeEy8O File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1912. WARNING:toil.leader:W/W/jobNeEy8O job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1913. WARNING:toil.leader:W/W/jobNeEy8O File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1914. WARNING:toil.leader:W/W/jobNeEy8O super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1915. WARNING:toil.leader:W/W/jobNeEy8O File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1916. WARNING:toil.leader:W/W/jobNeEy8O returnValues = self._run(jobGraph, fileStore)
  1917. WARNING:toil.leader:W/W/jobNeEy8O File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1918. WARNING:toil.leader:W/W/jobNeEy8O return self.run(fileStore)
  1919. WARNING:toil.leader:W/W/jobNeEy8O File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  1920. WARNING:toil.leader:W/W/jobNeEy8O makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  1921. WARNING:toil.leader:W/W/jobNeEy8O File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  1922. WARNING:toil.leader:W/W/jobNeEy8O parameters=["cactus_setup"] + args + sequences)
  1923. WARNING:toil.leader:W/W/jobNeEy8O File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1924. WARNING:toil.leader:W/W/jobNeEy8O raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1925. WARNING:toil.leader:W/W/jobNeEy8O RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.40" port="27754" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmp3Vx_bH/f641a8fb-7dab-4192-999e-56ed0bd2ff3f/tmp9rPwcN.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmp3Vx_bH/f641a8fb-7dab-4192-999e-56ed0bd2ff3f/tmp6g4Dhk.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmp3Vx_bH/f641a8fb-7dab-4192-999e-56ed0bd2ff3f/tmpKWnjh0.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmp3Vx_bH/f641a8fb-7dab-4192-999e-56ed0bd2ff3f/tmp4J7ZRh.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-e0c05519-3899-4a42-9065-c10aa0c432fb/tmp3Vx_bH/f641a8fb-7dab-4192-999e-56ed0bd2ff3f/tmpQGO55k.tmp'] failed with output:
  1926. WARNING:toil.leader:W/W/jobNeEy8O ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn40.grid.pub.ro
  1927. WARNING:toil.leader:W/W/jobNeEy8O WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' W/W/jobNeEy8O with ID W/W/jobNeEy8O to 5
  1928. INFO:toil.leader:Issued job 'CactusSetupPhase' W/W/jobNeEy8O with job batch system ID: 224 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1929. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' W/W/jobNeEy8O
  1930. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' W/W/jobNeEy8O
  1931. WARNING:toil.leader:W/W/jobNeEy8O INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1932. WARNING:toil.leader:W/W/jobNeEy8O INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1933. WARNING:toil.leader:W/W/jobNeEy8O WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1934. WARNING:toil.leader:W/W/jobNeEy8O INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  1935. WARNING:toil.leader:W/W/jobNeEy8O INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  1936. WARNING:toil.leader:W/W/jobNeEy8O INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.37" port="10193" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpMulWMg/b6bca5a9-71a6-4561-90e6-4315314d0487/tmpBhXErZ.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpMulWMg/b6bca5a9-71a6-4561-90e6-4315314d0487/tmpKtxRQI.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpMulWMg/b6bca5a9-71a6-4561-90e6-4315314d0487/tmpUhN7gv.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpMulWMg/b6bca5a9-71a6-4561-90e6-4315314d0487/tmpKT7_L5.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpMulWMg/b6bca5a9-71a6-4561-90e6-4315314d0487/tmp9hfIYu.tmp']
  1937. WARNING:toil.leader:W/W/jobNeEy8O Set log level to INFO
  1938. WARNING:toil.leader:W/W/jobNeEy8O Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  1939. WARNING:toil.leader:W/W/jobNeEy8O <kyoto_tycoon database_dir="fakepath" host="172.16.13.37" port="10193" />
  1940. WARNING:toil.leader:W/W/jobNeEy8O </st_kv_database_conf>
  1941. WARNING:toil.leader:W/W/jobNeEy8O
  1942. WARNING:toil.leader:W/W/jobNeEy8O Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpMulWMg/b6bca5a9-71a6-4561-90e6-4315314d0487/tmpBhXErZ.tmp
  1943. WARNING:toil.leader:W/W/jobNeEy8O Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpMulWMg/b6bca5a9-71a6-4561-90e6-4315314d0487/tmpKtxRQI.tmp
  1944. WARNING:toil.leader:W/W/jobNeEy8O Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpMulWMg/b6bca5a9-71a6-4561-90e6-4315314d0487/tmpUhN7gv.tmp
  1945. WARNING:toil.leader:W/W/jobNeEy8O Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpMulWMg/b6bca5a9-71a6-4561-90e6-4315314d0487/tmpKT7_L5.tmp
  1946. WARNING:toil.leader:W/W/jobNeEy8O Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpMulWMg/b6bca5a9-71a6-4561-90e6-4315314d0487/tmp9hfIYu.tmp
  1947. WARNING:toil.leader:W/W/jobNeEy8O Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.37 with error: network error
  1948. WARNING:toil.leader:W/W/jobNeEy8O Uncaught exception
  1949. WARNING:toil.leader:W/W/jobNeEy8O Traceback (most recent call last):
  1950. WARNING:toil.leader:W/W/jobNeEy8O File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  1951. WARNING:toil.leader:W/W/jobNeEy8O job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1952. WARNING:toil.leader:W/W/jobNeEy8O File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  1953. WARNING:toil.leader:W/W/jobNeEy8O super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  1954. WARNING:toil.leader:W/W/jobNeEy8O File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  1955. WARNING:toil.leader:W/W/jobNeEy8O returnValues = self._run(jobGraph, fileStore)
  1956. WARNING:toil.leader:W/W/jobNeEy8O File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  1957. WARNING:toil.leader:W/W/jobNeEy8O return self.run(fileStore)
  1958. WARNING:toil.leader:W/W/jobNeEy8O File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  1959. WARNING:toil.leader:W/W/jobNeEy8O makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  1960. WARNING:toil.leader:W/W/jobNeEy8O File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  1961. WARNING:toil.leader:W/W/jobNeEy8O parameters=["cactus_setup"] + args + sequences)
  1962. WARNING:toil.leader:W/W/jobNeEy8O File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  1963. WARNING:toil.leader:W/W/jobNeEy8O raise RuntimeError("Command %s failed with output: %s" % (call, output))
  1964. WARNING:toil.leader:W/W/jobNeEy8O RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.37" port="10193" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpMulWMg/b6bca5a9-71a6-4561-90e6-4315314d0487/tmpBhXErZ.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpMulWMg/b6bca5a9-71a6-4561-90e6-4315314d0487/tmpKtxRQI.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpMulWMg/b6bca5a9-71a6-4561-90e6-4315314d0487/tmpUhN7gv.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpMulWMg/b6bca5a9-71a6-4561-90e6-4315314d0487/tmpKT7_L5.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-9f42619a-b081-4af1-b44f-caae1d32fa0e/tmpMulWMg/b6bca5a9-71a6-4561-90e6-4315314d0487/tmp9hfIYu.tmp'] failed with output:
  1965. WARNING:toil.leader:W/W/jobNeEy8O ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn29.grid.pub.ro
  1966. WARNING:toil.leader:W/W/jobNeEy8O WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' W/W/jobNeEy8O with ID W/W/jobNeEy8O to 4
  1967. INFO:toil.leader:Issued job 'CactusSetupPhase' W/W/jobNeEy8O with job batch system ID: 225 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1968. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' t/N/job7paZYg
  1969. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' t/N/job7paZYg
  1970. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' t/N/job7paZYg with ID t/N/job7paZYg to 4
  1971. INFO:toil.leader:Issued job 'KtServerService' t/N/job7paZYg with job batch system ID: 226 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1972. WARNING:toil.leader:Job failed with exit value 1: 'CactusSetupPhase' W/W/jobNeEy8O
  1973. WARNING:toil.leader:No log file is present, despite job failing: 'CactusSetupPhase' W/W/jobNeEy8O
  1974. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' W/W/jobNeEy8O with ID W/W/jobNeEy8O to 3
  1975. INFO:toil.leader:Issued job 'CactusSetupPhase' W/W/jobNeEy8O with job batch system ID: 227 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1976. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' t/N/job7paZYg
  1977. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' t/N/job7paZYg
  1978. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' t/N/job7paZYg with ID t/N/job7paZYg to 3
  1979. INFO:toil.leader:Issued job 'KtServerService' t/N/job7paZYg with job batch system ID: 228 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1980. WARNING:toil.leader:Job failed with exit value 1: 'CactusSetupPhase' W/W/jobNeEy8O
  1981. WARNING:toil.leader:No log file is present, despite job failing: 'CactusSetupPhase' W/W/jobNeEy8O
  1982. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' W/W/jobNeEy8O with ID W/W/jobNeEy8O to 2
  1983. INFO:toil.leader:Issued job 'CactusSetupPhase' W/W/jobNeEy8O with job batch system ID: 229 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  1984. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' t/N/job7paZYg
  1985. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' t/N/job7paZYg
  1986. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' t/N/job7paZYg with ID t/N/job7paZYg to 2
  1987. INFO:toil.leader:Issued job 'KtServerService' t/N/job7paZYg with job batch system ID: 230 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1988. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' t/N/job7paZYg
  1989. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' t/N/job7paZYg
  1990. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' t/N/job7paZYg with ID t/N/job7paZYg to 1
  1991. INFO:toil.leader:Issued job 'KtServerService' t/N/job7paZYg with job batch system ID: 231 and cores: 0, disk: 2.0 G, and memory: 2.3 G
  1992. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' W/W/jobNeEy8O
  1993. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' W/W/jobNeEy8O
  1994. WARNING:toil.leader:W/W/jobNeEy8O INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  1995. WARNING:toil.leader:W/W/jobNeEy8O INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  1996. WARNING:toil.leader:W/W/jobNeEy8O WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  1997. WARNING:toil.leader:W/W/jobNeEy8O WARNING:toil.fileStore:Detected that job (a/n/jobKwRkH2/g/tmpwA1eA_-_serialiseJob-stream) prematurely terminated. Fixing the state of the job on disk.
  1998. WARNING:toil.leader:W/W/jobNeEy8O INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  1999. WARNING:toil.leader:W/W/jobNeEy8O INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  2000. WARNING:toil.leader:W/W/jobNeEy8O INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="25223" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-b02a3811-2b63-4208-851d-7815af46a62d/tmpgdkkV0/e8531448-cb33-44c4-8c40-f54f38fe3048/tmpanzxJ3.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-b02a3811-2b63-4208-851d-7815af46a62d/tmpgdkkV0/e8531448-cb33-44c4-8c40-f54f38fe3048/tmpGwzyhM.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-b02a3811-2b63-4208-851d-7815af46a62d/tmpgdkkV0/e8531448-cb33-44c4-8c40-f54f38fe3048/tmpzeq84X.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-b02a3811-2b63-4208-851d-7815af46a62d/tmpgdkkV0/e8531448-cb33-44c4-8c40-f54f38fe3048/tmp_GSXtr.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-b02a3811-2b63-4208-851d-7815af46a62d/tmpgdkkV0/e8531448-cb33-44c4-8c40-f54f38fe3048/tmp3veCng.tmp']
  2001. WARNING:toil.leader:W/W/jobNeEy8O Set log level to INFO
  2002. WARNING:toil.leader:W/W/jobNeEy8O Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  2003. WARNING:toil.leader:W/W/jobNeEy8O <kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="25223" />
  2004. WARNING:toil.leader:W/W/jobNeEy8O </st_kv_database_conf>
  2005. WARNING:toil.leader:W/W/jobNeEy8O
  2006. WARNING:toil.leader:W/W/jobNeEy8O Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-b02a3811-2b63-4208-851d-7815af46a62d/tmpgdkkV0/e8531448-cb33-44c4-8c40-f54f38fe3048/tmpanzxJ3.tmp
  2007. WARNING:toil.leader:W/W/jobNeEy8O Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-b02a3811-2b63-4208-851d-7815af46a62d/tmpgdkkV0/e8531448-cb33-44c4-8c40-f54f38fe3048/tmpGwzyhM.tmp
  2008. WARNING:toil.leader:W/W/jobNeEy8O Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-b02a3811-2b63-4208-851d-7815af46a62d/tmpgdkkV0/e8531448-cb33-44c4-8c40-f54f38fe3048/tmpzeq84X.tmp
  2009. WARNING:toil.leader:W/W/jobNeEy8O Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-b02a3811-2b63-4208-851d-7815af46a62d/tmpgdkkV0/e8531448-cb33-44c4-8c40-f54f38fe3048/tmp_GSXtr.tmp
  2010. WARNING:toil.leader:W/W/jobNeEy8O Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-b02a3811-2b63-4208-851d-7815af46a62d/tmpgdkkV0/e8531448-cb33-44c4-8c40-f54f38fe3048/tmp3veCng.tmp
  2011. WARNING:toil.leader:W/W/jobNeEy8O Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.29 with error: network error
  2012. WARNING:toil.leader:W/W/jobNeEy8O Uncaught exception
  2013. WARNING:toil.leader:W/W/jobNeEy8O Traceback (most recent call last):
  2014. WARNING:toil.leader:W/W/jobNeEy8O File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  2015. WARNING:toil.leader:W/W/jobNeEy8O job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2016. WARNING:toil.leader:W/W/jobNeEy8O File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  2017. WARNING:toil.leader:W/W/jobNeEy8O super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2018. WARNING:toil.leader:W/W/jobNeEy8O File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  2019. WARNING:toil.leader:W/W/jobNeEy8O returnValues = self._run(jobGraph, fileStore)
  2020. WARNING:toil.leader:W/W/jobNeEy8O File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  2021. WARNING:toil.leader:W/W/jobNeEy8O return self.run(fileStore)
  2022. WARNING:toil.leader:W/W/jobNeEy8O File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  2023. WARNING:toil.leader:W/W/jobNeEy8O makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  2024. WARNING:toil.leader:W/W/jobNeEy8O File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  2025. WARNING:toil.leader:W/W/jobNeEy8O parameters=["cactus_setup"] + args + sequences)
  2026. WARNING:toil.leader:W/W/jobNeEy8O File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  2027. WARNING:toil.leader:W/W/jobNeEy8O raise RuntimeError("Command %s failed with output: %s" % (call, output))
  2028. WARNING:toil.leader:W/W/jobNeEy8O RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.29" port="25223" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-b02a3811-2b63-4208-851d-7815af46a62d/tmpgdkkV0/e8531448-cb33-44c4-8c40-f54f38fe3048/tmpanzxJ3.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-b02a3811-2b63-4208-851d-7815af46a62d/tmpgdkkV0/e8531448-cb33-44c4-8c40-f54f38fe3048/tmpGwzyhM.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-b02a3811-2b63-4208-851d-7815af46a62d/tmpgdkkV0/e8531448-cb33-44c4-8c40-f54f38fe3048/tmpzeq84X.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-b02a3811-2b63-4208-851d-7815af46a62d/tmpgdkkV0/e8531448-cb33-44c4-8c40-f54f38fe3048/tmp_GSXtr.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-b02a3811-2b63-4208-851d-7815af46a62d/tmpgdkkV0/e8531448-cb33-44c4-8c40-f54f38fe3048/tmp3veCng.tmp'] failed with output:
  2029. WARNING:toil.leader:W/W/jobNeEy8O ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn39.grid.pub.ro
  2030. WARNING:toil.leader:W/W/jobNeEy8O WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' W/W/jobNeEy8O with ID W/W/jobNeEy8O to 1
  2031. INFO:toil.leader:Issued job 'CactusSetupPhase' W/W/jobNeEy8O with job batch system ID: 232 and cores: 1, disk: 2.0 G, and memory: 3.3 G
  2032. INFO:toil.leader:Job ended successfully: 'CactusSetupPhase' W/W/jobNeEy8O
  2033. WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'CactusSetupPhase' W/W/jobNeEy8O
  2034. WARNING:toil.leader:W/W/jobNeEy8O INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  2035. WARNING:toil.leader:W/W/jobNeEy8O INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
  2036. WARNING:toil.leader:W/W/jobNeEy8O WARNING:toil.resource:'JTRES_5d2f846cd67858267ed5af4717d96bda' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
  2037. WARNING:toil.leader:W/W/jobNeEy8O WARNING:toil.fileStore:Detected that job (a/n/jobKwRkH2/g/tmpkoE42E-_serialiseJob-stream) prematurely terminated. Fixing the state of the job on disk.
  2038. WARNING:toil.leader:W/W/jobNeEy8O INFO:toil.lib.bioio:Sequences in cactus setup: ['simHuman_chr6', 'simMouse_chr6', 'simRat_chr6', 'simCow_chr6', 'simDog_chr6']
  2039. WARNING:toil.leader:W/W/jobNeEy8O INFO:toil.lib.bioio:Sequences in cactus setup filenames: ['>id=1|simHuman.chr6|0\n', '>id=0|simMouse.chr6\n', '>id=2|simRat.chr6\n', '>id=4|simCow.chr6|0\n', '>id=3|simDog.chr6|0\n']
  2040. WARNING:toil.leader:W/W/jobNeEy8O INFO:cactus.shared.common:Running the command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.40" port="21837" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpPWAJ_7/fb2f54c6-b9f5-4d29-94aa-259a2fcfd81c/tmpmCgazS.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpPWAJ_7/fb2f54c6-b9f5-4d29-94aa-259a2fcfd81c/tmpTxsHCx.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpPWAJ_7/fb2f54c6-b9f5-4d29-94aa-259a2fcfd81c/tmphcpwrL.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpPWAJ_7/fb2f54c6-b9f5-4d29-94aa-259a2fcfd81c/tmpY_jO1n.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpPWAJ_7/fb2f54c6-b9f5-4d29-94aa-259a2fcfd81c/tmpWTj0VK.tmp']
  2041. WARNING:toil.leader:W/W/jobNeEy8O Set log level to INFO
  2042. WARNING:toil.leader:W/W/jobNeEy8O Flower disk name : <st_kv_database_conf type="kyoto_tycoon">
  2043. WARNING:toil.leader:W/W/jobNeEy8O <kyoto_tycoon database_dir="fakepath" host="172.16.13.40" port="21837" />
  2044. WARNING:toil.leader:W/W/jobNeEy8O </st_kv_database_conf>
  2045. WARNING:toil.leader:W/W/jobNeEy8O
  2046. WARNING:toil.leader:W/W/jobNeEy8O Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpPWAJ_7/fb2f54c6-b9f5-4d29-94aa-259a2fcfd81c/tmpmCgazS.tmp
  2047. WARNING:toil.leader:W/W/jobNeEy8O Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpPWAJ_7/fb2f54c6-b9f5-4d29-94aa-259a2fcfd81c/tmpTxsHCx.tmp
  2048. WARNING:toil.leader:W/W/jobNeEy8O Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpPWAJ_7/fb2f54c6-b9f5-4d29-94aa-259a2fcfd81c/tmphcpwrL.tmp
  2049. WARNING:toil.leader:W/W/jobNeEy8O Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpPWAJ_7/fb2f54c6-b9f5-4d29-94aa-259a2fcfd81c/tmpY_jO1n.tmp
  2050. WARNING:toil.leader:W/W/jobNeEy8O Sequence file/directory /export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpPWAJ_7/fb2f54c6-b9f5-4d29-94aa-259a2fcfd81c/tmpWTj0VK.tmp
  2051. WARNING:toil.leader:W/W/jobNeEy8O Exception: ST_KV_DATABASE_EXCEPTION: Opening connection to host: 172.16.13.40 with error: network error
  2052. WARNING:toil.leader:W/W/jobNeEy8O Uncaught exception
  2053. WARNING:toil.leader:W/W/jobNeEy8O Traceback (most recent call last):
  2054. WARNING:toil.leader:W/W/jobNeEy8O File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
  2055. WARNING:toil.leader:W/W/jobNeEy8O job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2056. WARNING:toil.leader:W/W/jobNeEy8O File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1096, in _runner
  2057. WARNING:toil.leader:W/W/jobNeEy8O super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  2058. WARNING:toil.leader:W/W/jobNeEy8O File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
  2059. WARNING:toil.leader:W/W/jobNeEy8O returnValues = self._run(jobGraph, fileStore)
  2060. WARNING:toil.leader:W/W/jobNeEy8O File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
  2061. WARNING:toil.leader:W/W/jobNeEy8O return self.run(fileStore)
  2062. WARNING:toil.leader:W/W/jobNeEy8O File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 641, in run
  2063. WARNING:toil.leader:W/W/jobNeEy8O makeEventHeadersAlphaNumeric=self.getOptionalPhaseAttrib("makeEventHeadersAlphaNumeric", bool, False))
  2064. WARNING:toil.leader:W/W/jobNeEy8O File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 220, in runCactusSetup
  2065. WARNING:toil.leader:W/W/jobNeEy8O parameters=["cactus_setup"] + args + sequences)
  2066. WARNING:toil.leader:W/W/jobNeEy8O File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/shared/common.py", line 1040, in cactus_call
  2067. WARNING:toil.leader:W/W/jobNeEy8O raise RuntimeError("Command %s failed with output: %s" % (call, output))
  2068. WARNING:toil.leader:W/W/jobNeEy8O RuntimeError: Command ['cactus_setup', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.16.13.40" port="21837" />\n\t\t</st_kv_database_conf>\n\t', '--logLevel', 'INFO', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpPWAJ_7/fb2f54c6-b9f5-4d29-94aa-259a2fcfd81c/tmpmCgazS.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpPWAJ_7/fb2f54c6-b9f5-4d29-94aa-259a2fcfd81c/tmpTxsHCx.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpPWAJ_7/fb2f54c6-b9f5-4d29-94aa-259a2fcfd81c/tmphcpwrL.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpPWAJ_7/fb2f54c6-b9f5-4d29-94aa-259a2fcfd81c/tmpY_jO1n.tmp', u'/export/home/ncit/external/a.mizeranschi/temp/cactus-test/cactusTemp/toil-25c65c76-293f-4da9-a908-fef81d76b1fc-6386a5c9-5d92-486a-9720-412b1ca610f6/tmpPWAJ_7/fb2f54c6-b9f5-4d29-94aa-259a2fcfd81c/tmpWTj0VK.tmp'] failed with output:
  2069. WARNING:toil.leader:W/W/jobNeEy8O ERROR:toil.worker:Exiting the worker because of a failed job on host haswell-wn37.grid.pub.ro
  2070. WARNING:toil.leader:W/W/jobNeEy8O WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusSetupPhase' W/W/jobNeEy8O with ID W/W/jobNeEy8O to 0
  2071. WARNING:toil.leader:Job 'CactusSetupPhase' W/W/jobNeEy8O with ID W/W/jobNeEy8O is completely failed
  2072. WARNING:toil.leader:Job failed with exit value 137: 'KtServerService' t/N/job7paZYg
  2073. WARNING:toil.leader:No log file is present, despite job failing: 'KtServerService' t/N/job7paZYg
  2074. WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' t/N/job7paZYg with ID t/N/job7paZYg to 0
  2075. WARNING:toil.leader:Job 'KtServerService' t/N/job7paZYg with ID t/N/job7paZYg is completely failed
  2076. INFO:toil.leader:Finished toil run with 10 failed jobs.
  2077. INFO:toil.leader:Failed jobs at end of the run: 'ProgressiveUp' O/t/jobBTKSzR 'CactusTrimmingBlastPhase' 6/c/job8rdfrw 'CactusSetupPhase' W/W/jobNeEy8O 'ProgressiveDown' w/1/jobSc89ZQ 'RunCactusPreprocessorThenProgressiveDown2' d/c/jobyROBMF 'StartPrimaryDB' a/n/jobKwRkH2 'KtServerService' t/N/job7paZYg 'RunCactusPreprocessorThenProgressiveDown' E/U/jobU_k7ZC 'ProgressiveNext' H/f/jobOf_7TQ 'CactusSetupCheckpoint' P/9/jobVgO4XI
  2078. Traceback (most recent call last):
  2079. File "/export/home/ncit/external/a.mizeranschi/toil_conda/bin/cactus", line 11, in <module>
  2080. sys.exit(main())
  2081. File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/cactus/progressive/cactus_progressive.py", line 520, in main
  2082. halID = toil.start(RunCactusPreprocessorThenProgressiveDown(options, project, memory=configWrapper.getDefaultMemory()))
  2083. File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/common.py", line 784, in start
  2084. return self._runMainLoop(rootJobGraph)
  2085. File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/common.py", line 1059, in _runMainLoop
  2086. jobCache=self._jobCache).run()
  2087. File "/export/home/ncit/external/a.mizeranschi/toil_conda/lib/python2.7/site-packages/toil/leader.py", line 237, in run
  2088. raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore)
  2089. toil.leader.FailedJobsException
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!