SHARE
TWEET

Untitled

a guest Jun 17th, 2019 67 Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. # test-conda-env-hpc
  2.  
  3. ## Question
  4.  
  5. Does the conda environment that you have activated on the login node also get used in the compute nodes when submitting a job?
  6.  
  7. ## Methods
  8.  
  9. * Used conda version 4.6.14
  10. 1. Activate a conda env with snakemake installed.
  11.     1. Submit the job & note the job id.
  12.     1. The log file should have snakemake output.
  13. 1. Activate a conda env without snakemake.
  14.     1. Submit the job & note the job id.
  15.     1. The log file should have an error message that the snakemake command was not found.
  16.  
  17. ### SLURM
  18. * HPC: `beta-login.stage.arc-ts.umich.edu`
  19. * Script: `submit-slurm.sh`
  20. * Conda environments:
  21.     * With snakemake: `slurm-with-smk.yaml`
  22.     * Without snakemake: `slurm-no-smk.yaml`
  23.    
  24. ### PBS-Torque
  25. * HPC: `flux-login.arc-ts.umich.edu`
  26. * Script: `submit-pbs.sh`
  27. * Conda environments:
  28.     * With snakemake: `pbs-with-smk.yaml`
  29.     * Without snakemake: `pbs-no-smk.yaml`
  30.  
  31. ## Results
  32.  
  33. ### SLURM job with snakemake: 33069
  34.  
  35. ```
  36. Building DAG of jobs...
  37. Using shell: /usr/bin/bash
  38. Provided cores: 1
  39. Rules claiming more threads will be scaled down.
  40. Job counts:
  41.     count   jobs
  42.         1   touch
  43.         1
  44.  
  45. [Mon Jun  3 14:06:20 2019]
  46. rule touch:
  47.     output: file.txt
  48.     jobid: 0
  49.  
  50. [Mon Jun  3 14:06:20 2019]
  51. Finished job 0.
  52. 1 of 1 steps (100%) done
  53. Complete log: /home/sovacool/test-conda-env/.snakemake/log/2019-06-03T140620.029171.snakemake.log
  54. ```
  55.  
  56. ### SLURM job without snakemake: 33070
  57.  
  58. Conda environment: `slurm-no-smk.yaml`
  59.  
  60. ```
  61. /var/spool/slurmd.spool/job33070/slurm_script: line 49: snakemake: command not found
  62. ```
  63.  
  64. ### PBS job with snakemake: 33732423
  65.  
  66. ```
  67. nyx5568
  68. /home/sovacool/test-conda-env-hpc
  69. Building DAG of jobs...
  70. Using shell: /usr/bin/bash
  71. Provided cores: 1
  72. Rules claiming more threads will be scaled down.
  73. Job counts:
  74.     count   jobs
  75.     1   touch
  76.     1
  77.  
  78. [Wed Jun  5 11:01:27 2019]
  79. rule touch:
  80.     output: file.txt
  81.     jobid: 0
  82.  
  83. [Wed Jun  5 11:01:27 2019]
  84. Finished job 0.
  85. 1 of 1 steps (100%) done
  86. Complete log: /home/sovacool/test-conda-env-hpc/.snakemake/log/2019-06-05T110127.477703.snakemake.log
  87. ```
  88.  
  89. ### PBS job without snakemake: 33732416
  90.  
  91. ```
  92. nyx5089
  93. /home/sovacool/test-conda-env-hpc
  94. /var/spool/torque/mom_priv/jobs/33732416.nyx.arc-ts.umich.edu.SC: line 21: snakemake: command not found
  95. ```
  96.  
  97. ## Conclusion
  98.  
  99. Yes!
RAW Paste Data
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy. OK, I Understand
 
Top