Snakemake: --use-conda 和 --cluster答案

【问题标题】：Snakemake: --use-conda with --clusterSnakemake: --use-conda 和 --cluster
【发布时间】：2021-03-25 20:09:26
【问题描述】：

当我使用--cluster 和--use-conda 运行时，Snakemake 在提交到集群之前似乎没有设置 conda 环境，因此我的作业会失败。在集群提交之前设置 conda 环境是否缺少技巧？

编辑：

我在 conda 环境中得到了 snakemake，例如：

channels:
  - bioconda
  - conda-forge
dependencies:
  - snakemake-minimal=5.19.3
  - xrootd=4.12.2

复制者：

我用Snakefile、dothing.py 和environment.yml 创建一个目录：

Snakefile:

shell.prefix('unset PYTHONPATH; unset LD_LIBRARY_PATH; unset PYTHONHOME; ')

rule dothing:
    conda: 'environment.yml'
    output: 'completed.out'
    log: 'thing.log'
    shell: 'python dothing.py &> {log} && touch {output}'

dothing.py:

import uncertainties

print('it worked!')

environment.yml:

name: testsnakeconda
channels:
  - conda-forge
dependencies:
  - uncertainties=3.1.4

如果我喜欢在本地运行

snakemake --cores all --use-conda

它运行没有问题：

Building DAG of jobs...
Creating conda environment environment.yml...
Downloading and installing remote packages.
Environment for environment.yml created (location: .snakemake/conda/e0fff47f)
Using shell: /usr/bin/bash
Provided cores: 10
Rules claiming more threads will be scaled down.
Job counts:
        count   jobs
        1       dothing
        1

[Tue Jun 30 16:19:38 2020]
rule dothing:
    output: completed.out
    log: thing.log
    jobid: 0

Activating conda environment: /path/to/environment.yml
[Tue Jun 30 16:19:39 2020]
Finished job 0.
1 of 1 steps (100%) done
Complete log: /path/to/.snakemake/log/2020-06-30T161824.906217.snakemake.log

如果我尝试使用--cluster 提交

snakemake --cores all --use-conda --cluster 'condor_qsub -V -l procs={threads}' --latency-wait 30 --max-jobs-per-second 100 --jobs 50

没有关于设置 conda 环境的消息，并且作业失败并出现错误：

Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cluster nodes: 50
Job counts:
        count   jobs
        1       dothing
        1

[Tue Jun 30 16:20:49 2020]
rule dothing:
    output: completed.out
    log: thing.log
    jobid: 0

Submitted job 0 with external jobid 'Your job 9246856 ("snakejob.dothing.0.sh") has been submitted'.
[Tue Jun 30 16:26:00 2020]
Error in rule dothing:
    jobid: 0
    output: completed.out
    log: thing.log (check log file(s) for error message)
    conda-env: /path/to/.snakemake/conda/e0fff47f
    shell:
        python dothing.py &> thing.log && touch completed.out
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
    cluster_jobid: Your job 9246856 ("snakejob.dothing.0.sh") has been submitted

Error executing rule dothing on cluster (jobid: 0, external: Your job 9246856 ("snakejob.dothing.0.sh") has been submitted, jobscript: /path/to/.snakemake/tmp.a7fpixla/snakejob.dothing.0.sh). For error details see the cluster log and the log files of the involved rule(s).
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /path/to/.snakemake/log/2020-06-30T162049.793041.snakemake.log

我可以看到问题是uncertainties 包不可用：

$ cat thing.log
Traceback (most recent call last):
  File "dothing.py", line 1, in <module>
    import uncertainties
ImportError: No module named uncertainties

编辑：没有--cluster的详细输出：

Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 10
Rules claiming more threads will be scaled down.
Job counts:
        count   jobs
        1       dothing
        1
Resources before job selection: {'_cores': 10, '_nodes': 9223372036854775807}
Ready jobs (1):
        dothing
Selected jobs (1):
        dothing
Resources after job selection: {'_cores': 9, '_nodes': 9223372036854775806}

[Thu Jul  2 21:51:18 2020]
rule dothing:
    output: completed.out
    log: thing.log
    jobid: 0

Activating conda environment: /path/to/workingdir/.snakemake/conda/e0fff47f
[Thu Jul  2 21:51:33 2020]
Finished job 0.
1 of 1 steps (100%) done
Complete log: /path/to/workingdir/.snakemake/log/2020-07-02T215117.964474.snakemake.log
unlocking
removing lock
removing lock
removed all locks

带有--cluster的详细输出：

Building DAG of jobs...
Checking status of 0 jobs.
Using shell: /usr/bin/bash
Provided cluster nodes: 50
Job counts:
    count   jobs
    1   dothing
    1
Resources before job selection: {'_cores': 9223372036854775807, '_nodes': 50}
Ready jobs (1):
    dothing
Selected jobs (1):
    dothing
Resources after job selection: {'_cores': 9223372036854775806, '_nodes': 49}

[Thu Jul  2 21:40:23 2020]
rule dothing:
    output: completed.out
    log: thing.log
    jobid: 0

Jobscript:
#!/bin/sh
# properties = {"type": "single", "rule": "dothing", "local": false, "input": [], "output": ["completed.out"], "wildcards": {}, "params": {}, "log": ["thing.log"], "threads": 1, "resources": {}, "jobid": 0, "cluster": {}}
 cd /path/to/workingdir && \
/path/to/miniconda/envs/envname/bin/python3.8 \
-m snakemake dothing --snakefile /path/to/workingdir/Snakefile \
--force -j --keep-target-files --keep-remote \
--wait-for-files /path/to/workingdir/.snakemake/tmp.5n32749i /path/to/workingdir/.snakemake/conda/e0fff47f --latency-wait 30 \
 --attempt 1 --force-use-threads \
--wrapper-prefix https://github.com/snakemake/snakemake-wrappers/raw/ \
   --allowed-rules dothing --nocolor --notemp --no-hooks --nolock \
--mode 2  --use-conda  && touch /path/to/workingdir/.snakemake/tmp.5n32749i/0.jobfinished || (touch /path/to/workingdir/.snakemake/tmp.5n32749i/0.jobfailed; exit 1)

Submitted job 0 with external jobid 'Your job 9253728 ("snakejob.dothing.0.sh") has been submitted'.
Checking status of 1 jobs.
...
Checking status of 1 jobs.
[Thu Jul  2 21:46:23 2020]
Error in rule dothing:
    jobid: 0
    output: completed.out
    log: thing.log (check log file(s) for error message)
    conda-env: /path/to/workingdir/.snakemake/conda/e0fff47f
    shell:
        python dothing.py &> thing.log && touch completed.out
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
    cluster_jobid: Your job 9253728 ("snakejob.dothing.0.sh") has been submitted

Error executing rule dothing on cluster (jobid: 0, external: Your job 9253728 ("snakejob.dothing.0.sh") has been submitted, jobscript: /path/to/workingdir/.snakemake/tmp.5n32749i/snakejob.dothing.0.sh). For error details see the cluster log and the log files of the involved rule(s).
Cleanup job metadata.
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /path/to/workingdir/.snakemake/log/2020-07-02T214022.614691.snakemake.log
unlocking
removing lock
removing lock
removed all locks

【问题讨论】：

如果在没有--cluster 模式的情况下运行会发生什么？您在工作环境中是否有可用的 conda？另外，添加--verbose 标志将帮助您调试。
@ManavalanGajapathy 如果我在没有--cluster 的情况下运行，它运行没有问题。当我使用--cluster 提交时，我收到与它在错误环境中运行相关的错误。
这很奇怪。带有错误消息的最小示例将有助于进一步调试。另请参阅if this is relevant。
@ManavalanGajapathy 我在上面添加了一个最小示例。其他帖子似乎不相关；我在尝试加载 conda 时没有收到错误——它似乎根本没有尝试设置 conda 环境。
快速查看，这似乎是一个蛇形错误。您可以尝试使用较旧的 snakemake 版本（

标签： snakemake

【解决方案1】：

对我有用的是在规则本身内设置 python 解释器的完整路径...

rule dothing:
    conda: 'environment.yml'
    output: 'completed.out'
    log: 'thing.log'
    shell: '/full_path_to_your_environment/bin/python dothing.py &> {log} && touch {output}'

如果它是安装在特定环境中的包（这是我的情况），则为您的 python 脚本的完整路径。

rule dothing:
    conda: 'environment.yml'
    output: 'completed.out'
    log: 'thing.log'
    shell: '/full_path_to_your_environment/bin/python /full_path_to_your_environment/package_dir/dothing.py &> {log} && touch {output}'

/full_path_to_your_environment/ 我的意思是哈希名称 conda 和 snakemake 在他们第一次安装时给你的环境（例如 /path/to/workingdir/.snakemake/conda/e0fff47f）

这有点难看，但还是成功了。

希望对你有帮助

【讨论】：