Nextflow+Singularity：没有这样的文件或目录，即使文件存在答案

【问题标题】：Nextflow+Singularity: No such file or directory, even though file existsNextflow+Singularity：没有这样的文件或目录，即使文件存在
【发布时间】：2019-09-17 11:47:55
【问题描述】：

我正在尝试运行这个pipeline，它使用 Nextflow 实现并使用 Docker 容器。不幸的是，我不能使用 Docker，因为它不兼容 HPC（没有 sudo），所以我使用 Singularity 而不是 Docker。但是，似乎路径未正确安装，因为我收到此错误：

Error executing process > 'truncate_input_headers'

Caused by:
  Process `truncate_input_headers` terminated with an error exit status (1)

Command executed:

  truncate_header.lua < phased.1_scaffolds_FINAL.fasta > truncated.fasta

Command exit status:
  1

Command output:
  (empty)

Command error:
  .command.sh: line 2: phased.1_scaffolds_FINAL.fasta: No such file or directory

Work dir:
  /work/project/ladsie_002/work/77/1854982bdacdd60fbe447554ab153b

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

但是，当我查看路径时，该文件确实存在：

$ ll /work/project/ladsie_002/work/77/1854982bdacdd60fbe447554ab153b
total 1
lrwxrwxrwx 1 bbrink users 76 30. Aug 13:44 phased.1_scaffolds_FINAL.fasta -> /work/project/ladsie_002/companion/input/pleo/phased.1_scaffolds_FINAL.fasta

这是我的配置：

env {
    GT_RETAINIDS = "yes"
    AUGUSTUS_CONFIG_PATH = "/opt/data/augustus"
    FILTER_SHORT_PARTIALS_RULE = "/opt/data/filters/filter_short_partials.lua"
    PFAM = "/opt/pfam/Pfam-A.hmm"
    PFAM2GO = "/opt/data/pfam2go/pfam2go.txt"
    RATT_CONFIG = "/opt/RATT/RATT.config_euk_NoPseudo_SpliceSite"
}

params.GO_OBO = "/opt/go.obo"
params.NCRNA_MODELS = "/opt/data/cm/rnas.cm"
params.CIRCOS_CONFIG_FILE = "/opt/data/circos/circos.debian.conf"
params.CIRCOS_BIN_CONFIG_FILE = "/opt/data/circos/circos.bin.debian.conf"
params.SPECFILE = "/opt/data/speck/output_check.lua"
params.AUGUSTUS_EXTRINSIC_CFG = "/opt/data/augustus/extrinsic.cfg"

process {
    container = 'sangerpathogens/companion:latest'
}

singularity {
    enabled = true
//    autoMounts = true
}

executor {
    name = 'local'
    queueSize = 2
    pollInterval = '3sec'
}

我试图为此找到解决方案，我唯一能找到的是autoMounts = true 选项（上面已注释掉），这会导致管道在启用时甚至找不到它自己的 lua 脚本：

Error executing process > 'truncate_input_headers'

Caused by:
  Process `truncate_input_headers` terminated with an error exit status (127)

Command executed:

  truncate_header.lua < phased.1_scaffolds_FINAL.fasta > truncated.fasta

Command exit status:
  127

Command output:
  (empty)

Command error:
  .command.sh: line 2: truncate_header.lua: command not found

Work dir:
  /work/project/ladsie_002/work/95/f03b31ed18a84f331b83cad0232bd5

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

编辑1：我听从了tsnowlan的建议，将--debug添加到singularity.engineOptions。但是，有问题的路径似乎已安装：

Error executing process > 'truncate_input_headers'

Caused by:
  Process `truncate_input_headers` terminated with an error exit status (1)

Command executed:

  truncate_header.lua < phased.1_scaffolds_FINAL.fasta > truncated.fasta

Command exit status:
  1

Command output:
  (empty)

Command error:
  DEBUG   [U=1038,P=29509]   mountGeneric()                Mounting tmpfs to /var/singularity/mnt/session
  DEBUG   [U=1038,P=29509]   mountImage()                  Mounting loop device /dev/loop0 to /var/singularity/mnt/session/rootfs
  DEBUG   [U=1038,P=29509]   mountGeneric()                Mounting overlay to /var/singularity/mnt/session/final
  DEBUG   [U=1038,P=29509]   setPropagationMount()         Set RPC mount propagation flag to SLAVE
  VERBOSE [U=1038,P=29509]   Passwd()                      Checking for template passwd file: /var/singularity/mnt/session/rootfs/etc/passwd
  VERBOSE [U=1038,P=29509]   Passwd()                      Creating passwd content
  VERBOSE [U=1038,P=29509]   Passwd()                      Creating template passwd file and appending user data: /var/singularity/mnt/session/rootfs/etc/passwd
  DEBUG   [U=1038,P=29509]   addIdentityMount()            Adding /etc/passwd to mount list
  VERBOSE [U=1038,P=29509]   addIdentityMount()            Default mount: /etc/passwd:/etc/passwd
  VERBOSE [U=1038,P=29509]   Group()                       Checking for template group file: /var/singularity/mnt/session/rootfs/etc/group
  VERBOSE [U=1038,P=29509]   Group()                       Creating group content
  DEBUG   [U=1038,P=29509]   addIdentityMount()            Adding /etc/group to mount list
  VERBOSE [U=1038,P=29509]   addIdentityMount()            Default mount: /etc/group:/etc/group
  DEBUG   [U=1038,P=29509]   mountGeneric()                Remounting /var/singularity/mnt/session/final
  DEBUG   [U=1038,P=29509]   mountGeneric()                Mounting /dev to /var/singularity/mnt/session/final/dev
  DEBUG   [U=1038,P=29509]   mountGeneric()                Mounting /etc/localtime to /var/singularity/mnt/session/final/usr/share/zoneinfo/UTC
  DEBUG   [U=1038,P=29509]   mountGeneric()                Mounting /etc/hosts to /var/singularity/mnt/session/final/etc/hosts
  DEBUG   [U=1038,P=29509]   mountGeneric()                Mounting /etc/singularity/actions to /var/singularity/mnt/session/final/.singularity.d/actions
  DEBUG   [U=1038,P=29509]   mountGeneric()                Remounting /var/singularity/mnt/session/final/.singularity.d/actions
  DEBUG   [U=1038,P=29509]   mountGeneric()                Mounting /proc to /var/singularity/mnt/session/final/proc
  DEBUG   [U=1038,P=29509]   mountGeneric()                Remounting /var/singularity/mnt/session/final/proc
  DEBUG   [U=1038,P=29509]   mountGeneric()                Mounting sysfs to /var/singularity/mnt/session/final/sys
  DEBUG   [U=1038,P=29509]   mountGeneric()                Mounting /home/bbrink to /var/singularity/mnt/session/home/bbrink
  DEBUG   [U=1038,P=29509]   mountGeneric()                Remounting /var/singularity/mnt/session/home/bbrink
  DEBUG   [U=1038,P=29509]   mountGeneric()                Mounting /var/singularity/mnt/session/home/bbrink to /var/singularity/mnt/session/final/home/bbrink
  DEBUG   [U=1038,P=29509]   mountGeneric()                Remounting /var/singularity/mnt/session/final/home/bbrink
  DEBUG   [U=1038,P=29509]   mountGeneric()                Mounting /tmp to /var/singularity/mnt/session/final/tmp
  DEBUG   [U=1038,P=29509]   mountGeneric()                Remounting /var/singularity/mnt/session/final/tmp
  DEBUG   [U=1038,P=29509]   mountGeneric()                Mounting /var/tmp to /var/singularity/mnt/session/final/var/tmp
  DEBUG   [U=1038,P=29509]   mountGeneric()                Remounting /var/singularity/mnt/session/final/var/tmp
  DEBUG   [U=1038,P=29509]   mountGeneric()                Mounting /work/project/ladsie_002/work/de/74a8bbd8bd11ad1524800fb7e71556 to /var/singularity/mnt/session/final/work/project/ladsie_002/work/de/74a8bbd8bd11ad1524800fb7e71556
  DEBUG   [U=1038,P=29509]   mountGeneric()                Remounting /var/singularity/mnt/session/final/work/project/ladsie_002/work/de/74a8bbd8bd11ad1524800fb7e71556
  DEBUG   [U=1038,P=29509]   mountGeneric()                Mounting /var/singularity/mnt/session/etc/resolv.conf to /var/singularity/mnt/session/final/etc/resolv.conf
  DEBUG   [U=1038,P=29509]   mountGeneric()                Mounting /var/singularity/mnt/session/etc/passwd to /var/singularity/mnt/session/final/etc/passwd
  DEBUG   [U=1038,P=29509]   mountGeneric()                Mounting /var/singularity/mnt/session/etc/group to /var/singularity/mnt/session/final/etc/group
  DEBUG   [U=1038,P=29509]   create()                      Chroot into /var/singularity/mnt/session/final
  DEBUG   [U=0,P=29543]      Chroot()                      Change current directory to /var/singularity/mnt/session/final
  DEBUG   [U=0,P=29543]      Chroot()                      Hold reference to host / directory
  DEBUG   [U=0,P=29543]      Chroot()                      Called pivot_root on /var/singularity/mnt/session/final
  DEBUG   [U=0,P=29543]      Chroot()                      Change current directory to host / directory
  DEBUG   [U=0,P=29543]      Chroot()                      Apply slave mount propagation for host / directory
  DEBUG   [U=0,P=29543]      Chroot()                      Called unmount(/, syscall.MNT_DETACH)
  DEBUG   [U=0,P=29543]      Chroot()                      Changing directory to / to avoid getpwd issues
  DEBUG   [U=1038,P=29509]   create()                      Chdir into / to avoid errors
  VERBOSE [U=1038,P=29542]   startup()                     Execute stage 2
  DEBUG   [U=1038,P=29542]   Stage()                       Entering stage 2
  DEBUG   [U=1038,P=29509]   PostStartProcess()            Post start process
  .command.sh: line 2: phased.1_scaffolds_FINAL.fasta: No such file or directory
  DEBUG   [U=1038,P=29509]   CleanupContainer()            Cleanup container
  DEBUG   [U=1038,P=29509]   Master()                      Child exited with exit status 1

Work dir:
  /work/project/ladsie_002/work/de/74a8bbd8bd11ad1524800fb7e71556

编辑2：我修复了错误的挂载点，导致另一个错误：

gt: error: could not execute script ...rk/project/ladsie_002/companion/bin/gff3_to_embl.lua:74: bad argument #1 to 'lines' (/opt/go.obo: Permission denied)

我猜是容器内的权限设置不正确。

【问题讨论】：

你使用什么版本的singularity来构建镜像以及运行它的版本是什么？你是直接在 VM 上运行 nextflow 还是使用 slurm/PBS 提交到集群？
奇点版本 3.2.1-1.1.el7。现在我正在测试直接在集群的头/主节点上运行 nextflow。解决此问题后，我将修改配置以使用 SLURM 提交作业。

标签： docker filesystems singularity-container nextflow

【解决方案1】：

正如 pditommaso 所提到的，您需要确保奇点图像内的环境设置正确。您可以通过 Singularity 定义文件中的 %environment 块或在主机环境中导出 SINGULARITY_$VARNAME 来执行此操作。例如，SINGULARITY_PERL5LIB=/opt/ORTHOMCLV1.4/:/opt/RATT/:/opt/ABACAS2/:$PERL5LIB 在 nextflow 配置的 env 块中设置容器内的 PERL5LIB 变量。

您还可能需要在singularity.runOptions 中指定要挂载到容器中的数据位置。那里的细节取决于您使用的 Singularity 版本以及主机操作系统是否在其内核中启用了 OverlayFS。

编辑 1：绑定/安装

基本形式是-B /host/path:/image/path。如果奇点镜像中不存在/image/path，并且操作系统内核中未启用OverlayFS，您将收到错误消息：WARNING: Skipping user bind, non existent bind point (directory/file) in container: '/image/path'。 directory/file 是正在安装的位置。

您可以在版本 3+ 中使用 --debug 标志或在版本 2 中使用 -vv 来查看正在挂载的内容。在 nextflow 中，这将添加到 singularity.engineOptions。

请注意，如果挂载目录中的任何符号链接引用了奇点映像中不可用的路径，它们都会被破坏。

如果挂载的数据包含符号链接，则它们通常很有可能引用未挂载到容器中的文件，并且在运行奇点映像时似乎是断开的链接。

编辑 2：文件权限

构建奇点图像的最简单方法是sudo singularity build companion.sif docker://sangerpathogens/companion:latest。如果您需要修改任何内容，例如 /opt/go.obo 文件，您需要创建一个定义文件。

Bootstrap: docker
From: sangerpathogens/companion:latest
# note: for reproducibility, it is best to use a tag whose target won't change over time

%post
    chmod 644 /opt/go.obo

如果需要任何其他更改，它们会进入%post 块。与 Docker 不同，singularity 不会为每个命令使用层来创建映像，因此您不必担心链接 some_command && some_other_command && so_on && etc。

【讨论】：

我正在运行奇异版本 3.2.1-1.1.el7。如何确定主机操作系统是否启用了 OverlayFS，以及如何正确挂载数据？ PATH好像设置好了，就是nextflow链接的数据看不到了。
将--debug 添加到singularity.runOptions。在 nextflow 日志中，使用 grep 查找 Mounting /。这将向您显示容器内安装的所有主机位置。如果未启用 OverlayFS 并且您尝试将数据挂载到的目录不存在，您将在进程日志中看到警告“跳过使用绑定，不存在的绑定点...”。
更新了答案，增加了绑定/安装说明。 OverlayFS 可能没问题。如果您的集群运行的是较旧的操作系统（cento/rhel 6、ubuntu 14.04 等），这主要是一个问题
感谢您的帮助。实际上，必须将--debug 添加到singularity.engineOptions。但是，路径似乎已安装。我更新了问题。
我想我一开始就错过了这个。 work 目录中的文件是指向不同文件的符号链接吗？如果是这样，则链接到的文件很可能不在图像中，并且程序只是看到一个损坏的链接。尝试直接在链接文件上运行或将-B /work 添加到singularity.runOptions。

【解决方案2】：

Singulary 可能无法正确转换用户环境定义的 Docker 容器。确保 Singularity 映像定义 PATH 和项目 Dockerfile 中定义的其他变量 https://github.com/sanger-pathogens/companion/blob/master/Dockerfile#L107

【讨论】：

PATH 设置正确（至少在第一种情况下），因为它在 Docker 容器中找到了工具和脚本。在我的文件系统上找不到的是容器外的文件。

【解决方案3】：

此修复将有助于 truncate_headers.lua 脚本不存在。

https://github.com/sanger-pathogens/companion/issues/93

【讨论】：