扭矩工作随机死亡答案

【问题标题】：Torque job randomly dying扭矩工作随机死亡
【发布时间】：2018-02-27 22:03:35
【问题描述】：

我正在以大约 27 个批次运行 Python3 脚本，所有这些脚本都具有不同的输入。然后将结果保存到 results/$sizex$size 文件夹中。工作目录也必须更改为该文件夹，以便程序可以保存一些图像和数据。

这是我的 pbs 脚本：

#!/bin/bash
#PBS -l nodes=1:ppn=28
#PBS -l mem=16gb
#PBS -l walltime=120:00:00

cd $PBS_O_WORKDIR

mkdir -p results

module purge
module load newmodules/1.0-Lmod  GCC/6.3.0-2.27  OpenMPI/2.0.2
module load Python/3.6.1

j=0
for i in $(seq 2 2 1024); do
    if [ "$j" -gt "28" ]; then
        wait;
        j=0;
    fi
    cd results
    mkdir -p $i"x"$i
    cd $i"x"$i
    time python3 $PBS_O_WORKDIR/model.py $i > result.txt &
    cd $PBS_O_WORKDIR
    ((j++))
    ((j++))
done

wait

这些是我从运行tracejob 得到的日志：

kill_task: not killing process (pid=142457/state=Z) with sig 15
02/27/2018 21:42:26.758 M    kill_task: killing pid 142458 task 1 with sig 15
02/27/2018 21:42:26.758 M    kill_task: not killing process (pid=142460/state=Z) with sig 15
02/27/2018 21:42:26.758 M    kill_task: killing pid 142461 task 1 with sig 15
02/27/2018 21:42:26.758 M    kill_task: not killing process (pid=142463/state=Z) with sig 15
02/27/2018 21:42:26.758 M    kill_task: killing pid 142464 task 1 with sig 15
02/27/2018 21:42:26.758 M    kill_task: not killing process (pid=142466/state=Z) with sig 15
02/27/2018 21:42:26.758 M    kill_task: killing pid 142467 task 1 with sig 15
02/27/2018 21:42:26.758 M    kill_task: not killing process (pid=142469/state=Z) with sig 15
02/27/2018 21:42:26.758 M    kill_task: killing pid 142470 task 1 with sig 15
02/27/2018 21:42:26.758 M    kill_task: not killing process (pid=142472/state=Z) with sig 15
02/27/2018 21:42:26.758 M    kill_task: killing pid 142473 task 1 with sig 15
02/27/2018 21:42:26.758 M    kill_task: not killing process (pid=142475/state=Z) with sig 15
02/27/2018 21:42:26.758 M    kill_task: killing pid 142476 task 1 with sig 15
02/27/2018 21:42:26.758 M    kill_task: not killing process (pid=142478/state=Z) with sig 15
02/27/2018 21:42:26.758 M    kill_task: killing pid 142479 task 1 with sig 15
02/27/2018 21:42:26.758 M    kill_task: not killing process (pid=142481/state=Z) with sig 15
02/27/2018 21:42:26.758 M    kill_task: killing pid 142482 task 1 with sig 15
02/27/2018 21:42:26.758 M    kill_task: not killing process (pid=142483/state=Z) with sig 15
02/27/2018 21:42:26.788 M    kill_task: not killing process (pid=142442/state=Z) with sig 9
02/27/2018 21:42:26.788 M    kill_task: not killing process (pid=142445/state=Z) with sig 9
02/27/2018 21:42:26.788 M    kill_task: not killing process (pid=142448/state=Z) with sig 9
02/27/2018 21:42:26.788 M    kill_task: not killing process (pid=142451/state=Z) with sig 9
02/27/2018 21:42:26.788 M    kill_task: not killing process (pid=142454/state=Z) with sig 9
02/27/2018 21:42:26.788 M    kill_task: not killing process (pid=142457/state=Z) with sig 9
02/27/2018 21:42:26.788 M    kill_task: not killing process (pid=142460/state=Z) with sig 9
02/27/2018 21:42:26.788 M    kill_task: not killing process (pid=142463/state=Z) with sig 9
02/27/2018 21:42:26.788 M    kill_task: not killing process (pid=142466/state=Z) with sig 9
02/27/2018 21:42:26.788 M    kill_task: not killing process (pid=142469/state=Z) with sig 9
02/27/2018 21:42:26.788 M    kill_task: not killing process (pid=142472/state=Z) with sig 9
02/27/2018 21:42:26.788 M    kill_task: not killing process (pid=142475/state=Z) with sig 9
02/27/2018 21:42:26.788 M    kill_task: not killing process (pid=142478/state=Z) with sig 9
02/27/2018 21:42:26.788 M    kill_task: not killing process (pid=142481/state=Z) with sig 9
02/27/2018 21:42:26.788 M    kill_task: not killing process (pid=142483/state=Z) with sig 9
02/27/2018 21:42:26.788 M    scan_for_terminated: job 2205476.example.com task 1 terminated, sid=121918
02/27/2018 21:42:26.788 M    job was terminated
02/27/2018 21:42:26.818 M    obit sent to server
02/27/2018 21:42:26.882 M    removed job scrip

作业运行一个小时左右，然后随机死亡。我不确定为什么。我曾尝试增加挂墙时间，但这无济于事。

基本上，我的 python 脚本从 2 到 1024 读取 2 的倍数，并且每个脚本并行运行（以 27 个为一组，以避免服务器崩溃/换出）。谁能建议为什么会这样？

【问题讨论】：

标签： bash data-science scientific-computing pbs torque

【解决方案1】：

所以我通过使用 GNU 并行解决了这个问题。在某些服务器上，您可能需要像这样加载模块：module load gnu-parallel

然后在 pbs 脚本中我只是删除了 for 循环：

parallel -j28 python3 model.py {1} > results{1}.txt ::: $(seq 100 -2 2)

我还必须更改程序中的工作目录，以免结果被覆盖。

【讨论】：