【问题标题】:Error message when running jobs using torque. read_tcp_reply, Mismatching protocols. Expected protocol 4 but read reply for 0使用扭矩运行作业时出现错误消息。 read_tcp_reply,协议不匹配。预期协议 4,但读取 0 的回复
【发布时间】:2016-12-06 12:49:39
【问题描述】:

我的系统是 Cent OS7,我安装了torque-6.1.0 配置./configure --prefix=/opt/pbs --with-debug --with-scp --disable-gcc-warnings

我的服务器名称是“node00”,我添加了一个名为“node01”的从节点

[root@node00 torque]# pbsnodes
node01
     state = free
     power_state = Running
     np = 16
     ntype = cluster
     status = opsys=linux,uname=Linux node01 2.6.32-358.el6.x86_64 #1 SMP Fri Feb 22 00:31:26 UTC 2013 x86_64,nsessions=0,nusers=0,idletime=7057,totmem=98382176kb,availmem=97993700kb,physmem=32846184kb,ncpus=16,loadave=0.00,gres=,netload=286314300,state=free,varattr= ,cpuclock=Fixed,macaddr=0c:c4:7a:02:ba:98,version=6.1.0,rectime=1481028058,jobs=
     mom_service_port = 15002
     mom_manager_port = 15003

我提交了一份简单的工作echo "sleep 5" | qsub 然后它在qstat -f中返回了一条错误消息

queue_type = E
sched_hint = Unable to copy files back - please see the mother superior's
    log for exact details.
comment = Job started on Tue Dec 06 at 21:35

于是我看了女上司的日志vi /var/spool/torque/mom_logs/20161206

12/06/2016 21:35:33.397;02;   pbs_mom.14693;Svr;Log;Log opened
12/06/2016 21:35:33.397;02;   pbs_mom.14693;Svr;pbs_mom;Torque Mom Version = 6.1.0, loglevel = 0
12/06/2016 21:35:33.404;02;   pbs_mom.14693;Svr;setpbsserver;node00
12/06/2016 21:35:33.404;02;   pbs_mom.14693;Svr;mom_server_add;server node00 added
12/06/2016 21:35:33.405;02;   pbs_mom.14694;n/a;initialize;independent
12/06/2016 21:35:33.405;02;   pbs_mom.14694;Svr;dep_initialize;mom is now oom-killer safe
12/06/2016 21:35:33.405;02;   pbs_mom.14694;Svr;read_mom_hierarchy;No local mom hierarchy file found, will request from server.
12/06/2016 21:35:33.407;128;   pbs_mom.14694;Svr;pbs_mom;before init_abort_jobs
12/06/2016 21:35:33.410;02;   pbs_mom.14694;Svr;pbs_mom;Is up
12/06/2016 21:35:33.410;02;   pbs_mom.14694;Svr;setup_program_environment;MOM executable path and mtime at launch: /opt/pbs/sbin/pbs_mom 1481027487
12/06/2016 21:35:33.414;02;   pbs_mom.14694;Svr;pbs_mom;Torque Mom Version = 6.1.0, loglevel = 0
12/06/2016 21:35:33.419;01;   pbs_mom.14706;Svr;pbs_mom;LOG_ERROR::read_tcp_reply, Mismatching protocols. Expected protocol 4 but read reply for 0
12/06/2016 21:35:33.419;01;   pbs_mom.14706;Svr;pbs_mom;LOG_ERROR::read_tcp_reply, Could not read reply for protocol 4 command 4: End of File
12/06/2016 21:35:33.419;01;   pbs_mom.14706;Svr;pbs_mom;LOG_ERROR::mom_server_update_stat, Couldn't read a reply from the server
12/06/2016 21:35:33.419;01;   pbs_mom.14706;Svr;pbs_mom;LOG_ERROR::send_update_to_a_server, Could not contact any of the servers to send an update
12/06/2016 21:35:33.419;01;   pbs_mom.14706;Svr;pbs_mom;LOG_ERROR::send_update_to_a_server, Status not successfully updated for 1 MOM status update intervals
12/06/2016 21:36:18.445;01;   pbs_mom.14795;Svr;pbs_mom;LOG_ERROR::read_tcp_reply, Mismatching protocols. Expected protocol 4 but read reply for 0
12/06/2016 21:36:18.445;01;   pbs_mom.14795;Svr;pbs_mom;LOG_ERROR::read_tcp_reply, Could not read reply for protocol 4 command 4: End of File
12/06/2016 21:36:18.445;01;   pbs_mom.14795;Svr;pbs_mom;LOG_ERROR::mom_server_update_stat, Couldn't read a reply from the server
12/06/2016 21:36:18.445;01;   pbs_mom.14795;Svr;pbs_mom;LOG_ERROR::send_update_to_a_server, Could not contact any of the servers to send an update
12/06/2016 21:36:18.445;01;   pbs_mom.14795;Svr;pbs_mom;LOG_ERROR::send_update_to_a_server, Status not successfully updated for 2 MOM status update intervals

node01node00 似乎无法互相发送数据。这样对吗?我该如何解决这个问题?

【问题讨论】:

  • 要检查的东西:确认服务器和计算是相同的版本。确认您没有运行额外的 pbs_mom 进程。在服务器和节点上禁用 ipchains 和 iptables。在 mom_priv/config 中增加 $loglevel,重启 pbs_mom,然后查看 mom 的 log 和 syslog。
  • 你提到的检查清单做得正确。我安装了较低扭矩的版本,然后错误消息更像Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password)。它看起来像 ssh 权限问题。但是我用./configure --with-rcp=/usr/bin/rcp安装了这个版本,那为什么我的扭矩还想用ssh呢? ..
  • 无论如何.. 我生成了授权密钥,然后它运行良好。但是为什么torque 不使用rsh 仍然让我感到困惑。谢谢!

标签: pbs torque


【解决方案1】:

关于标题文本:“read_tcp_reply, Mismatching protocols. Expected protocol 4 but read reply for 0” 这是在以下情况下显示在系统上的错误:

  1. pbs_mom 在 pbs_server 未知的节点上运行(从节点文件中排除)
  2. 当 /var/spool/torque/server_priv/jobs 目录被本应在作业终止时删除的作业文件堵塞时(这很容易增长到数千个文件,因为 pbs_server 在清理方面是出了名的糟糕)。同样的事情也适用于 /var/spool/torque/server_priv/arrays 目录。
  3. 清除上述两种情况后,在具有 400 个节点和 1000 个作业(排队和/或运行)的系统上仍然可以看到它。在这种情况下,它每小时发生 5-10 次。

在所有情况下,tcpdump 都会在 pbs_server 端显示 mom 在发送状态更新后收到 tcp reset。它很容易被追踪到:

    tcpdump -i <interface> tcp port 15001 and tcp[13]=4

    08:15:25.162128 IP 10.44.0.94.15001 > 10.44.1.220.215: Flags [R], seq 2437471647, win 0, length 0

On the node this is logged:
    10/13/2018 08:15:25.161;01; pbs_mom.17548;Svr;pbs_mom;LOG_ERROR::read_tcp_reply, Mismatching protocols. Expected protocol 4 but read reply for 0
    10/13/2018 08:15:25.161;01;   pbs_mom.17548;Svr;pbs_mom;LOG_ERROR::read_tcp_reply, Could not read reply for protocol 4 command 4: End of File
    10/13/2018 08:15:25.161;01;   pbs_mom.17548;Svr;pbs_mom;LOG_ERROR::mom_server_update_stat, Couldn't read a reply from the server
    10/13/2018 08:15:25.161;01;   pbs_mom.17548;Svr;pbs_mom;LOG_ERROR::send_update_to_a_server, Could not contact any of the servers to send an update
    10/13/2018 08:15:25.161;01;   pbs_mom.17548;Svr;pbs_mom;LOG_ERROR::send_update_to_a_server, Status not successfully updated for 1 MOM status update intervals

更新: 我们最终通过在文件 /var/spool/torque/server_priv/mom_hierarchy 中实现 MOM 层次结构解决了这个问题。 对于一个 500 个节点的集群,我们定义了 8 个组(mom_hierarchy 中的路径),其中顶层为 2 个节点,一层为该组中的其余节点。像这样的:

<path>
<level>node1,node2</level>
<level> comma separated list of some 60 nodes</level>
</path>
<path> 
<level>node2,node1</level>
<level comma separated list of some 60 nodes</level>
</path>
<path>
<level>node3,node4</level>
<level>comma separated list of some 60 nodes</level>
</path>
<path>
<level>node4,node3</level>
<level>comma separated list of some 60 nodes</level>
</path>
.....

【讨论】:

    猜你喜欢
    • 1970-01-01
    • 1970-01-01
    • 2023-03-30
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 2023-03-12
    • 1970-01-01
    • 2020-11-07
    相关资源
    最近更新 更多