反省到永远运行的盐高状态？答案

【问题标题】：Introspection into forever-running salt highstate?反省到永远运行的盐高状态？
【发布时间】：2014-10-01 23:55:20
【问题描述】：

我一直在尝试使用 Salt，并且成功地锁定了我的 highstate 命令。它已经运行了好几个小时了，尽管没有什么可以保证这样的时间。

我所做的最后一个更改是修改 nginx 的 service.watch 状态。目前为：

nginx:
  pkg.installed:
    - name: nginx
  service:
    - running
    - enable: True
    - restart: True
    - watch:
      - file: /etc/nginx/nginx.conf
      - file: /etc/nginx/sites-available/default.conf
      - pkg: nginx

我所做的最后一个更改是将第二个 file: 参数添加到 watch。

在让它运行一整夜之后，状态没有任何变化，我随后 Ctrl-C'd 进程。 sudo salt -v 'web*' state.highstate -l debug 的最后输出是：

[DEBUG   ] Checking whether jid 20140403022217881027 is still running
[DEBUG   ] get_returns for jid 20140403103702550977 sent to set(['web1.mysite.com']) will timeout at 10:37:04
[DEBUG   ] jid 20140403103702550977 found all minions
Execution is still running on web1.mysite.com
^CExiting on Ctrl-C
This job's jid is:                                                                                                                                     
20140403022217881027
The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later run:
salt-run jobs.lookup_jid 20140403022217881027

立即再次运行，我得到了这个：

$ sudo salt -v 'web*' state.highstate -l debug
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Missing configuration file: /home/eykd/.salt
[DEBUG   ] Configuration file path: /etc/salt/master
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Missing configuration file: /home/eykd/.salt
[DEBUG   ] LocalClientEvent PUB socket URI: ipc:///var/run/salt/master/master_event_pub.ipc
[DEBUG   ] LocalClientEvent PULL socket URI: ipc:///var/run/salt/master/master_event_pull.ipc
Executing job with jid 20140403103715454952
-------------------------------------------

[DEBUG   ] Checking whether jid 20140403103715454952 is still running
[DEBUG   ] get_returns for jid 20140403103720479720 sent to set(['web1.praycontinue.com']) will timeout at 10:37:22
[INFO    ] jid 20140403103720479720 minions set(['web1.mysite.com']) did not return in time
[DEBUG   ] Loaded no_out as virtual quiet
[DEBUG   ] Loaded json_out as virtual json
[DEBUG   ] Loaded yaml_out as virtual yaml
[DEBUG   ] Loaded pprint_out as virtual pprint
web1.praycontinue.com:
    Minion did not return

然后我运行相同的命令，并收到 this：

$ sudo salt -v 'web*' state.highstate -l debug
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Missing configuration file: /home/eykd/.salt
[DEBUG   ] Configuration file path: /etc/salt/master
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Missing configuration file: /home/eykd/.salt
[DEBUG   ] LocalClientEvent PUB socket URI: ipc:///var/run/salt/master/master_event_pub.ipc
[DEBUG   ] LocalClientEvent PULL socket URI: ipc:///var/run/salt/master/master_event_pull.ipc
Executing job with jid 20140403103729848942
-------------------------------------------

[DEBUG   ] Loaded no_out as virtual quiet
[DEBUG   ] Loaded json_out as virtual json
[DEBUG   ] Loaded yaml_out as virtual yaml
[DEBUG   ] Loaded pprint_out as virtual pprint
web1.mysite.com:
    Data failed to compile:
----------
    The function "state.highstate" is running as PID 4417 and was started at 2014, Apr 03 02:22:17.881027 with jid 20140403022217881027

在 PID 4417 下没有运行任何进程。运行 sudo salt-run jobs.lookup_jid 20140403022217881027 不会显示任何内容。

不幸的是，我无法通过 ssh 连接到 minion，因为 salt 尚未配置我的 authorized_keys。 :\

那么，我的问题是：到底出了什么问题，我到底是怎么发现的？

【问题讨论】：

标签： salt-stack

【解决方案1】：

所以，经过大量调试，这是由于 Nginx 服务配置不当造成的。 service nginx start 挂了，salt-minion 也挂了。

【讨论】：

【解决方案2】：

当我使用 Ctrl-C 中止 salt-master 上的 state.highstate 运行时，我遇到了这种情况。原来，错误信息中引用的 PID 实际上是 minion 机器上的 salt-minion 进程的 PID。

我能够通过在 minion 上重新启动 salt-minion 进程，然后在 master 上重新执行 state.highstate 来解决问题。

【讨论】：