分布式张量流错误/

Question

在运行分布式张量流 (TF v0.9.0rc0) 设置时，我启动了 3 个参数服务器，然后启动了 6 个工作器。参数服务器似乎很好，给出消息Started server with target: grpc://localhost:2222。但是工作人员给出了我有疑问的其他错误（如下）。

在我看来，有时计算机无法相互通信，从而产生socket error, connection refused 错误。似乎工作人员在初始化变量时无法找到参数服务器并给出Cannot assign a device 错误。

谁能帮我理解这些错误分别意味着什么，每个问题有多大，如果需要的话，也许可以给我一些解决方法的建议？

具体来说：

1. 为什么我收到socket errors？
2. 为什么会有Master init: Unavailable 问题/它们是什么意思？
3. 如何确保请求的设备可用？
4. 这看起来像是我应该发布到 tensorflow 的 github account 的问题页面吗？

设置注意事项：

• 所有计算机报告 TensorFlow 版本：0.9.0rc0 (python -c "import tensorflow as tf; print(tf.__version__);"), 虽然有一些可能是从源代码而不是 pip 包安装的，如果这很重要的话。
• 所有计算机都在同一个 1Gb 以太网交换机上。
• 硬件基本相同，一些工作人员运行双 GPU。

---

他们都给出这个错误（IP地址改变）：

E0719 12:06:17.711635677    2543 tcp_client_posix.c:173]  
 failed to connect to 'ipv4:192.168.xx.xx:2222': socket error: connection refused

---

但所有非首席员工也都给予：

E tensorflow/core/distributed_runtime/master.cc:202] Master init: Unavailable: 

---

此外，一些非首席员工崩溃，出现此错误：

Traceback (most recent call last):  
    File "main.py", line 219, in <module>  
        r.main()  
    File "main.py", line 119, in main  
        with sv.prepare_or_wait_for_session(server.target, config=tf.ConfigProto(gpu_options=gpu_options)) as sess:  
    File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/supervisor.py", line 691, in prepare_or_wait_for_sessionn max_wait_secs=max_wait_secs)  
    File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/session_manager.py", line 282, in wait_for_session  
        sess.run([self._local_init_op])  
    File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 372, in run
        run_metadata_ptr)  
    File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 636, in _run  
        feed_dict_string, options, run_metadata)  
    File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 708, in _do_run  
        target_list, options, run_metadata)  
    File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 728, in _do_call  
        raise type(e)(node_def, op, message)  
    tensorflow.python.framework.errors.InvalidArgumentError: Cannot assign a device to node 'save/restore_slice_23':
        Could not satisfy explicit device specification '/job:ps/task:3/device:CPU:0'
        because no devices matching that specification are registered in this process; available devices: 
            /job:ps/replica:0/task:0/cpu:0,
            /job:ps/replica:0/task:1/cpu:0,
            /job:ps/replica:0/task:2/cpu:0,
            /job:ps/replica:0/task:4/cpu:0,
            /job:worker/replica:0/task:0/cpu:0,
            /job:worker/replica:0/task:0/gpu:0,
            /job:worker/replica:0/task:1/cpu:0,
            /job:worker/replica:0/task:1/gpu:0,
            /job:worker/replica:0/task:2/cpu:0,
            /job:worker/replica:0/task:2/gpu:0 
[[Node: save/restore_slice_23 = RestoreSlice[dt=DT_FLOAT, preferred_shard=-1, _device="/job:ps/task:3/device:CPU:0"](save/Const, save/restore_slice_23/tensor_name, save/restore_slice_23/shape_and_slice)]]
Caused by op u'save/restore_slice_23', defined at:  
    File "main.py", line 219, in <module>  
        r.main()  
    File "main.py", line 101, in main  
        saver = tf.train.Saver()  
    File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 845, in __init__  
        restore_sequentially=restore_sequentially)  
    File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 515, in build  
        filename_tensor, vars_to_save, restore_sequentially, reshape)  
    File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 271, in _AddRestoreOps  
        values = self.restore_op(filename_tensor, vs, preferred_shard)  
    File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 186, in restore_op
        preferred_shard=preferred_shard)  
    File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/io_ops.py", line 202, in _restore_slice  
        preferred_shard, name=name)  
    File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_io_ops.py", line 358, in _restore_slice  
        preferred_shard=preferred_shard, name=name)  
    File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/op_def_library.py", line 704, in apply_op  
        op_def=op_def)  
    File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2260, in create_op  
        original_op=self._default_original_op, op_def=op_def)  
    File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1230, in __init__  
        self._traceback = _extract_stack()

Answer 1

我知道我的问题是什么。

TL;DR：负责人需要了解所有变量，以便初始化它们所有。非首席员工不能创建自己的变量。

我正在转换一个旧程序，其中所有工作人员都有一些独立变量，但需要共享一些变量（我使用 ZMQ 来传递这些）到分布式 TensorFlow 设置，并且忘记初始化所有变量上的所有变量工人。我有类似的东西

# Create worker specific variable
with tf.variable_scope("world_{}".format(**worker_id**)):
    w1 = tf.get_variable("weight", shape=(input_dim, hidden_dim), dtype=tf.float32, initializer=tf.truncated_normal_initializer())

而不是做这样的事情：

# Create all worker specific variables
all_w1 = {}
for worker in worker_cnt:
    with tf.variable_scope("world_{}".format(**worker_id**)):  
        all_w1[worker] = tf.get_variable("weight", shape=(input_dim, hidden_dim), dtype=tf.float32, initializer=tf.truncated_normal_initializer())

# grab worker specific variable
w1 = all_w1[**worker_id**] 

---

至于错误...

我怀疑这会导致一些工人因上面的Master init: Unavailable: 错误消息而死亡，因为酋长从来不知道工人想要创建的变量。

对于设备不可用（第 3 次）错误没有找到该设备的原因，我没有可靠的解释，但我认为又是这样，因为只有主人才能创建它，而且他不知道新的变量。

第一个错误似乎是因为计算机在出现故障后还没有准备好说话，因为我在修复后没有看到该错误。如果我杀死一个工人并重新启动他，我仍然可以看到它，但如果他们都一起启动，这似乎不是问题。

---

无论如何，如果以后有人遇到同样的错误，我希望这会有所帮助。