【问题标题】:Dask multi-stage resource setup causes Failed to Serialize ErrorDask 多阶段资源设置导致 Failed to Serialize 错误
【发布时间】:2022-08-04 14:21:30
【问题描述】:

使用 Dask 文档中的确切代码 https://jobqueue.dask.org/en/latest/examples.html

如果页面发生变化,代码如下:

from dask_jobqueue import SLURMCluster
from distributed import Client
from dask import delayed

cluster = SLURMCluster(memory=\'8g\',
                       processes=1,
                       cores=2,
                       extra=[\'--resources ssdGB=200,GPU=2\'])

cluster.scale(2)
client = Client(cluster)

def step_1_w_single_GPU(data):
    return \"Step 1 done for: %s\" % data


def step_2_w_local_IO(data):
    return \"Step 2 done for: %s\" % data


stage_1 = [delayed(step_1_w_single_GPU)(i) for i in range(10)]
stage_2 = [delayed(step_2_w_local_IO)(s2) for s2 in stage_1]

result_stage_2 = client.compute(stage_2,
                                resources={tuple(stage_1): {\'GPU\': 1},
                                           tuple(stage_2): {\'ssdGB\': 100}})

这会导致这样的错误:

distributed.protocol.core - CRITICAL - Failed to Serialize
Traceback (most recent call last):
  File \"/opt/eagleseven/pyenv/e7cloudv0/lib/python3.8/site-packages/distributed/protocol/core.py\", line 76, in dumps
    frames[0] = msgpack.dumps(msg, default=_encode_default, use_bin_type=True)
  File \"/opt/eagleseven/pyenv/e7cloudv0/lib/python3.8/site-packages/msgpack/__init__.py\", line 38, in packb
    return Packer(**kwargs).pack(o)
  File \"msgpack/_packer.pyx\", line 294, in msgpack._cmsgpack.Packer.pack
  File \"msgpack/_packer.pyx\", line 300, in msgpack._cmsgpack.Packer.pack
  File \"msgpack/_packer.pyx\", line 297, in msgpack._cmsgpack.Packer.pack
  File \"msgpack/_packer.pyx\", line 264, in msgpack._cmsgpack.Packer._pack
  File \"msgpack/_packer.pyx\", line 231, in msgpack._cmsgpack.Packer._pack
  File \"msgpack/_packer.pyx\", line 231, in msgpack._cmsgpack.Packer._pack
  File \"msgpack/_packer.pyx\", line 264, in msgpack._cmsgpack.Packer._pack
  File \"msgpack/_packer.pyx\", line 231, in msgpack._cmsgpack.Packer._pack
  File \"msgpack/_packer.pyx\", line 231, in msgpack._cmsgpack.Packer._pack
  File \"msgpack/_packer.pyx\", line 229, in msgpack._cmsgpack.Packer._pack
  File \"msgpack/_packer.pyx\", line 264, in msgpack._cmsgpack.Packer._pack
  File \"msgpack/_packer.pyx\", line 291, in msgpack._cmsgpack.Packer._pack
TypeError: can not serialize \'Delayed\' object
distributed.comm.utils - ERROR - can not serialize \'Delayed\' object
Traceback (most recent call last):
  File \"/opt/eagleseven/pyenv/e7cloudv0/lib/python3.8/site-packages/distributed/comm/utils.py\", line 33, in _to_frames
    return list(protocol.dumps(msg, **kwargs))
  File \"/opt/eagleseven/pyenv/e7cloudv0/lib/python3.8/site-packages/distributed/protocol/core.py\", line 76, in dumps
    frames[0] = msgpack.dumps(msg, default=_encode_default, use_bin_type=True)
  File \"/opt/eagleseven/pyenv/e7cloudv0/lib/python3.8/site-packages/msgpack/__init__.py\", line 38, in packb
    return Packer(**kwargs).pack(o)
  File \"msgpack/_packer.pyx\", line 294, in msgpack._cmsgpack.Packer.pack
  File \"msgpack/_packer.pyx\", line 300, in msgpack._cmsgpack.Packer.pack
  File \"msgpack/_packer.pyx\", line 297, in msgpack._cmsgpack.Packer.pack
  File \"msgpack/_packer.pyx\", line 264, in msgpack._cmsgpack.Packer._pack
  File \"msgpack/_packer.pyx\", line 231, in msgpack._cmsgpack.Packer._pack
  File \"msgpack/_packer.pyx\", line 231, in msgpack._cmsgpack.Packer._pack
  File \"msgpack/_packer.pyx\", line 264, in msgpack._cmsgpack.Packer._pack
  File \"msgpack/_packer.pyx\", line 231, in msgpack._cmsgpack.Packer._pack
  File \"msgpack/_packer.pyx\", line 231, in msgpack._cmsgpack.Packer._pack
  File \"msgpack/_packer.pyx\", line 229, in msgpack._cmsgpack.Packer._pack
  File \"msgpack/_packer.pyx\", line 264, in msgpack._cmsgpack.Packer._pack
  File \"msgpack/_packer.pyx\", line 291, in msgpack._cmsgpack.Packer._pack
TypeError: can not serialize \'Delayed\' object
distributed.batched - ERROR - Error in batched write
Traceback (most recent call last):
  File \"/opt/eagleseven/pyenv/e7cloudv0/lib/python3.8/site-packages/distributed/batched.py\", line 94, in _background_send
    nbytes = yield self.comm.write(
  File \"/opt/eagleseven/pyenv/e7cloudv0/lib/python3.8/site-packages/tornado/gen.py\", line 762, in run
    value = future.result()
  File \"/opt/eagleseven/pyenv/e7cloudv0/lib/python3.8/site-packages/distributed/comm/tcp.py\", line 250, in write
    frames = await to_frames(
  File \"/opt/eagleseven/pyenv/e7cloudv0/lib/python3.8/site-packages/distributed/comm/utils.py\", line 50, in to_frames
    return _to_frames()
  File \"/opt/eagleseven/pyenv/e7cloudv0/lib/python3.8/site-packages/distributed/comm/utils.py\", line 33, in _to_frames
    return list(protocol.dumps(msg, **kwargs))
  File \"/opt/eagleseven/pyenv/e7cloudv0/lib/python3.8/site-packages/distributed/protocol/core.py\", line 76, in dumps
    frames[0] = msgpack.dumps(msg, default=_encode_default, use_bin_type=True)
  File \"/opt/eagleseven/pyenv/e7cloudv0/lib/python3.8/site-packages/msgpack/__init__.py\", line 38, in packb
    return Packer(**kwargs).pack(o)
  File \"msgpack/_packer.pyx\", line 294, in msgpack._cmsgpack.Packer.pack
  File \"msgpack/_packer.pyx\", line 300, in msgpack._cmsgpack.Packer.pack
  File \"msgpack/_packer.pyx\", line 297, in msgpack._cmsgpack.Packer.pack
  File \"msgpack/_packer.pyx\", line 264, in msgpack._cmsgpack.Packer._pack
  File \"msgpack/_packer.pyx\", line 231, in msgpack._cmsgpack.Packer._pack
  File \"msgpack/_packer.pyx\", line 231, in msgpack._cmsgpack.Packer._pack
  File \"msgpack/_packer.pyx\", line 264, in msgpack._cmsgpack.Packer._pack
  File \"msgpack/_packer.pyx\", line 231, in msgpack._cmsgpack.Packer._pack
  File \"msgpack/_packer.pyx\", line 231, in msgpack._cmsgpack.Packer._pack
  File \"msgpack/_packer.pyx\", line 229, in msgpack._cmsgpack.Packer._pack
  File \"msgpack/_packer.pyx\", line 264, in msgpack._cmsgpack.Packer._pack
  File \"msgpack/_packer.pyx\", line 291, in msgpack._cmsgpack.Packer._pack
TypeError: can not serialize \'Delayed\' object

Python 版本:3.8.10 黎明:2022.2.0 工作队列:0.7.3 问题不言而喻。设置就像在文档中一样。我没有什么可以解释的了,但是stackoverflow说我的细节到代码太低了,所以我需要写更多的东西来发布这个问题。

  • 这似乎是文档中的一个错误。您可以将其直接发布到dask-jobqueue issues board。我不知道这样做的正确方法是什么,但我确信有人会这样做,并且文档当然不应该让你误入歧途。我在dask.distributed API docs 中看不到任何内容,表明您可以提供一个字典,其中期货/延迟作为键,普通资源规范作为值。

标签: python python-3.x dask dask-distributed dask-delayed


【解决方案1】:

正如@Michael Delgado 在 cmets 中所指出的,这似乎是文档的问题(提出了here)。资源是一个字典,每个键是资源的名称,值表示任务使用的数量。

在对相关问题的回答中,最初的提交作者 Matt Rocklin 提到经常请求此功能(指定任务级资源),但目前不可用:https://stackoverflow.com/a/63310721/10693596

一种可能性是将annotation 用于图形的特定组件,请参阅this answer。

【讨论】:

  • 分层注释将是适当的方法。
猜你喜欢
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 2018-10-24
  • 1970-01-01
  • 1970-01-01
相关资源
最近更新 更多