【发布时间】:2022-08-04 14:21:30
【问题描述】:
使用 Dask 文档中的确切代码 https://jobqueue.dask.org/en/latest/examples.html
如果页面发生变化,代码如下:
from dask_jobqueue import SLURMCluster
from distributed import Client
from dask import delayed
cluster = SLURMCluster(memory=\'8g\',
processes=1,
cores=2,
extra=[\'--resources ssdGB=200,GPU=2\'])
cluster.scale(2)
client = Client(cluster)
def step_1_w_single_GPU(data):
return \"Step 1 done for: %s\" % data
def step_2_w_local_IO(data):
return \"Step 2 done for: %s\" % data
stage_1 = [delayed(step_1_w_single_GPU)(i) for i in range(10)]
stage_2 = [delayed(step_2_w_local_IO)(s2) for s2 in stage_1]
result_stage_2 = client.compute(stage_2,
resources={tuple(stage_1): {\'GPU\': 1},
tuple(stage_2): {\'ssdGB\': 100}})
这会导致这样的错误:
distributed.protocol.core - CRITICAL - Failed to Serialize
Traceback (most recent call last):
File \"/opt/eagleseven/pyenv/e7cloudv0/lib/python3.8/site-packages/distributed/protocol/core.py\", line 76, in dumps
frames[0] = msgpack.dumps(msg, default=_encode_default, use_bin_type=True)
File \"/opt/eagleseven/pyenv/e7cloudv0/lib/python3.8/site-packages/msgpack/__init__.py\", line 38, in packb
return Packer(**kwargs).pack(o)
File \"msgpack/_packer.pyx\", line 294, in msgpack._cmsgpack.Packer.pack
File \"msgpack/_packer.pyx\", line 300, in msgpack._cmsgpack.Packer.pack
File \"msgpack/_packer.pyx\", line 297, in msgpack._cmsgpack.Packer.pack
File \"msgpack/_packer.pyx\", line 264, in msgpack._cmsgpack.Packer._pack
File \"msgpack/_packer.pyx\", line 231, in msgpack._cmsgpack.Packer._pack
File \"msgpack/_packer.pyx\", line 231, in msgpack._cmsgpack.Packer._pack
File \"msgpack/_packer.pyx\", line 264, in msgpack._cmsgpack.Packer._pack
File \"msgpack/_packer.pyx\", line 231, in msgpack._cmsgpack.Packer._pack
File \"msgpack/_packer.pyx\", line 231, in msgpack._cmsgpack.Packer._pack
File \"msgpack/_packer.pyx\", line 229, in msgpack._cmsgpack.Packer._pack
File \"msgpack/_packer.pyx\", line 264, in msgpack._cmsgpack.Packer._pack
File \"msgpack/_packer.pyx\", line 291, in msgpack._cmsgpack.Packer._pack
TypeError: can not serialize \'Delayed\' object
distributed.comm.utils - ERROR - can not serialize \'Delayed\' object
Traceback (most recent call last):
File \"/opt/eagleseven/pyenv/e7cloudv0/lib/python3.8/site-packages/distributed/comm/utils.py\", line 33, in _to_frames
return list(protocol.dumps(msg, **kwargs))
File \"/opt/eagleseven/pyenv/e7cloudv0/lib/python3.8/site-packages/distributed/protocol/core.py\", line 76, in dumps
frames[0] = msgpack.dumps(msg, default=_encode_default, use_bin_type=True)
File \"/opt/eagleseven/pyenv/e7cloudv0/lib/python3.8/site-packages/msgpack/__init__.py\", line 38, in packb
return Packer(**kwargs).pack(o)
File \"msgpack/_packer.pyx\", line 294, in msgpack._cmsgpack.Packer.pack
File \"msgpack/_packer.pyx\", line 300, in msgpack._cmsgpack.Packer.pack
File \"msgpack/_packer.pyx\", line 297, in msgpack._cmsgpack.Packer.pack
File \"msgpack/_packer.pyx\", line 264, in msgpack._cmsgpack.Packer._pack
File \"msgpack/_packer.pyx\", line 231, in msgpack._cmsgpack.Packer._pack
File \"msgpack/_packer.pyx\", line 231, in msgpack._cmsgpack.Packer._pack
File \"msgpack/_packer.pyx\", line 264, in msgpack._cmsgpack.Packer._pack
File \"msgpack/_packer.pyx\", line 231, in msgpack._cmsgpack.Packer._pack
File \"msgpack/_packer.pyx\", line 231, in msgpack._cmsgpack.Packer._pack
File \"msgpack/_packer.pyx\", line 229, in msgpack._cmsgpack.Packer._pack
File \"msgpack/_packer.pyx\", line 264, in msgpack._cmsgpack.Packer._pack
File \"msgpack/_packer.pyx\", line 291, in msgpack._cmsgpack.Packer._pack
TypeError: can not serialize \'Delayed\' object
distributed.batched - ERROR - Error in batched write
Traceback (most recent call last):
File \"/opt/eagleseven/pyenv/e7cloudv0/lib/python3.8/site-packages/distributed/batched.py\", line 94, in _background_send
nbytes = yield self.comm.write(
File \"/opt/eagleseven/pyenv/e7cloudv0/lib/python3.8/site-packages/tornado/gen.py\", line 762, in run
value = future.result()
File \"/opt/eagleseven/pyenv/e7cloudv0/lib/python3.8/site-packages/distributed/comm/tcp.py\", line 250, in write
frames = await to_frames(
File \"/opt/eagleseven/pyenv/e7cloudv0/lib/python3.8/site-packages/distributed/comm/utils.py\", line 50, in to_frames
return _to_frames()
File \"/opt/eagleseven/pyenv/e7cloudv0/lib/python3.8/site-packages/distributed/comm/utils.py\", line 33, in _to_frames
return list(protocol.dumps(msg, **kwargs))
File \"/opt/eagleseven/pyenv/e7cloudv0/lib/python3.8/site-packages/distributed/protocol/core.py\", line 76, in dumps
frames[0] = msgpack.dumps(msg, default=_encode_default, use_bin_type=True)
File \"/opt/eagleseven/pyenv/e7cloudv0/lib/python3.8/site-packages/msgpack/__init__.py\", line 38, in packb
return Packer(**kwargs).pack(o)
File \"msgpack/_packer.pyx\", line 294, in msgpack._cmsgpack.Packer.pack
File \"msgpack/_packer.pyx\", line 300, in msgpack._cmsgpack.Packer.pack
File \"msgpack/_packer.pyx\", line 297, in msgpack._cmsgpack.Packer.pack
File \"msgpack/_packer.pyx\", line 264, in msgpack._cmsgpack.Packer._pack
File \"msgpack/_packer.pyx\", line 231, in msgpack._cmsgpack.Packer._pack
File \"msgpack/_packer.pyx\", line 231, in msgpack._cmsgpack.Packer._pack
File \"msgpack/_packer.pyx\", line 264, in msgpack._cmsgpack.Packer._pack
File \"msgpack/_packer.pyx\", line 231, in msgpack._cmsgpack.Packer._pack
File \"msgpack/_packer.pyx\", line 231, in msgpack._cmsgpack.Packer._pack
File \"msgpack/_packer.pyx\", line 229, in msgpack._cmsgpack.Packer._pack
File \"msgpack/_packer.pyx\", line 264, in msgpack._cmsgpack.Packer._pack
File \"msgpack/_packer.pyx\", line 291, in msgpack._cmsgpack.Packer._pack
TypeError: can not serialize \'Delayed\' object
Python 版本:3.8.10 黎明:2022.2.0 工作队列:0.7.3 问题不言而喻。设置就像在文档中一样。我没有什么可以解释的了,但是stackoverflow说我的细节到代码太低了,所以我需要写更多的东西来发布这个问题。
-
这似乎是文档中的一个错误。您可以将其直接发布到dask-jobqueue issues board。我不知道这样做的正确方法是什么,但我确信有人会这样做,并且文档当然不应该让你误入歧途。我在dask.distributed API docs 中看不到任何内容,表明您可以提供一个字典,其中期货/延迟作为键,普通资源规范作为值。
标签: python python-3.x dask dask-distributed dask-delayed