【发布时间】:2019-10-03 03:08:56
【问题描述】:
我有一个在本地运行的 python 2.7 脚本来启动 Apache Beam / Google Dataflow 作业(SDK 2.12.0)。该作业从 Google 存储桶中获取一个 csv 文件,对其进行处理,然后在 Google Datastore 中为每一行创建一个实体。该脚本多年来运行良好......但现在它失败了:
INFO:root:2019-05-15T22:07:11.481Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:root:2019-05-15T21:47:13.370Z: JOB_MESSAGE_ERROR: Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 773, in run
self._load_main_session(self.local_staging_directory)
File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 489, in _load_main_session
pickler.load_session(session_file)
File "/usr/local/lib/python2.7/dist-packages/apache_beam/internal/pickler.py", line 280, in load_session
return dill.load_session(file_path)
File "/usr/local/lib/python2.7/dist-packages/dill/_dill.py", line 410, in load_session
module = unpickler.load()
File "/usr/lib/python2.7/pickle.py", line 864, in load
dispatch[key](self)
File "/usr/lib/python2.7/pickle.py", line 1139, in load_reduce
value = func(*args)
File "/usr/local/lib/python2.7/dist-packages/dill/_dill.py", line 827, in _import_module
return __import__(import_name)
ImportError: No module named idlelib
我相信此错误发生在工作人员级别(而不是本地)。我没有在我的脚本中提及它。为了确保不是我,我已经在本地安装了所有 google-cloud 包、apache-beam[gcp] 等的更新——以防万一。我尝试将 idlelib 导入到我的脚本中,我得到了同样的错误。有什么建议吗?
【问题讨论】:
标签: python-2.7 google-cloud-platform google-cloud-dataflow apache-beam