缺少可选依赖项“gcsfs”。处理 GCS 文件需要 gcsfs 库使用 pip 或 conda 安装 gcsfs

Question

我正在尝试使用 Dataflow 将存储在 GCS 存储桶中的 CSV 文件读入数据帧。作业因错误而失败：

raise_with_traceback raise exc.with_traceback(traceback) File "apache_beam/runners/common.py", line 1213, in apache_beam.runners.common.DoFnRunner.process File "apache_beam/runners/common.py", line 570, in apache_beam.runners.common.SimpleInvoker.invoke_process File "/home/curate_try_final.py", line 71, in <lambda> File "/home/curate_try_final.py", line 66, in convert_to_parquet File "/usr/local/lib/python3.7/site-packages/pandas/io/parsers.py", line 685, in parser_f return _read(filepath_or_buffer, kwds) File "/usr/local/lib/python3.7/site-packages/pandas/io/parsers.py", line 440, in _read filepath_or_buffer, encoding, compression File "/usr/local/lib/python3.7/site-packages/pandas/io/common.py", line 213, in get_filepath_or_buffer from pandas.io import gcs File "/usr/local/lib/python3.7/site-packages/pandas/io/gcs.py", line 5, in <module> "gcsfs", extra="The gcsfs library is required to handle GCS files" File "/usr/local/lib/python3.7/site-packages/pandas/compat/_optional.py", line 93, in import_optional_dependency raise ImportError(message.format(name=name, extra=extra)) from None ImportError: Missing optional dependency 'gcsfs'. The gcsfs library is required to handle GCS files Use pip or conda to install gcsfs. [while running 'ConvertToParquet']

即使使用安装后也显示相同的错误

sudo pip3 install gcsfs

有人可以帮忙吗？

Answer 1

可以使用 conda 或 pip 安装 GCSFS 库：

conda install -c conda-forge gcsfs

或

pip install gcsfs

或通过克隆存储库：

git clone https://github.com/dask/gcsfs/
cd gcsfs/
pip install .

更多详细信息请见here。