Geeting ModuleNotFoundError：在作曲家 DAG 运行中没有名为“pytz”的模块答案

【问题标题】：Geeting ModuleNotFoundError: No module named 'pytz' in composer DAG runGeeting ModuleNotFoundError：在作曲家 DAG 运行中没有名为“pytz”的模块
【发布时间】：2021-08-19 15:44:47
【问题描述】：

我在 Composer DAG 中进行 Bigquery 操作并收到以下错误：

Event with job id 36cc0fe962103bf2bb7a9c Failed
[2021-08-19 15:13:07,807] {base_task_runner.py:113} INFO - Job 263145: Subtask sample [2021-08-19 15:13:07,806] {pod_launcher.py:125} INFO - b'    from google.cloud.bigquery.routine import RoutineReference\n'
[2021-08-19 15:13:07,807] {base_task_runner.py:113} INFO - Job 263145: Subtask sample [2021-08-19 15:13:07,807] {pod_launcher.py:125} INFO - b'  File "/usr/local/lib/python3.7/dist-packages/google/cloud/bigquery/routine/__init__.py", line 18, in <module>\n'
[2021-08-19 15:13:07,810] {base_task_runner.py:113} INFO - Job 263145: Subtask sample [2021-08-19 15:13:07,808] {pod_launcher.py:125} INFO - b'    from google.cloud.bigquery.enums import DeterminismLevel\n'
[2021-08-19 15:13:07,810] {base_task_runner.py:113} INFO - Job 263145: Subtask sample [2021-08-19 15:13:07,808] {pod_launcher.py:125} INFO - b'  File "/usr/local/lib/python3.7/dist-packages/google/cloud/bigquery/enums.py", line 21, in <module>\n'
[2021-08-19 15:13:07,811] {base_task_runner.py:113} INFO - Job 263145: Subtask sample [2021-08-19 15:13:07,808] {pod_launcher.py:125} INFO - b'    from google.cloud.bigquery.query import ScalarQueryParameterType\n'
[2021-08-19 15:13:07,812] {base_task_runner.py:113} INFO - Job 263145: Subtask sample [2021-08-19 15:13:07,809] {pod_launcher.py:125} INFO - b'  File "/usr/local/lib/python3.7/dist-packages/google/cloud/bigquery/query.py", line 23, in <module>\n'
[2021-08-19 15:13:07,812] {base_task_runner.py:113} INFO - Job 263145: Subtask sample [2021-08-19 15:13:07,809] {pod_launcher.py:125} INFO - b'    from google.cloud.bigquery.table import _parse_schema_resource\n'
[2021-08-19 15:13:07,812] {base_task_runner.py:113} INFO - Job 263145: Subtask sample [2021-08-19 15:13:07,809] {pod_launcher.py:125} INFO - b'  File "/usr/local/lib/python3.7/dist-packages/google/cloud/bigquery/table.py", line 23, in <module>\n'
[2021-08-19 15:13:07,813] {base_task_runner.py:113} INFO - Job 263145: Subtask sample [2021-08-19 15:13:07,809] {pod_launcher.py:125} INFO - b'    import pytz\n'
[2021-08-19 15:13:07,813] {base_task_runner.py:113} INFO - Job 263145: Subtask sample [2021-08-19 15:13:07,809] {pod_launcher.py:125} INFO - b"ModuleNotFoundError: No module named 'pytz'\n"

我没有在气流或 BQ 方法中使用 pytz。我曾尝试在我的作曲家环境中添加 pytz，但没有奏效。请提出建议。

【问题讨论】：

似乎这是您正在使用的软件包的依赖项之一，而不是您专门使用它。你是如何安装google-cloud-bigquery的？尝试重新安装。
嗨，OP。让我们知道答案是否有帮助。如果它有用，请考虑支持它。如果它回答了你的问题，那么请接受它。这样其他人就知道你已经（充分地）得到了帮助。另见What should I do when someone answers my question?
是的，答案对我有用。谢谢
@PrithwirajSamanta 请通过单击 Ricco 答案左侧的复选标记接受答案来关闭此问题。

标签： airflow google-cloud-composer

【解决方案1】：

此错误是由于 https://github.com/googleapis/python-bigquery/issues/885 损坏造成的，Bigquery version 2.24.1 上发布了对此的修复。

您可以使用命令 gcloud composer environments update 更新您的 bigquery 依赖项。详情请见Installing a Python dependency from PyPI。

出于测试目的，环境具有 Composer Version=1.16.14 和 Airflow Version=1.10.15。使用Cloud Composer pre installed packages 中列出的预安装包，我只复制了 Google 依赖项并更新了下面的包。这是因为根据Bigquery constraints，它们是 Bigquery 的依赖项。

google-cloud-bigquery==2.24.1
google-api-core==1.29.0
grpcio==1.38.1

完整要求.txt：

google-ads==4.0.0
google-api-core==1.29.0
google-api-python-client==1.12.8
google-apitools==0.5.31
google-auth==1.28.0
google-auth-httplib2==0.1.0
google-auth-oauthlib==0.4.3
google-cloud-automl==2.2.0
google-cloud-bigquery==2.24.1
google-cloud-bigquery-datatransfer==3.1.0
google-cloud-bigquery-storage==2.1.0
google-cloud-bigtable==1.7.0
google-cloud-build==2.0.0
google-cloud-container==1.0.1
google-cloud-core==1.6.0
google-cloud-datacatalog==3.1.0
google-cloud-dataproc==2.3.0
google-cloud-datastore==1.15.3
google-cloud-dlp==1.0.0
google-cloud-kms==2.2.0
google-cloud-language==1.3.0
google-cloud-logging==2.2.0
google-cloud-memcache==0.3.0
google-cloud-monitoring==2.0.0
google-cloud-os-login==2.1.0
google-cloud-pubsub==2.3.0
google-cloud-pubsublite==0.3.0
google-cloud-redis==2.1.0
google-cloud-secret-manager==1.0.0
google-cloud-spanner==1.19.1
google-cloud-speech==1.3.2
google-cloud-storage==1.36.2
google-cloud-tasks==2.2.0
google-cloud-texttospeech==1.0.1
google-cloud-translate==1.7.0
google-cloud-videointelligence==1.16.1
google-cloud-vision==1.0.0
google-cloud-workflows==0.2.0
google-crc32c==1.1.2
google-pasta==0.2.0
google-resumable-media==1.2.0
googleapis-common-protos==1.53.0
grpc-google-iam-v1==0.12.3
grpcio==1.38.1
grpcio-gcp==0.2.2

命令：

gcloud composer environments update your-composer-environment-name --update-pypi-packages-from-file requirements.txt --location your-composer-location

【讨论】：