【发布时间】:2017-11-16 17:13:40
【问题描述】:
我们正在尝试使用 Pandas 从 BigQuery 中提取一些数据,并遇到关于导入的绝对庞大的跟踪。据我所知,所有正确的依赖项都已安装。
Traceback (most recent call last):
File "<frozen importlib._bootstrap>", line 879, in _find_spec
AttributeError: 'PyxImporter' object has no attribute 'find_spec'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "run_forecasts.py", line 117, in <module>
run_forecasts()
File "run_forecasts.py", line 50, in run_forecasts
a.run()
File "./budgetforecastmodel/prepare/funnel_entrance.py", line 37, in run
self._project_id
File "./budgetforecastmodel/prepare/funnel_entrance.py", line 60, in _read_data
funnel_entrances = fe_query.run()
File "./budgetforecastmodel/extract/bigquery/biq_query_base.py", line 41, in run
self.read_data()
File "./budgetforecastmodel/extract/bigquery/biq_query_base.py", line 141, in read_data
self.df = pd.read_gbq(self.query, self.project_id)
File "./venv/lib/python3.6/site-packages/pandas/io/gbq.py", line 100, in read_gbq
**kwargs)
File "./venv/lib/python3.6/site-packages/pandas_gbq/gbq.py", line 906, in read_gbq
dialect=dialect, auth_local_webserver=auth_local_webserver)
File "./venv/lib/python3.6/site-packages/pandas_gbq/gbq.py", line 202, in __init__
self.credentials = self.get_credentials()
File "./venv/lib/python3.6/site-packages/pandas_gbq/gbq.py", line 214, in get_credentials
credentials = self.get_application_default_credentials()
File "./venv/lib/python3.6/site-packages/pandas_gbq/gbq.py", line 243, in get_application_default_credentials
credentials, _ = google.auth.default(scopes=[self.scope])
File "./venv/lib/python3.6/site-packages/google/auth/_default.py", line 281, in default
credentials, project_id = checker()
File "./venv/lib/python3.6/site-packages/google/auth/_default.py", line 158, in _get_gae_credentials
from google.auth import app_engine
File "./venv/lib/python3.6/site-packages/google/auth/app_engine.py", line 32, in <module>
from google.appengine.api import app_identity
File "<frozen importlib._bootstrap>", line 961, in _find_and_load
File "<frozen importlib._bootstrap>", line 946, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 881, in _find_spec
File "<frozen importlib._bootstrap>", line 855, in _find_spec_legacy
File "./venv/lib/python3.6/site-packages/pyximport/pyximport.py", line 253, in find_module
fp, pathname, (ext,mode,ty) = imp.find_module(fullname,package_path)
File "~/.pyenv/versions/3.6.1/lib/python3.6/imp.py", line 270, in find_module
"not {}".format(type(path)))
RuntimeError: 'path' must be None or a list, not <class '_frozen_importlib_external._NamespacePath'>
我们的需求文件如下所示:
cachetools==2.0.1
certifi==2017.7.27.1
chardet==3.0.4
et-xmlfile==1.0.1
fbprophet==0.2
google-api-python-client==1.6.4
google-auth==1.1.1
google-auth-httplib2==0.0.2
google-auth-oauthlib==0.1.1
httplib2==0.10.3
idna==2.6
jdcal==1.3
numpy==1.12.1
oauth2client==4.1.2
oauthlib==2.0.4
openpyxl==2.4.8
pandas==0.20.1
pandas-gbq==0.2.0
psycopg2==2.7.3.1
py==1.4.34
pyasn1==0.3.7
pyasn1-modules==0.1.5
pyodbc==4.0.17
pytest==3.2.2
python-dateutil==2.6.0
pytz==2017.2
PyYAML==3.12
requests==2.18.4
requests-oauthlib==0.8.0
rpy2==2.8.5
rsa==3.4.2
six==1.10.0
SQLAlchemy==1.1.9
sqlalchemy-redshift==0.6.0
uritemplate==3.0.0
urllib3==1.22
xlrd==1.1.0
这发生在具有全新虚拟环境的全新克隆上,在 OS X 上运行 Python 3.6.1。在两台机器上重现。老实说,甚至有点不知所措。它以前有效,但有些东西退步导致它失败,但它死亡的方式表明它不完全是因为我们?
从字面上看,任何线索都会有所帮助! :)
【问题讨论】:
-
据我所见,您正在 appengine 中运行此代码,对吗?可以分享一下代码吗?是标准环境吗?
-
不,这是独立的并且在本地运行。我们只使用
pandas.read_gbq(query, project_id),而后者又使用BigQuery API library。除此之外没有什么特别的。 -
有趣...不知道为什么,但它似乎使用来自 appengine 的
app_identity进行身份验证(AFAIK 这应该只在 appengine 标准环境中发生) -
原来 Pandas/Google BQ/FB 的 Prophet 之间存在一些冲突。 I raised an issue。不确定这是否可以作为答案?
-
如果确认存在错误,那么我认为肯定存在。希望其他有同样问题的人也能找到这个。
标签: python python-3.x pandas google-bigquery