【问题标题】:app engine standard environment call bigquery python应用引擎标准环境调用bigquery python
【发布时间】:2018-04-29 01:14:11
【问题描述】:

我正在尝试在 python 中部署一个简单的标准应用引擎,并从那里通过 python bigquery 客户端进行 bigquery 查询。

代码就这么简单:

from __future__ import absolute_import
import webapp2
import os
from google.cloud import bigquery

class MainPage(webapp2.RequestHandler):
  def get(self):


  client = bigquery.Client(project = "ancient-ceiling-125223")
  project_name = str(client.project)

  query_job = client.query("select 1")

  assert query_job.state == 'RUNNING'

  iterator = query_job.result(timeout= 30)

  rows = list(iterator)

  self.response.write('nothing to see %s' % (project_name))


app = webapp2.WSGIApplication(
[('/', MainPage)], debug=True)

错误日志:错误出现在虚拟查询请求中

*('Connection broken: IncompleteRead(209 bytes read)', IncompleteRead(209 bytes read)) (/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py:1528)
Traceback (most recent call last):
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1511, in __call__
    rv = self.handle_exception(request, response, e)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1505, in __call__
    rv = self.router.dispatch(request, response)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
    return route.handler_adapter(request, response)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1077, in __call__
    return handler.dispatch()
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 547, in dispatch
    return self.handle_exception(e, self.app.debug)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 545, in dispatch
    return method(*args, **kwargs)
  File "/base/data/home/apps/s~ancient-ceiling-125223/20171115t104156.405543624689752939/main.py", line 35, in get
    query_job = client.query("select 1")
  File "/base/data/home/apps/s~ancient-ceiling-125223/20171115t104156.405543624689752939/lib/google/cloud/bigquery/client.py", line 986, in query
    job._begin(retry=retry)
  File "/base/data/home/apps/s~ancient-ceiling-125223/20171115t104156.405543624689752939/lib/google/cloud/bigquery/job.py", line 397, in _begin
    method='POST', path=path, data=self._build_resource())
  File "/base/data/home/apps/s~ancient-ceiling-125223/20171115t104156.405543624689752939/lib/google/cloud/bigquery/client.py", line 271, in _call_api
    return call()
  File "/base/data/home/apps/s~ancient-ceiling-125223/20171115t104156.405543624689752939/lib/google/api_core/retry.py", line 260, in retry_wrapped_func
    on_error=on_error,
  File "/base/data/home/apps/s~ancient-ceiling-125223/20171115t104156.405543624689752939/lib/google/api_core/retry.py", line 177, in retry_target
    return target()
  File "/base/data/home/apps/s~ancient-ceiling-125223/20171115t104156.405543624689752939/lib/google/cloud/_http.py", line 290, in api_request
    headers=headers, target_object=_target_object)
  File "/base/data/home/apps/s~ancient-ceiling-125223/20171115t104156.405543624689752939/lib/google/cloud/_http.py", line 183, in _make_request
    return self._do_request(method, url, headers, data, target_object)
  File "/base/data/home/apps/s~ancient-ceiling-125223/20171115t104156.405543624689752939/lib/google/cloud/_http.py", line 212, in _do_request
    url=url, method=method, headers=headers, data=data)
  File "/base/data/home/apps/s~ancient-ceiling-125223/20171115t104156.405543624689752939/lib/google/auth/transport/requests.py", line 186, in request
    method, url, data=data, headers=request_headers, **kwargs)
  File "/base/data/home/apps/s~ancient-ceiling-125223/20171115t104156.405543624689752939/lib/requests/sessions.py", line 502, in request
    resp = self.send(prep, **send_kwargs)
  File "/base/data/home/apps/s~ancient-ceiling-125223/20171115t104156.405543624689752939/lib/requests/sessions.py", line 652, in send
    r.content
  File "/base/data/home/apps/s~ancient-ceiling-125223/20171115t104156.405543624689752939/lib/requests/models.py", line 825, in content
    self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
  File "/base/data/home/apps/s~ancient-ceiling-125223/20171115t104156.405543624689752939/lib/requests/models.py", line 750, in generate
    raise ChunkedEncodingError(e)
ChunkedEncodingError: ('Connection broken: IncompleteRead(209 bytes read)', IncompleteRead(209 bytes read))*

【问题讨论】:

  • 这似乎不是身份验证问题。只是为了测试,如果你尝试一个非常简单的查询,例如"select 1",它是否有效?另外,我想你是在 flex 环境下运行的,对吗?
  • 一个简单的查询也不行。我在标准环境中运行。
  • 您显示的代码与回溯不匹配,回溯显示for row in query_job: 语句-该遗漏可能会掩盖问题。如果您有一个更简单的示例,请发布与之匹配的回溯。
  • 我想知道一个对象是否超出范围(在省略的代码中)并导致连接过早关闭。
  • 我刚刚为测试部署的更简化的代码:client = bigquery.Client(project = "ancient-ceiling-125223") project_name = str(client.project) query_job = client.query("select 1") 断言 query_job.state == 'RUNNING' 迭代器 = query_job.result(timeout= 30) 行 = 列表(迭代器)

标签: python-2.7 google-app-engine google-bigquery


【解决方案1】:

我无法使用 python bigquery 客户端库使其工作,这是我在标准应用引擎环境中发现的工作,

from __future__ import absolute_import
import webapp2

from googleapiclient.discovery import build
from oauth2client.client import GoogleCredentials


class MainPage(webapp2.RequestHandler):
  def get(self):

    credentials = GoogleCredentials.get_application_default()
    service = build('bigquery', 'v2', credentials=credentials)

    datasets = service.datasets().list(projectId="ancient-ceiling-125223").execute()

    self.response.write('datasets: %s' % datasets)


app = webapp2.WSGIApplication(
    [('/', MainPage)], debug=True)

【讨论】:

  • 要走的路!这就是为什么我向您询问环境的原因,我也无法使 google-cloud API 适用于 Standard,只有 googleapiclient 也适用。不过,我建议在请求之外创建您的服务(在这个repo 中有一个我如何做到这一点的示例)。还要注意,如果您的查询花费的时间超过 60 秒,那么这种方法将不起作用。届时您可能需要安排查询时间。
  • 非常感谢。回购真的很有帮助
  • 我可以确认 BigQuery 客户端 API 仍然无法在 App Engine 标准环境中使用。这篇不完整且令人困惑的博文 (cloud.google.com/blog/products/gcp/…) 展示了一种使用服务发现 API 查询 BigQuery 表的方法,尽管它没有明确说明它正在做什么。 GAE 不适用于标准客户端 API,这似乎是一个令人发指的疏忽……
  • 我们一直在使用服务发现 API 在应用引擎中进行 BigQuery 调用。它工作得非常健壮。有一段时间没有尝试标准客户端 api。
猜你喜欢
  • 2018-06-03
  • 2019-01-25
  • 1970-01-01
  • 2019-05-12
  • 1970-01-01
  • 2021-06-22
  • 1970-01-01
  • 1970-01-01
  • 2018-10-10
相关资源
最近更新 更多