App Engine 批量数据上传 URLError: "Connection reset by peer"

【问题标题】：App Engine Bulk Data Upload URLError: "Connection reset by peer"App Engine 批量数据上传 URLError: "Connection reset by peer"
【发布时间】：2010-08-08 16:10:42
【问题描述】：

我正在尝试使用 appcfg.py 将一些记录上传到我的本地数据存储中

实际上只插入了少量记录，我得到以下输出（有大量错误）：

       $ appcfg.py upload_data --config_file=bulkloader.yaml --filename=/output.csv --kind=AutoCompleteIndex --url=http://localhost:8084/remote_api .
        Application: theapp1; version: dev.
        Uploading data records.
        [INFO    ] Logging to bulkloader-log-20100808.170223
        [INFO    ] Throttling transfers:
        [INFO    ] Bandwidth: 250000 bytes/second
        [INFO    ] HTTP connections: 8/second
        [INFO    ] Entities inserted/fetched/modified: 20/second
        [INFO    ] Batch Size: 10
        [INFO    ] Opening database: bulkloader-progress-20100808.170223.sql3
        Please enter login credentials for localhost
        Email: kjuyg
        Password for kjuyg: 
        [INFO    ] Connecting to localhost:8084/remote_api
        [INFO    ] Starting import; maximum 10 entities per post
        .......................................[ERROR   ] Retrying on non-fatal URL error: [Errno 54] Connection reset by peer
        Traceback (most recent call last):
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 695, in PerformWork
            transfer_time = self._TransferItem(thread_pool)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 852, in _TransferItem
            self.request_manager.PostEntities(self.content)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 1296, in PostEntities
            datastore.Put(entities)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 289, in Put
            'datastore_v3', 'Put', req, datastore_pb.PutResponse(), rpc)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 193, in _MakeSyncCall
            rpc.check_success()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 474, in check_success
            self.__rpc.CheckSuccess()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_rpc.py", line 149, in _WaitImpl
            self.request, self.response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 223, in MakeSyncCall
            handler(request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 349, in _Dynamic_Put
            'datastore_v3', 'Put', put_request, put_response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 155, in MakeSyncCall
            self._MakeRealSyncCall(service, call, request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 167, in _MakeRealSyncCall
            encoded_response = self._server.Send(self._path, encoded_request)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appengine_rpc.py", line 346, in Send
            f = self.opener.open(req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 391, in open
            response = self._open(req, data)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _open
            '_open', req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 369, in _call_chain
            result = func(*args)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1173, in http_open
            return self.do_open(httplib.HTTPConnection, req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1148, in do_open
            raise URLError(err)
        URLError: <urlopen error [Errno 54] Connection reset by peer>
        ................................................[ERROR   ] Retrying on non-fatal URL error: [Errno 54] Connection reset by peer
        Traceback (most recent call last):
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 695, in PerformWork
            transfer_time = self._TransferItem(thread_pool)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 852, in _TransferItem
            self.request_manager.PostEntities(self.content)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 1296, in PostEntities
            datastore.Put(entities)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 289, in Put
            'datastore_v3', 'Put', req, datastore_pb.PutResponse(), rpc)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 193, in _MakeSyncCall
            rpc.check_success()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 474, in check_success
            self.__rpc.CheckSuccess()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_rpc.py", line 149, in _WaitImpl
            self.request, self.response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 223, in MakeSyncCall
            handler(request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 349, in _Dynamic_Put
            'datastore_v3', 'Put', put_request, put_response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 155, in MakeSyncCall
            self._MakeRealSyncCall(service, call, request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 167, in _MakeRealSyncCall
            encoded_response = self._server.Send(self._path, encoded_request)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appengine_rpc.py", line 346, in Send
            f = self.opener.open(req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 391, in open
            response = self._open(req, data)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _open
            '_open', req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 369, in _call_chain
            result = func(*args)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1173, in http_open
            return self.do_open(httplib.HTTPConnection, req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1148, in do_open
            raise URLError(err)
        URLError: <urlopen error [Errno 54] Connection reset by peer>
        .............[ERROR   ] Retrying on non-fatal URL error: [Errno 54] Connection reset by peer
        Traceback (most recent call last):
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 695, in PerformWork
            transfer_time = self._TransferItem(thread_pool)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 852, in _TransferItem
            self.request_manager.PostEntities(self.content)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 1296, in PostEntities
            datastore.Put(entities)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 289, in Put
            'datastore_v3', 'Put', req, datastore_pb.PutResponse(), rpc)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 193, in _MakeSyncCall
            rpc.check_success()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 474, in check_success
            self.__rpc.CheckSuccess()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_rpc.py", line 149, in _WaitImpl
            self.request, self.response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 223, in MakeSyncCall
            handler(request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 349, in _Dynamic_Put
            'datastore_v3', 'Put', put_request, put_response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 155, in MakeSyncCall
            self._MakeRealSyncCall(service, call, request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 167, in _MakeRealSyncCall
            encoded_response = self._server.Send(self._path, encoded_request)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appengine_rpc.py", line 346, in Send
            f = self.opener.open(req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 391, in open
            response = self._open(req, data)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _open
            '_open', req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 369, in _call_chain
            result = func(*args)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1173, in http_open
            return self.do_open(httplib.HTTPConnection, req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1148, in do_open
            raise URLError(err)
        URLError: <urlopen error [Errno 54] Connection reset by peer>
        ................[ERROR   ] Retrying on non-fatal URL error: [Errno 54] Connection reset by peer
        Traceback (most recent call last):
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 695, in PerformWork
            transfer_time = self._TransferItem(thread_pool)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 852, in _TransferItem
            self.request_manager.PostEntities(self.content)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 1296, in PostEntities
            datastore.Put(entities)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 289, in Put
            'datastore_v3', 'Put', req, datastore_pb.PutResponse(), rpc)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 193, in _MakeSyncCall
            rpc.check_success()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 474, in check_success
            self.__rpc.CheckSuccess()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_rpc.py", line 149, in _WaitImpl
            self.request, self.response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 223, in MakeSyncCall
            handler(request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 349, in _Dynamic_Put
            'datastore_v3', 'Put', put_request, put_response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 155, in MakeSyncCall
            self._MakeRealSyncCall(service, call, request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 167, in _MakeRealSyncCall
            encoded_response = self._server.Send(self._path, encoded_request)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appengine_rpc.py", line 346, in Send
            f = self.opener.open(req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 391, in open
            response = self._open(req, data)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _open
            '_open', req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 369, in _call_chain
            result = func(*args)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1173, in http_open
            return self.do_open(httplib.HTTPConnection, req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1148, in do_open
            raise URLError(err)
        URLError: <urlopen error [Errno 54] Connection reset by peer>
        .......................................................................................................................................................................[ERROR   ] Retrying on non-fatal URL error: [Errno 54] Connection reset by peer
        Traceback (most recent call last):
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 695, in PerformWork
            transfer_time = self._TransferItem(thread_pool)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 852, in _TransferItem
            self.request_manager.PostEntities(self.content)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 1296, in PostEntities
            datastore.Put(entities)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 289, in Put
            'datastore_v3', 'Put', req, datastore_pb.PutResponse(), rpc)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 193, in _MakeSyncCall
            rpc.check_success()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 474, in check_success
            self.__rpc.CheckSuccess()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_rpc.py", line 149, in _WaitImpl
            self.request, self.response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 223, in MakeSyncCall
            handler(request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 349, in _Dynamic_Put

.... etc .....
            return self.do_open(httplib.HTTPConnection, req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1148, in do_open
            raise URLError(err)
        URLError: <urlopen error [Errno 54] Connection reset by peer>
        .......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................^C..[INFO    ] An error occurred. Shutting down...
        ..........
        [INFO    ] 19580 entites total, 0 previously transferred
        [INFO    ] 18570 entities (3593152 bytes) transferred in 225.1 seconds
        [INFO    ] Some entities not successfully transferred

知道可能是什么问题吗？

【问题讨论】：

    标签： python google-app-engine google-cloud-datastore bigtable urlfetch


    【解决方案1】：

    我在我的 Mac 上遇到了完全相同的问题。我通过创建代理解决了这个问题，将以下内容放入我的 httpd-vhost.conf 文件（在 etc/apache2/extra/ 中）：

    <Location /remote_api>
  ProxyPass http://localhost:8888/remote_api
</Location>

    假设您的测试服务器在端口 8888 上运行。

    然后我运行：

    sudo apachectl restart

    重启apache服务器。

    现在这个命令（“Survey”是数据项之一）：

    /Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/appcfg.py upload_data /
--kind=Survey --file=Survey.csv /
--url=http://localhost/remote_api --email=test@example.com /
--config_file=config.yml

    给出这个输出：

    [INFO    ] Logging to bulkloader-log-20121114.100837
[INFO    ] Throttling transfers:
[INFO    ] Bandwidth: 250000 bytes/second
[INFO    ] HTTP connections: 8/second
[INFO    ] Entities inserted/fetched/modified: 20/second
[INFO    ] Batch Size: 10
Password for test@example.com: 
[INFO    ] Opening database: bulkloader-progress-20121114.100837.sql3
[INFO    ] Connecting to localhost/remote_api
[INFO    ] Starting import; maximum 10 entities per post
....
[INFO    ] 36 entities total, 0 previously transferred
[INFO    ] 36 entities (15402 bytes) transferred in 1.7 seconds
[INFO    ] All entities successfully transferred

    应该如此。

    【讨论】：

      【解决方案2】：

      查看您的服务器日志。您是否收到超出堆大小或 OutOfMemory 异常？如果您的上传量很大，这将在本地开发服务器上发生，因为开发服务器将数据存储保存在主内存中。服务器将挂起一段时间，然后您将重置连接。我的开发服务器内存不足，只有 10MB 的数据存储数据，因为它使用大量额外的 RAM 来组织它和存储元数据。您可以使用 -Xmx 开关增加堆的大小。

      【讨论】：

      • 我们应该去哪里设置-Xmx开关？
      猜你喜欢
      • 2019-02-02
      • 1970-01-01
      • 2018-10-15
      • 2012-01-18
      • 1970-01-01
      • 2019-03-29
      • 1970-01-01
      • 2020-06-18
      • 2013-03-13
      相关资源
      最近更新 更多
      热门标签
      Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode