python ssl eof 发生违反协议，wantwriteerror，zeroreturnerror

Question

我正在运行许多 celery 任务 (20,000)，使用 gevent 作为池（也猴子修补所有）。这些任务中的每一项都需要第三方服务（如 adwords）来提取数据。

由于潜在的 SSL 错误，我一直有任务失败。下面是一些异常的堆栈跟踪（没有特别的顺序，这些都是来自不同任务的失败）。我偶尔也会遇到 WantWriteError 和 ZeroReturnError，但 EOF 错误似乎出现得最多。

这些错误发生在使用不同的客户端库时，例如 googleads（用于肥皂通信的 suds 库）以及 requests 和 elasticsearch。我猜其中一些库使用 urllib3 而其他库使用 urllib2 等。

有很多关于 EOF 问题和强制 TLSv1 的信息，但我似乎找不到有效的解决方案。

我不确定我是否一次运行了太多请求，是否有阻塞或什么；任何帮助将不胜感激，我正在把头发拉出来。

Traceback (most recent call last):
  ...
  File "/srv/reporting/src/reporting/stats/adwords/client.py", line 58, in _awql_report
    downloader = self._get_client(client_id).GetReportDownloader(version=self.REPORT_DOWNLOADER_VERSION)
  File "/usr/local/lib/python2.7/dist-packages/googleads/adwords.py", line 283, in GetReportDownloader
    return ReportDownloader(self, version, server)
  File "/usr/local/lib/python2.7/dist-packages/googleads/adwords.py", line 400, in __init__
    proxy=proxy_option, cache=self._adwords_client.cache).wsdl.schema
  File "/usr/local/lib/python2.7/dist-packages/suds/client.py", line 115, in __init__
    self.wsdl = reader.open(url)
  File "/usr/local/lib/python2.7/dist-packages/suds/reader.py", line 150, in open
    d = self.fn(url, self.options)
  File "/usr/local/lib/python2.7/dist-packages/suds/wsdl.py", line 136, in __init__
    d = reader.open(url)
  File "/usr/local/lib/python2.7/dist-packages/suds/reader.py", line 74, in open
    d = self.download(url)
  File "/usr/local/lib/python2.7/dist-packages/suds/reader.py", line 92, in download
    fp = self.options.transport.open(Request(url))
  File "/usr/local/lib/python2.7/dist-packages/suds/transport/https.py", line 62, in open
    return HttpTransport.open(self, request)
  File "/usr/local/lib/python2.7/dist-packages/suds/transport/http.py", line 67, in open
    return self.u2open(u2request)
  File "/usr/local/lib/python2.7/dist-packages/suds/transport/http.py", line 132, in u2open
    return url.open(u2request, timeout=tm)
  File "/usr/lib/python2.7/urllib2.py", line 400, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 418, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1216, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1178, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno 8] _ssl.c:504: EOF occurred in violation of protocol>



Traceback (most recent call last):
  ...
  File "/srv/reporting/src/reporting/stats/analytics/client.py", line 57, in get_access_token
    response = requests.post('https://accounts.google.com/o/oauth2/token', data)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 88, in post
    return request('post', url, data=data, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 456, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 559, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 382, in send
    raise SSLError(e, request=request)
SSLError: [Errno bad handshake] (-1, 'Unexpected EOF')




Traceback (most recent call last):
  ...
    self.es.index(index=self.INDICE, doc_type=self.ROOT_CLASS.__name__, body=self.export(obj), id=obj.id)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/utils.py", line 68, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/__init__.py", line 213, in index
    _make_path(index, doc_type, id), params=params, body=body)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py", line 284, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_requests.py", line 44, in perform_request
    response = self.session.request(method, url, data=body, timeout=timeout or self.timeout)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 456, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 559, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 327, in send
    timeout=timeout
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 493, in urlopen
    body=body, headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 319, in _make_request
    httplib_response = conn.getresponse(buffering=True)
  File "/usr/lib/python2.7/httplib.py", line 1030, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 407, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 365, in _read_status
    line = self.fp.readline()
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 273, in readline
    data = self._sock.recv(self._rbufsize)
  File "/usr/local/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 995, in recv
    self._raise_ssl_error(self._ssl, result)
  File "/usr/local/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 851, in _raise_ssl_error
    raise ZeroReturnError()
ZeroReturnError

Answer 1

所以让我们按每个回溯块来分解它。第一个结尾是：

  File "/usr/lib/python2.7/urllib2.py", line 1178, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno 8] _ssl.c:504: EOF occurred in violation of protocol>

这来自 urllib2。这接收到 EOF 的事实使我认为服务器在您等待该“线程”再次从套接字读取时关闭了连接。您可能想使用更多 time.sleep(0) 来让步给 gevent。

第二个回溯来自请求：

  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 382, in send
    raise SSLError(e, request=request)
SSLError: [Errno bad handshake] (-1, 'Unexpected EOF')

[Errno bad handshake] 会让我倾向于认为这是建立连接的问题，这可能是由意外的 EOF 引起的。这是使用gevent 引起的吗？我不确定。

最终的回溯肯定也来自请求，但它也来自 PyOpenSSL，并且没有被 urllib3 或请求捕获。

  File "/usr/local/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 851, in _raise_ssl_error
    raise ZeroReturnError()
ZeroReturnError

我进行了一些搜索，发现 "According to the pyOpenSSL docs ZeroReturnError means that the SSL connection has been closed cleanly." 这告诉我服务器再次关闭了连接，因为您花了很长时间从套接字读取任何内容。

简而言之，我认为您需要更频繁地显式让步，以确保不会出现这些套接字问题。不过，这只是一个猜测，因此请持保留态度。