带有请求模块的 ConnectionResetError 在异常内部引发异常，导致程序崩溃

Question

我的程序崩溃的异常如下：

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/urllib3/response.py", line 397, in _error_catcher
    yield
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/urllib3/response.py", line 479, in read
    data = self._fp.read(amt)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 447, in read
    n = self.readinto(b)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 491, in readinto
    n = self.fp.readinto(b)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [Errno 54] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/requests/models.py", line 750, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/urllib3/response.py", line 531, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/urllib3/response.py", line 496, in read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/urllib3/response.py", line 415, in _error_catcher
    raise ProtocolError('Connection broken: %r' % e, e)
urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(54, 'Connection reset by peer')", ConnectionResetError(54, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/me/Documents/CBB/scrape.py", line 301, in <module>
    main([2018, 11, 6, 2018, 12, 1])
  File "/Users/me/Documents/CBB/scrape.py", line 207, in main
    get_range(int(argv[0]), int(argv[1]), int(argv[2]), int(argv[3]), int(argv[4]), int(argv[5]))
  File "/Users/me/Documents/CBB/scrape.py", line 250, in get_range
    rg.get_bsc()
  File "/Users/me/Documents/CBB/scrape.py", line 103, in get_bsc
    self.get_pbp()
  File "/Users/me/Documents/CBB/scrape.py", line 120, in get_pbp
    soup = self.scraper.open_page(url=url)
  File "/Users/me/Documents/CBB/scrape_util.py", line 209, in open_page
    response = self.session.get(url, proxies={'https': ip, 'http': ip}, headers=headers)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/requests/sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/requests/sessions.py", line 686, in send
    r.content
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/requests/models.py", line 828, in content
    self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/requests/models.py", line 753, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: ConnectionResetError(54, 'Connection reset by peer')", ConnectionResetError(54, 'Connection reset by peer'))

我正在从 Internet 偶尔一次中断一两秒钟的位置运行网络爬虫，在这些情况下，我预计会收到某种异常。我对这些异常的处理方法是等待几秒钟，然后再次尝试访问该页面。对于此问题导致的大多数类型的异常，我可以轻松地except 异常并重试。但是，每当这导致 ConnectionResetError 时，程序就会因为异常处理程序中引发的异常而崩溃。我不知道如何解决这个问题，因为引发任何异常的代码都不是我编写的。我没有提出任何例外，我想扼杀所有这些，以便我可以重试。有什么办法可以忽略这些异常内的异常崩溃？

我已经将ConnectionResetErrors 排除在外，并尽快离开它们，但这是行不通的。我正在使用请求模块来打开这些页面。

Answer 1

这里有 3 个不同的异常，它们按以下顺序发生：

1. ConnectionResetError
2. urllib3.exceptions.ProtocolError
3. requests.exceptions.ChunkedEncodingError

前两个例外在您正在使用的库中除外，您不应该关心它们。最后一个 (requests.exceptions.ChunkedEncodingError) 也不例外，因为你也不例外，所以它被提出来了。所以你应该有：

except requests.exceptions.ChunkedEncodingError:
    # retry like you do for other exceptions

为了说明发生了什么，请检查以下代码：

def external_function():
    try:
        1 / 0    #  this produces ZeroDivisionError
    except ZeroDivisionError:
        a = b    #  this produces NameError

try:
    external_function()
except ZeroDivisionError:
    print('This line is never printed')
except NameError:
    print('NamerError is caught')

external_function() 说明requests.get()。首先，出现ZeroDivisionError 错误。但是，它不会被引发，因为它是由一些外部代码处理的。在处理此异常期间，外部代码会产生另一个异常，这次是NameError。此异常不由外部代码处理，因此会返回给您。然后由你来处理它。如果您不处理它，它会引发并打印沿途发生的所有异常的 Traceback，即使它们被排除在外。