从http更改为https后Python请求没有得到响应[重复]

Question

从 http 更改为 https 后，我在抓取网站时遇到了麻烦，不知道如何解决这个问题。我试图从中抓取的网站是https://www.boldsystems.org。两天前还是http://www.boldsystems.org，我的刮刀运行良好。

示例代码：

import requests
requests.get('https://www.boldsystems.org')

我返回的错误代码：

Traceback (most recent call last):
  File "C:\Users\dommi\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\contrib\pyopenssl.py", line 488, in wrap_socket
    cnx.do_handshake()
  File "C:\Users\dommi\AppData\Local\Programs\Python\Python37-32\lib\site-packages\OpenSSL\SSL.py", line 1934, in do_handshake
    self._raise_ssl_error(self._ssl, result)
  File "C:\Users\dommi\AppData\Local\Programs\Python\Python37-32\lib\site-packages\OpenSSL\SSL.py", line 1671, in _raise_ssl_error
    _raise_current_error()
  File "C:\Users\dommi\AppData\Local\Programs\Python\Python37-32\lib\site-packages\OpenSSL\_util.py", line 54, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\dommi\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "C:\Users\dommi\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "C:\Users\dommi\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 976, in _validate_conn
    conn.connect()
  File "C:\Users\dommi\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connection.py", line 370, in connect
    ssl_context=context,
  File "C:\Users\dommi\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\util\ssl_.py", line 377, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "C:\Users\dommi\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\contrib\pyopenssl.py", line 494, in wrap_socket
    raise ssl.SSLError("bad handshake: %r" % e)
ssl.SSLError: ("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])",)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\dommi\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\adapters.py", line 449, in send
    timeout=timeout
  File "C:\Users\dommi\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 725, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "C:\Users\dommi\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\util\retry.py", line 439, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='boldsystems.org', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_
certificate', 'certificate verify failed')])")))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\dommi\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "C:\Users\dommi\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Users\dommi\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\dommi\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\dommi\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='boldsystems.org', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_cert
ificate', 'certificate verify failed')])")))

我找到了一些建议禁用验证的解决方案，例如：

requests.get('https://boldsystems.org', verify = False)

但我认为这是一种不好的方法，因为 SSL 验证是有原因的。

我已经更新了certify、requests和urllib3。我还尝试将 SSL 证书保存到 .pem 文件并将其交给请求函数，但实际上我不确定它的作用，它没有帮助。

我可以在 Windows 和 Ubuntu 以及不同的计算机上重现该问题，所以我认为问题出在我尝试请求的网站的某个地方。

非常感谢我的问题的解决方案或解释这里发生的事情。

Answer 1

我在自己的系统（Ubuntu 18.04，Python 3.6.9）上测试了 boldsystems.org，得到了相同的结果。普通浏览器虽然工作正常。 SSLLabs 的免费 ssltest 工具报告“此服务器的证书链不完整....”。

不完整的证书链只是意味着服务器没有在链中发送中间证书。与 Python 不同，浏览器可能已经缓存了整个链，因此可以正常工作。

解决方法是向验证请求提供一个证书包，以便它能够评估整个链。丑陋，但它应该工作。您将需要下载链中的所有证书，将它们连接起来并将它们呈现给请求。 https://blogs.gnome.org/danni/2015/11/26/using-an-ssl-intermediate-as-your-ca-cert-with-python-requests/ 对此进行了解释。

Answer 2

我不太清楚为什么会这样，但我必须手动将证书信息添加到 certifi 的 cacert.pem 文件中才能使其正常工作。

按照此处给出的步骤操作：Unable to get local issuer certificate when using requests in python

然后它与 200 一起工作：

>>> requests.get('https://www.boldsystems.org')
<Response [200]>