Beautifulsoup: requests.exceptions.ConnectionError: (\'Connection aborted.\', RemoteDisconnected(\'远程端关闭连接无响应答案

【问题标题】：Beautifulsoup: requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without responseBeautifulsoup: requests.exceptions.ConnectionError: (\'Connection aborted.\', RemoteDisconnected(\'远程端关闭连接无响应
【发布时间】：2023-01-10 12:14:00
【问题描述】：

我正在尝试用 beautifulsoup4 构建一个 python webscraper。如果我在我的 Macbook 上运行代码，脚本可以运行，但如果我让脚本在我的家庭服务器 (ubuntu vm) 上运行，我会收到以下错误信息（见下文）。我尝试了一个 vpn 连接和多个标头但没有成功。

非常感谢您对如何使脚本正常工作的反馈。谢谢！

这里的错误消息：

{'User-Agent': 'Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/534.7 ChromePlus/1.5.0.0alpha1'}
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.10/http/client.py", line 1374, in getresponse
    response.begin()
  File "/usr/lib/python3.10/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.10/http/client.py", line 287, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

[...]

requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
[Finished in 15.9s with exit code 1]

这是我的代码：

from bs4 import BeautifulSoup
import requests
import pyuser_agent

URL = f"https://www.edmunds.com/inventory/srp.html?radius=5000&sort=publishDate%3Adesc&pagenumber=2"

ua = pyuser_agent.UA()
headers = {'User-Agent': ua.random}
print(headers)

response = requests.get(url=URL, headers=headers)
soup = BeautifulSoup(response.text, 'lxml')
overview = soup.find()
print(overview)

我尝试了多个标头，但没有得到结果

【问题讨论】：

你能用这个标题（不是随机的）试试吗：headers = {"User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:108.0) Gecko/20100101 Firefox/108.0"}？
太棒了！谢谢你！你能给我一个简短的解释吗？
我已经发布了一个答案，所以你可以关闭这个问题。

标签： python web-scraping web beautifulsoup

【解决方案1】：

尝试使用真正的网络浏览器用户代理，而不是来自 pyuser_agent 的随机用户代理。例如：

import requests
from bs4 import BeautifulSoup

URL = f"https://www.edmunds.com/inventory/srp.html?radius=5000&sort=publishDate%3Adesc&pagenumber=2"


headers = {"User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:108.0) Gecko/20100101 Firefox/108.0"}


response = requests.get(url=URL, headers=headers)
soup = BeautifulSoup(response.text, "lxml")
overview = soup.find()
print(overview)

可能的解释可能是服务器保留了一个真实世界用户代理列表，并且不向某些不存在的用户代理提供任何页面。

【讨论】：

【解决方案2】：

我很不擅长找出正确的标头和 cookie 集，所以在这些情况下，我经常最终求助于：

cloudscraper

response = cloudscraper.create_scraper().get(URL)

或 HTMLSession - 这特别漂亮，因为它还解析 HTML 并具有一些 JavaScript 支持
```
response = HTMLSession().get(URL)
```

【讨论】：