将`host`添加到标头后超过30个重定向[python请求]

Question

在标头中获取带有host 的url 将引发异常Exceeded 30 redirects。
这太奇怪了，我无法弄清楚。
下面是测试代码：

>>> url = 'http://bbs.duchang8.com/forum-29-1.html'
>>> r = requests.get(url)
>>> print r.status_code
200
>>> headers = {
...     'Host': 'bbs.duchang8.com',
... }
>>> r = requests.get(url, headers=headers)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/www/article_fetcher/venv/local/lib/python2.7/site-packages/requests/api.py", line 69, in get
    return request('get', url, params=params, **kwargs)
  File "/data/www/article_fetcher/venv/local/lib/python2.7/site-packages/requests/api.py", line 50, in request
    response = session.request(method=method, url=url, **kwargs)
  File "/data/www/article_fetcher/venv/local/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "/data/www/article_fetcher/venv/local/lib/python2.7/site-packages/requests/sessions.py", line 594, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "/data/www/article_fetcher/venv/local/lib/python2.7/site-packages/requests/sessions.py", line 114, in resolve_redirects
    raise TooManyRedirects('Exceeded %s redirects.' % self.max_redirects)
requests.exceptions.TooManyRedirects: Exceeded 30 redirects.

Answer 1

简答：

不要覆盖 Host: 标头。

或者，用客户端重定向到的主机覆盖它。

长答案

通过显式设置 Host 标头，您是在告诉 requests 在所有后续请求中使用该标头，包括由于服务器重定向响应而重新发出的任何请求。 p>

在这种情况下，requests 客户端被重定向到由不同服务器托管的位置 http://www.duchang8.com/forum-29-1.html； www.duchang8.com 与 bbs.duchang8.com。尽管两个主机名都解析为相同的 IP 地址，但远程 HTTP 服务器对它们的处理方式不同。

nett 结果是requests 继续使用您提供的Host: 标头，而不是服务器返回的正确标头。然后，由于 URL/服务器主机和 Host: 标头不匹配，随后对新位置的请求被拒绝（通过重定向）。

>>> import requests
>>> url = 'http://bbs.duchang8.com/forum-29-1.html'
>>> r = requests.get(url)
>>> r
<Response [200]>
>>> r.history
[<Response [301]>]
>>> r.history[0].headers
{'content-length': '178', 'server': 'nginx', 'connection': 'keep-alive', 'location': 'http://www.duchang8.com/forum-29-1.html', 'date': 'Mon, 03 Aug 2015 12:20:31 GMT', 'content-type': 'text/html'}

在这里，我们看到客户端通过 HTTP 301 响应和 location: 标头重定向到 http://www.duchang8.com/forum-29-1.html。

使用curl，您可以看到如果在获取新位置时尝试提供不同的Host: 标头会发生什么：

$ curl -v -L -H 'Host: bbs.duchang8.com' http://www.duchang8.com/forum-29-1.html
*   Trying 61.160.249.39...
* Connected to www.duchang8.com (61.160.249.39) port 80 (#0)
> GET /forum-29-1.html HTTP/1.1
> User-Agent: curl/7.40.0
> Accept: */*
> Host: bbs.duchang8.com
> 
< HTTP/1.1 301 Moved Permanently
< Server: nginx
< Date: Mon, 03 Aug 2015 12:27:33 GMT
< Content-Type: text/html
< Content-Length: 178
< Connection: keep-alive
< Location: http://www.duchang8.com/forum-29-1.html
< 
* Ignoring the response-body
* Connection #0 to host www.duchang8.com left intact
* Issue another request to this URL: 'http://www.duchang8.com/forum-29-1.html'
* Found bundle for host www.duchang8.com: 0x21b54c0
* Re-using existing connection! (#0) with host www.duchang8.com
* Connected to www.duchang8.com (61.160.249.39) port 80 (#0)
> GET /forum-29-1.html HTTP/1.1
> User-Agent: curl/7.40.0
> Accept: */*
> Host: bbs.duchang8.com
> 
< HTTP/1.1 301 Moved Permanently
< Server: nginx
< Date: Mon, 03 Aug 2015 12:27:33 GMT
< Content-Type: text/html
< Content-Length: 178
< Connection: keep-alive
< Location: http://www.duchang8.com/forum-29-1.html
<
# and so so, and so on....

它最终进入一个重定向循环。 requests 会发生相同的请求和响应序列，最终将决定永远不会结束并中止请求。