from tornado.httpclient import HTTPClient
client = HTTPClient()
response = client.fetch('https://www.baidu.com/')
print(response.body)
client.close()
'''
b'<html>\r\n<head>\r\n\t<script>\r\n\t\tlocation.replace(location.href.replace("https://","http://"));\r\n\t</script>\r\n</head>\r\n<body>\r\n\t<noscript><meta http-equiv="refresh" content="0;url=http://www.baidu.com/"></noscript>\r\n</body>\r\n</html>'
'''用tornado的HTTPClient抓取百度页面的时候，返回的是“网页跳转”？？应该是百度的防爬虫手段吧。
解决的方法是，伪装成浏览器。
from tornado.httpclient import HTTPClient
client = HTTPClient()
response = client.fetch('https://www.baidu.com/',headers = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/17.17134'})
print(response.body)
client.close()
就是在http头里添加User-Agent字段说明（覆盖掉原本的User-Agent）
GET / HTTP/1.1
Host: www.baidu.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/17.17134这个User-Agent是我从edge上拿的，可是，明明只是edge啊，为什么也要伪装成其他浏览器？？