python爬虫request模块详解

wlx97e6

requests模块

使用requests可以模拟浏览器的请求，比起之前用到的urllib，requests模块的api更加便捷（本质就是封装了urllib3）

注意：requests库发送请求将网页内容下载下来以后，并不会执行js代码，这需要我们自己分析目标站点然后发起新的request请求

官方文档：http://cn.python-requests.org/zh_CN/latest/

安装：pip3 install requests

requests模块的各种请求方式

源码构成如下

# 以上方法均是在此方法的基础上构建

requests.request(method, url, **kwargs)

其中最常用的请求方式就是post和get请求，泵智商，post和get就是封装了request请求的请求方式

>>> r = requests.get(\'https://api.github.com/events\')
相当于requests,request(method=\'get\', \'https://api.github.com/events\')
>>> r = requests.post(\'http://httpbin.org/post\', data = {\'key\':\'value\'})
相当于requests,request(method=\'post\', \'https://api.github.com/events\', data = {\'key\':\'value\'})

requests,request方法详解

request（）源码

def request(method, url, **kwargs):
    """Constructs and sends a :class:`Request <Request>`.

    :param method: method for the new :class:`Request` object.
    :param url: URL for the new :class:`Request` object.
    :param params: (optional) Dictionary or bytes to be sent in the query string for the :class:`Request`.
    :param data: (optional) Dictionary or list of tuples ``[(key, value)]`` (will be form-encoded), bytes, or file-like object to send in the body of the :class:`Request`.
    :param json: (optional) json data to send in the body of the :class:`Request`.
    :param headers: (optional) Dictionary of HTTP Headers to send with the :class:`Request`.
    :param cookies: (optional) Dict or CookieJar object to send with the :class:`Request`.
    :param files: (optional) Dictionary of ``\'name\': file-like-objects`` (or ``{\'name\': file-tuple}``) for multipart encoding upload.
        ``file-tuple`` can be a 2-tuple ``(\'filename\', fileobj)``, 3-tuple ``(\'filename\', fileobj, \'content_type\')``
        or a 4-tuple ``(\'filename\', fileobj, \'content_type\', custom_headers)``, where ``\'content-type\'`` is a string
        defining the content type of the given file and ``custom_headers`` a dict-like object containing additional headers
        to add for the file.
    :param auth: (optional) Auth tuple to enable Basic/Digest/Custom HTTP Auth.
    :param timeout: (optional) How many seconds to wait for the server to send data
        before giving up, as a float, or a :ref:`(connect timeout, read
        timeout) <timeouts>` tuple.
    :type timeout: float or tuple
    :param allow_redirects: (optional) Boolean. Enable/disable GET/OPTIONS/POST/PUT/PATCH/DELETE/HEAD redirection. Defaults to ``True``.
    :type allow_redirects: bool
    :param proxies: (optional) Dictionary mapping protocol to the URL of the proxy.
    :param verify: (optional) Either a boolean, in which case it controls whether we verify
            the server\'s TLS certificate, or a string, in which case it must be a path
            to a CA bundle to use. Defaults to ``True``.
    :param stream: (optional) if ``False``, the response content will be immediately downloaded.
    :param cert: (optional) if String, path to ssl client cert file (.pem). If Tuple, (\'cert\', \'key\') pair.
    :return: :class:`Response <Response>` object
    :rtype: requests.Response

    Usage::

      >>> import requests
      >>> req = requests.request(\'GET\', \'http://httpbin.org/get\')
      <Response [200]>
    """

    # By using the \'with\' statement we are sure the session is closed, thus we
    # avoid leaving sockets open which can trigger a ResourceWarning in some
    # cases, and look like a memory leak in others.
    with sessions.Session() as session:
        return session.request(method=method, url=url, **kwargs)

　　下面对源码中的各个属性进行分析

method和url

指名请求方式和请求路径

requests.request(method=\'get\', url=\'http://127.0.0.1:8000/test/\')
requests.request(method=\'post\', url=\'http://127.0.0.1:8000/test/\')

params

requests模块发送请求有data、json、params三种携带参数的方法。

params在get请求中使用，data、json在post请求中使用。

params可以接收的参数：

- 可以是字典
- 可以是字符串
字典字符串都会被自动编码发送到url
- 可以是字节（必须是ascii编码以内）

接收字典字符串都会被自动编码发送到url，如下

import requests
wd=\'egon老师\'
pn=1

response=requests.get(\'https://www.baidu.com/s\',
                      params={
                          \'wd\':wd,
                          \'pn\':pn
                      },
                      headers={
                        \'User-Agent\':\'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.75 Safari/537.36\',
                      })
print(response.url)
# 输出为：https://www.baidu.com/s?wd=egon%E8%80%81%E5%B8%88&pn=1
# 可见url已被自动编码

　　上面代码相当于如下代码，params编码转换本质上是用urlencode

import requests
from urllib.parse import urlencode
wd=\'egon老师\'
encode_res=urlencode({\'k\':wd},encoding=\'utf-8\')
keyword=encode_res.split(\'=\')[1]
print(keyword)
# 然后拼接成url
url=\'https://www.baidu.com/s?wd=%s&pn=1\' %keyword

response=requests.get(url,
                      headers={
                        \'User-Agent\':\'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.75 Safari/537.36\',
                      })
print(response.url)
# 输出为：https://www.baidu.com/s?wd=egon%E8%80%81%E5%B8%88&pn=1

　　还有一点注意的就是接收字节数据时，不能传非ASCII码外的字符，如下就是错误的

import requests

# re = requests.request(method=\'get\',
#                  url=\'http://127.0.0.1:8000/test/\',
#                  params=bytes("k1=v1&k2=水电费&k3=v3&k3=vv3", encoding=\'utf8\'))

data

requests模块发送请求有data、json、params三种携带参数的方法。params在get请求中使用，data、json在post请求中使用。

data可以接收的参数为：字典，字符串，字节，文件对象，data和json两者的区别在于data的请求体为name=alex&age=18格式而json请求体为‘{\'k1\': \'v1\', \'k2\': \'水电费\'}’（字符串）

requests.request(method=\'POST\',
                 url=\'http://127.0.0.1:8000/test/\',
                 data={\'k1\': \'v1\', \'k2\': \'水电费\'})

requests.request(method=\'POST\',
                 url=\'http://127.0.0.1:8000/test/\',
                 data="k1=v1; k2=v2; k3=v3; k3=v4"
                 )

requests.request(method=\'POST\',
                 url=\'http://127.0.0.1:8000/test/\',
                 data="k1=v1;k2=v2;k3=v3;k3=v4",
                 headers={\'Content-Type\': \'application/x-www-form-urlencoded\'}
                 )

requests.request(method=\'POST\',
                 url=\'http://127.0.0.1:8000/test/\',
                 data=open(\'data_file.py\', mode=\'r\', encoding=\'utf-8\'),  # 文件内容是：k1=v1;k2=v2;k3=v3;k3=v4
                 headers={\'Content-Type\': \'application/x-www-form-urlencoded\'}
                 )

json

将json中对应的数据进行序列化成一个字符串，json.dumps(...)

然后发送到服务器端的body中，并且Content-Type是 {\'Content-Type\': \'application/json\'}

标志：payload

requests.request(method=\'POST\',
                 url=\'http://127.0.0.1:8000/test/\',
                 json={\'k1\': \'v1\', \'k2\': \'水电费\'})

headers

发送请求头到服务器

requests.request(method=\'POST\',
                 url=\'http://127.0.0.1:8000/test/\',
                 json={\'k1\': \'v1\', \'k2\': \'水电费\'},
                 headers={\'Content-Type\': \'application/x-www-form-urlencoded\'}
                 )

cookies

# 发送Cookie到服务器端
requests.request(method=\'POST\',
                 url=\'http://127.0.0.1:8000/test/\',
                 data={\'k1\': \'v1\', \'k2\': \'v2\'},
                 cookies={\'cook1\': \'value1\'},
                 )
# 也可以使用CookieJar（字典形式就是在此基础上封装）
from http.cookiejar import CookieJar
from http.cookiejar import Cookie

obj = CookieJar()
obj.set_cookie(Cookie(version=0, name=\'c1\', value=\'v1\', port=None, domain=\'\', path=\'/\', secure=False, expires=None,
                      discard=True, comment=None, comment_url=None, rest={\'HttpOnly\': None}, rfc2109=False,
                      port_specified=False, domain_specified=False, domain_initial_dot=False, path_specified=False)
               )
requests.request(method=\'POST\',
                 url=\'http://127.0.0.1:8000/test/\',
                 data={\'k1\': \'v1\', \'k2\': \'v2\'},
                 cookies=obj)

files

发送文件
file_dict = {
    \'f1\': open(\'readme\', \'rb\')
}
requests.request(method=\'POST\',
                 url=\'http://127.0.0.1:8000/test/\',
                 files=file_dict)

发送文件，定制文件名
file_dict = {
    \'f1\': (\'test.txt\', open(\'readme\', \'rb\'))
}
requests.request(method=\'POST\',
                 url=\'http://127.0.0.1:8000/test/\',
                 files=file_dict)

发送文件，定制文件名
file_dict = {
    \'f1\': (\'test.txt\', "hahsfaksfa9kasdjflaksdjf")
}
requests.request(method=\'POST\',
                 url=\'http://127.0.0.1:8000/test/\',
                 files=file_dict)

发送文件，定制文件名
file_dict = {
    \'f1\': (\'test.txt\', "hahsfaksfa9kasdjflaksdjf", \'application/text\', {\'k1\': \'0\'})
}
requests.request(method=\'POST\',
                 url=\'http://127.0.0.1:8000/test/\',
                 files=file_dict)

auth认证

解决浏览器的自带认证问题

认证设置:登陆网站是,弹出一个框,要求你输入用户名密码（与alter很类似），此时是无法获取html的，但本质原理是拼接成请求头发送

r.headers[\'Authorization\'] = _basic_auth_str(self.username, self.password)

一般的网站都不用默认的加密方式，都是自己写，那么我们就需要按照网站的加密方式，自己写一个类似于_basic_auth_str的方法
得到加密字符串后添加到请求头：r.headers[\'Authorization\'] =func(\'.....\')

HTTPBasicAuth实际是向浏览器发一个带有Authorization:.................的请求

HTTPBasicAuth
from requests.auth import HTTPBasicAuth, HTTPDigestAuth

ret = requests.get(\'https://api.github.com/user\', auth=HTTPBasicAuth(\'wupeiqi\', \'sdfasdfasdf\'))
print(ret.text)

　　auth别的使用方式

# ret = requests.get(\'http://192.168.1.1\',
# auth=HTTPBasicAuth(\'admin\', \'admin\'))
# ret.encoding = \'gbk\'
# print(ret.text)

# ret = requests.get(\'http://httpbin.org/digest-auth/auth/user/pass\', auth=HTTPDigestAuth(\'user\', \'pass\'))
# print(ret)

timeout

两种超时:float or tuple
timeout=0.1 #代表接收数据的超时时间
timeout=(0.1,0.2)#0.1代表链接超时 0.2代表接收数据的超时时间

import requests
respone=requests.get(\'https://www.baidu.com\',
                     timeout=0.0001)

redirects

ret = requests.get(\'http://127.0.0.1:8000/test/\', allow_redirects=False)
print(ret.text)

proxies

代理设置

# 根据协议来确定发送请求时候的ip地址
proxies = {
    "http": "61.172.249.96:80",
    "https": "http://61.185.219.126:3128",
}

# 根据接收请求的地址来确定用什么地址发送

proxies = {\'http://10.20.1.128\': \'http://10.10.1.10:5323\'}

ret = requests.get("http://www.proxy360.cn/Proxy", proxies=proxies)
print(ret.headers)

from requests.auth import HTTPProxyAuth

proxyDict = {
    \'http\': \'77.75.105.165\',
    \'https\': \'77.75.105.165\'
}
auth = HTTPProxyAuth(\'username\', \'mypassword\')

r = requests.get("http://www.google.com", proxies=proxyDict, auth=auth)
print(r.text)

#支持socks代理,安装:pip install requests[socks]
import requests
proxies = {
    \'http\': \'socks5://user:pass@host:port\',
    \'https\': \'socks5://user:pass@host:port\'
}
respone=requests.get(\'https://www.12306.cn\',
                     proxies=proxies)

print(respone.status_code)

stream

ret = requests.get(\'http://127.0.0.1:8000/test/\', stream=True)
print(ret.content)
ret.close()

# from contextlib import closing
# with closing(requests.get(\'http://httpbin.org/get\', stream=True)) as r:
# # 在此处理响应。
# for i in r.iter_content():
# print(i)

session

import requests

session = requests.Session()

### 1、首先登陆任何页面，获取cookie

i1 = session.get(url="http://dig.chouti.com/help/service")

### 2、用户登陆，携带上一次的cookie，后台对cookie中的 gpsd 进行授权
i2 = session.post(
    url="http://dig.chouti.com/login",
    data={
        \'phone\': "8615131255089",
        \'password\': "xxxxxx",
        \'oneMonth\': ""
    }
)

i3 = session.post(
    url="http://dig.chouti.com/link/vote?linksId=8589623",
)
print(i3.text)

编码问题

import requests
response=requests.get(\'http://www.autohome.com/news\')
# response.encoding=\'gbk\' #汽车之家网站返回的页面内容为gb2312编码的，而requests的默认编码为ISO-8859-1，如果不设置成gbk则中文乱码
print(response.text)

　　

1

分类：

技术点：

相关文章：

猜你喜欢
相关资源
相似解决方案
热门标签
Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode