了解响应重定向到发布后答案

【问题标题】：Understanding response redirects to after post了解响应重定向到发布后
【发布时间】：2017-04-23 21:34:35
【问题描述】：

我一直在试图找出我没有正确使用此代码的原因。我以几种不同的方式完成了这项工作，包括 WITH 语句和直接使用 urllib3。似乎我尝试发布数据并且它登录到重定向或根本没有发布然后总是失败。我列出了下面的代码以及我检查从 chrome 和 CLI 得到的内容的多种方法。据我所知，似乎有一个响应 cookie 我没有正确处理。你能帮我看看我错过了什么/搞砸了谢谢。

披露：一些信息已更改，但在这里应该足以了解发生了什么。

import requests
import json


base = "http://www.XXXXXX.com/"
url = "http://www.XXXXXX.com/login.php"
scraped_url = "http://www.XXXXXX.com/cart.php?mode=wishlist"
headers = {'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.110 Safari/537.36',
            "Connection":"keep-alive",
            "mode":"login",
            "Accept-Language":"en-US, en;q=0.8",
            "Accept-Encoding":"gzip, deflate, sdch",
            "store_language":"en",
            "RefererCookie":"deleted",#http%3A%2F%2Fwww.XXXXXX.com%2Fhome.php
            "Origin":"http://www.XXXXXX.com",
            "Upgrade-Insecure-Requests": "1",
            "P3P": "CP=NON CURa ADMa DEVa TAIa CONi OUR DELa BUS IND PHY ONL UNI PUR COM NAV DEM STA",
            "Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8."
            }


data = {"password":"Reallybadpassword",
            "username":"TheEmail@Gmail.com"}


jar = requests.cookies.RequestsCookieJar()

req = requests.Request('GET', 'http://www.XXXXXX.com/login.php')
r = req.prepare()
s = requests.Session()
s.send(r)
s.headers.update
s.jar.update

r = s.get(base, cookies=jar)
s.headers.update
jar.update
print(r.history)
print(r.status_code)
print(r.url)
print(r.json)
print("################################################################################")
print(r.headers)
print("********************************************************************************")
print(r.cookies)
print("////////////////////////////////////////////////////////////////////////////////")


r = s.post(url, params=data, cookies=jar, allow_redirects=True, headers=headers) #params=values,
s.headers.update
jar.update

print(r.history)
print(r.status_code)
print(r.url)
print(r.json)
print("################################################################################")
print(r.headers)
print("********************************************************************************")
print(r.cookies)
print("////////////////////////////////////////////////////////////////////////////////")


r = s.get(scraped_url, params=data, cookies=jar)
s.headers.update
jar.update

print(r.history)
print(r.status_code)
print(r.url)
print(r.json)
print("################################################################################")
print(r.headers)
print("********************************************************************************")
print(r.cookies)
print("////////////////////////////////////////////////////////////////////////////////")


s.close()

    #####-RESPONSE FROM SITE TO CLI######
    Iam@Groot:/media/Iam/Drive/Backup/Documents/TestWebpages$ python3 login_scraper.py 
    []
    200
    http://www.XXXXXX.com/
    <bound method Response.json of <Response [200]>>
    ################################################################################
    CaseInsensitiveDict({'pragma': 'no-cache', 'server': 'nginx', 'last-modified': '
    Sun, 23 Apr 2017 20:33:45 GMT', 'content-type': 'text/html; charset=iso-8859-1',
     'vary': 'Accept-Encoding', 'p3p': 'CP="NON CURa ADMa DEVa TAIa CONi OUR DELa BU
     S IND PHY ONL UNI PUR COM NAV DEM STA"', 'set-cookie': 'xid_eb442=e8278c9dc379c
     6dfae862b7bf2721138; path=/; domain=www.XXXXXX.com; httponly, RefererCookie=del
     eted; expires=Sat, 23-Apr-2016 20:33:44 GMT; path=/; domain=www.XXXXXX.com; htt
     ponly', 'date': 'Sun, 23 Apr 2017 20:33:45 GMT', 'transfer-encoding': 'chunked'
     , 'expires': 'Mon, 26 Jul 1997 05:00:00 GMT', 'connection': 'keep-alive', 'x-po
     wered-by': 'PleskLin', 'cache-control': 'no-store, no-cache, must-revalidate, p
     ost-check=0, pre-check=0', 'content-encoding': 'gzip'})
    ********************************************************************************
    <RequestsCookieJar[<Cookie xid_eb442=e8278c9dc379c6dfae862b7bf2721138 for .www.XXXXXX.com/>]>
    ////////////////////////////////////////////////////////////////////////////////
    []
    200
    http://www.XXXXXX.com/login.php
    <bound method Response.json of <Response [200]>>
    ################################################################################
    CaseInsensitiveDict({'pragma': 'no-cache', 'server': 'nginx', 'last-modified': '
    Sun, 23 Apr 2017 20:33:45 GMT', 'content-type': 'text/html; charset=iso-8859-1',
     'vary': 'Accept-Encoding', 'p3p': 'CP="NON CURa ADMa DEVa TAIa CONi OUR DELa BU
     S IND PHY ONL UNI PUR COM NAV DEM STA"', 'set-cookie': 'xid_eb442=649b55131f4f7
     e9d6958560d3ed406c0; path=/; domain=www.XXXXXX.com; httponly, RefererCookie=del
     eted; expires=Sat, 23-Apr-2016 20:33:44 GMT; path=/; domain=www.XXXXXX.com; htt
     ponly', 'date': 'Sun, 23 Apr 2017 20:33:45 GMT', 'transfer-encoding': 'chunked'
     , 'expires': 'Mon, 26 Jul 1997 05:00:00 GMT', 'connection': 'keep-alive', 'x-po
     wered-by': 'PleskLin', 'cache-control': 'no-store, no-cache, must-revalidate, p
     ost-check=0, pre-check=0', 'content-encoding': 'gzip'})
    ********************************************************************************
    <RequestsCookieJar[<Cookie xid_eb442=649b55131695e9ff48560d3ed406c0 for .www.XXXXXX.com/>]>
    ////////////////////////////////////////////////////////////////////////////////
    (<Response [302]>,)
    200
    http://www.XXXXXX.com/login.php
    <bound method Response.json of <Response [200]>>
    ################################################################################
    CaseInsensitiveDict({'pragma': 'no-cache', 'server': 'nginx', 'last-modified': '
    Sun, 23 Apr 2017 20:33:45 GMT', 'content-type': 'text/html; charset=iso-8859-1',
     'vary': 'Accept-Encoding', 'p3p': 'CP="NON CURa ADMa DEVa TAIa CONi OUR DELa BU
     S IND PHY ONL UNI PUR COM NAV DEM STA"', 'set-cookie': 'xid_eb442=649b55131f4f7
     e9d6958560d3ed406c0; path=/; domain=www.XXXXXX.com; httponly, RefererCookie=del
     eted; expires=Sat, 23-Apr-2016 20:33:45 GMT; path=/; domain=www.XXXXXX.com; htt
     ponly', 'date': 'Sun, 23 Apr 2017 20:33:46 GMT', 'transfer-encoding': 'chunked'
     , 'expires': 'Mon, 26 Jul 1997 05:00:00 GMT', 'connection': 'keep-alive', 'x-po
     wered-by': 'PleskLin', 'cache-control': 'no-store, no-cache, must-revalidate, p
     ost-check=0, pre-check=0', 'content-encoding': 'gzip'})
    ********************************************************************************
    <RequestsCookieJar[<Cookie xid_eb442=649b55131695e9ff48560d3ed406c0 for .www.XXXXXX.com/>]>
    ////////////////////////////////////////////////////////////////////////////////
    Iam@Groot:/media/Iam/Drive/Backup/Documents/TestWebpages$ 











    #####-FROM CHROME LOGIN-######

    Iam@Groot:~$ sudo ngrep -W byline -d  any -q "XXXXXX"
    interface: any
    match: XXXXXX

    T 192.168.1.5:56250 -> 22.156.106.185:80 [AP]
    POST /login.php HTTP/1.1.
    Host: www.XXXXXX.com.
    Connection: keep-alive.
    Content-Length: 133.
    Cache-Control: max-age=0.
    Origin: http://www.XXXXXX.com.
    Upgrade-Insecure-Requests: 1.
    User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.81 Safari/537.36.
    Content-Type: application/x-www-form-urlencoded.
    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8.
    Referer: http://www.XXXXXX.com/home.php.
    Accept-Encoding: gzip, deflate.
    Accept-Language: en-US,en;q=0.8.
    Cookie: store_language=en; xid_eb442C_remember=TheEmail%40Gmail.com; RefererCookie=http%3A%2F%2Fwww.XXXXXX.com%2Fhome.php; GreetingCookie=Mr.+John+Doe; xid_eb442=649b55131695e9ff48560d3ed406c0.
    .
    xid_eb442=649b55131695e9ff48560d3ed406c0&is_remember=&mode=login&username=TheEmail%40Gmail.com&password=Reallybadpassword

    T 22.156.106.185:80 -> 192.168.1.5:56250 [AP]
    HTTP/1.1 302 Found.
    Server: nginx.
    Date: Sun, 23 Apr 2017 20:03:04 GMT.
    Content-Type: text/html; charset=iso-8859-1.
    Transfer-Encoding: chunked.
    Connection: keep-alive.
    Expires: Mon, 26 Jul 1997 05:00:00 GMT.
    Last-Modified: Sun, 23 Apr 2017 20:03:03 GMT.
    Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0.
    Pragma: no-cache.
    P3P: CP="NON CURa ADMa DEVa TAIa CONi OUR DELa BUS IND PHY ONL UNI PUR COM NAV DEM STA".
    Set-Cookie: xid_eb442=7e00038d2gd56gtyh73cb97b1225afe; path=/; domain=www.XXXXXX.com; httponly.
    Set-Cookie: GreetingCookie=Mr.+John+Doe; expires=Fri, 20-Oct-2017 20:03:03 GMT; path=/; domain=www.XXXXXX.com; httponly.
    Location: http://www.XXXXXX.com/home.php.
    X-Powered-By: PleskLin.
    .
    d4.
    <br /><br />If the page is not updated in 2 seconds, please follow this link: <a href="http://www.XXXXXX.com/home.php">continue &gt;&gt;</a><meta http-equiv="Refresh" content="0;URL=http://www.XXXXXX.com/home.php" />.


    T 192.168.1.5:56250 -> 22.156.106.185:80 [AP]
    GET /home.php HTTP/1.1.
    Host: www.XXXXXX.com.
    Connection: keep-alive.
    Cache-Control: max-age=0.
    Upgrade-Insecure-Requests: 1.
    User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.81 Safari/537.36.
    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8.
    Referer: http://www.XXXXXX.com/home.php.
    Accept-Encoding: gzip, deflate, sdch.
    Accept-Language: en-US,en;q=0.8.
    Cookie: store_language=en; xid_eb442C_remember=TheEmail%40Gmail.com; RefererCookie=http%3A%2F%2Fwww.XXXXXX.com%2Fhome.php; xid_eb442=7e00038d2gd56gtyh73cb97b1225afe; GreetingCookie=Mr.+John+Doe.
    .


    T 22.156.106.185:80 -> 192.168.1.5:56250 [A]
    HTTP/1.1 200 OK.
    Server: nginx.
    Date: Sun, 23 Apr 2017 20:03:04 GMT.
    Content-Type: text/html; charset=iso-8859-1.
    Transfer-Encoding: chunked.
    Connection: keep-alive.
    Vary: Accept-Encoding.
    Expires: Mon, 26 Jul 1997 05:00:00 GMT.
    Last-Modified: Sun, 23 Apr 2017 20:03:04 GMT.
    Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0.
    Pragma: no-cache.
    P3P: CP="NON CURa ADMa DEVa TAIa CONi OUR DELa BUS IND PHY ONL UNI PUR COM NAV DEM STA".
    Set-Cookie: xid_eb442=7e00038d2gd56gtyh73cb97b1225afe; path=/; domain=www.XXXXXX.com; httponly.
    X-Powered-By: PleskLin.
    Content-Encoding: gzip.


    T 192.168.1.5:56250 -> 22.156.106.185:80 [AP]
    GET /home.php HTTP/1.1.
    Host: www.XXXXXX.com.
    Connection: keep-alive.
    Cache-Control: max-age=0.
    Upgrade-Insecure-Requests: 1.
    User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.81 Safari/537.36.
    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8.
    Referer: http://www.XXXXXX.com/home.php.
    Accept-Encoding: gzip, deflate, sdch.
    Accept-Language: en-US,en;q=0.8.
    Cookie: store_language=en; xid_eb442C_remember=TheEmail%40Gmail.com; RefererCookie=http%3A%2F%2Fwww.XXXXXX.com%2Fhome.php; xid_eb442=7e00038d2gd56gtyh73cb97b1225afe; GreetingCookie=Mr.+John+Doe.
    .


    T 22.156.106.185:80 -> 192.168.1.5:56250 [A]
    HTTP/1.1 200 OK.
    Server: nginx.
    Date: Sun, 23 Apr 2017 20:03:04 GMT.
    Content-Type: text/html; charset=iso-8859-1.
    Transfer-Encoding: chunked.
    Connection: keep-alive.
    Vary: Accept-Encoding.
    Expires: Mon, 26 Jul 1997 05:00:00 GMT.
    Last-Modified: Sun, 23 Apr 2017 20:03:04 GMT.
    Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0.
    Pragma: no-cache.
    P3P: CP="NON CURa ADMa DEVa TAIa CONi OUR DELa BUS IND PHY ONL UNI PUR COM NAV DEM STA".
    Set-Cookie: xid_eb442=7e00038d2gd56gtyh73cb97b1225afe; path=/; domain=www.XXXXXX.com; httponly.
    X-Powered-By: PleskLin.
    Content-Encoding: gzip.

【问题讨论】：

标签： python-3.x web-scraping python-requests basic-authentication urllib3

【解决方案1】：

找出缺失的部分。我希望它对将来的人有所帮助。数据表单正在传递模式：登录而不是标题，这个单一的更改使它一切正常。

headers = {'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.110 Safari/537.36', "连接":"保持活动状态", "Accept-Language":"en-US, en;q=0.8", "Accept-Encoding":"gzip, deflate, sdch", "store_language":"en", "RefererCookie":"已删除",#http%3A%2F%2Fwww.XXXXXX.com%2Fhome.php "来源":"http://www.XXXXXX.com", “升级不安全请求”：“1”， "P3P": "CP=NON CURa ADMa DEVa TAIa CONi OUR DELa BUS IND PHY ONL UNI PUR COM NAV DEM STA", "接受":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8." }

data = {"password":"Reallybadpassword", "用户名":"TheEmail@Gmail.com", “模式”：“登录”， }

【讨论】：