使用 Python 请求 POST 提交表单数据不起作用

Question

我正在编写一个 python 脚本来自动检查狗重新归巢网站上我们可能会在它们可用时采用的狗，但是我无法完成此网站上的表单数据并且无法弄清楚为什么。

表单属性表明它应该有一个 post 方法，我已经检查了表单的所有输入并创建了一个有效负载。

我希望返回包含搜索结果的页面，并从结果页面中抓取 html，以便我可以开始处理它，但抓取的只是表单页面，永远不会有结果。

我尝试使用带有有效负载的 .get 作为参数，带有有效负载的 url 并使用 requests-html 库来呈现任何 Java 脚本元素，但均未成功。

如果您将 url_w_payload 粘贴到浏览器中，它会加载页面并显示其中一个字段为空。如果您然后在 url 栏中再次按 enter 以重新加载页面而不修改它加载的 url...可能与 cookie 有关？

import requests
from requests_html import HTMLSession

session = HTMLSession()

form_url =  "https://www.rspca.org.uk/findapet?p_p_id=petSearch2016_WAR_ptlPetRehomingPortlets&p_p_lifecycle=1&p_p_state=normal&p_p_mode=view&_petSearch2016_WAR_ptlPetRehomingPortlets_action=search"

url_w_payload = "https://www.rspca.org.uk/findapet?p_p_id=petSearch2016_WAR_ptlPetRehomingPortlets&p_p_lifecycle=1&p_p_state=normal&p_p_mode=view&_petSearch2016_WAR_ptlPetRehomingPortlets_action=search&noPageView=false&animalType=DOG&freshSearch=false&arrivalSort=false&previousAnimalType=&location=WC2N5DU&previousLocation=&prevSearchedPostcode=&postcode=WC2N5DU&searchedLongitude=-0.1282688&searchedLatitude=51.5072106"

payload = {'noPageView': 'false','animalType': 'DOG', 'freshSearch': 'false', 'arrivalSort': 'false', 'previousAnimalType': '', 'location': 'WC2N5DU', 'previousLocation': '','prevSearchedPostcode': '', 'postcode': 'WC2N5DU', 'searchedLongitude': '-0.1282688',  'searchedLatitude': '51.5072106'}

#req = requests.post(form_url, data = payload)

#with open("requests_output.txt", "w") as f:
 #   f.write(req.text)

ses = session.post(form_url, data = payload)

ses.html.render()

with open("session_output.txt", "w") as f:
    f.write(ses.text)

print("Done")

Answer 1

Cookie 和标头有一些障碍，但一旦你做对了，你就会得到正确的响应。

这是怎么做的：

import time
from urllib.parse import urlencode

import requests
from bs4 import BeautifulSoup

query_string = {
    "p_p_id": "petSearch2016_WAR_ptlPetRehomingPortlets",
    "p_p_lifecycle": 1,
    "p_p_state": "normal",
    "p_p_mode": "view",
    "_petSearch2016_WAR_ptlPetRehomingPortlets_action": "search",
}

payload = {
    'noPageView': 'false',
    'animalType': 'DOG',
    'freshSearch': 'false',
    'arrivalSort': 'false',
    'previousAnimalType': '',
    'location': 'WC2N5DU',
    'previousLocation': '',
    'prevSearchedPostcode': '',
    'postcode': 'WC2N5DU',
    'searchedLongitude': '-0.1282688',
    'searchedLatitude': '51.5072106',
}


def make_cookies(cookie_dict: dict) -> str:
    return "; ".join(f"{k}={v}" for k, v in cookie_dict.items())


with requests.Session() as connection:
    main_url = "https://www.rspca.org.uk"
    
    connection.headers["User-Agent"] = "Mozilla/5.0 (X11; Linux x86_64) " \
                                       "AppleWebKit/537.36 (KHTML, like Gecko) " \
                                       "Chrome/90.0.4430.212 Safari/537.36"
    r = connection.get(main_url)
    
    cookies = make_cookies(r.cookies.get_dict())
    additional_string = f"; cb-enabled=enabled; " \
                        f"LFR_SESSION_STATE_10110={int(time.time())}"
    
    post_url = f"https://www.rspca.org.uk/findapet?{urlencode(query_string)}"
    connection.headers.update(
        {
            "cookie": cookies + additional_string,
            "referer": post_url,
            "content-type": "application/x-www-form-urlencoded",
        }
    )
    response = connection.post(post_url, data=urlencode(payload)).text
    dogs = BeautifulSoup(response, "lxml").find_all("a", class_="detailLink")
    print("\n".join(f"{main_url}{dog['href']}" for dog in dogs))

输出（为简洁起见而缩短，无需对页面进行分页，因为所有狗都在响应中）：

https://www.rspca.org.uk/findapet/details/-/Animal/JAY_JAY/ref/217747/rehome/
https://www.rspca.org.uk/findapet/details/-/Animal/STORM/ref/217054/rehome/
https://www.rspca.org.uk/findapet/details/-/Animal/DASHER/ref/205702/rehome/
https://www.rspca.org.uk/findapet/details/-/Animal/EVE/ref/205701/rehome/
https://www.rspca.org.uk/findapet/details/-/Animal/SEBASTIAN/ref/178975/rehome/
https://www.rspca.org.uk/findapet/details/-/Animal/FIJI/ref/169578/rehome/
https://www.rspca.org.uk/findapet/details/-/Animal/ELLA/ref/154419/rehome/
https://www.rspca.org.uk/findapet/details/-/Animal/BEN/ref/217605/rehome/
https://www.rspca.org.uk/findapet/details/-/Animal/SNOWY/ref/214416/rehome/
https://www.rspca.org.uk/findapet/details/-/Animal/BENSON/ref/215141/rehome/
https://www.rspca.org.uk/findapet/details/-/Animal/BELLA/ref/207716/rehome/

and much more ...

PS。我真的很喜欢这个挑战，因为我有两只来自收容所的狗。坚持下去，伙计！