Python 在帖子上请求 422 错误

Question

我一直在尝试抓取像 GitHub 这样需要登录身份验证的网站，但与 Github 不同的是，它没有 API。我遵循了these 指令和许多其他指令，但似乎没有任何效果，只是返回 422 错误。

from lxml import html

url = "https://github.com/login"
user = "my email"
pas = "associated password"

sess = requests.Session()
r = sess.get(url)

rhtml = html.fromstring(r.text)

#get all hidden input fields and make a dict of them
hidden = rhtml.xpath(r'//form//input[@type="hidden"]')
form = {x.attrib["name"]: x.attrib["value"] for x in hidden}

#add login creds to the dict
form['login'] = user
form['password'] = pas

#post
res = sess.post(url, data=form)

print(res)
# <Response [422]>

我也试过sess.post(url, data={'login':user, 'password':pas})，结果相同。 @987654324 先@cookie 并在帖子中使用它们似乎也不起作用。

我怎样才能获得我的登录页面，最好不使用 Selenium？

Answer 1

那是因为action 的表单与登录页面不同。

这就是你可以使用requests 和BeautifulSoup 的方法：

import requests
from bs4 import BeautifulSoup

url = "https://github.com/login"
user = "<username>"
pwd = "<password>"

with requests.Session() as s:

    r = s.get(url)
    soup = BeautifulSoup(r.content, "lxml")

    hidden = soup.find_all("input", {'type':'hidden'})
    target = "https://github.com" + soup.find("form")['action']
    payload = {x["name"]: x["value"] for x in hidden}

    #add login creds to the dict
    payload['login'] = user
    payload['password'] = pwd

    r = s.post(target, data=payload)
    print(r)