如何向网站发出请求并下载搜索数据

Question

我正在尝试为个人列表下载一些汽车 VIN 数据。这涉及通过 python3.7 中的 requests.get 向网站“http://vin.place/search.php”发出请求。

我的代码如下所示：

import requests
import pandas as pd
from bs4 import BeautifulSoup

payload = {'first name':'JOHN','last name':'DOE'}
webpage_response = requests.get("http://vin.place/search.php",data = payload)

webpage = webpage_response.content
soup = BeautifulSoup(webpage,"html.parser")

results = soup.select(".search-content")

for x in results:
  print(x.get_text())

不幸的是，我发现 results = [] 。我怀疑我编写requests.get 命令的方式有问题。我不确定网站的正确键应该是多少。有人可以帮忙吗？谢谢。

Answer 1

你的payload错了，应该是first，last。以及小写的名字：

import requests
from bs4 import BeautifulSoup

payload = {'first':'john','last':'doe'}
webpage_response = requests.post("http://vin.place/search.php", data=payload)

webpage = webpage_response.content
soup = BeautifulSoup(webpage,"html.parser")

results = soup.select(".search-content")

for x in results:
    print(x.text)

打印：

JOHN DOE  Purchase of a  2007 HONDA ACCORD
JOHN DOE  Purchase of a  2007 CHRYSLER TOWN AND COUNTRY
JOHN DOE  Purchase of a  2007 DODGE RAM PICKUP 2500
JOHN DOE  Purchase of a  2007 DODGE RAM PICKUP 1500
JOHN DOE  Purchase of a  2007 VOLKSWAGEN PASSAT
JOHN DOE  Purchase of a  2007 VOLKSWAGEN JETTA
JOHN DOE  Purchase of a  2007 CHRYSLER SEBRING
JOHN DOE  Purchase of a  2007 DODGE RAM PICKUP 1500
JOHN DOE  Purchase of a  2007 JEEP PATRIOT
JOHN DOE  Purchase of a  2007 Chevrolet TrailBlazer
JOHN DOE  Purchase of a  2007 FORD EDGE
JOHN DOE  Purchase of a  2007 JEEP WRANGLER UNLIMITED
JOHN DOE  Purchase of a  2007 DODGE RAM PICKUP 2500
JOHN DOE  Purchase of a  2007 FORD MUSTANG
JOHN DOE  Purchase of a  2007 JEEP WRANGLER
JOHN DOE  Purchase of a  2007 CHRYSLER 300
JOHN DOE  Purchase of a  2007 JEEP WRANGLER UNLIMITED
JOHN DOE  Purchase of a  2007 DODGE MAGNUM
JOHN DOE  Purchase of a  2007 CADILLAC ESCALADE ESV
JOHN DOE  Purchase of a  2007 FORD F-150
JOHN DOE  Purchase of a  2007 HONDA ACCORD
JOHN DOE  Purchase of a  2007 JEEP LIBERTY
JOHN DOE  Purchase of a  2007 CADILLAC ESCALADE
JOHN DOE  Purchase of a  2007 DODGE RAM PICKUP 1500
JOHN DOE  Purchase of a  2007 JEEP WRANGLER

Answer 2

手动搜索时查看chrome发送的表单数据，数据为：first=JOHN&last=DOE，另外，网站使用POST

请试试这个：

import requests
import pandas as pd
from bs4 import BeautifulSoup

payload = {'first':'JOHN','last':'DOE'}
webpage_response = requests.post("http://vin.place/search.php",data = payload)

webpage = webpage_response.content
soup = BeautifulSoup(webpage,"html.parser")

results = soup.select(".search-content")

for x in results:
  print(x.get_text())