WEB SCRAPING - python 请求会话无法收集数据

Question

我看过一些类似的帖子，但都没有给我答案。我只需要从一个网站获取 html 内容。我正在发送带有特定案例数据的 POST 请求，然后使用 GET 请求我想 scrape 从 html 中获取文本。问题是我总是收到第一页的内容。不知道我做错了什么。

import requests

headers = {
    'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3',
    'Accept-Encoding':'gzip, deflate, br',
    'Accept-Language':'pl-PL,pl;q=0.9,en-US;q=0.8,en;q=0.7',
    'Connection':'keep-alive',
    'Content-Type':'application/x-www-form-urlencoded',
    'Origin':'https://przegladarka-ekw.ms.gov.pl',
    'Referer':'https://przegladarka-ekw.ms.gov.pl/eukw_prz/KsiegiWieczyste/wyszukiwanieKW',
    'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36',

}

data = {
'kodWydzialu':'PT1R', 
'nrKw':'00037314', 
'cyfraK':'9',
}

url = 'https://przegladarka-ekw.ms.gov.pl/eukw_prz/KsiegiWieczyste/wyszukiwanieKW'

r = requests.session()
r.post(url, data=data, headers=headers)
final_content = r.get(url, headers=headers)
print(final_content.text)

GET 请求来自 ("https://przegladarka-ekw.ms.gov.pl/eukw_prz/eukw201906070952/js/jquery-1.11.0_min.js ") 但它会返回一堵代码。我的目标是抓取将上面的数据提供给搜索菜单后出现的页面。

Answer 1

试试这个

import json
import urllib.request



headers = {
    'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3',
    'Accept-Encoding':'gzip, deflate, br',
    'Accept-Language':'pl-PL,pl;q=0.9,en-US;q=0.8,en;q=0.7',
    'Connection':'keep-alive',
    'Content-Type':'application/x-www-form-urlencoded',
    'Origin':'https://przegladarka-ekw.ms.gov.pl',
    'Referer':'https://przegladarka-ekw.ms.gov.pl/eukw_prz/KsiegiWieczyste/wyszukiwanieKW',
    'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36',

}

data = {
'kodWydzialu':'PT1R',
'nrKw':'00037314',
'cyfraK':'9',
}

url = 'https://przegladarka-ekw.ms.gov.pl/eukw_prz/KsiegiWieczyste/wyszukiwanieKW'


r=urllib.request.urlopen(url, data=bytes(json.dumps(data), encoding="utf-8"))
final_content = r
for i in r:
    print(i)