无法从发出帖子请求的站点中获取预期结果

Question

我正在尝试使用下面的脚本从webpage 获取一些 json 响应。以下是在该站点中填充结果的步骤。点击位于webpage 底部的同意 按钮，然后点击EDIT SEARCH 按钮并最后在 SHOW RESULTS 按钮上，无需更改任何内容。

我试过这样：

import requests
from bs4 import BeautifulSoup

url = 'http://finra-markets.morningstar.com/BondCenter/Results.jsp'
post_url = 'http://finra-markets.morningstar.com/bondSearch.jsp'

payload = {
    'postData': {'Keywords':[]},
    'ticker': '',
    'startDate': '',
    'endDate': '',
    'showResultsAs': 'B',
    'debtOrAssetClass': '1,2',
    'spdsType': ''
}

payload_second = {
    'count': '20',
    'searchtype': 'B',
    'query': {"Keywords":[{"Name":"debtOrAssetClass","Value":"3,6"},{"Name":"showResultsAs","Value":"B"}]},
    'sortfield': 'issuerName',
    'sorttype': '1',
    'start': '0',
    'curPage': '1'
}

with requests.Session() as s:
    s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1; ) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.122 Safari/537.36'
    s.headers['Referer'] = 'http://finra-markets.morningstar.com/BondCenter/UserAgreement.jsp'
    r = s.post(url,json=payload)
    s.headers['Access-Control-Allow-Headers'] = r.headers['Access-Control-Allow-Headers']
    s.headers['cf-request-id'] = r.headers['cf-request-id']
    s.headers['CF-RAY'] = r.headers['CF-RAY']
    s.headers['X-Requested-With'] = 'XMLHttpRequest'
    s.headers['Origin'] = 'http://finra-markets.morningstar.com'
    s.headers['Referer'] = 'http://finra-markets.morningstar.com/BondCenter/Results.jsp'
    r = s.post(post_url,json=payload_second)
    print(r.content)

这是我运行上面的脚本时得到的结果：

b'\n\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\n\n\n{}'

如何使脚本填充该站点的预期结果？

附：我不希望使用 selenium 来完成这项工作。

Answer 1

http://finra-markets.morningstar.com/BondCenter/Results.jsp 的响应不包含搜索结果。它必须是异步获取数据。

找出哪些网络请求返回了搜索结果的一种简单方法是使用 Firefox 的 Dev Tools 搜索请求以找到其中一个搜索结果：

为了将 HTTP 请求转换为 Python 请求，我将请求从 Firefox 复制为 CURL 代码方法，将其导入 Postman，然后将其导出为 Python 代码（我知道有点啰嗦（和懒惰）！ ):

所有这些导致以下代码：

import requests

url = "http://finra-markets.morningstar.com/bondSearch.jsp"

payload = "count=20&searchtype=B&query=%7B%22Keywords%22%3A%5B%7B%22Name%22%3A%22debtOrAssetClass%22%2C%22Value%22%3A%223%2C6%22%7D%2C%7B%22Name%22%3A%22showResultsAs%22%2C%22Value%22%3A%22B%22%7D%5D%7D&sortfield=issuerName&sorttype=1&start=0&curPage=1"
headers = {
    'User-Agent': "...",
    'Accept': "text/plain, */*; q=0.01",
    'Accept-Language': "en-US,en;q=0.5",
    'Content-Type': "application/x-www-form-urlencoded; charset=UTF-8",
    'X-Requested-With': "XMLHttpRequest",
    'Origin': "http://finra-markets.morningstar.com",
    'DNT': "1",
    'Connection': "keep-alive",
    'Referer': "http://finra-markets.morningstar.com/BondCenter/Results.jsp",
    'Cookie': "...",
    'cache-control': "no-cache"
    }

response = requests.request("POST", url, data=payload, headers=headers)

print(response.text)

响应不是 100% JSON。所以我只是去掉了外部空白和{B:..} 部分：

>>> text = response.text.strip()[3:-1]
>>> import json
>>> data = json.loads(text)
>>> data['Columns'][0]                                                                                                             
{'moodyRating': {'ratingText': '', 'ratingNumber': 0},
 'fitchRating': {'ratingText': None, 'ratingNumber': None},
 'standardAndPoorRating': {'ratingText': '', 'ratingNumber': 0},