使用 python 从 ajax 请求中抓取 XML 响应

Question

我正在尝试获取在点击最大（时间范围）按钮时加载到this page 图表中的数据。数据通过 ajax 请求加载。

我检查了请求并尝试使用 requests python 库重现它，但我只能从该图表中检索 1 年的数据。

这是我使用的代码：

r = requests.get("https://www.justetf.com/en/etf-profile.html?0-4.0-tabs-panel-chart-dates-ptl_max&groupField=none&sortField=ter&sortOrder=asc&from=search&isin=IE00B3VWN518&tab=chart&_=1576272593482")
r.content

我也尝试过使用Session：

from requests import Session
session = Session()

session.head('http://justetf.com')

response = session.get(
    url='https://www.justetf.com/en/etf-profile.html?0-4.0-tabs-panel-chart-dates-ptl_max&groupField=none&sortField=ter&sortOrder=asc&from=search&isin=IE00B3VWN518&tab=chart&_=1575929227619',
    data = {"0-4.0-tabs-panel-chart-dates-ptl_max":"",
            "groupField":"none","sortField":"ter",
            "sortOrder":"asc","from":"search",
            "isin":"IE00B3VWN518",
            "tab":"chart",
            "_":"1575929227619"
           },

    headers={
        'Host': 'www.justetf.com',
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko/20100101 Firefox/70.0',
        'Accept': 'application/xml, text/xml, */*; q=0.01',
        'Accept-Language': 'en-US,en;q=0.5',
        'Accept-Encoding': 'gzip, deflate, br',
        'Wicket-Ajax': 'true',
        'Wicket-Ajax-BaseURL': 'en/etf-profile.html?0&groupField=none&sortField=ter&sortOrder=asc&from=search&isin=IE00B3VWN518&tab=chart',
        'Wicket-FocusedElementId': 'id28',
        'X-Requested-With': 'XMLHttpRequest',
        'Connection': 'keep-alive',
        'Referer': 'https://www.justetf.com/en/etf-profile.html?groupField=none&sortField=ter&sortOrder=asc&from=search&isin=IE00B3VWN518&tab=chart',
        'Cookie': 'locale_=en; _ga=GA1.2.1297456970.1574289342; cookieconsent_status=dismiss; AWSALB=QMWHJxgfcpLXJLqX0i0FgBuLn+mpVHVeLRQ6upH338LdggA4/thXHT2vVWQX7pdBd1r486usZXgpAF8RpDsGJNtf6ei8e5NHTsg0hzVHR9C+Fj89AWuQ7ue+fzV2; JSESSIONID=ABB2A35B91751CA9B2D293F5A04505BE; _gid=GA1.2.1029531470.1575928527; _gat=1',
        'TE': 'Trailer'


        },

    cookies = {"_ga":"GA1.2.1297456970.1574289342","_gid":"GA1.2.1411779365.1574289342","AWSALB":"5v+tPMgooQC0deJBlEGl2wVeUSmwVGJdydie1D6dAZSRAK5eBsmg+DQCdBj8t25YRytC5NIi0TbU3PmDcNMjiyFPTp1xKHgwNjZcDvMRePZjTxthds5DsvelzE2I","JSESSIONID":"310F346AED94D1A345207A3489DCF83D","locale_":"en"}
)

但我得到了这个回应

<ajax-response><redirect><![CDATA[/en/etf-profile.html?0&groupField=none&sortField=ter&sortOrder=asc&from=search&isin=IE00B3VWN518&tab=chart]]></redirect></ajax-response>

为什么当我点击 MAX 时，我在浏览器上得到的同一个 XML 文件没有得到响应？

Answer 1

好的，下面是我获取您寻求的数据的解决方案：

url = "https://www.justetf.com/en/etf-profile.html"

querystring = {
  # Modify this string to get the timeline you want
  # Currently it is set to "max" as you can see
  "0-1.0-tabs-panel-chart-dates-ptl_max":"",
  "groupField":"none",
  "sortField":"ter",
  "sortOrder":"asc",
  "from":"search",
  "isin":"IE00B3VWN518",
  "tab":"chart",
  "_":"1576627890798"}

# Not all of these headers may be necessary
headers = {
    'authority': "www.justetf.com",
    'accept': "application/xml, text/xml, */*; q=0.01",
    'x-requested-with': "XMLHttpRequest",
    'wicket-ajax-baseurl': "en/etf-profile.html?0&groupField=none&sortField=ter&sortOrder=asc&from=search&isin=IE00B3VWN518&tab=chart",
    'wicket-ajax': "true",
    'wicket-focusedelementid': "id27",
    'Connection': "keep-alive",
}

session = requests.Session()

# The first request won't return what we want but it sets the cookies
response = session.get( url, params=querystring)

# Cookies have been set now we can make the 2nd request and get the data we want
response = session.get( url, headers=headers, params=querystring)

print(response.text)

作为奖励，我包含了一个 repl.it 的链接，我在其中实际解析数据并获取每个单独的数据点。你可以找到这个here。

如果有帮助，请告诉我！