抓取：从图表中提取数据

Question

我找到了一种从chart page of the Wall Street Journal 下载数据的方法，方法是查看网络选项卡（在开发工具面板中）并重现刷新图表时创建的请求。它的工作原理如下：

import requests
import json
import time
from datetime import datetime as dt
from urllib.parse import urlencode 

data = {
        "Step":"PT5M",
        "TimeFrame":"D1",
        "StartDate":int(dt(2019, 5, 1).timestamp()*1000),
        "EndDate":int(dt(2019, 5, 5).timestamp()*1000),
        "EntitlementToken":"57494d5ed7ad44af85bc59a51dd87c90",
        "IncludeMockTick":True,
        "FilterNullSlots":False,
        "FilterClosedPoints":True,
        "IncludeClosedSlots":False,
        "IncludeOfficialClose":True,
        "InjectOpen":False,
        "ShowPreMarket":False,
        "ShowAfterHours":False,
        "UseExtendedTimeFrame":True,
        "WantPriorClose":False,
        "IncludeCurrentQuotes":False,
        "ResetTodaysAfterHoursPercentChange":False,
        "Series":[{"Key":"STOCK/US/XNYS/ABB","Dialect":"Charting","Kind":"Ticker","SeriesId":"s1","DataTypes":["Last"],"Indicators":[{"Parameters":[{"Name":"Period","Value":"50"}],"Kind":"SimpleMovingAverage","SeriesId":"i2"},{"Parameters":[],"Kind":"Volume","SeriesId":"i3"}]}]
    }

data = {
    'json' : json.dumps(data)
}

data = urlencode(data)

headers = {
    'Accept': 'application/json, text/javascript, */*; q=0.01',
    'Dylan2010.EntitlementToken': '57494d5ed7ad44af85bc59a51dd87c90',
    'Origin': 'https://quotes.wsj.com',
    'Referer': 'https://quotes.wsj.com/ABB/advanced-chart',
    'Sec-Fetch-Mode': 'cors',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36'
}

url = 'https://api.wsj.net/api/michelangelo/timeseries/history?' + data + '&ckey=57494d5ed7'
print(url)
r = requests.get(url, headers = headers)
r.text

这很棒，因为它非常简单且有效，但是我只能检索最长 25 天左右的分钟数据。

另一方面，morningstar charting 似乎有更多可用的分钟数据，我想用它做同样的事情：只需通过查找在更新图表时的背景。但是当我查看网络选项卡时，我在更改日期范围时看不到任何呼叫。我对javascript不太了解，想知道他们使用什么替代机制来实现这一点。 （也许是异步/获取？）

有谁知道我怎么会看到这些电话？

Answer 1

您将需要调查是否有任何其他查询字符串参数是基于时间的（或依赖于股票代码）。我已经用一个似乎仍然有效的固定值替换了几个参数。

import requests

start_date = '20170814'
end_date = '20190102'

r = requests.get(f'https://quotespeed.morningstar.com/ra/uniqueChartData?instid=MSSAL&sdkver=2.39.0&CToken=1&productType=sdk&cdt=7&ed={end_date}&f=d&hasPreviousClose=true&ipoDates=19801212&pids=0P000000GY&sd={start_date}&tickers=126.1.AAPL&qs_wsid=27E31E614F74FC7D8828E941CAC2D319&tmpid=1&instid=MSSAL&_=1').json()

print(r)

---

fiddler 观察到的原始参数：