我是抓取的新手，想从以下网址抓取“历史数据”表：https://coinmarketcap.com/currencies/bitcoin/historical-data/
我曾尝试使用 bs4，但似乎没有什么对我有用，因为它只是返回一个空列表... 据我了解，我需要做的是在容器中找到所有“tr” - 还是什么？ 我没有那么多代码，但我认为向您展示它是有意义的，所以有一些东西可以使用：
我的代码：
page = requests.get("https://coinmarketcap.com/currencies/bitcoin/historical-data/")
soup = BeautifulSoup(page.content, 'html.parser')
soup.find_all('tr')
【问题讨论】：
标签： python pandas web-scraping beautifulsoup python-requests
您要查找的数据通过 XHR/Fetch 调用添加到页面。你可以像下面这样得到它
import requests
r = requests.get('https://api.coinmarketcap.com/data-api/v3/cryptocurrency/historical?id=1&convertId=2781&timeStart=1633910400&timeEnd=1639180800')
if r.status_code == 200:
print(r.json())
【讨论】：
扩展@balderman 的答案，您可以尝试将其正确转换为 pandas 数据框格式：
output = pd.DataFrame(requests.get('https://api.coinmarketcap.com/data-api/v3/cryptocurrency/historical?id=1&convertId=2781&timeStart=1633910400&timeEnd=1639180800').json()['data']['quotes'])
返回
timeOpen ... quote
0 2021-10-11T00:00:00.000Z ... {'open': 54734.124840616, 'high': 57793.039249...
1 2021-10-12T00:00:00.000Z ... {'open': 57526.8320114193, 'high': 57627.87860...
2 2021-10-13T00:00:00.000Z ... {'open': 56038.2567881108, 'high': 57688.66010...
3 2021-10-14T00:00:00.000Z ... {'open': 57372.8320788954, 'high': 58478.73549...
4 2021-10-15T00:00:00.000Z ... {'open': 57345.9019791856, 'high': 62757.12970...
.. ... ... ...
56 2021-12-06T00:00:00.000Z ... {'open': 49413.4790992129, 'high': 50929.51909...
57 2021-12-07T00:00:00.000Z ... {'open': 50581.8300495181, 'high': 51934.78189...
58 2021-12-08T00:00:00.000Z ... {'open': 50667.6476830609, 'high': 51171.37531...
59 2021-12-09T00:00:00.000Z ... {'open': 50450.0820524109, 'high': 50797.16544...
60 2021-12-10T00:00:00.000Z ... {'open': 47642.1435531841, 'high': 50015.25298...
最后使用join() 操作，我们可以取消包含具有值的字典的quote 列：
output = output.join(pd.concat([pd.DataFrame([x]) for x in output['quote']]).reset_index(drop=True)).drop(columns='quote')
要以漂亮清晰的格式获取它：
timeOpen timeClose timeHigh timeLow open high low close volume marketCap timestamp
0 2021-10-11T00:00:00.000Z 2021-10-11T23:59:59.999Z 2021-10-11T19:47:02.000Z 2021-10-11T00:04:02.000Z 54734.124841 57793.039249 54519.765520 57484.789465 4.263733e+10 1.083079e+12 2021-10-11T23:59:59.999Z
1 2021-10-12T00:00:00.000Z 2021-10-12T23:59:59.999Z 2021-10-12T06:14:02.000Z 2021-10-12T20:09:02.000Z 57526.832011 57627.878602 54477.974468 56041.056838 4.108376e+10 1.055926e+12 2021-10-12T23:59:59.999Z
2 2021-10-13T00:00:00.000Z 2021-10-13T23:59:59.999Z 2021-10-13T21:43:02.000Z 2021-10-13T09:10:02.000Z 56038.256788 57688.660104 54370.973228 57401.097527 4.168425e+10 1.081612e+12 2021-10-13T23:59:59.999Z
3 2021-10-14T00:00:00.000Z 2021-10-14T23:59:59.999Z 2021-10-14T02:27:02.000Z 2021-10-14T18:30:02.000Z 57372.832079 58478.735499 56957.076136 57321.525280 3.661579e+10 1.080160e+12 2021-10-14T23:59:59.999Z
4 2021-10-15T00:00:00.000Z 2021-10-15T23:59:59.999Z 2021-10-15T20:28:02.000Z 2021-10-15T01:20:02.000Z 57345.901979 62757.129703 56868.142693 61593.950061 5.178008e+10 1.160726e+12 2021-10-15T23:59:59.999Z
【讨论】：