使用 BeautifulSoup 进行抓取 - 类名相同的问题

Question

我正在尝试一个学校项目从以下网站抓取数据：https://www.coingecko.com/en/coins/bitcoin/historical_data/usd?start_date=2021-01-01&end_date=2021-09-30。我的目标是为以下列获取单独的列表：关闭、打开、交易量和日期。我的问题是，对于列卷，打开和关闭，类名（文本中心）是相同的（第一行的示例）：

<th scope="row" class="font-semibold text-center">2021-09-30</th>
<td class="text-center">
$782,626,384,092
</td>
<td class="text-center">
$30,068,690,312
</td>
<td class="text-center">
$41,588
</td>
<td class="text-center">
N/A
</td>

我尝试使用以下代码解决它，但没有成功（对于关闭值）：

from bs4 import BeautifulSoup
import requests
import pandas as pd

website = 'https://www.coingecko.com/en/coins/bitcoin/historical_data/usd?start_date=2021-01-01&end_date=2021-09-30#panel'

response = requests.get(website)

soup = BeautifulSoup(response.content, 'html.parser')

results = soup.find('table', {'class':'table-striped'}).find('tbody').find_all('tr')

close = []
volume = []
open = []
date = []

all_tr = soup.find_all('tr')
print('rows:', len(all_tr))

for row in all_tr:
    all_td = row.find_all('td', recursive=False)
    print('columns:', len(all_td))
    for column in all_td:
        print(column.text)

    close.append(all_td[4].text)

如果有人可以帮助我，我将非常感激！

Answer 1

这是使用defaultdict 的答案。不确定您是否覆盖了collections：

from collections import defaultdict
from bs4 import BeautifulSoup
import requests

website = 'https://www.coingecko.com/en/coins/bitcoin/historical_data/usd?start_date=2021-01-01&end_date=2021-09-30#panel'

response = requests.get(website)

soup = BeautifulSoup(response.content, 'html.parser')
table = soup.find('table', {'class':'table-striped'})
columns = [th.text for th in table.find('thead').find_all('th')]
rows = table.find('tbody').find_all('tr')

data = defaultdict(list)
[data[columns[i]].append(col.text.strip()) for row in rows for i, col in enumerate(row.find_all('td'))]

print(data.keys())
print(data['Date'][:5])

打印出来：

dict_keys(['Date', 'Market Cap', 'Volume', 'Open'])
['$782,626,384,092', '$775,534,111,089', '$794,889,951,096', '$825,341,135,636', '$808,279,417,023']

即使您不使用defaultdict，您也可以从中看出您需要两个循环，一个循环遍历您的行，然后一个内部循环循环遍历行中的每一列。如果你想保持它非常基本，你可以通过以下方式消除内部循环：

r = 0
for row in rows:
   cols = row.find_all('td')
   date[r] = cols[0]
   cap[r] = cols[1]
   ...
   r += 1

Answer 2

这是使用 BeautifulSoup 和 css 选择器的解决方案。

from bs4 import BeautifulSoup
import requests
import pandas as pd

website = 'https://www.coingecko.com/en/coins/bitcoin/historical_data/usd?start_date=2021-01-01&end_date=2021-09-30#panel'

response = requests.get(website)

soup = BeautifulSoup(response.content, 'html.parser')

results = soup.select('table.table-striped tbody tr')

# close = []
# volume = []
# datum = []
# open = []
data=[]
for result in results:
    close = result.select_one('td.text-center:nth-child(5)').get_text(strip=True)
    volume = result.select_one('td.text-center:nth-child(3)').get_text(strip=True)
    open = result.select_one('td.text-center:nth-child(4)').get_text(strip=True)
    date = result.select_one('th[scope="row"]').get_text(strip=True)
    data.append([close,volume,open,date])


cols = ["close", "volume","open","datum"]

df = pd.DataFrame(data, columns= cols)
print(df)

输出：

     close            volume     open       datum
0        N/A   $30,068,690,312  $41,588  2021-09-30
1    $41,588   $29,691,944,223  $41,010  2021-09-29
2    $41,010   $30,483,144,439  $42,247  2021-09-28
3    $42,247   $30,462,815,705  $43,337  2021-09-27
4    $43,337   $30,898,116,660  $42,857  2021-09-26
..       ...               ...      ...         ...
268  $34,082   $74,657,165,356  $31,516  2021-01-05
269  $31,516  $178,894,068,361  $33,008  2021-01-04
270  $33,008   $57,273,436,641  $32,164  2021-01-03
271  $32,164   $34,089,717,988  $29,352  2021-01-02
272  $29,352   $43,503,516,563  $29,022  2021-01-01

[273 rows x 4 columns]

Answer 3

您可以使用 pandas 来做到这一点，如下所示：

代码：

import requests
import pandas as pd

headers = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36'}

url = "https://www.coingecko.com/en/coins/bitcoin/historical_data/usd?start_date=2021-01-01&end_date=2021-09-30#panel"

req = requests.get(url,headers=headers)

table = pd.read_html(req.text, attrs = {"class":"table-striped"} )

df = table[0]#.to_csv('score.csv',index = False)

print(df)

输出：

 Date        Market Cap            Volume     Open    Close
0    2021-09-30  $782,626,384,092   $30,068,690,312  $41,588      NaN
1    2021-09-29  $775,534,111,089   $29,691,944,223  $41,010  $41,588
2    2021-09-28  $794,889,951,096   $30,483,144,439  $42,247  $41,010
3    2021-09-27  $825,341,135,636   $30,462,815,705  $43,337  $42,247
4    2021-09-26  $808,279,417,023   $30,898,116,660  $42,857  $43,337
..          ...               ...               ...      ...      ...
268  2021-01-05  $585,726,270,249   $74,657,165,356  $31,516  $34,082
269  2021-01-04  $613,616,917,626  $178,894,068,361  $33,008  $31,516
270  2021-01-03  $597,887,713,054   $57,273,436,641  $32,164  $33,008
271  2021-01-02  $545,593,282,215   $34,089,717,988  $29,352  $32,164
272  2021-01-01  $539,438,036,436   $43,503,516,563  $29,022  $29,352

[273 rows x 5 columns]