【发布时间】:2022-01-12 16:31:47
【问题描述】:
我正在从https://www.aos.wisc.edu/~sco/lakes/Mendota-ice.html 抓取门多塔湖的冰层数据。我写了一个可以工作的刮板,但我很好奇代码是否可以被清理。
这是来自 HTML 的代表 sn-p。有时您想要的数据在
之前,有时在 <br\> 之后。
<td align="center" valign="TOP" width="12%">
<p>
<font face="Arial,Helvetica" size="2">
2010-11
<br/>
2011-12
<br/>
2012-13
<br/>
2013-14
<br/>
2014-15
<br/>
2015-16
<br/>
2016-17
<br/>
2017-18
<br/>
2018-19
<br/>
"
<br/>
2019-20
<br/>
2020-21
</font>
</p>
</td>
这是我编写的用于抓取数据的函数。我查看每个<br/> 以获取数据并使用一些逻辑来处理最后一个。我也使用了太多的 for 循环。是的,代码有效,但这不是好看的代码,这也很重要
理想情况下,我只想获取字体标签之间的内容并沿<br/> 拆分。我努力做到这一点,最终得到了一堆 \n \r 和 \xa0 字符。
下面的代码是当你暴力破解解决方案时会发生什么。
from collections import defaultdict
from bs4 import BeautifulSoup
import pandas as pd
import numpy as np
import requests
def get_mendota_data():
# read html
url = 'https://www.aos.wisc.edu/~sco/lakes/Mendota-ice.html'
r = requests.get(url)
html = r.text
# create soup object
soup = BeautifulSoup(html, 'lxml')
# gets headers
headers = []
match = soup.find('table')
for cell in match.find_all('td'):
try:
val = cell.p.text.strip()
headers.extend([val])
# duplicated, break out at blank space
except Exception as e:
break
# this used to put data into default_dict
len_headers_minus_one = len(headers)-1
# default dict to store data
d = defaultdict(list)
# outside loop over headers
num=0
for match in soup.find_all('table'):
for col in match.find_all('tr'):
for val in col.find_all('font'):
val_list = []
for enum, br in enumerate(val.find_all('br'), start=1):
try:
# see if data before the <br/>
next_s = br.previousSibling.strip()
# make sure not empty string ''
if len(next_s)>0:
val_list.extend([next_s])
# see if data behind the last <br/>
if (enum==len(val.find_all('br'))) and len(br.nextSibling.strip())>0:
next_s = br.nextSibling.strip()
val_list.extend([next_s])
except:
pass
# make sure list contains data
if len(val_list)>0:
# puts val_list into dict
d[headers[num]].extend(val_list)
# and updates num
if num<len_headers_minus_one:
num = num +1
else:
num=0
# conver to tdataframe
return pd.DataFrame.from_dict(dict(d))
你可以这样运行函数
df = get_mendota_data()
任何有效地抓取这些数据的想法都将不胜感激。谢谢!
更新
我了解了 stripped_strings 并能够大幅清理代码。这是我能做到的最干净的,但很想摆脱嵌套的 for 循环
def get_mendota_data():
# read html
url = 'https://www.aos.wisc.edu/~sco/lakes/Mendota-ice.html'
r = requests.get(url)
html = r.text
# create soup object
soup = BeautifulSoup(html, 'lxml')
# soup.find('table') gets all the first table
# but I know the first four are the headers
headers = [val for val in soup.find('table').stripped_strings][0:4]
# initate a dictionary to store data
def_dict = defaultdict(list)
for match in soup.find_all('tr'):
col_list = [] # will store all the columns
for font in match.find_all('font'):
tmp_list = [data for data in font.stripped_strings]
# append to col_list
if len(tmp_list)>0:
col_list.append(tmp_list)
# put first half in dict
for enum, col in enumerate(col_list[0:4]):
def_dict[headers[enum]].extend(col)
# put second half in def_dict
for enum, col in enumerate(col_list[4:]):
def_dict[headers[enum]].extend(col)
# conver to tdataframe
return pd.DataFrame.from_dict(dict(def_dict))
【问题讨论】:
标签: python web-scraping beautifulsoup