【问题标题】:Parsing ugly html with BeautifulSoup用 BeautifulSoup 解析丑陋的 html
【发布时间】:2022-01-12 16:31:47
【问题描述】:

我正在从https://www.aos.wisc.edu/~sco/lakes/Mendota-ice.html 抓取门多塔湖的冰层数据。我写了一个可以工作的刮板,但我很好奇代码是否可以被清理。

这是来自 HTML 的代表 sn-p。有时您想要的数据在
之前,有时在 <br\> 之后。

     <td align="center" valign="TOP" width="12%">
      <p>
       <font face="Arial,Helvetica" size="2">
        2010-11
        <br/>
        2011-12
        <br/>
        2012-13
        <br/>
        2013-14
        <br/>
        2014-15
        <br/>
        2015-16
        <br/>
        2016-17
        <br/>
        2017-18
        <br/>
        2018-19
        <br/>
        "
        <br/>
        2019-20
        <br/>
        2020-21
       </font>
      </p>
     </td>

这是我编写的用于抓取数据的函数。我查看每个&lt;br/&gt; 以获取数据并使用一些逻辑来处理最后一个。我也使用了太多的 for 循环。是的,代码有效,但这不是好看的代码,这也很重要

理想情况下,我只想获取字体标签之间的内容并沿&lt;br/&gt; 拆分。我努力做到这一点,最终得到了一堆 \n \r 和 \xa0 字符。

下面的代码是当你暴力破解解决方案时会发生什么。

from collections import defaultdict
from bs4 import BeautifulSoup
import pandas as pd
import numpy as np
import requests
    
def get_mendota_data():
    # read html
    url = 'https://www.aos.wisc.edu/~sco/lakes/Mendota-ice.html'
    r = requests.get(url)
    html = r.text

    # create soup object
    soup = BeautifulSoup(html, 'lxml')

    # gets headers
    headers = []
    match = soup.find('table')
    for cell in match.find_all('td'):
        try:
            val = cell.p.text.strip()
            headers.extend([val])
        # duplicated, break out at blank space
        except Exception as e:
            break

    # this used to put data into default_dict
    len_headers_minus_one = len(headers)-1
    
    # default dict to store data
    d = defaultdict(list)

    # outside loop over headers
    num=0 
    for match in soup.find_all('table'):
        for col in  match.find_all('tr'): 
            for val in col.find_all('font'):
                val_list = []
                for enum, br in enumerate(val.find_all('br'), start=1):
                    try:

                        # see if data before the <br/>
                        next_s = br.previousSibling.strip()

                        # make sure not empty string ''
                        if len(next_s)>0:
                            val_list.extend([next_s])

                        # see if data behind the last <br/>
                        if (enum==len(val.find_all('br'))) and len(br.nextSibling.strip())>0:
                            next_s = br.nextSibling.strip()
                            val_list.extend([next_s])
                    except:
                        pass
                
                # make sure list contains data
                if len(val_list)>0:
                    # puts val_list into dict 
                    d[headers[num]].extend(val_list)
                    # and updates num
                    if num<len_headers_minus_one:
                        num = num +1
                    else:
                        num=0

    # conver to tdataframe 
    return pd.DataFrame.from_dict(dict(d)) 

你可以这样运行函数

df = get_mendota_data()

任何有效地抓取这些数据的想法都将不胜感激。谢谢!

更新

我了解了 stripped_strings 并能够大幅清理代码。这是我能做到的最干净的,但很想摆脱嵌套的 for 循环

def get_mendota_data():
    # read html
    url = 'https://www.aos.wisc.edu/~sco/lakes/Mendota-ice.html'
    r = requests.get(url)
    html = r.text

    # create soup object
    soup = BeautifulSoup(html, 'lxml')

    # soup.find('table') gets all the first table
    # but I know the first four are the headers
    headers = [val for val in soup.find('table').stripped_strings][0:4]

    # initate a dictionary to store data
    def_dict = defaultdict(list)

    for match in soup.find_all('tr'):
        col_list = [] # will store all the columns 
        for font in match.find_all('font'):
            tmp_list = [data for data in font.stripped_strings]

            # append to col_list
            if len(tmp_list)>0:
                col_list.append(tmp_list)

        # put first half in dict
        for enum, col in enumerate(col_list[0:4]):
            def_dict[headers[enum]].extend(col)

        # put second half in def_dict
        for enum, col in enumerate(col_list[4:]):
            def_dict[headers[enum]].extend(col)


    # conver to tdataframe 
    return pd.DataFrame.from_dict(dict(def_dict)) 

【问题讨论】:

    标签: python web-scraping beautifulsoup


    【解决方案1】:

    这是一个设计非常混乱的页面,但提取数据是一项有趣的任务。我决定使用 xpath 和列表推导来实现:

    import lxml.html as lh
    doc = lh.fromstring(r.text)
    
    entries = []
    targets = doc.xpath('//table//tr')
    
    for target in targets:
        subs = target.xpath('.//td')
        del subs[4] #this one is an empty column
        entries.extend([[s.strip() for s in sub.xpath('.//*/text()')] for sub in subs])
    
    #some trailing empty elements needed to be removed before proceeding
    entries = [[list(filter(lambda x: x != "", entry)) for entry in entries[r::4]] for r in range(4)]
    
    #unpack to the correct columns
    [Winter,Closed,Opened,Days] = [[x for list in entry for x in list] for entry in entries]
    
    #and now create a dataframe:
    final = pd.DataFrame(list(zip(Winter,Closed,Opened,Days)),
                   columns =['Winter', 'Closed',"Opened","Days"])
    final
    

    输出是 176 个冬日的开启和关闭事件:

        Winter   Closed     Opened  Days
    0   1852-53     ---     5 Apr   ---
    1   1853-54     27 Dec  ---     ---
    2   1854-55     ---     ---     ---
    3   1855-56     18 Dec  14 Apr  118
    4   1856-57     6 Dec   6 May   151
    

    等等

    【讨论】:

      猜你喜欢
      • 1970-01-01
      • 2021-07-24
      • 1970-01-01
      • 2012-12-13
      • 2020-04-26
      • 1970-01-01
      • 2013-03-10
      • 2019-09-30
      • 1970-01-01
      相关资源
      最近更新 更多