Beautifulsoup4 - 不选择跨度类的所有实例

Question

我正在尝试从使用非特定跨度类来格式化/显示内容的网站上抓取数据。这些页面提供有关化学产品的信息，每个产品都在一个 div 类中进行描述。 我首先由那个 div 类解析，并正在努力从那里提取我需要的数据。我已经能够得到很多东西，但我似乎无法提取的部分在跨度类“ppisreportspan”中 如果您查看代码，您会注意到它在每个化学描述中出现多次。

<tr>
    <td><h4 id='stateprod'>MAINE STATE PRODUCT REPORT</h4><hr class='report'><span style="color:Maroon;" Class="subtitle">Company Number: </span><span style='color:black;' Class="subtitle">38</span><br /><span Class="subtitle">MONSANTO COMPANY                                                                                    <br/>800 N. LINDBERGH BOULEVARD                                  <br/>MAIL STOP FF4B                                              <br/>ST LOUIS&nbsp;MO&nbsp;63167-0001<br/></span><br/><span style="color:Maroon;" Class="subtitle">Number of Currently Registered Products: </span><span style='color:black; font-size:14px' class="subtitle">80</span><br /><br/><p class='noprint'><img alt='' src='images/epalogo.png' />&nbsp;&nbsp;View the label in the US EPA Pesticide Product Label System (PPLS).<br /><img alt='' src='images/alstar.png' />&nbsp;&nbsp;View the label in the Accepted Labels State Tracking and Repository (ALSTAR).<br /></p>
        <hr class='report'>
            <div class='nopgbrk'>
                <span class='ppisreportspanprodname'>PRECEPT INSECTICIDE </span>
                <br/>EPA Registration Number: <a href = "http://iaspub.epa.gov/apex/pesticides/f?p=PPLS:102:::NO::P102_REG_NUM:100-1075" target='blank'>100-1075-524 <img alt='EPA PPLS Link' src='images/pplslink.png'/></a>
                <span class='line-break'></span>
                <span class=ppisProd>ME Product Number: </span>
                <**span class="ppisreportspan"**>2014000996</span>
                <br />Registration Year: &nbsp;&nbsp;<**span class="ppisreportspan"**>2019</span>
                &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Type: &nbsp;&nbsp;<span class="ppisreportspan">RESTRICTED</span><br/><br/>
                <table width='100%'>
                    <tr>
                        <td width='13%'>Percent</td>
                        <td style='width:87%;align:left'>Active Ingredient</td>
                    </tr>
                    <tr>
                        <td><span class="ppisreportspan">3.0000</span></td>
                        <td><span class="ppisreportspan">Tefluthrin (128912)</span></td>
                    </tr>
                </table><hr />
        </div>

        <div class='nopgbrk'>
            <span class='ppisreportspanprodname' >ACCELERON IC-609 INSECTICIDE SEED TREATMENT FOR CORN</span>
            <br/>EPA Registration Number: <a href = "http://iaspub.epa.gov/apex/pesticides/f?p=PPLS:102:::NO::P102_REG_NUM:264-789" target='blank'>264-789-524 <img alt='EPA PPLS Link' src='images/pplslink.png'/>
            </a><span class='line-break'></span>
            <span class=ppisProd>ME Product Number: <a href = "alstar_label.aspx?LabelId=116671" target = 'blank'>2009005053</span>
                <img alt='ALSTAR Link' src='images/alstar.png'/></a>
                <br />Registration Year: &nbsp;&nbsp;<span class="ppisreportspan">2019</span>
                <br/>
                <table width='100%'>
                    <tr>
                        <td width='13%'>Percent</td>
                        <td style='width:87%;align:left'>Active Ingredient</td>
                    </tr>
                    <tr>
                        <td><span class="ppisreportspan">48.0000</span></td>
                        <td><span class="ppisreportspan">Clothianidin (44309)</span></td>
                    </tr>
                </table><hr />
            </div>

此样本包含两种化学品。一个有“alstar” ID 和链接，一个没有。两者都有注册年份。这些是让我感到困惑的数据点。

您可能还注意到，在第一个示例中，“ppisreportspan”中存储了一个 10 位代码。我能够将其提取为“ppisProd”跨度的一部分，用于没有 Alstar 链接的 nay 记录。我不明白为什么，但它强调了我的解析过程似乎忽略了跨度类。

在过去的 2 天里，我根据 SO 上的各种不同答案尝试了各种方法，所以我不可能全部列出。我似乎能够从第一个“跨度”到最后一个跨度的末尾得到任何东西，或者我得到“无类型”错误或空列表。

这个最接近：

它为许多 div 块返回正确的跨度，但它仍然跳过（返回空白元组 []）对于任何具有 alstar 链接（如示例中的第二个）的跨度。 picture showing data and then a series of three sets of empty brackets where the data should be

import urllib.request, urllib.parse, urllib.error
from bs4 import BeautifulSoup
import ssl
import re

url = input('Enter URL:')
hand = open(url)
soup = BeautifulSoup(hand, 'html.parser')
#create a list of chunks by product (div)
products = soup.find_all('div' , class_ ='nopgbrk')
print(type(products))
print(len(products))

tempalstars =[]
rptspanclasses = []
regyears = []
alstarIDs = []
asltrlinks = []
# read the span tags
for product in products:
    tempalstar = product.find_all('span', class_= "ppisreportspan")
    tempalstars.append(tempalstar)

print(tempalstar)

最终，我希望能够为每个 div 块从这些 span 语句中选择年份的文本以及 Alstar 链接，但是当我可以获得找到所有实例的代码时，我将跨越那座桥那个班的。

或者 - 有没有更简单的方法可以让我获得注册年份和 Alstar 链接（例如 <a href = "alstar_label.aspx?LabelId=116671" target = 'blank'>2009005053</span> <img alt='ALSTAR Link' src='images/alstar.png'/></a>），而不是我想要做的？

我正在使用 Python 3.7.2，谢谢！

Answer 1

我设法从该站点获取了一些数据。您只需要知道公司编号，如果是monsanto，则编号为38（此编号在选择Maine 并在搜索框中输入monsanto 后显示：

import re
import requests
from bs4 import BeautifulSoup

url_1 = 'http://npirspublic.ceris.purdue.edu/state/state_menu.aspx?state=ME'
url_2 = 'http://npirspublic.ceris.purdue.edu/state/company.aspx'

company_name = 'monsanto'
company_number = '38'

with requests.session() as s:
    r = s.get(url_1)

    soup = BeautifulSoup(r.text, 'lxml')

    data = {i['name']: '' for i in soup.select('input[name]')}
    for i in soup.select('input[value]'):
        data[i['name']] = i['value']

    data['ctl00$ContentPlaceHolder1$search'] = 'company'
    data['ctl00$ContentPlaceHolder1$TextBoxInput1'] = company_name

    r = s.post(url_1, data=data)

    soup = BeautifulSoup(r.text, 'lxml')

    data = {i['name']: '' for i in soup.select('input[name]')}
    for i in soup.select('input[value]'):
        data[i['name']] = i['value']

    data = {k: v for k, v in data.items() if not k.startswith('ctl00$ContentPlaceHolder1$')}
    data['ctl00$ContentPlaceHolder1${}'.format(company_number)] = 'Display+Products'

    r = s.post(url_2, data=data)
    soup = BeautifulSoup(r.text, 'lxml')

    for div in soup.select('.nopgbrk'):
        #extract name
        print(div.select_one('.ppisreportspanprodname').text)

        #extract ME product number:
        s = ''.join(re.findall(r'\d{10}', div.text))
        print(s)

        #extract alstar link
        s = div.select_one('a[href*="alstar_label.aspx"]')
        if s:
            print(s['href'])
        else:
            print('No ALSTAR link')

        #extract Registration year:
        s = div.find(text=lambda t: 'Registration Year:' in t)
        if s:
            print(s.next.text)
        else:
            print('No registration year.')

        print('-' * 80)

打印：

PRECEPT INSECTICIDE                                                                                                                                                                                     
2014000996
No ALSTAR link
2019
--------------------------------------------------------------------------------
ACCELERON IC-609 INSECTICIDE SEED TREATMENT FOR CORN                                                                                                                                                    
2009005053
alstar_label.aspx?LabelId=117531
2019
--------------------------------------------------------------------------------
ACCELERON D-342 FUNGICIDE SEED TREATMENT                                                                                                                                                                
2015000498
alstar_label.aspx?LabelId=117538
2019
--------------------------------------------------------------------------------
ACCELERON DX-309                                                                                                                                                                                        
2009005026
alstar_label.aspx?LabelId=117559
2019
--------------------------------------------------------------------------------

... and so on.