【问题标题】：BeautifulSoup: Extracting a Title and adjacent <a> tagsBeautifulSoup：提取标题和相邻的 <a> 标签
【发布时间】：2022-01-13 08:21:28
【问题描述】：

我正在尝试使用以下内容从“当前事件”页面上的 Wikipedias 侧边栏中获取数据。目前这会产生一个对象数组，每个对象的值都为title 和url。

我还想为从<h3> id 或文本内容派生的数组headline 中的对象提供一个新值。这将导致每个对象具有三个值：headline、url 和 title。但是，我不确定如何遍历这些。

美汤密码

        soup = BeautifulSoup(response, "html.parser").find('div', {'aria-labelledby': 'Ongoing_events'})
        links = soup.findAll('a')


        for item in links:
            title = item.text
            url = ("https://en.wikipedia.org"+item['href'])

            eo = CurrentEventsObject(title, url)
            eventsArray.append(eo)

维基百科时事列表

<div class="mw-collapsible-content">
    <h3><span class="mw-headline" id="Disasters">Disasters</span</h3>
    <ul>
        <li><a href="/wiki/Climate_crisis" title="Climate crisis">Climate crisis</a></li>
        <li><a href="/wiki/COVID-19_pandemic" title="COVID-19 pandemic">COVID-19 pandemic</a></li>
        <li><a href="/wiki/2021%E2%80%9322_European_windstorm_season" title="2021–22 European windstorm season">2021–22 European windstorm season</a></li>
        <li><a href="/wiki/2020%E2%80%9321_H5N8_outbreak" class="mw-redirect" title="2020–21 H5N8 outbreak">2020–21 H5N8 outbreak</a></li>
        <li><a href="/wiki/2021_Pacific_typhoon_season" title="2021 Pacific typhoon season">2021 Pacific typhoon season</a></li>
        <li><a href="/wiki/2021_Madagascar_food_crisis" class="mw-redirect" title="2021 Madagascar food crisis">Madagascar food crisis</a></li>
        <li><a href="/wiki/Water_scarcity_in_Iran" title="Water scarcity in Iran">Water crisis in Iran</a></li>
        <li><a href="/wiki/Famine_in_Yemen_(2016%E2%80%93present)" title="Famine in Yemen (2016–present)">Yemeni famine</a></li>
        <li><a href="/wiki/2021_Cumbre_Vieja_volcanic_eruption" title="2021 Cumbre Vieja volcanic eruption">2021 La Palma eruption</a></li>
    </ul>
    <h3><span class="mw-headline" id="Economic">Economic</span></h3>
    <ul>
        <li><a href="/wiki/2020%E2%80%932021_global_chip_shortage" title="2020–2021 global chip shortage">2020–2021 global chip shortage</a></li>
        <li><a href="/wiki/2021_global_supply_chain_crisis" title="2021 global supply chain crisis">2021 global supply chain crisis</a></li>
        <li><a href="/wiki/COVID-19_recession" title="COVID-19 recession">COVID-19 recession</a></li>
        <li><a href="/wiki/Lebanese_liquidity_crisis" title="Lebanese liquidity crisis">Lebanese liquidity crisis</a></li>
        <li><a href="/wiki/Pandora_Papers" title="Pandora Papers">Pandora Papers leak</a></li>
        <li><a href="/wiki/2021_Sri_Lankan_economic_crisis" title="2021 Sri Lankan economic crisis">Sri Lankan economic and food crisis</a></li>
        <li><a href="/wiki/2018%E2%80%932021_Turkish_currency_and_debt_crisis" title="2018–2021 Turkish currency and debt crisis">Turkish currency and debt crisis</a></li>
        <li><a href="/wiki/2021_United_Kingdom_natural_gas_supplier_crisis" title="2021 United Kingdom natural gas supplier crisis">United Kingdom natural gas supplier crisis</a></li>
    </ul>
    <h3><span class="mw-headline" id="Politics">Politics</span></h3>
    <ul>
        <li><a href="/wiki/2021_Belarus%E2%80%93European_Union_border_crisis" title="2021 Belarus–European Union border crisis">Belarus−European Union border crisis</a></li>
        <li><a href="/wiki/2021_Brazilian_protests" title="2021 Brazilian protests">Brazilian protests</a></li>
        <li><a href="/wiki/2021_Colombian_protests" title="2021 Colombian protests">Colombian tax reform protests</a></li>
        <li><a href="/wiki/2021_Eswatini_protests" title="2021 Eswatini protests">Eswatini protests</a></li>
        <li><a href="/wiki/2018%E2%80%932021_Haitian_protests" title="2018–2021 Haitian protests">Haitian protests</a></li>
        <li><a href="/wiki/2020%E2%80%932021_Indian_farmers%27_protest" title="2020–2021 Indian farmers' protest">Indian farmers' protests</a></li>
        <li><a href="/wiki/Insulate_Britain_protests" title="Insulate Britain protests">Insulate Britain protests</a></li>
        <li><a href="/wiki/2021_Jersey_dispute" title="2021 Jersey dispute">Jersey dispute</a></li>
        <li><a href="/wiki/Libyan_peace_process" title="Libyan peace process">Libyan peace process</a></li>
        <li><a href="/wiki/2020%E2%80%9321_Malaysian_political_crisis" title="2020–21 Malaysian political crisis">Malaysian political crisis</a></li>
        <li><a href="/wiki/2021_Myanmar_protests" title="2021 Myanmar protests">Myanmar protests</a></li>
        <li><a href="/wiki/2018%E2%80%932021_Nicaraguan_protests" title="2018–2021 Nicaraguan protests">Nicaraguan protests</a></li>
        <li><a href="/wiki/End_SARS" title="End SARS">Nigerian protests</a></li>
        <li><a href="/wiki/2019%E2%80%932021_Persian_Gulf_crisis" title="2019–2021 Persian Gulf crisis">Persian Gulf crisis</a></li>
        <li><a href="/wiki/2017%E2%80%93present_Peruvian_political_crisis" title="2017–present Peruvian political crisis">Peruvian crisis</a></li>
        <li><a href="/wiki/2021_Russian_election_protests" title="2021 Russian election protests">Russian election protests</a></li>
        <li><a href="/wiki/2021_Solomon_Islands_unrest" title="2021 Solomon Islands unrest">Solomon Islands unrest</a></li>
        <li><a href="/wiki/Tigrayan_peace_process" title="Tigrayan peace process">Tigrayan peace process</a></li>
        <li><a href="/wiki/2020%E2%80%932021_Thai_protests" title="2020–2021 Thai protests">Thai protests</a></li>
        <li><a href="/wiki/2021_Tunisian_political_crisis" title="2021 Tunisian political crisis">Tunisian political crisis</a></li>
        <li><a href="/wiki/2020%E2%80%932021_United_States_racial_unrest" title="2020–2021 United States racial unrest">United States racial unrest</a></li>
        <li><a href="/wiki/Venezuelan_presidential_crisis" title="Venezuelan presidential crisis">Venezuelan presidential crisis</a></li>
    </ul>
    <div class="editlink noprint plainlinks"><a class="external text" href="https://en.wikipedia.org/w/index.php?title=Portal:Current_events/Sidebar&amp;action=edit">edit section</a></div>
</div>

【问题讨论】：

标签： python html beautifulsoup

【解决方案1】：

注意：尝试选择更具体的元素，以便在一个进程中获取所有信息 - 在循环之外定义一个列表将避免覆盖

以下步骤将创建一个字典列表，例如可以简单地迭代或转换为数据框。

#1 选择<h3> 中直接siblings 的所有<ul>

soup.select('h3 + ul')

#2 选择<h3> 并获取其文本：

e.find_previous_sibling('h3').text.strip()

#3 选择<ul> 中的所有<a> 并在创建字典列表时迭代结果：

for a in e.select('a'):
    data.append({
        'headline':headline,
        'title': a['title'],
        'url':'https://en.wikipedia.org'+a['href']
})

示例

soup = BeautifulSoup(response, "html.parser").find('div', {'aria-labelledby': 'Ongoing_events'})

data = []

for e in soup.select('h3 + ul'):
    headline = e.find_previous_sibling('h3').text.strip()
    for a in e.select('a'):
        data.append({
            'headline':headline,
            'title': a['title'],
            'url':'https://en.wikipedia.org'+a['href']
        })

data

输出

[{'headline': 'Disasters',
  'title': 'Climate crisis',
  'url': 'https://en.wikipedia.org/wiki/Climate_crisis'},
 {'headline': 'Disasters',
  'title': 'COVID-19 pandemic',
  'url': 'https://en.wikipedia.org/wiki/COVID-19_pandemic'},
 {'headline': 'Disasters',
  'title': '2021–22 European windstorm season',
  'url': 'https://en.wikipedia.org/wiki/2021%E2%80%9322_European_windstorm_season'},
 {'headline': 'Disasters',
  'title': '2020–21 H5N8 outbreak',
  'url': 'https://en.wikipedia.org/wiki/2020%E2%80%9321_H5N8_outbreak'},
 {'headline': 'Disasters',
  'title': '2021 Pacific typhoon season',
  'url': 'https://en.wikipedia.org/wiki/2021_Pacific_typhoon_season'},
 {'headline': 'Disasters',
  'title': '2021 Madagascar food crisis',
  'url': 'https://en.wikipedia.org/wiki/2021_Madagascar_food_crisis'},
 {'headline': 'Disasters',
  'title': 'Water scarcity in Iran',
  'url': 'https://en.wikipedia.org/wiki/Water_scarcity_in_Iran'},
 {'headline': 'Disasters',
  'title': 'Famine in Yemen (2016–present)',
  'url': 'https://en.wikipedia.org/wiki/Famine_in_Yemen_(2016%E2%80%93present)'},
 {'headline': 'Disasters',
  'title': '2021 Cumbre Vieja volcanic eruption',
  'url': 'https://en.wikipedia.org/wiki/2021_Cumbre_Vieja_volcanic_eruption'},
 {'headline': 'Economic',
  'title': '2020–2021 global chip shortage',
  'url': 'https://en.wikipedia.org/wiki/2020%E2%80%932021_global_chip_shortage'},
 {'headline': 'Economic',
  'title': '2021 global supply chain crisis',
  'url': 'https://en.wikipedia.org/wiki/2021_global_supply_chain_crisis'},
 {'headline': 'Economic',
  'title': 'COVID-19 recession',
  'url': 'https://en.wikipedia.org/wiki/COVID-19_recession'},
 {'headline': 'Economic',
  'title': 'Lebanese liquidity crisis',
  'url': 'https://en.wikipedia.org/wiki/Lebanese_liquidity_crisis'},
 {'headline': 'Economic',
  'title': 'Pandora Papers',
  'url': 'https://en.wikipedia.org/wiki/Pandora_Papers'},
 {'headline': 'Economic',
  'title': '2021 Sri Lankan economic crisis',
  'url': 'https://en.wikipedia.org/wiki/2021_Sri_Lankan_economic_crisis'},
 {'headline': 'Economic',
  'title': '2018–2021 Turkish currency and debt crisis',
  'url': 'https://en.wikipedia.org/wiki/2018%E2%80%932021_Turkish_currency_and_debt_crisis'},
 {'headline': 'Economic',
  'title': '2021 United Kingdom natural gas supplier crisis',
  'url': 'https://en.wikipedia.org/wiki/2021_United_Kingdom_natural_gas_supplier_crisis'},
 {'headline': 'Politics',
  'title': '2021 Belarus–European Union border crisis',
  'url': 'https://en.wikipedia.org/wiki/2021_Belarus%E2%80%93European_Union_border_crisis'},
 {'headline': 'Politics',
  'title': '2021 Brazilian protests',
  'url': 'https://en.wikipedia.org/wiki/2021_Brazilian_protests'},
 {'headline': 'Politics',
  'title': '2021 Colombian protests',
  'url': 'https://en.wikipedia.org/wiki/2021_Colombian_protests'},
 {'headline': 'Politics',
  'title': '2021 Eswatini protests',
  'url': 'https://en.wikipedia.org/wiki/2021_Eswatini_protests'},
 {'headline': 'Politics',
  'title': '2018–2021 Haitian protests',
  'url': 'https://en.wikipedia.org/wiki/2018%E2%80%932021_Haitian_protests'},
 {'headline': 'Politics',
  'title': "2020–2021 Indian farmers' protest",
  'url': 'https://en.wikipedia.org/wiki/2020%E2%80%932021_Indian_farmers%27_protest'},
 {'headline': 'Politics',
  'title': 'Insulate Britain protests',
  'url': 'https://en.wikipedia.org/wiki/Insulate_Britain_protests'},
 {'headline': 'Politics',
  'title': '2021 Jersey dispute',
  'url': 'https://en.wikipedia.org/wiki/2021_Jersey_dispute'},
 {'headline': 'Politics',
  'title': 'Libyan peace process',
  'url': 'https://en.wikipedia.org/wiki/Libyan_peace_process'},
 {'headline': 'Politics',
  'title': '2020–21 Malaysian political crisis',
  'url': 'https://en.wikipedia.org/wiki/2020%E2%80%9321_Malaysian_political_crisis'},
 {'headline': 'Politics',
  'title': '2021 Myanmar protests',
  'url': 'https://en.wikipedia.org/wiki/2021_Myanmar_protests'},
 {'headline': 'Politics',
  'title': '2018–2021 Nicaraguan protests',
  'url': 'https://en.wikipedia.org/wiki/2018%E2%80%932021_Nicaraguan_protests'},
 {'headline': 'Politics',
  'title': 'End SARS',
  'url': 'https://en.wikipedia.org/wiki/End_SARS'},
 {'headline': 'Politics',
  'title': '2019–2021 Persian Gulf crisis',
  'url': 'https://en.wikipedia.org/wiki/2019%E2%80%932021_Persian_Gulf_crisis'},
 {'headline': 'Politics',
  'title': '2017–present Peruvian political crisis',
  'url': 'https://en.wikipedia.org/wiki/2017%E2%80%93present_Peruvian_political_crisis'},
 {'headline': 'Politics',
  'title': '2021 Russian election protests',
  'url': 'https://en.wikipedia.org/wiki/2021_Russian_election_protests'},
 {'headline': 'Politics',
  'title': '2021 Solomon Islands unrest',
  'url': 'https://en.wikipedia.org/wiki/2021_Solomon_Islands_unrest'},
 {'headline': 'Politics',
  'title': 'Tigrayan peace process',
  'url': 'https://en.wikipedia.org/wiki/Tigrayan_peace_process'},
 {'headline': 'Politics',
  'title': '2020–2021 Thai protests',
  'url': 'https://en.wikipedia.org/wiki/2020%E2%80%932021_Thai_protests'},
 {'headline': 'Politics',
  'title': '2021 Tunisian political crisis',
  'url': 'https://en.wikipedia.org/wiki/2021_Tunisian_political_crisis'},
 {'headline': 'Politics',
  'title': '2020–2021 United States racial unrest',
  'url': 'https://en.wikipedia.org/wiki/2020%E2%80%932021_United_States_racial_unrest'},
 {'headline': 'Politics',
  'title': 'Venezuelan presidential crisis',
  'url': 'https://en.wikipedia.org/wiki/Venezuelan_presidential_crisis'}]

【讨论】：