使用 BeautifulSoup 从 HTML 中提取几个“下一个兄弟姐妹”答案

【问题标题】：Extracting several "next siblings" from HTML with BeautifulSoup使用 BeautifulSoup 从 HTML 中提取几个“下一个兄弟姐妹”
【发布时间】：2023-03-06 19:48:01
【问题描述】：

我有一组共享以下结构的 HTML 文件：

<h1>ITEM NAME</h1>
<span class="standardLabel">Place of publication: </span>PLACENAME
<br /><span class="standardLabel">Publication dates: </span>DATE
<br /><span class="standardLabel">Notes: </span>NOTES
<br /><span class="standardLabel">Frequency: </span>FREQUENCY

我要提取的是所有以粗体表示的信息，但我只能编写一个捕获“项目名称”和“地名”的脚本：

# import packages

from bs4 import BeautifulSoup
import os
from os.path import dirname, join
directory=("C:\\Users\\mobarget\\Google Drive\\ACADEMIA\\10_Data analysis_PhD\\NLI Newspaper DB")

# search information in each file

for infile in os.listdir(directory):
    filename=join(directory, infile)
    indata=open(filename,"r", encoding="utf-8", errors="ignore") 
    contents = indata.read()
    soup = BeautifulSoup(contents,'html')
    newspaper=soup.find('h1')
    if newspaper:
        print("Title of file no.", str(infile), ": ", newspaper)
        place=soup.find("span",{"class":"standardLabel"}).next_sibling
        print(place)
    else:
        continue

输出是：

Title of file no. 1 :  <h1>About Town</h1>
Dungannon, Co. Tyrone

Title of file no. 10 :  <h1>Amárach: Guth na Gaeltachta</h1>
Dublin, Co. Dublin

Title of file no. 100 :  <h1>Belfast Election</h1>
Belfast, Co. Antrim

[等]

有什么想法可以在不使代码过于冗余的情况下提取丢失的数据吗？

【问题讨论】：

标签： html python-3.x beautifulsoup

【解决方案1】：

您可以使用 CSS 选择器 span:contains("<YOUR STRING>") 找到特定的 <span> 标签，然后执行 .next_sibling。

例如：

from bs4 import BeautifulSoup

txt = '''<h1>ITEM NAME</h1>
<span class="standardLabel">Place of publication: </span>PLACENAME
<br /><span class="standardLabel">Publication dates: </span>DATE
<br /><span class="standardLabel">Notes: </span>NOTES
<br /><span class="standardLabel">Frequency: </span>FREQUENCY'''

soup = BeautifulSoup(txt, 'html.parser')

title = soup.h1.text
place = soup.select_one('span:contains("Place of publication:")').next_sibling.strip()
dates = soup.select_one('span:contains("Publication dates:")').next_sibling.strip()
notes = soup.select_one('span:contains("Notes:")').next_sibling.strip()
freq = soup.select_one('span:contains("Frequency:")').next_sibling.strip()

print(title)
print(place)
print(dates)
print(notes)
print(freq)

打印：

ITEM NAME
PLACENAME
DATE
NOTES
FREQUENCY

【讨论】：

谢谢，CSS 选择器运行良好。但是，soup = BeautifulSoup(txt, 'html.parser') 在 Python3 中似乎不起作用。我收到未定义“txt”的通知，必须坚持使用“内容”。
@OnceUponATime 是的，txt 仅在我的示例中定义。如果您的程序中有contents，请使用它！

【解决方案2】：

使用 Andrej Kesely 的回答中的代码，我还添加了缺失属性的异常处理：

# import packages

from bs4 import BeautifulSoup
import os
from os.path import dirname, join
directory=("C:\\Users\\mobarget\\Google Drive\\ACADEMIA\\10_Data analysis_PhD\\NLI Newspaper DB")

# read downloaded HTML files

for infile in os.listdir(directory):
    filename=join(directory, infile)
    indata=open(filename,"r", encoding="utf-8", errors="ignore") 
    contents = indata.read()
    soup = BeautifulSoup(contents, 'html.parser')
    newspaper=soup.find('h1')
    if newspaper:
        try:
            # read data from tags
        
            title = soup.h1.text
            place = soup.select_one('span:contains("Place of publication:")').next_sibling.strip()
            dates = soup.select_one('span:contains("Publication dates:")').next_sibling.strip()
            notes = soup.select_one('span:contains("Notes:")').next_sibling.strip()
            freq = soup.select_one('span:contains("Frequency:")').next_sibling.strip()

            # print results

            print("Title of file no.", str(infile), ": ", title)
            print(place)
            print(dates)
            print(notes)
            print(freq)

            # exception handling if attributes are missing

        except AttributeError:
            print("no data")
            
    else:
        continue

【讨论】：