BeautifulSoup：从嵌套部分和 div 元素中选择 div

Question

如何提取 div['field-item'] 内容，它是具有类 "field-name-field-mpd-total-capacity" 的部分的子项？我正在处理https://rbnenergy.com/node/6081 以供参考。

<section class="field-name-field-mpd-total-capacity">
    <h2 class="field-label">Total Capacity:&nbsp;</h2>
    <div class="field-items">
        <div class="field-item even">125 Mb/d</div>
    </div>
</section>
</td>

也许现在对我动脑筋已经太晚了。这是我的示例代码：

import requests
from bs4 import BeautifulSoup
html = """
    <section class="field-name-field-mpd-total-capacity"><h2 class="field-label">Total Capacity:&nbsp;</h2><div class="field-items"><div class="field-item even">125 Mb/d</div></div></section>          </td>
    """
soup = BeautifulSoup(html, 'lxml')
out = soup.find("section", { "class" : "field-item" })
print(out)

Answer 1

试试这个：

import requests
from bs4 import BeautifulSoup

html = """
    <section class="field-name-field-mpd-total-capacity"><h2 class="field-label">Total Capacity:&nbsp;</h2><div class="field-items"><div class="field-item even">125 Mb/d</div></div></section>          </td>
    """

soup = BeautifulSoup(html, 'lxml')
for allz in soup.findAll("section", { "class" : "field-name-field-mpd-total-capacity" }):
    print(allz.find("div", { "class" : "field-item"}).string)

它也可以直接从网络源中工作。使用类似的东西

page = requests.get("https://example.com/node/") 给它

Answer 2

试试这个：

>>> from bs4 import BeautifulSoup
>>> 
>>> html = """
...     <section class="field-name-field-mpd-total-capacity"><h2 class="field-label">Total Capacity:&nbsp;</h2><div class="field-items"><div class="field-item     even">125 Mb/d</div></div></section>          </td>
...     """
>>>
>>> soup = BeautifulSoup(html, 'lxml')
>>> out = soup.find("div", { "class" : "field-item" })
>>> print(out)
<div class="field-item even">125 Mb/d</div>
>>> out.text
'125 Mb/d'

find 的第一个参数是（usually）要查找的元素的名称。它将在提供的示例中失败，因为没有具有特定类的 section 元素。您可以将其更改为div 以达到预期的效果。

要使用 field-name-field-mpd-total-capacity 类元素从 section 中提取数据项，您可以使用：

>>> from bs4 import BeautifulSoup
>>>
>>> html = '''<section class="field-name-field-mpd-total-capacity"><h2 class="field-label">Total Capacity:&nbsp;</h2><div class="field-items"><div class="field-item even">125 Mb/d</div></div></section>          </td>'''
>>> soup = BeautifulSoup(html, 'lxml')
>>> section = soup.find('section', {'class': 'field-name-field-mpd-total-capacity'})
>>> [x.text for x in section.find_all('div', {'class': 'field-item'})]
['125 Mb/d']

我个人发现将我正在抓取的页面转换为 dicts 以便于处理非常有用。根据您提供的页面，我认为这可能会对您有所帮助：

import requests
from bs4 import BeautifulSoup

response = requests.get('https://rbnenergy.com/node/6081')
soup = BeautifulSoup(response.text, 'lxml')

data = {}

for element in soup.find_all("section", { "class" : "field" }):
    key = element.find('h2', {'class': 'field-label'})
    content = element.find('div', {'class': 'field-items'}).text

    data[key.text.rstrip(':\xa0')] = content

print(data)

样本输出：

{'Operator': 'Rangeland', 'Commodity': 'Crude Oil', 'Stage': 'Operational', 'Project Type': 'New Build', 'In Service Date': 'Q3/2016', 'Diameter': '12 inches', 'Length': '109 miles', 'Base Capacity': '125 Mb/d', 'Total Capacity': '125 Mb/d', 'Origin': 'Orla, TXUnited States', 'Destination': 'Midland, TXUnited States'}