如何在 Python & Beautifulsoup 中抓取具有相同类名的不同值？答案

【问题标题】：How to scrape different values that share the same class name in Python & Beautifulsoup?如何在 Python & Beautifulsoup 中抓取具有相同类名的不同值？
【发布时间】：2022-01-12 21:52:45
【问题描述】：

我正在尝试抓取一个房地产网站，但我遇到了一个具有相同类名 (class="number") 但具有 3 个不同值（“卧室”、“浴室”和“车库”）的 div。这是我正在尝试抓取的数据：

<div class="infoHolder">
        
        <div class="title">3 Bed Townhouse in Ferndale</div>
        <div class="priceDescription">R 1 280 000</div>
        <div class="priceAdditionalDescriptor"></div>

        
            <div class="propertyType">Townhouse</div>
                    <div class="suburb">Ferndale</div>
                <div class="features row">
                <div class="number">3</div>
                <div class="icon bedroom"></div>                
                            <div class="number">2</div>
                <div class="icon bathroom"></div>
                            <div class="number">1</div>
                <div class="icon garage"></div>

        </div>

这是我正在使用的代码，但它没有产生预期的结果：

import requests
from bs4 import BeautifulSoup as bs

page = 1
prices = []
property_type = []
locations = []
str_address= []
bedrooms = []
bathrooms = []
parking_space = []
while page != 57:
    url = f"https://www.privateproperty.co.za/for-sale/gauteng/johannesburg/randburg-and-ferndale/35?page={page}"
    response = requests.get(url)
    soup = bs(response.content, "html.parser")
    for price in soup.findAll('div', class_="priceDescription"):
        prices.append(price.get_text(strip=True))
    for p_type in soup.findAll('div', class_="propertyType"):
        property_type.append(p_type.get_text(strip=True))
    for location in soup.findAll("div", class_="suburb"):
        locations.append(location.get_text(strip=True))
    for street_address in soup.findAll('div', class_="address"):
        str_address.append(street_address.get_text(strip=True))
    for el in soup.findAll("div", class_="number"):
        bedrooms.append(el.get_text(strip=True))
    for br in soup.findAll("div", class_="number"):
        bathrooms.append(br.get_text(strip=True))
    for parking in soup.findAll("div", class_="number"):
        parking_space.append(parking.get_text(strip=True))
    page = page + 1

非常感谢您的帮助。提前致谢。

【问题讨论】：

标签： python beautifulsoup

【解决方案1】：

您可以使用findNextSibling 来查找指定标签之后的标签。例如。首先搜索类"icon bedroom" 的div，然后使用findNextSibling 查找类"number" 的div：

from bs4 import BeautifulSoup as bs

html = '''<div class="infoHolder">
        
        <div class="title">3 Bed Townhouse in Ferndale</div>
        <div class="priceDescription">R 1 280 000</div>
        <div class="priceAdditionalDescriptor"></div>

        
            <div class="propertyType">Townhouse</div>
                    <div class="suburb">Ferndale</div>
                <div class="features row">
                <div class="number">3</div>
                <div class="icon bedroom"></div>                
                            <div class="number">2</div>
                <div class="icon bathroom"></div>
                            <div class="number">1</div>
                <div class="icon garage"></div>

        </div>'''

soup = bs(html, "html.parser")

bedrooms = soup.find('div', class_="icon bedroom").findNextSibling('div', class_="number").get_text()

输出：2

【讨论】：