如何使用漂亮的汤为页面抓取数据答案

【问题标题】：How to scrape data for the page using beautiful soup如何使用漂亮的汤为页面抓取数据
【发布时间】：2021-11-27 17:23:48
【问题描述】：

这些代码会给我地址，但不会给我电话号码

    import requests

    from bs4 import BeautifulSoup
            
    headers ={'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36'}
    r=requests.get('https://www.houzz.com/professionals/general-contractor') 
    soup=BeautifulSoup(r.content, 'html.parser')
    tra = soup.find_all('div',class_='hz-pro-search-result__info')
    for pro in tra:
       address=pro.find('span',class_='hz-pro-search-result__location-info__text').text
        try:
           phone=pro.select('div.hz-pro-search-result__right-info__contact-info > span > span').text
        except:
           phone=''
        print(address,phone)

从这里获取Phone Number的数据，页面的链接是https://www.houzz.com/professionals/general-contractor

<div class="hz-pro-search-result__right-info__contact-info"><span class="hz-pro-search-result__contact-info"><span class="icon-font icon-phone hz-pro-search-result__contact-info__icon" aria-hidden="true"></span>(800) 310-7154</span></div>

<span class="hz-pro-search-result__contact-info"><span class="icon-font icon-phone hz-pro-search-result__contact-info__icon" aria-hidden="true"></span>(800) 310-7154</span>

【问题讨论】：

您可能需要 selenium，因为我们只有在单击 View Phone Number 按钮后才能获得电话号码
我试过 selenium，click 电话号码并提供电话号码，但它不适合我
你也可以给那个代码吗？
他们什么都不给
不只是给出你尝试使用 selenium 的代码

标签： python selenium web-scraping beautifulsoup

【解决方案1】：

该网站需要您点击“查看电话号码”以显示电话号码。您可以使用 Selenium 来执行此操作。我也在使用 pandas，但如果你愿意，可以跳过它。

安装硒：

pip install selenium

Install Selenium Driver 适用于您的 Chrome 版本

安装熊猫：

pip install pandas

代码如下：

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
import time
import pandas as pd

# location of chromedriver.exe
driver = webdriver.Chrome("D:/chromedriver/94/chromedriver.exe")

# opening website
driver.get("https://www.houzz.com/professionals/general-contractor")

# waiting for the DIVs to load
WebDriverWait(driver, 60).until(EC.presence_of_element_located((By.XPATH, '//div[@class="hz-pro-search-result__right-info"]')))

# getting all the relevant DIVs
info_divs = driver.find_elements(By.XPATH,  '//div[@class="hz-pro-search-result__right-info"]')

house_details = {
    "address": [],
    "phone": []
}

for row in info_divs:
    try:
        address = row.find_element(By.CLASS_NAME, "hz-pro-search-result__right-info__full-address")
        phone = row.find_element(By.CLASS_NAME, "hz-pro-search-result__right-info__contact-info")
        phone.click()
        time.sleep(0.5)
        house_details['address'].append(address.text)
        house_details['phone'].append(phone.text)
    except Exception as ex:
        print(f"something went wrong. {ex}")
        
# saving to dataframe
house_df = pd.DataFrame.from_dict(house_details)
# exporting to .csv
house_df.to_csv('house_details.csv', index=False)

print(house_df)

【讨论】：