用漂亮的汤刮不充分？答案

【问题标题】：Scraping using beautiful soup not working fully?用漂亮的汤刮不充分？
【发布时间】：2021-09-10 13:41:53
【问题描述】：

我试图使用 Python 上的 BeautifulSoup 从网站“https://www.geappliances.com/ge-appliances/kitchen/ranges/”中抓取一些数据，该网站有一些产品。

import unittest, time, random
from selenium import webdriver
from webdriver_manager.firefox import GeckoDriverManager
from bs4 import BeautifulSoup
import pandas as pd

links = []
browser = webdriver.Firefox(executable_path="C:\\Users\\drivers\\geckodriver\\win64\\v0.29.1\\geckodriver.exe")
browser.get("https://www.geappliances.com/ge-appliances/kitchen/ranges/")
content = browser.page_source
soup = BeautifulSoup(content, "html.parser")

for main in soup.findAll('li', attrs = {'class' : 'product'}):    
    name=main.find('a', href=True)
    if (name != ''):
        links.append((name.get('href')).strip())
   
print("Got links : ", len(links))
exit()

在输出中我得到：- 获得链接：0

我打印了汤，发现汤里没有这部分。我一直试图解决这个问题无济于事。难道我做错了什么？任何建议表示赞赏。谢谢。

【问题讨论】：

标签： python beautifulsoup

【解决方案1】：

研究网页的来源。检查你的findAll函数。

【讨论】：

请添加更多详细信息以扩展您的答案，例如工作代码或文档引用。

【解决方案2】：

您应该等到页面加载完毕。使用time.sleep() 暂停执行一段时间。

你可以这样试试。

from bs4 import BeautifulSoup
from selenium import webdriver
import time

url = 'https://www.geappliances.com/ge-appliances/kitchen/ranges/'
driver = webdriver.Chrome("chromedriver.exe")
driver.get(url)
time.sleep(5)

soup = BeautifulSoup(driver.page_source, 'lxml')
u = soup.find('ul', class_='productGrid')
for item in u.find_all('li', class_='product'):
    print(item.find('a')['href'])

driver.close()

/appliance/GE-Profile-30-Smart-Slide-In-Front-Control-Induction-Fingerprint-Resistant-Range-with-In-Oven-Camera-PHS93XYPFS
/appliance/GE-Profile-30-Electric-Pizza-Oven-PS96PZRSS
/appliance/GE-Profile-30-Smart-Slide-In-Front-Control-Gas-Double-Oven-Convection-Fingerprint-Resistant-Range-PGS960YPFS
/appliance/GE-Profile-30-Smart-Dual-Fuel-Slide-In-Front-Control-Fingerprint-Resistant-Range-P2S930YPFS
/appliance/GE-Profile-30-Smart-Slide-In-Electric-Double-Oven-Convection-Fingerprint-Resistant-Range-PS960YPFS
/appliance/GE-Profile-30-Smart-Slide-In-Front-Control-Induction-and-Convection-Range-with-No-Preheat-Air-Fry-PHS930BPTS
/appliance/GE-Profile-30-Smart-Slide-In-Fingerprint-Resistant-Front-Control-Induction-and-Convection-Range-with-No-Preheat-Air-Fry-PHS930YPFS
/appliance/GE-Profile-30-Smart-Slide-In-Front-Control-Gas-Range-with-No-Preheat-Air-Fry-PGS930BPTS
/appliance/GE-Profile-30-Smart-Slide-In-Front-Control-Gas-Fingerprint-Resistant-Range-with-No-Preheat-Air-Fry-PGS930YPFS
/appliance/GE-Profile-30-Smart-Slide-In-Electric-Convection-Range-with-No-Preheat-Air-Fry-PSS93BPTS
/appliance/GE-Profile-30-Smart-Slide-In-Electric-Convection-Fingerprint-Resistant-Range-with-No-Preheat-Air-Fry-PSS93YPFS
/appliance/GE-30-Slide-In-Front-Control-Gas-Double-Oven-Range-JGSS86SPSS

【讨论】：