如何通过 Selenium 从网站上抓取产品名称？

Question

我正在尝试抓取此页面：https://redmart.com/fresh-produce/fresh-vegetables。但我面临的问题是它只返回一些元素。 我使用的代码如下：

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium import webdriver

# Start the WebDriver and load the page
wd = webdriver.Chrome(executable_path=r"C:\Chrome\chromedriver.exe")
wd.get('https://redmart.com/fresh-produce/fresh-vegetables')

# Wait for the dynamically loaded elements to show up
WebDriverWait(wd, 300).until(
EC.visibility_of_element_located((By.CLASS_NAME, "productDescriptionAndPrice")))

# And grab the page HTML source
html_page = wd.page_source
wd.quit()

# Now you can use html_page as you like
from bs4 import BeautifulSoup
soup = BeautifulSoup(html_page, 'lxml')
print(soup)

我需要使用 Selenium，因为页面是 JAVAscript 生成的，所以源代码没有用。如果您打开该页面，它有大约 60 行产品（总共大约 360 种产品）。运行此代码只会给我 6 行产品。停在黄洋葱上。

谢谢！

Answer 1

根据您的问题和网站https://redmart.com/fresh-produce/fresh-vegetables，Selenium 可以单独轻松地抓取所有产品名称。正如您所提到的，总共有大约 360 种产品，但只有 大约 35 种产品来自一个特定的类，我为您提供了一个解决方案如下：

• 代码块：

  from selenium import webdriver
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.support import expected_conditions as EC
  
  item_names = []
  options = webdriver.ChromeOptions() 
  options.add_argument("start-maximized")
  options.add_argument('disable-infobars')
  driver=webdriver.Chrome(chrome_options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
  driver.get("https://redmart.com/fresh-produce/fresh-vegetables")
  titles = WebDriverWait(driver, 5).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@class='productDescriptionAndPrice']//h4/a")))
  for title in titles:
      item_names.append(title.text)
  try:
      driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
      titles = WebDriverWait(driver, 5).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@class='productDescriptionAndPrice']//h4/a")))
      for title in titles:
      item_names.append(title.text)
  except:
      pass
  for item_name in item_names:
      print(item_name)
  driver.quit()
  

• 控制台输出：

  Eco Leaf Baby Spinach Fresh Vegetable
  Eco Leaf Kale Fresh Vegetable
  Sustenir Agriculture Almighty Arugula
  Sustenir Fresh Toscano Black Kale
  Sustenir Fresh Kinky Green Curly Kale
  ThyGrace Honey Cherry Tomato
  Australian Broccoli
  Sustenir Agriculture Italian Basil
  GIVVO Japanese Cucumbers
  YUVVO Red Onions
  Australian Cauliflower
  YUVVO Spring Onion
  GIVVO Old Ginger
  GIVVO Cherry Grape Tomatoes
  YUVVO Holland Potato
  ThyGrace Traffic Light Capsicum Bell Peppers
  GIVVO Whole Garlic
  GIVVO Celery
  Eco Leaf Baby Spinach Fresh Vegetable
  Eco Leaf Kale Fresh Vegetable
  Sustenir Agriculture Almighty Arugula
  Sustenir Fresh Toscano Black Kale
  Sustenir Fresh Kinky Green Curly Kale
  ThyGrace Honey Cherry Tomato
  Australian Broccoli
  Sustenir Agriculture Italian Basil
  GIVVO Japanese Cucumbers
  YUVVO Red Onions
  Australian Cauliflower
  YUVVO Spring Onion
  GIVVO Old Ginger
  GIVVO Cherry Grape Tomatoes
  YUVVO Holland Potato
  ThyGrace Traffic Light Capsicum Bell Peppers
  GIVVO Whole Garlic
  GIVVO Celery
  

注意：您可以构建更健壮的 XPATH 或 CSS-SELECTOR 以包含更多产品并提取相关的产品名称。

Answer 2

这里有一些 Java 代码可以运行。测试等待 30 个元素。

@Test
public void test1() {
    List<WebElement> found = new WebDriverWait(driver, 300).until(wd -> {
        List<WebElement> elements = driver.findElements(By.className("productDescriptionAndPrice"));
        if(elements.size() > 30)
            return elements ;
        ((JavascriptExecutor) driver).executeScript("window.scrollTo(0, document.body.offsetHeight)");
        return null;
    });
    for (WebElement e : found) {
        System.out.println(e.getText());
    }
}

Answer 3

您好 DebanjanB，感谢您的帮助。我一整天都在尝试这个。真正的问题在于将完整的产品列表放入源代码中。如果一切都在源头中，我认为可以将其提取出来。我相信当你向下滚动时源会发生变化，也许这就是为什么我们都只能提取 36 个项目。

考虑到这一点，我的暂定解决方案如下。它并不完美，因为我必须稍后进行进一步处理以删除重复项。如果您有其他想法或可以进一步优化，我将非常感激。

一般的想法是向下滚动，抓取源代码并添加 1 大长的重叠源代码。对于 360 产品页面，我有 1400 多种产品这样做，这就是为什么我说这是一个糟糕的解决方案。

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
import time
from bs4 import BeautifulSoup

# Start the WebDriver and load the page
wd = webdriver.Chrome(executable_path=r"C:\Chrome\chromedriver.exe")
wd.delete_all_cookies()
wd.set_page_load_timeout(30)

wd.get('https://redmart.com/fresh-produce/fresh-vegetables#toggle=all')
time.sleep(5)

html_page = wd.page_source
soup = BeautifulSoup(html_page, 'lxml')

while True:
    wd.execute_script("window.scrollTo(0, document.body.scrollHeight)")
    time.sleep(3)
    html_page = wd.page_source
    soup2 = BeautifulSoup(html_page, 'lxml')

    for element in soup2.body:
         soup.body.append(element) 
    time.sleep(2)

    #break condition
    new_height = wd.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break
    last_height = new_height
wd.quit()

results = soup.findAll('div', attrs='class':'productDescriptionAndPrice'})
len(results)
results[0] # this tally with the first product
results[-1] # this tallies with the last

说实话，对这个解决方案非常失望。谢谢，请随时让他们来，让他们来！