使用 python 和 selenium 从网页获取产品详细信息

Question

我需要从网页中抓取一些数据。我正在使用 selenium 和 Python，但无法获得所需的结果。如果有人可以建议我该做什么或如何做我可以更改代码以获得所需的结果，这将非常有帮助。

我需要按以下顺序获取结果位置、产品名称、URL 和公司名称。

下面的脚本能够获取所有标题名称，但带有额外的“\n”字符，无法获取其他详细信息。有人可以帮我解决这个问题或就此提出建议吗？

from bs4 import BeautifulSoup
from selenium import webdriver 
from selenium.webdriver.common.by import By 
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC 
from selenium.common.exceptions import TimeoutException

option = webdriver.ChromeOptions()
option.add_argument(" — incognito")
#browser = webdriver.Chrome(executable_path='/usr/local/bin/chromedriver/', chrome_options=option)
browser = webdriver.Chrome(executable_path='/users/user_1566/downloads/chrome_driver/chromedriver', chrome_options=option)

browser.get('https://www.google.com/search?q=samsung+note10')
#items = len(browser.find_elements_by_class_name("cu-container"))
#items = len(browser.find_elements_by_class_name("mnr-c pla-unit"))
#print(items)
timeout = 20
try:
    WebDriverWait(browser, timeout).until(EC.visibility_of_element_located((By.XPATH, "//div[@class='top-pla-group-inner']")))
except TimeoutException:
    print('Timed out waiting for page to load')
    #browser.quit()

titles_element = browser.find_elements_by_xpath("//div[@class='mnr-c pla-unit']")
# use list comprehension to get the actual repo titles and not the selenium objects.
titles = [x.text for x in titles_element]
# print out all the titles.
print('titles:')
print(titles, '\n')

language_element = browser.find_elements_by_xpath("//a[@class=''plantl pla-unit-single-clickable-target clickable-card']")
print(language_element)
# same concept as for list-comprehension above.
languages = [x.text for x in language_element]
print("languages:")
print(languages, "\n")


for title, language in zip(titles, languages):
    print("RepoName : Language")
    print(title + ": " + language, "\n")        

                                                                     

来自 URL https://www.google.com/search?q=samsung+note10 的预期输出

    Position  Company_Name     Product_Name                                 URL 

    1         CHECK24          Samsung Galaxy Note10 256 GB 
                               glow mit Vertrag + Allnet Flat bei                                                                     

Answer 1

试试

details=browser.find_elements_by_class_name("pymv4e")
for i in details:
    print(str(i))

之后，您可以根据需要显示