网络从亚马逊搜索中抓取标题到终端答案

【问题标题】：Web scraping out Titles from Amazon search into termina网络从亚马逊搜索中抓取标题到终端
【发布时间】：2021-03-04 07:12:35
【问题描述】：

我正在尝试修改我的代码，以便将亚马逊搜索中的标题打印到终端上。我该怎么做？经过大量尝试错误后，我最近才真正将其从亚马逊打印到终端中。

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
import time

PATH = "/Users/fuadhafiz/Documents/Python Projects/Selenium Automation /Web Scraping (1)/chromedriver"
driver = webdriver.Chrome(PATH)
driver.get("https://www.amazon.co.uk/")

search = driver.find_element_by_id("twotabsearchtextbox")
search.send_keys("usb c to hdmi")
search.send_keys(Keys.RETURN)

try:
    main = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, ".s-main-slot")))
    print(main.text)
except TimeoutException:
    print("Timeout")

print(main)

driver.quit()

到目前为止，它会将整个页面打印到终端上。这是打印到终端的结果中的一小部分：

USB C to HDMI
Shop BENFEI 
USB C to HDMI Cable(4K@60Hz), Benfei USB Type-C to HDMI Cable [Thunderbolt 3] Compatible for MacBook Pro 2019/2018/2017, Samsung Galaxy S9/S8, Surface Book 2, Dell XPS 13/15 and More - 1.8M, Grey
268
USB C to HDMI Adapter 4K@60Hz, BENFEI USB Type-C to HDMI Adapter [Thunderbolt 3 Compatible] for MacBook Pro 2018/2017, MacBook Air/iPad Pro 2018, Samsung Galaxy S10/S9, Surface Book 2 and More - Slim
133
USB C Hub, BENFEI 5 in 1 Type C to HDMI 4K Adapter with 2 USB Ports, SD/Micro SD Card Reader, for MacBook Pro 2019/2018/2017, iPad Pro 2019/2018, Pixelbook, XPS, and More
142
Sponsored
Sponsored
iVANKY USB C to HDMI Cable, [2020 Updated Version] 4K@60Hz Nylon Braided Cable, Type C [Thunderbolt 3 Compatible] to HDMI, for MacBook Pro/Air, iPad, Surface Book, Samsung Galaxy and More - 6.6ft, 2M
352
£14
99
£15.99

【问题讨论】：

标签： python python-3.x selenium web-scraping automation

【解决方案1】：

为什么不用CSS 选择器选择所有结果并打印出来？

方法如下：

from selenium import webdriver
from selenium.webdriver.common.keys import Keys

from selenium.webdriver.chrome.options import Options

options = Options()
options.headless = False
driver = webdriver.Chrome(options=options)

driver.get("https://www.amazon.co.uk/")

search = driver.find_element_by_id("twotabsearchtextbox")
search.send_keys("usb c to hdmi")
search.send_keys(Keys.RETURN)

search_results = driver.find_elements_by_css_selector('h1:last-child, h2:last-child, h3:last-child, h4:last-child')

for result in search_results[1:]:
    print(result.text)

driver.quit()

输出：

iVANKY USB C to HDMI Cable, [2020 Updated Version] 4K@60Hz Nylon Braided Cable, Type C [Thunderbolt 3 Compatible] to HDMI, for MacBook Pro/Air, iPad, Surface Book, Samsung Galaxy and More - 6.6ft, 2M
AmazonBasics Premium Aluminium USB-C to HDMI Cable Adapter (Thunderbolt 3 Compatible) 4K@60Hz, 0.9 metres
USB C to HDMI Adapter, Type c to HDMI 4K Adapter (Thunderbolt 3 Compatible) with Video Audio Output for MacBook Pro 2018/2017/2016, Samsung Note 9/S9/Note 8/S8, Huawei Mate 20 and More (Black)
USB C to HDMI Cable 4K, uni【2020 New Updated】USB Type C to HDMI Cable(Thunderbolt 3 Compatible), Compatible with iPad Pro 2020, MacBook Pro, Samsung S20, Huawei Mate40, XPS and More - 6ft/1.8m
USB C to HDMI Cable【4K@60Hz High Refresh Rate】uni Type-C [Thunderbolt 3] to HDMI Cable Compatible with MacBook Pro, MacBook Air, iPad Pro, Surface Book 2, Galaxy S20, Dell XPS and More - 0.9m/3ft
QGeeM USB C to HDMI Cable Adapter 6ft 4K@30Hz, USB Type C to HDMI Cable Thunderbolt 3 Compatible with MacBook Pro 2018 IPad pro,Samsung S9 S10,Surface Book 2,Dell XPS 13/15,Pixelbook More
USB C to Dual HDMI Adapter, Thunderbolt 3 USB Type C to 2 HDMI Monitor Adapter for Surface Book 2/Go/Pro 7,Dell XPS 13/15, Lenovo Yoga 720/900/910/920,Macbook Pro Air, iPad Pro and more
USB C to HDMI Adapter 4K for MacBook Pro, HDMI Adapter Hub Dock with USB 3.0 Port, Type-C PD Charging Port Compatible w/Nintendo Switch/MacBook Air 2018/Chromebook Pixel/Dell XPS13 - Space Grey
and so on ...

【讨论】：

【解决方案2】：

从搜索结果中获取每个项目的标题。诱导WebDriverWait() 并等待presence_of_all_elements_located() 并关注css selector。

allprodtitle =WebDriverWait(driver,10).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR,".s-main-slot h2>a>span")))
for item in allprodtitle:
    print("Title: " + item.get_attribute("innerHTML"))

控制台输出：

Title: iVANKY USB C to HDMI Cable, [2020 Updated Version] 4K@60Hz Nylon Braided Cable, Type C [Thunderbolt 3 Compatible] to HDMI, for MacBook Pro/Air, iPad, Surface Book, Samsung Galaxy and More - 6.6ft, 2M
Title: AmazonBasics Premium Aluminium USB-C to HDMI Cable Adapter (Thunderbolt 3 Compatible) 4K@60Hz, 0.9 metres
Title: USB C to HDMI Adapter, Type c to HDMI 4K Adapter (Thunderbolt 3 Compatible) with Video Audio Output for MacBook Pro 2018/2017/2016, Samsung Note 9/S9/Note 8/S8, Huawei Mate 20 and More (Black)
Title: USB C to HDMI Cable 4K, uni【2020 New Updated】USB Type C to HDMI Cable(Thunderbolt 3 Compatible), Compatible with iPad Pro 2020, MacBook Pro, Samsung S20, Huawei Mate40, XPS and More - 6ft/1.8m
Title: USB C to HDMI Cable【4K@60Hz High Refresh Rate】uni Type-C [Thunderbolt 3] to HDMI Cable Compatible with MacBook Pro, MacBook Air, iPad Pro, Surface Book 2, Galaxy S20, Dell XPS and More - 0.9m/3ft
Title: QGeeM USB C to HDMI Cable Adapter 6ft 4K@30Hz, USB Type C to HDMI Cable Thunderbolt 3 Compatible with MacBook Pro 2018 IPad pro,Samsung S9 S10,Surface Book 2,Dell XPS 13/15,Pixelbook More
Title: USB C to Dual HDMI Adapter, Thunderbolt 3 USB Type C to 2 HDMI Monitor Adapter for Surface Book 2/Go/Pro 7,Dell XPS 13/15, Lenovo Yoga 720/900/910/920,Macbook Pro Air, iPad Pro and more
Title: USB C to HDMI Adapter 4K for MacBook Pro, HDMI Adapter Hub Dock with USB 3.0 Port, Type-C PD Charging Port Compatible w/Nintendo Switch/MacBook Air 2018/Chromebook Pixel/Dell XPS13 - Space Grey
Title: USB C Hub for iPad Pro 2018 2019 2020, Baseus 6 in 1 USB C to 4K HDMI Adapter, PD 100W Power Delivery, USB3.0, SD/TF Card Reader, 3.5mm Headphone Jack Compatible with MacBook Pro/iPad Pro - Grey
Title: USB C Hub, Aceele 10 in 1 USB Ultra Adapter with 4K HDMI, Power Delivery, 6 USB3.0 Ports, SD/TF Card Reader for MacBook Pro/Air, iPad Pro, Surface Pro 7, Samsung S20+ and More Type C Devices
Title: USB C to HDMI Adapter, Type c to HDMI 4K Adapter (Thunderbolt 3 Compatible) with Video Audio Output for MacBook Pro 2018/2017/2016, Samsung Note 9/S9/Note 8/S8, Huawei Mate 20 and More (Black)
Title: USB C to HDMI Cable for Home Office 6ft, Anker Type C to HDMI Adapter Cable 4K 60Hz for MacBook Pro 2020, iPad Pro 2020, Samsung Galaxy S20/ S10, Dell XPS 13/15, and More [Thunderbolt 3 Compatible]
Title: USB C to HDMI Adapter, Syntech Thunderbolt 3 to HDMI Adapter Compatible with MacBook Pro 2019/2018/2017, MacBook Air, iPad Pro 2020, iPad Air 2020, Samsung Galaxy S10/S9,Dell XPS 13/15 and More
Title: Apple USB-C Digital AV Multi-port Adaptor - White (Latest Model)
Title: Syntech USB C to HDMI Cable (4K@60Hz),Thunderbolt 3 to HDMI Cable Compatible with MacBook Pro 2019/2018/2017, MacBook Air, iPad Pro 2020,Samsung Galaxy S10/S9, Dell XPS and More - 6ft/1.8m
Title: USB C to HDMI Cable(4K@60Hz), Bauihr Type-C to HDMI Cable (Thunderbolt 3) Compatible for iPad Pro 2018, MacBook Pro/Air, Galaxy S10/Note 8/S9/S8, Huawei P20/Pro/P30/Pro, Mate20/30/30 pro - 6ft/1.8m
Title: USB C to HDMI Adapter 4K for MacBook Pro, HDMI Adapter Hub Dock with USB 3.0 Port, Type-C PD Charging Port Compatible w/Nintendo Switch/MacBook Air 2018/Chromebook Pixel/Dell XPS13 - Space Grey
Title: USB C to HDMI Adapter, Syntech Thunderbolt 3 to HDMI Adapter Compatible with MacBook Pro 2019/2018/2017, MacBook Air, iPad Pro 2020, iPad Air 2020, Samsung Galaxy S10/S9,Dell XPS 13/15 and More
Title: COVVY USB-C to HDMI Adapter USB C Charging Port Converter Adapter USB C Hub Compatible with Chromebook/Samsung Galaxy S8/S9, MacBook Air, iPad Pro 2020,Dell XPS 13/15 and More (Silver)
Title: USB C to HDMI Cable,CHOETECH 4K@60Hz (Thunderbolt 3 Compatible) HDMI Cable Compatible with Apple iPad Pro,2018/2017/2016 MacBookPro,iMac,MacBook,ChromeBookl,Note10/S10/Note 9/S9,Huawei P30/P20/Mate20
Title: USB C to HDMI Adapter, 3 in 1 Multiport USB Type C to 4K HDMI, USB3.0 and USB C Power Delivery Port Converter Compatible with MacBook/Chromebook Pixel/Dell XPS13/Samsung S10/S10+ and More
Title: USB Type C(Thunderbolt 3) to HDMI 4K UHD 1.8M Cable, Benfei USB 3.1 (USB-C) to HDMI Adapter Male to Male Gold-Plated Cord
Title: USB C to HDMI Cable [3m/10Ft], JSAUX 4K@60Hz Type C to HDMI Cord Thunderbolt 3 Compatible for Apple iPad Pro 2019/2018, MacBook Pro/Air, iMac, Samsung S20/S10/S9/S8 Plus/Note 10, Huawei P30/P20 - Red
Title: Belkin USB-C to HDMI Adapter + Charge (Supports 4K UHD Video, Passthrough Power up to 60W for Connected Devices) MacBook Pro HDMI Adapter
Title: USB C to HDMI Cable for Home Office 6ft, Anker Type C to HDMI Adapter Cable 4K 60Hz for MacBook Pro 2020, iPad Pro 2020, Samsung Galaxy S20/ S10, Dell XPS 13/15, and More [Thunderbolt 3 Compatible]
Title: USB C to HDMI Adapter, Thunderbolt 3 Type C to HDMI Adapter 4K for MacBook Pro 2019/2018/2017/2016, MacBook Air, Samsung Note 9/S9/Note 8/S8, Huawei P30/Mate 30
Title: QGeeM USB C to HDMI Cable Adapter 6ft 4K@30Hz, USB Type C to HDMI Cable Thunderbolt 3 Compatible with MacBook Pro 2018 IPad pro,Samsung S9 S10,Surface Book 2,Dell XPS 13/15,Pixelbook More
Title: USB C to HDMI Adapter, Aluminum Type C/Thunderbolt 3 to HDMI 4K Adapter for MacBook Pro 2017-2019, MacBook Air, Samsung Note 9/S9/Note 8/S8, Huawei Mate 20 and More (Black)
Title: USB C to HDMI Cable, uni USB Type C to HDMI Cable[Thunderbolt 3 Compatible] for Home Office, 4K, Compatible for iPad Air 4, iPad Pro 2020, MacBook Pro, Galaxy S20, XPS and More -6ft/1.8m
Title: USB C to HDMI Cable(4K@60Hz), Benfei USB Type-C to HDMI Cable [Thunderbolt 3] Compatible for MacBook Pro 2019/2018/2017, Samsung Galaxy S9/S8, Surface Book 2, Dell XPS 13/15 and More - 1.8M, Grey
Title: YEHUA USB C/Micro USB to HDMI Cable 2-in-1 Support Netflix, 6.6ft MHL to HDMI Adapter 1080P HD HDTV TikTok Mirroring for Android All Smartphone to TV/Projector/Monitor.

【讨论】：