Python 爬虫仅在前几个 div 中找到所需的标签

Question

我正在尝试从购物网站 (https://www.grailed.com/shop/EkpEBRw4rw) 上抓取一些图片，但我遇到了一些问题，因为列表会随着您滚动而更新。我正在尝试在下面的 HTML 标记中获取图像源：

现在我一直在使用的代码如下所示：

from bs4 import BeautifulSoup
from selenium import webdriver

url = 'https://www.grailed.com/shop/EkpEBRw4rw'
driver = webdriver.Chrome(executable_path='chromedriver.exe')
driver.get(url)
soup = BeautifulSoup(driver.page_source, 'html.parser')

listing = soup.select('.listing-cover-photo ')
for item in listing:
    print(item.select('img'))

问题在于，虽然它确实找到了每个列表的标签，但它只能找到前 6 个列表的标签。我的代码的输出如下所示：

输出：

[<img alt="Off-White Off White Caravaggio Hoodie" src="https://process.fs.grailed.com/AJdAgnqCST4iPtnUxiGtTz/cache=expiry:max/rotate=deg:exif/resize=width:480,height:640,fit:crop/output=format:webp,quality:70/compress/https://cdn.fs.grailed.com/api/file/yX8vvvBsTaugadX0jssT"/>]
(...a few more of these...)
[<img alt="Off-White Off-White Arrows Hoodie Black" src="https://process.fs.grailed.com/AJdAgnqCST4iPtnUxiGtTz/cache=expiry:max/rotate=deg:exif/resize=width:480,height:640,fit:crop/output=format:webp,quality:70/compress/https://cdn.fs.grailed.com/api/file/9CMvJoQIRaqgtK0u9ov0"/>]
[]
[]
[]
[]
(...many more empty lists...)

即使遍历侧面的所有页面（将 '?page = n' 添加到 url）并且仅显示每个页面的前 6 个条目，这种情况仍然存在。

Answer 1

要使用Selenium 在shopping site 中抓取<img> 标记的src 属性，您需要为visibility_of_all_elements_located() 诱导WebDriverWait，您可以使用以下解决方案：

• 代码块：

  from selenium import webdriver
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC
  
  options = webdriver.ChromeOptions()
  options.add_argument('start-maximized')
  options.add_argument('disable-infobars')
  options.add_argument('--disable-extensions')
  driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
  driver.get('https://www.grailed.com/shop/EkpEBRw4rw')
  print([my_image.get_attribute("src") for my_image in WebDriverWait(driver, 5).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.listing-cover-photo>img")))])
  

• 控制台输出：

  ['https://process.fs.grailed.com/AJdAgnqCST4iPtnUxiGtTz/cache=expiry:max/rotate=deg:exif/resize=width:240,height:320,fit:crop/output=format:webp,quality:70/compress/https://cdn.fs.grailed.com/api/file/yX8vvvBsTaugadX0jssT', 'https://process.fs.grailed.com/AJdAgnqCST4iPtnUxiGtTz/cache=expiry:max/rotate=deg:exif/resize=width:240,height:320,fit:crop/output=format:webp,quality:70/compress/https://cdn.fs.grailed.com/api/file/YjiErjJNQrarKGDuGr3S', 'https://process.fs.grailed.com/AJdAgnqCST4iPtnUxiGtTz/cache=expiry:max/rotate=deg:exif/resize=width:240,height:320,fit:crop/output=format:webp,quality:70/compress/https://cdn.fs.grailed.com/api/file/G9CwIli8QUW3uGgZeirk', 'https://process.fs.grailed.com/AJdAgnqCST4iPtnUxiGtTz/cache=expiry:max/rotate=deg:exif/resize=width:240,height:320,fit:crop/output=format:webp,quality:70/compress/https://cdn.fs.grailed.com/api/file/Ta9DAxg4SeKAT6kBLyJo', 'https://process.fs.grailed.com/AJdAgnqCST4iPtnUxiGtTz/cache=expiry:max/rotate=deg:exif/resize=width:240,height:320,fit:crop/output=format:webp,quality:70/compress/https://cdn.fs.grailed.com/api/file/QglmTKyTxu31PeDFWFnw', 'https://process.fs.grailed.com/AJdAgnqCST4iPtnUxiGtTz/cache=expiry:max/rotate=deg:exif/resize=width:240,height:320,fit:crop/output=format:webp,quality:70/compress/https://cdn.fs.grailed.com/api/file/9CMvJoQIRaqgtK0u9ov0', 'https://process.fs.grailed.com/AJdAgnqCST4iPtnUxiGtTz/cache=expiry:max/rotate=deg:exif/resize=width:240,height:320,fit:crop/output=format:webp,quality:70/compress/https://cdn.fs.grailed.com/api/file/MCJY9cSQsiSU4TlSTcD7', 'https://process.fs.grailed.com/AJdAgnqCST4iPtnUxiGtTz/cache=expiry:max/rotate=deg:exif/resize=width:240,height:320,fit:crop/output=format:webp,quality:70/compress/https://cdn.fs.grailed.com/api/file/L4NHu1ByT3Kwn8dRsdBX']