如何处理延迟加载？答案

【问题标题】：How to handle lazy loading?如何处理延迟加载？
【发布时间】：2020-03-11 21:49:50
【问题描述】：

在我的程序的这一部分，我试图获取网页上图像的所有链接。但是，这些图片是延迟加载的。即使它不像：向下滚动时出现图片，但图片已经在这里。每个页面上有 30 个产品。向下滚动页面也没有什么魔力。我该如何处理？

from selenium import webdriver
import os 
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
from selenium.webdriver.common.action_chains import ActionChains


url= 'https://www.bookdepository.com/category/2/Art-Photography/browse/viewmode/all'

driver = webdriver.Chrome('blabla')

driver.get(url)


a = 1
while a != 100:
    try:                                                                                                                                                                   
        link_picture = WebDriverWait(driver, 2).until(EC.visibility_of_element_located((By.XPATH, "/html/body/div[4]/div[5]/div[2]/div[3]/div/div/div/div/div["+ str(a) +"]/div[1]/a/img"))).get_attribute('src')                                                                                                    
        print(link_picture)
    except:
        print("\nno products left")
        #e = a - 1
        #print(a)
        #print(e)
        break
    a = a + 1

【问题讨论】：

你想获取每件商品的产品图片吗？

标签： python selenium web-scraping web-crawler lazy-loading

【解决方案1】：

实际上页面是通过JavaScript加载的，一旦页面加载，internally就会在host本身上呈现，因此Lazy-Load将是通过Implicit, Explicit, & Fluent处理selenium的一个非常糟糕的方法请在Selenium WebDriver 中等待，这将花费大量时间。

我们可以使用 requests 和 bs4 以一种棘手的方式。我们将收集imgIDs#，然后我们会将它们匹配为通常作为网站上的那个。

注意：我已经手动检查过，我可以为您确认该站点的CloudFront 的token id 是static，即d1w7fb2mkkr3kw。

from bs4 import BeautifulSoup
import requests

r = requests.get(
    "https://www.bookdepository.com/category/2/Art-Photography/browse/viewmode/all")

soup = BeautifulSoup(r.text, 'html.parser')

url = "https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/"

for item in soup.findAll("meta", itemprop="isbn"):
    item = item.get("content")
    print(
        f"{url}{item[:4]}/{item[4:8]}/{item}.jpg")

输出：

https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9781/4087/9781408708989.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9781/5098/9781509853311.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9780/7522/9780752265629.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9780/1410/9780141014081.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9781/2500/9781250038821.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9780/1410/9780141035796.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9781/8499/9781849941679.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9780/0995/9780099539551.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9781/4087/9781408711705.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9788/8837/9788883701153.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9780/7475/9780747568766.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9781/8609/9781860969423.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9781/4711/9781471157790.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9781/5098/9781509829477.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9780/7515/9780751535662.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9780/5005/9780500513606.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9781/4088/9781408890769.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9780/8771/9780877180128.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9788/8837/9788883705601.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9781/4722/9781472200341.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9781/8478/9781847807717.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9781/8609/9781860969430.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9780/3409/9780340936177.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9780/4862/9780486254500.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9780/7160/9780716022237.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9780/0994/9780099457046.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9781/4521/9781452106557.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9780/2411/9780241184837.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9781/7423/9781742372389.jpg
https://d1w7fb2mkkr3kw.cloudfront.net/assets/images/book/mid/9781/8499/9781849942850.jpg

更新：

import requests
from bs4 import BeautifulSoup

r = requests.get(
    "https://www.bookdepository.com/category/2/Art-Photography/browse/viewmode/all")
soup = BeautifulSoup(r.text, 'html.parser')

for item in soup.findAll("img", class_="lazy"):
    print(item.get("data-lazy"))

【讨论】：