使用 BeautifulSoup 进行 Selenium 滚动和抓取会产生重复的结果

Question

我有这个脚本可以从 Instagram 下载图片。我遇到的唯一问题是，当 Selenium 开始向下滚动到网页底部时，BeautifulSoup 在循环请求后开始抓取相同的 img src 链接。

虽然它会继续向下滚动并下载图片，但在完成之后，我最终会有 2 或 3 个重复。所以我的问题是有没有办法防止这种重复发生？

import requests
from bs4 import BeautifulSoup
import selenium.webdriver as webdriver


url = ('https://www.instagram.com/kitties')
driver = webdriver.Firefox()
driver.get(url)

scroll_delay = 0.5
last_height = driver.execute_script("return document.body.scrollHeight")
counter = 0

print('[+] Downloading:\n')

def screens(get_name):
    with open("/home/cha0zz/Desktop/photos/img_{}.jpg".format(get_name), 'wb') as f:
        r = requests.get(img_url)
        f.write(r.content)

while True:

    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    time.sleep(scroll_delay)
    new_height = driver.execute_script("return document.body.scrollHeight")

    soup = BeautifulSoup(driver.page_source, 'lxml')
    imgs = soup.find_all('img', class_='_2di5p')
    for img in imgs:
        img_url = img["src"]
        print('=> [+] img_{}'.format(counter))
        screens(counter)
        counter = counter + 1

    if new_height == last_height:
        break
    last_height = new_height

更新： 所以我把这部分代码放在while True之外，让selenium首先加载整个页面，希望bs4能刮掉所有的图像。它只工作到 30 号，然后停止。

soup = BeautifulSoup(driver.page_source, 'lxml')
imgs = soup.find_all('img', class_='_2di5p')
for img in imgs:
    #tn = datetime.now().strftime('%H:%M:%S')
    img_url = img["src"]
    print('=> [+] img_{}'.format(counter))
    screens(counter)
    counter = counter + 1

Answer 1

我会使用 os 库来检查文件是否已经存在

import os


def screens(get_name):
    with open("/home/cha0zz/Desktop/photos/img_{}.jpg".format(get_name), 'wb') as f:
        if os.path.isfile(path/to/the/file):      #checks file exists. Gives false on directory
    # or if os.path.exists(path/to/the/file): #checks file/directory exists
            pass
        else:
            r = requests.get(img_url)
            f.write(r.content)

*我可能弄乱了 if 和 with 语句的顺序

Answer 2

它在您的脚本的第二个版本中仅加载 30 个的原因是因为其余元素已从页面 DOM 中删除，并且不再是 BeautifulSoup 看到的源的一部分。解决方案是继续做你第一次做的事情，但在遍历列表并调用screens()之前删除所有重复的元素。你可以这样做using sets，如下所示，但我不确定这是否是绝对最有效的方法：

import requests
import selenium.webdriver as webdriver
import time

driver = webdriver.Firefox()

url = ('https://www.instagram.com/cats/?hl=en')
driver.get(url)

scroll_delay = 3
last_height = driver.execute_script("return document.body.scrollHeight")
counter = 0

print('[+] Downloading:\n')

def screens(get_name):
    with open("test_images/img_{}.jpg".format(get_name), 'wb') as f:
        r = requests.get(img_url)
        f.write(r.content)

old_imgs = set()

while True:

    imgs = driver.find_elements_by_class_name('_2di5p')

    imgs_dedupe = set(imgs) - set(old_imgs)

    for img in imgs_dedupe:
        img_url = img.get_attribute("src")
        print('=> [+] img_{}'.format(counter))
        screens(counter)
        counter = counter + 1

    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    time.sleep(scroll_delay)
    new_height = driver.execute_script("return document.body.scrollHeight")

    old_imgs = imgs

    if new_height == last_height:
        break
    last_height = new_height

driver.quit()

如您所见，我使用不同的页面进行测试，其中包含 420 张猫的图片。结果是 420 张图片，即该帐户上的帖子数，其中没有重复。