Scraper 不拾取图片 URL Beautiful Soup

Question

我的抓取工具没有始终如一地拾取页面上的图片 URL。有时会，大多数时候不会。当它没有获取 URL 时，这就是我在 CSV 中得到的内容：data:

我看不出有什么问题，谁能帮忙？

我已经尝试增加睡眠时间以确保页面上的所有元素都已加载，我在同一个网站上有其他页面，这是同样的事情，有时它可以工作，有时它不会。

我应该使用不同的方法将元素拾取到 session_image = session_soup.img['src']吗？

我也多次用这种方法爬过其他网站，从来没有出现过这个问题。是否与这个特定网站有关？

我的代码：

from selenium import webdriver
from bs4 import BeautifulSoup as soup
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait as browser_wait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.options import Options
from selenium.common.exceptions import ElementClickInterceptedException
import time
import re
import csv

# initialize the chrome browser
browser = webdriver.Chrome(executable_path=r'./chromedriver')
browser.implicitly_wait(20)

# URL
class_pass_url = 'https://www.classpass.com'

# Create file and writes the first row, added encoding type as write was giving errors
f = open('ClassPass.csv', 'w', encoding='utf-8')
headers = 'IMAGE URL\n'
f.write(headers)

# classpass results page
page = "https://classpass.com/studios/sum-yoga-london"

browser.get(page)

# Browser waits

#browser_wait(browser, 10).until(EC.visibility_of_element_located((By.CLASS_NAME, "line")))
time.sleep(4)

# Scrolls to bottom of page to reveal all classes
# browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")

studio_page_source = browser.page_source
studio_soup = soup(studio_page_source, "html.parser")

try:
    studio_name = studio_soup.h2.text
except (AttributeError, TypeError,) as e:
    pass

sessions = studio_soup.find_all('h3', {'class': '_4Fnd4DwToJFbbU5jAfqSv'})

for session in sessions:
    twitter, facebook, instagram, session_website, telnumber, session_description = '', '', '', '', '', ''
    session_link = class_pass_url + session.a['href']
    browser.get(session_link)

    #browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    #browser.execute_script("window.scrollTo(0,0);")
    #time.sleep(2)

    browser_wait(browser, 10).until(EC.presence_of_element_located((By.CLASS_NAME, '_1ruz3nW6mOnylv99BOA_tm')))

    # parses individual class page
    session_page_source = browser.page_source
    session_soup = soup(session_page_source, "html.parser")


    try:
        session_image = session_soup.img['src']
    except (AttributeError, TypeError,) as e:
        pass

    print(session_image)

    f.write(

        session_image +

        "\n")
    

Answer 1

要获取图像，您可以在代码中进行以下更改：

session_image = session_soup.find('meta', {'property': "og:image"})
session_image = session_image.get('content')