硒没有加载整个页面进行抓取答案

【问题标题】：selenium not loading whole page for scraping硒没有加载整个页面进行抓取
【发布时间】：2022-01-26 06:17:06
【问题描述】：

from selenium import webdriver  # pip install selenium
import time
import re
# importing the module
from pytube import YouTube

def dnload(lnk):
    # where to save
    SAVE_PATH = "C:/Python/amazingrussian" #to_do
    # link of the video to be downloaded
    link=lnk
    yt = YouTube(link)  

    try:
        yt.streams.filter(progressive = True, file_extension = "mp4").first().download(output_path = SAVE_PATH,  filename=None)
    except:
        print("Some Error!"+l)
    time.sleep(15)    
    print('Task Completed!')



# make sure you download chrome driver from https://chromedriver.chromium.org/downloads and put it in folder 'driver'
driver = webdriver.Chrome('driver\chromedriver.exe')
driver.get('https://www.youtube.com/c/AmazingRussian/videos')  # put here your link

# scroll page down
old_position = 0
new_position = None
position_script = """return (window.pageYOffset !== undefined) ?
          window.pageYOffset : (document.documentElement ||
          document.body.parentNode || document.body);"""
while new_position != old_position:
    old_position = driver.execute_script(position_script)
    time.sleep(1)
    driver.execute_script(
        """var scrollingElement = (document.scrollingElement ||
         document.body);scrollingElement.scrollTop =
         scrollingElement.scrollHeight;""")    
    new_position = driver.execute_script(position_script) 
time.sleep(100)    
source_page = driver.execute_script("return document.documentElement.outerHTML")
driver.close()


pattern = "/watch?(.*?)webPageType"
with open('quotes.txt', 'w', encoding='utf-8') as f:
  for x in re.findall(pattern, source_page):
     lnk = "https://www.youtube.com/watch"+x.replace('"', "")
     f.write(lnk)
     f.write('\n')

for x in re.findall(pattern, source_page):
 lnk = "https://www.youtube.com/watch"+x.replace('"', "")
 dnload(lnk)

我正在尝试从 youtube 播放列表下载视频，但似乎整个网页都没有被 selenium 加载

我是否应该添加更多代码来修复它主要问题是页面完全加载但没有反映。当我自己在浏览器中加载页面时，一切都很好。

为什么会这样？

【问题讨论】：

标签： python selenium

【解决方案1】：

您可以通过使用WebDriverWait 等到加载所需的元素来解决此问题。

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

WebDriverWait(driver, 5).until(EC.presence_of_element_located((Your Element)))

这将等待您选择的元素被加载，然后再继续获取它。如果页面加载时间过长，您必须将号码 5 更改为您选择的时间。目前，它会等待 5 秒，然后再给我们一个错误，但这对于其他人来说可能太小了。

【讨论】：