从播放列表中抓取视频标题

Question

我想写一个从 YouTube 音乐播放列表中收集视频标题的爬虫，因为有时视频会被删除。我是 python 新手。我通过一篇文章写了代码：

我检查了许多网站上的代码功能（通过更改链接、标签和类），一切正常，但不知何故，YouTube 却没有。

如何从播放列表中获取视频标题？

import requests
from bs4 import BeautifulSoup

url = 'https://www.youtube.com/playlist?list=PLuDh46ey2oy-qmIqPH0o1ZUZ9BFuqvtBn'
response = requests.get(url)
soup = BeautifulSoup(response.text, 'lxml')
quotes = soup.find_all('a', class_='yt-simple-endpoint style-scope ytd-playlist-video-renderer')

for quote in quotes:
    print(quote.text)

Answer 1

您的代码没有产生任何结果的主要原因是 soup = BeautifulSoup(response.text, 'lxml') 不包含您的标签。

您可以使用 print(soup.prettify())

进行检查

我建议使用pytube 来提取播放列表标题

import re
from pytube import YouTube
from pytube import Playlist


playlist = Playlist("https://www.youtube.com/playlist?list=PLuDh46ey2oy-qmIqPH0o1ZUZ9BFuqvtBn")
playlist._video_regex = re.compile(r"\"url\":\"(/watch\?v=[\w-]*)")
print('Number of videos in playlist: %s' % len(playlist.video_urls))
for url in playlist.video_urls:
    yt = YouTube(url)
    print(yt.title)
    #output
    Busta Rhymes - Touch It (TikTok Remix) Lyrics | touch it clean busta rhymes remix tik tok
    Shaggy - Boombastic (Official Music Video)
    Nobody - Mitski (slowed + reverb)
    Mitski - Nobody (Official Video)
    ...truncated
    

Answer 2

您可能已经从堆栈溢出中阅读过，因为您提到了 YouTube 使用 JavaScript 的相同主题，因此您可以试用 selenium 包，它提供了自动化浏览器的能力，您可以从中提取数据以获取更多信息，您可以从 @987654321 阅读@

代码如下：

from selenium import webdriver

path="you're path of driver"
driver=webdriver.Chrome(path)

url = 'https://www.youtube.com/playlist?list=PLuDh46ey2oy-qmIqPH0o1ZUZ9BFuqvtBn'
response = driver.get(url)    

main_a=driver.find_elements_by_id("video-title")
lst=[]

for a in main_a:
    lst.append(a.get_attribute("aria-label"))
print(lst)

Answer 3

这个答案是对 Bhavya Parikh 答案的增强，它使用了 selenium。下面的代码添加了我在对 Bhavya 的回答的评论中提到的滚动功能。

有几种方法可以向下滚动页面，答案显示了其中一种方法。该代码也使用无头模式，因此 Chrome 浏览器窗口不显示。

from time import sleep

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options


chrome_options = Options()
chrome_options.add_argument("--disable-infobars")
chrome_options.add_argument("--disable-extensions")
chrome_options.add_argument("--disable-popup-blocking")
chrome_options.add_argument('--headless')

# window size as an argument is required in headless mode
chrome_options.add_argument('window-size=1920x1080')

# Hide the "Chrome is being controlled by automated test software" banner
chrome_options.add_experimental_option("useAutomationExtension", False)
chrome_options.add_experimental_option("excludeSwitches", ['enable-automation'])

driver = webdriver.Chrome('/usr/local/bin/chromedriver', options=chrome_options)

url = 'https://www.youtube.com/playlist?list=PLuDh46ey2oy-qmIqPH0o1ZUZ9BFuqvtBn'
response = driver.get(url)
driver.implicitly_wait(15)

# finds the body tag
elem = driver.find_element_by_tag_name("body")

# you can also use the html tag
# elem = driver.find_element_by_tag_name("html")

no_of_pagedowns = 100
while no_of_pagedowns:
    elem.send_keys(Keys.PAGE_DOWN)
    sleep(0.2)
    no_of_pagedowns -= 1

title_tags = driver.find_elements_by_id("video-title")
video_titles = []
for title_tag in title_tags:
    video_titles.append(title_tag.get_attribute("aria-label"))

# do something with the list of titles.

driver.close()