网页抓取 - 不显示 html 代码的文本部分

Question

当我尝试通过 python 使用 Selenium 库抓取网站时遇到问题。 关键是我想获得一些有关收集到此站点的歌曲的信息：https://bandcamp.com/?g=all&s=top&p=0&gn=0&f=all&w=0。

但是，当我尝试从相应的 html 代码中提取文本时，该过程返回一个空列表。

如果我从浏览器 (Chrome) 中查看 html 代码，我会看到文本部分，但是当我在 python 中查看相同的代码时，文本部分不会出现。

这是我的代码：

browser = webdriver.Chrome()
browser.get("https://bandcamp.com/?g=all&s=top&p=0&gn=0&f=all&w=0")

name_song = browser.find_elements_by_css_selector("a.item-title")
name_artist = browser.find_elements_by_css_selector("a.item-artist")

genre = browser.find_elements_by_class_name("item-genre")
print(name_song, name artist, genre)

当我打印这三个变量时，我得到了 html 代码，但我无法从中提取任何内容。我怎么解决这个问题？非常感谢您的帮助。

我的目标是让“Apocalypticists”、“Kriegsmachine”和“metal”，分别分配给一个不同的变量。

Answer 1

你离得很近。您只需要诱导 WebDriverWait 以使所需的 元素可见 并将 WebElements 存储在三个不同的 Lists 和遍历它们以打印所需的文本，您可以使用以下解决方案：

• 代码块：

  from selenium import webdriver
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.support import expected_conditions as EC
  
  options = webdriver.ChromeOptions() 
  options.add_argument("start-maximized")
  options.add_argument('disable-infobars')
  browser = webdriver.Chrome(chrome_options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
  browser.get("https://bandcamp.com/?g=all&s=top&p=0&gn=0&f=all&w=0")
  name_song = WebDriverWait(browser, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "a.item-title")))
  name_artist = WebDriverWait(browser, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR,"a.item-artist")))
  genre = WebDriverWait(browser, 20).until(EC.visibility_of_all_elements_located((By.XPATH,"//a[@class='item-artist']//following::span[1]")))
  for song, artist, gen in zip(name_song, name_artist, genre):
      print("%s song is by %s and is of %s genre" % (song.text, artist.text, gen.text))
  

• 控制台输出：

  Apocalypticists song is by Kriegsmaschine and is of metal genre
  The Path song is by Carbon Based Lifeforms and is of ambient genre
  Christmas Time Is Here (N & S America Edition) song is by Khruangbin and is of funk genre
  Christmas Time Is Here (Excluding N & S America) song is by Khruangbin and is of funk genre
  Snailchan Adventure song is by Ujico*/Snail's House and is of electronic genre
  O God who avenges, shine forth. Rise up, Judge of the Earth; pay back to the proud what they deserve. song is by the body and is of metal genre
  T-Rex EP song is by Ben Prunty and is of soundtrack genre
  Woodland Womp (24bit 96kHz) song is by Kalya Scintilla and is of electronic genre
  

Answer 2

Element 对象不会给出 innerText 值。您需要调用 element.text 来获取它。 browser.find_elements_by_class_name("item-genre") 正在返回 23 个元素。定位器也必须更改以获得适当的 8 元素。

browser = webdriver.Chrome()
browser.get("https://bandcamp.com/?g=all&s=top&p=0&gn=0&f=all&w=0")

name_song = browser.find_elements_by_css_selector("a.item-title")
name_artist = browser.find_elements_by_css_selector("a.item-artist")
genre = browser.find_elements_by_css_selector("span.item-genre")

for i in range(len(name_song)-1):
  print(name_song[i].text)
  print(name_artist[i].text)
  print(genre[i].text)

Answer 3

您只需要进入每个元素即可获得所需的内容。您上面的代码返回三个硒元素注入列表。每个对象都有可以访问的属性，其中一个属性是.text

如果我运行上面的代码，我就可以访问name_song：

[<selenium.webdriver.remote.webelement.WebElement (session="83853054732fa0a5bfbc8a7e32a1e05b", element="0.4629143928625561-1")>,...

但是，如果我只想从这些元素中获取文本，我可以从每个元素中调用 text 属性：

[i.text for i in name_song]

['Apocalypticists',
 'The Path',
 'Christmas Time Is Here (N & S America Edition)',
 'Christmas Time Is Here (Excluding N & S America)',
 'Snailchan Adventure',
 'O God who avenges, shine forth. Rise up, Judge of the Earth; pay back to the proud what they deserve.',
 'T-Rex EP',
 'Woodland Womp (24bit 96kHz)']

然后索引到该列表中：

[i.text for i in name_song]
'Apocalypticists'