抓取信息 - BeautifulSoup/Python答案

【问题标题】：Scrape information - BeautifulSoup/Python抓取信息 - BeautifulSoup/Python
【发布时间】：2020-12-01 15:11:41
【问题描述】：

我的代码转到一个网站，提取 URL，然后转到它抓取的 URL（到这里都可以正常工作）

现在在这个新页面上，我想获取一些信息（作者姓名），但是它正在打印空白

代码如下：

from selenium import webdriver
from bs4 import BeautifulSoup
import time
import requests
driver = webdriver.Chrome()
eachLink=[]
baseurl='https://meetinglibrary.asco.org'
for x in range (1,2):
  driver.get(f'https://meetinglibrary.asco.org/results?meetingView=2020%20ASCO%20Virtual%20Scientific%20Program&page={x}')
  time.sleep(3)
  page_source = driver.page_source
  soup = BeautifulSoup(page_source,'html.parser')
  productlist=soup.find_all('a',class_='ng-star-inserted')
  for item in productlist:
     for link in item.find_all('a',href=True):
         eachLink.append(baseurl+link['href'])
print(eachLink)
infobox=[]
for b in eachLink:
    r=requests.get(b)
    time.sleep(1)
    soup1=BeautifulSoup(r.content,'html.parser')
    auth=soup1.find('a',class_='asset-metadata-value link ng-star-inserted')
    print(auth)

【问题讨论】：

那么究竟是什么问题？
它在帖子中的字面意思是，它正在打印空白......

标签： python html web-scraping beautifulsoup

【解决方案1】：

也许 time.sleep(1) 在 eachLink 循环中不够长，页面仍在加载中。您可以使用显式等待来检查预期条件，而不是使用 time.sleep（隐式等待）。

https://selenium-python.readthedocs.io/waits.html

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

path = "//div[@id='YOURIDHERE']"   #change to something that should be present for eachLink

button = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located(
        (By.XPATH, path) 
    )

【讨论】：

【解决方案2】：

我认为它的帮助。无需第二个 for 循环即可提取作者。

from selenium import webdriver
from bs4 import BeautifulSoup
import time
import requests
driver = webdriver.Chrome()
eachLink=[]
authors = []
baseurl='https://meetinglibrary.asco.org'
for x in range (1,2):
  driver.get(f'https://meetinglibrary.asco.org/results?meetingView=2020%20ASCO%20Virtual%20Scientific%20Program&page={x}')
  time.sleep(120)
  page_source = driver.page_source

  soup = BeautifulSoup(page_source,'html.parser')
  productlist=soup.find_all('a',class_='ng-star-inserted')
  for auth in soup.find_all('div', {'class':'record__ellipsis'}):
      authors.append(auth.text)

  for item in productlist:
     for link in item.find_all('a',href=True):
         eachLink.append(baseurl+link['href'])
print(eachLink)
print('\n', authors, '\n')

driver.quit()

【讨论】：

谢谢，但我想最终获取更多信息。所以我想从 href 页面而不是当前页面中提取