使用 Python 从网页中抓取 URL 时返回 {{link}}

Question

我正在从如下网页中抓取网址

from bs4 import BeautifulSoup
import requests

url = "https://www.investing.com/search/?q=Axon&tab=news"
response = requests.get(url, headers={'User-Agent': 'Mozilla/5.0'})
soup = BeautifulSoup(response.content, "html.parser")

for s in soup.find_all('div',{'class':'articleItem'}):

    for a in s.find_all('div',{'class':'textDiv'}):
        for b in a.find_all('a',{'class':'title'}):
            print(b.get('href'))

输出如下所示

/news/stock-market-news/axovant-updates-on-parkinsons-candidate-axolentipd-1713474
/news/stock-market-news/digital-alley-up-24-on-axon-withdrawal-from-patent-challenge-1728115
/news/stock-market-news/axovant-sciences-misses-by-009-763209
/analysis/microns-mu-shares-gain-on-q3-earnings-beat-upbeat-guidance-200529289
/analysis/axon,-espr,-momo,-zyne-200182141
/analysis/factors-likely-to-impact-axon-enterprises-aaxn-q4-earnings-200391393
{{link}}
{{link}}

问题是

1. 未提取所有 URL
2. 看到最后两条，为什么会这样？

以上两个问题有什么解决办法吗？

Answer 1

那是因为您正在发出 HTTP 请求，而 youtube 使用 JavaScript 呈现视频数据。为了能够解析 JS 内容，你必须使用支持发出请求然后用 JS 渲染它的库。尝试使用模块requests_html。 pypi.org/project/requests-html

Answer 2

解决此问题的一种方法是使用 selenium：

driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

当 selenium 向下滚动到页面底部时，您会阅读 pagesource 并关闭 selenium 并使用 Beautifulsoup 解析 pagesource。也可以用 selenium 解析到。

先硒和bs4：

from selenium import webdriver
from bs4 import BeautifulSoup

import time

PAUSE_TIME = 1
driver = webdriver.Firefox(executable_path='c:/program/geckodriver.exe')
driver.get('https://www.investing.com/search/?q=Axon&tab=news')
lh = driver.execute_script("return document.body.scrollHeight")

while True:

    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")


    time.sleep(PAUSE_TIME)


    nh = driver.execute_script("return document.body.scrollHeight")
    if nh == lh:
        break
    lh = nh
pagesourece = driver.page_source
driver.close()

soup = BeautifulSoup(pagesourece, "html.parser")

for s in soup.find_all('div',{'class':'articleItem'}):

    for a in s.find_all('div',{'class':'textDiv'}):
        for b in a.find_all('a',{'class':'title'}):
            print(b.get('href'))

只有硒版本：

from selenium import webdriver
from bs4 import BeautifulSoup

import time

PAUSE_TIME = 1
driver = webdriver.Firefox(executable_path='c:/program/geckodriver.exe')
driver.get('https://www.investing.com/search/?q=Axon&tab=news')
lh = driver.execute_script("return document.body.scrollHeight")

while True:

    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")


    time.sleep(PAUSE_TIME)


    nh = driver.execute_script("return document.body.scrollHeight")
    if nh == lh:
        break
    lh = nh
pagesourece = driver.page_source




for s in driver.find_elements_by_css_selector('div.articleItem'):

    for a in s.find_elements_by_css_selector('div.textDiv'):
        for b in a.find_elements_by_css_selector('a.title'):
            print(b.get_attribute('href'))
driver.close()

注意你必须安装selenium 并下载geckodriver 来运行它。如果您想在其他路径中使用 geckodriver，则必须更改 c:/program：

driver = webdriver.Firefox(executable_path='c:/program/geckodriver.exe')

到你的 geckodriver 路径。