使用 Beautiful Soup 时无法理解空数组输出

Question

我编写了一个非常小的 Python 脚本，用于从 CNN 网站上抓取文章标题。

import requests
from bs4 import BeautifulSoup

url='https://edition.cnn.com/'
topics=['world','politics','business']
r=requests.get(url+topics[1])
soup=BeautifulSoup(r.content,'html.parser')
spans=soup.find_all('span',{'class':"cd__headline-text"})
print(spans)

执行此代码后，我只是得到一个空列表作为输出。这不是我所期望或寻找的，因为我试图刮掉标签后面的文本。我试图引用的 html 块的 sn-p 是-

<span class="cd__headline-text">
Bernie Sanders faces pivotal clash as Democratic establishment joins forces against him
</span>

请帮助澄清我的代码似乎做错了什么和/或我可能犯的任何逻辑错误。

Answer 1

您的代码运行良好。它只是不会为 politics 页面产生结果。

试试这个：

import requests
from bs4 import BeautifulSoup

url='https://edition.cnn.com/'
topics = ['world','politics','business']

headlines = []

for topic in topics:

    r = requests.get(url+topic)
    soup=BeautifulSoup(r.content,'html.parser')

    for span in soup.find_all('span',{'class':"cd__headline-text"}):
        headlines.append(span.text)
        print(span.text)
        print()

headlines 打印到：

The bizarre ways that coronavirus is changing etiquette
Over half of all virus cases in one country are linked to this group
Trump's Middle East plan could jeopardize Jordan-Israel peace treaty, Jordan PM says
Irish duo's win marks rare victory for women in the 'Nobel of architecture'
After more than 240 days, Australia's New South Wales is finally free from bushfires
Child drowns off Greek coast after Turkey opens border with Europe 
A migration crisis and disagreement with Turkey is the last thing Europe needs right now
Vatican to open controversial WW2-era files on Pope Pius XII
Netanyahu projected to win Israeli election, but exit polls suggest bloc just short of majority
Adviser to Iran's Supreme Leader dies after contracting coronavirus
Israeli election exit polls project Netanyahu in lead
She became pregnant at the age of 12. Now, Kenya's Christine Ongare is an Olympic boxing qualifier
Nigeria says it is ready and more than capable of dealing with coronavirus
Kenya bans commercial slaughter of donkeys following a rise in animal theft 
Violence forces Haiti to cancel Carnival
....

您不会得到politics 的结果，因为内容是在浏览器中使用Javascript 动态呈现的（正如G. Anderson 在他的cmets 中解释的那样）。但是，使用requests，您只能获得原始 HTML。

在浏览器中打开该站点并将View page source 与Inspect element 进行比较。前者产生原始 HTML，后者产生呈现的 HTML。