我无法使用 BeautifulSoup 从网页中获取列表中的项目

Question

我最近一直在阅读 BeautifulSoup 文档以熟悉如何解析网站。所以这对我来说很新鲜。谁能告诉我为什么我无法获得网站 (http://www.news-record.com) 上列出的最新头条新闻？

代码如下：

import urllib2
import BeautifulSoup


page = urllib2.urlopen("http://www.news-record.com/")

soup = BeautifulSoup.BeautifulSoup(page)



headlines = []



headline = soup.find('a', ({"class" : "nrcTxt_headline"}))
while headline:
    url = headline.findParent('div')['id']
    headlines.append([url, headline.string])
    headline = headline.findNext('span', {'class' : "nrcTxt_headline"})


print soup.headline

这是网站中感兴趣的部分：

<div class="nrcNav_menu">
   <ul>
    <li class="nrcTxt_menu1 nrcBlk_comboModTab nrc_default nrc_active">
     <a href="#nrcMod_FP_Breaking">
      Latest Headlines
     </a>
    </li>
    <li class="nrcTxt_menu2 nrcBlk_comboModTab nrc_itemLast">
     <a href="#nrcMod_FP_MostRead">
      Most Read
     </a>
    </li>
   </ul>
  </div>
  <div id="nrcMod_FP_Breaking" class="nrcBlk_comboModPage nrc_default nrc_active">
   <h4 class="nrcTxt_modHed">
    <span class="nrcTxt_label">
     Latest Headlines
    </span>
   </h4>
   <ul class="nrcBlk_artList">
    <li class="nrcBlk_artHedOnly nrcBlk_art nrcBlk_art4">
     <a class="nrcTxt_headline" href="/content/2012/02/28/article/city_hosts_earth_day_recycling_contest">
      City hosts Earth Day recycling contest
     </a>
     <span class="nrcBlk_pubdate">
      <span class="nrc_sep">
       (
      </span>
      <!-- COLLAPSE WHITESPACE

                                -->
      <span class="nrc_val">
       3:53 pm
      </span>
      <!-- COLLAPSE WHITESPACE

                                -->
      <span class="nrc_sep">
       )
      </span>
     </span>
    </li>
    <li class="nrcBlk_artHedOnly nrcBlk_art nrcBlk_art5">
     <a class="nrcTxt_headline" href="/content/2012/02/28/article/got_bull_rockingham_deputies_seek_900_pound_beast">
      Got bull? Rockingham deputies seek 900-pound beast
     </a>

下面的代码基本上可以满足我的要求。但是文本在括号中：

[u'Daily Deal: strength coaching sessions ']
[u'Teen dating safety tips']
[u'Splitting up the family']

如果我删除代码中的括号，我会收到错误：

import urllib2
import BeautifulSoup






site = "http://www.news-record.com/"

page = urllib2.urlopen("http://www.news-record.com/")
soup = BeautifulSoup.BeautifulSoup(page)

headlines = []


for headline in soup.findAll('a', {"class" : "nrcTxt_headline"}):
    url = headline.findParent('div', {"class":"nrcNav_menu"})
    print ([headline.string])

Answer 1

您将 find 参数包装在一个元组中，并没有真正尽可能地使用生成器..

soup.find('a', ({"class" : "nrcTxt_headline"}))

你想要

soup.find('a', {"class" : "nrcTxt_headline"})

例子：

for headline in soup.find('a', {"class" : "nrcTxt_headline"}):
    url = headline.findParent('div')['id']
    headlines.append([url, headline.string])

Answer 2

试试下面的代码：

import urllib2
import BeautifulSoup

site = "http://www.news-record.com/"

page = urllib2.urlopen("http://www.news-record.com/")
soup = BeautifulSoup.BeautifulSoup(page)

headlines = []

headlineList = soup.findAll('a', {"class" : "nrcTxt_headline"})
for headline in headlineList:
    headlines.append(site+str(headline.get('href')))

print headlines