抓取以某个单词开头的网页句子

Question

我正在用 BeautifulSoup 和 python 抓取一个网页，想抓取页面上以“Text Start:”开头的每个句子，如下面的代码所示。每个句子还以逗号结尾，后跟日期形式为月-日（下面是 5-4） 有很多这样的实例，所以我想浏览页面并返回以开头的每个句子“文本开始：”。

我一直在尝试使用 BeautifulSoup 包执行此操作，但遇到了麻烦。我认为我应该使用正则表达式，所以我一直在尝试，但还没有真正实现。

<div class="class">
<div class="time">
<span class="date">07/02/13</span>
<span class="sep">|</span>
<span class="duration">02:15</span>
<div class="clear"></div>
</div>
Text Start: This text changes each time, 5-4
</div>

Answer 1

使用regular expression 匹配specific text contents：

import re

soup.find_all(text=re.compile('^\s*Text Start:.*'))

演示：

>>> from bs4 import BeautifulSoup
>>> soup = BeautifulSoup('''\
... <div class="class">
... <div class="time">
... <span class="date">07/02/13</span>
... <span class="sep">|</span>
... <span class="duration">02:15</span>
... <div class="clear"></div>
... </div>
... Text Start: This text changes each time, 5-4
... </div>
... ''')
>>> import re
>>> soup.find_all(text=re.compile('^\s*Text Start:.*'))
[u'\nText Start: This text changes each time, 5-4\n']

Answer 2

您不需要为这么简单的任务使用 BeautifulSoup。它比直接使用正则表达式要慢得多。

做re.findall('^\s*Text Start:.*',page)。

在抓取网页时，我发现非常准确地了解页面内容很有用。 就个人而言，我这样做：

from httplib import HTTPConnection

hypr = HTTPConnection(host='stackoverflow.com',
                      timeout = 3)
rekete = ('/questions/17503336/'
          'scraping-webpage-sentences-'
          'beginning-with-certain-word')

hypr.request('GET',rekete)
page = hypr.getresponse().read()


print '\n'.join('%d %r' % (i,line)
                for i,line in enumerate(page.splitlines(True)))

显示是

0 '<!DOCTYPE html>\r\n'
1 '<html>\r\n'
2 '<head>\r\n'
3 '        \r\n'
4 '    <title>python - Scraping webpage sentences beginning with certain word - Stack Overflow</title>\r\n'
5 '    <link rel="shortcut icon" href="https://cdn.sstatic.net/stackoverflow/img/favicon.ico">\r\n'
6 '    <link rel="apple-touch-icon image_src" href="https://cdn.sstatic.net/stackoverflow/img/apple-touch-icon.png">\r\n'
7 '    <link rel="search" type="application/opensearchdescription+xml" title="Stack Overflow" href="/opensearch.xml">\r\n'
8 '    \r\n'
9 '    <script type="text/javascript" src="//ajax.googleapis.com/ajax/libs/jquery/1.7.1/jquery.min.js"></script>\r\n'
10 '    <script type="text/javascript" src="https://cdn.sstatic.net/js/stub.js?v=d2c9bad99c24"></script>\r\n'
11 '    <link rel="stylesheet" type="text/css" href="https://cdn.sstatic.net/stackoverflow/all.css?v=2079d4ae31a4">\r\n'
12 '    \r\n'
13 '    <link rel="canonical" href="http://stackoverflow.com/questions/17503336/scraping-webpage-sentences-beginning-with-certain-word">\r\n'
14 '    <link rel="alternate" type="application/atom+xml" title="Feed for question \'Scraping webpage sentences beginning with certain word\'" href="/feeds/question/17503336">\r\n'
15 '    <script type="text/javascript">\r\n'
16 '        \r\n'
17 '        StackExchange.ready(function () {\r\n'
18 '            StackExchange.using("postValidation", function () {\r\n'
19 "                StackExchange.postValidation.initOnBlurAndSubmit($('#post-form'), 2, 'answer');\r\n"
20 '            });\r\n'
21 '\r\n'
22 '            \r\n'
23 "            StackExchange.question.init({showAnswerHelp:true,totalCommentCount:0,shownCommentCount:0,highlightColor:'#F4A83D',backgroundColor:'#FFF',questionId:17503336});\r\n"
24 '\r\n'
25 '            styleCode();\r\n'
26 '\r\n'
27 "                StackExchange.realtime.subscribeToQuestion('1', '17503336');\r\n"
28 '            \r\n'
29 '                    });\r\n'
30 '    </script>\r\n'
31 '\r\n'

etc etc