html评论中的python网络抓取

Question

以下不一定是问题。我创建了一小段代码来从网页中提取数据，我想知道您对代码有什么看法以及如何改进它。

我需要知道博士面试的日期。他们不向我们发送电子邮件。 Here 他们会发布日期。我意识到我感兴趣的两个博士职位都在 HTML cmets 内部。它们都以字符串 URBAN 开头。

我创建了一个正则表达式来查找所有 cmets

regex = r"<!--(.*?)-->"

并使用 for 循环检查这些 cmets 内部是否存在 URBAN 字样。希望评论中没有字符串意味着他们发布了日期。

这是我的代码：

import requests, re, time, smtplib

url = "http://dottorato.polito.it/Esami_accesso.html"

DEBUG = False

foundInComment = True

""" 
. matches anything but \n   
* 0 or more occurrences of the pattern to its left
() groups
? for non-greedy 
"""
    regex = r"<!--(.*?)-->"

while foundInComment:
    try:
        r = requests.get(url)
        html = r.text 

        result = re.findall(regex,html,re.DOTALL) # re.DOTALL makes . match also \n 

        for match in result:
            if len(re.findall("URBAN",match)) > 1: #One of the commets has to have at least two URBAN
                foundInComment = True
                print("\"URBAN AND REGIONAL DEVELOPMEN\" found more than once in a comment at " 
                                       + time.strftime("%H:%M:%S"))
                break
            foundInComment = False

        time.sleep(600)

    except KeyboardInterrupt:
        raise
    except Exception as e:
        print e
        print "Going to sleep for 1 min"
        time.sleep(60)

if not DEBUG:
    fromaddr = 'someMail@gmail.com'
    toaddrs  = ['otherMail@gmail.com', fromaddr]

    msg = 'Subject: PHD polito\n\n Go to %s' % url 

    # Credentials
    username = 'someone'
    password = 'password'

    server = smtplib.SMTP('smtp.gmail.com:587')
    server.starttls()
    server.login(username,password)
    server.sendmail(fromaddr, toaddrs, msg)
    server.quit()

    print "End of program"

那么，你怎么看？

提前致谢！

PS：这是 HTML 注释的一部分，其中包含 URBAN：

<li><a href="colloqui/Architettura_Storia_Progetto2.pdf">URBAN AND REGIONAL DEVELOPMEN</a></li>
<li><a href="colloqui/Architettura_Storia_Progetto2.pdf">URBAN AND REGIONAL DEVELOPMEN - Cluster Tecnologie per le Smart Communities - Progetto Edifici a Zero Consumo Energetico in Distretti Urbani Intelligenti</a></li>
-->

我几乎可以肯定他们会复制此内容并将其粘贴到网页内的评论中。

Answer 1

另一种（我认为更可靠）的方法是使用专门的工具来完成这项工作——HTML 解析器。例如，使用BeautifulSoup，打印出所有包含URBAN 字样的comments：

import requests
from bs4 import BeautifulSoup, Comment

url = "http://dottorato.polito.it/Esami_accesso.html"
response = requests.get(url)

soup = BeautifulSoup(response.content)
print soup.find_all(text=lambda text:isinstance(text, Comment) and 'URBAN' in text)