将 URL 复制到包含特定术语的文件答案

【问题标题】：Copying URLs to file that contain specific term将 URL 复制到包含特定术语的文件
【发布时间】：2013-08-06 13:22:53
【问题描述】：

所以我试图获取页面包含术语“食谱改编自”或“食谱来自”的范围内的所有网址。这会将文件的所有链接复制到大约 7496，然后它会吐出 HTTPError 404。我做错了什么？我尝试实现 BeautifulSoup 和 requests，但仍然无法正常工作。

import urllib2
with open('recipes.txt', 'w+') as f:
    for i in range(14477):
        url = "http://www.tastingtable.com/entry_detail/{}".format(i)
        page_content = urllib2.urlopen(url).read()
        if "Recipe adapted from" in page_content:
            print url
            f.write(url + '\n')
        elif "Recipe from" in page_content:
            print url
            f.write(url + '\n')
        else:
            pass

【问题讨论】：

标签： python python-2.7 web-crawler urllib2

【解决方案1】：

您尝试抓取的某些 URL 不存在。忽略异常，或许可以跳过：

import urllib2
with open('recipes.txt', 'w+') as f:
    for i in range(14477):
        url = "http://www.tastingtable.com/entry_detail/{}".format(i)
        try:
            page_content = urllib2.urlopen(url).read()
        except urllib2.HTTPError as error:
            if 400 < error.code < 500:
                continue  # not found, unauthorized, etc.
            raise   # other errors we want to know about
        if "Recipe adapted from" in page_content or "Recipe from" in page_content:
            print url
            f.write(url + '\n')

【讨论】：