Python 3.4 - 遍历 n 个 URL，其中 n 不固定答案

【问题标题】：Python 3.4 - Looping through n URLS where n is not fixedPython 3.4 - 遍历 n 个 URL，其中 n 不固定
【发布时间】：2016-02-23 03:08:44
【问题描述】：

遍历一系列 URL 直到没有更多结果返回的最简单方法是什么？

如果 URL 的数量是固定的，例如 9，类似下面的代码就可以了

for i in range(1,10):
    print('http://www.trademe.co.nz/browse/categorylistings.aspx?v=list&rptpath=4-380-50-7145-&mcatpath=sports%2fcycling%2fmountain-bikes%2ffull-suspension&page='+ str(i)+'&sort_order=default ')

但是，URL 的数量是动态的，我看到一个页面显示“抱歉，目前此类别中没有列表。”当我过冲。下面的例子。

http://www.trademe.co.nz/browse/categorylistings.aspx?v=list&rptpath=4-380-50-7145-&mcatpath=sports%2fcycling%2fmountain-bikes%2ffull-suspension&page=10&sort_order=default

只返回带有结果的页面的最简单方法是什么？

干杯史蒂夫

【问题讨论】：

if err in response: break 怎么样，err 是您上面提到的错误？不过，使用 trademe API 很可能会更干净
我建议像一个优秀的互联网公民一样使用他们的 API，而不是窃取他们的数据：developer.trademe.co.nz/api-terms/terms-and-conditions

标签： python loops url web-scraping

【解决方案1】：

# count is an iterator that just keeps going
# from itertools import count
# but I'm not going to use it, because you want to set a reasonable limit
# otherwise you'll loop endlessly if your end condition fails

# requests is third party but generally better than the standard libs
import requests

base_url = 'http://www.trademe.co.nz/browse/categorylistings.aspx?v=list&rptpath=4-380-50-7145-&mcatpath=sports%2fcycling%2fmountain-bikes%2ffull-suspension&page={}&sort_order=default'

for i in range(1, 30):
    result = requests.get(base_url.format(i))
    if result.status_code != 200:
        break
    content = result.content.decode('utf-8')
    # Note, this is actually quite fragile
    # For example, they have 2 spaces between 'no' and 'listings'
    # so looking for 'no listings' would break
    # for a more robust solution be more clever.
    if 'Sorry, there are currently no' in content:
        break

    # do stuff with your content here
    print(i)

【讨论】：