Beautifulsoup 提取周期答案

【问题标题】：beaufitul soup extract cycleBeautifulsoup 提取周期
【发布时间】：2019-12-27 11:52:22
【问题描述】：

我想从以下网站的所有页面中提取城市数据。我有以下代码，但循环继续运行并一遍又一遍地提取数据。好像我漏了什么，你能帮忙吗

cities = []
with requests.Session() as session:
    session.headers = {
        'x-requested-with': 'XMLHttpRequest'
    }
    page = 1
    while True:
        url = f'https://www.kununu.com/de/volkswagen/kommentare/{page}'
        response = session.get(url)
        soup = BeautifulSoup(response.text, 'html.parser')
        new_comments = [
            cities.find_next_sibling('div').text.strip()
            for cities in soup.find_all('div', text=re.compile('Stadt'))
        ]
        cities += new_comments
        print(cities)
        page += 1
#print(cities)

【问题讨论】：

您的代码中没有退出条件。当page==99 或类似的东西时，你需要break。另外我不知道您正在抓取的网站的结构，但看起来页码不会改变结果？

标签： python html beautifulsoup

【解决方案1】：

您没有退出条件。你需要在某个时候从循环中break。

例如：

cities = []
with requests.Session() as session:
    session.headers = {
        'x-requested-with': 'XMLHttpRequest'
    }
    page = 1
    while True:
        if page >= 99:
            break
        url = f'https://www.kununu.com/de/volkswagen/kommentare/{page}'
        response = session.get(url)
        soup = BeautifulSoup(response.text, 'html.parser')
        new_comments = [
            cities.find_next_sibling('div').text.strip()
            for cities in soup.find_all('div', text=re.compile('Stadt'))
        ]
        cities += new_comments
        print(cities)
        page += 1

print(cities)  # this will print after 98 pages

【讨论】：

我们可以有超过 100 页。而且它还不断复制每个城市2次

【解决方案2】：

如果不是200，请检查status code of the response，从循环中中断。

我建议使用 css 选择器来搜索项目。

代码：

cities = []
with requests.Session() as session:
    session.headers = {
        'x-requested-with': 'XMLHttpRequest'
    }
    page = 1
    while True:
        url = 'https://www.kununu.com/de/volkswagen/kommentare/{}'.format(page)
        print(url)
        response = session.get(url)
        if response.status_code!=200:
             break
        soup = BeautifulSoup(response.text, 'html.parser')
        new_comments = [
            cities.find_next_sibling('div').text.strip()
            for cities in soup.select('.review-details.user-content.hidden-xs div.text-uppercase:contains(Stadt)')
        ]
        print(new_comments)
        cities += new_comments
        #print(cities)
        page += 1


print(cities)

在控制台上，您可以看到您的网址和城市。

https://www.kununu.com/de/volkswagen/kommentare/1
['Wolfsburg', 'Salzgitter', 'Wolfsburg', 'Wolfsburg', 'Kassel', 'Wolfsburg', 'Wolfsburg', 'Wolfsburg', 'Wolfsburg', 'Wolfsburg']
https://www.kununu.com/de/volkswagen/kommentare/2
['Wolfsburg', 'Wolfsburg', 'Wolfsburg', 'Kassel', 'braunschweig', 'Wolfsburg', 'Wolfsburg', 'Wolfsburg', 'Wolfsburg', 'Zwickau']
https://www.kununu.com/de/volkswagen/kommentare/3
['Dresden', 'Emden', 'Wolfsburg', 'Wolfsburg', 'Wolfsburg', 'Wolfsburg', 'Wolfsburg', 'Wolfsburg', 'Wolfsburg', 'Hamburg']
https://www.kununu.com/de/volkswagen/kommentare/4
['Emden', 'Hannover', 'Wolfsburg', 'Hannover', 'Kassel', 'Hamburg', 'Baunatal', 'Wolfsburg', 'Wolfsburg', 'Emden']
https://www.kununu.com/de/volkswagen/kommentare/5
['Wolfsburg', 'Wolfsburg', 'Wolfsburg', 'Wolfsburg', 'Wolfsburg', 'Wolfsburg', 'Osnabrück', 'Kassel', 'Wolfsburg', 'braunschweig']
https://www.kununu.com/de/volkswagen/kommentare/6
['Wolfsburg', 'Wolfsburg', 'Kassel', 'Wolfsburg', 'Wolfsburg', 'Wolfsburg', 'Kassel', 'Berlin', 'Wolfsburg', 'Emden']
https://www.kununu.com/de/volkswagen/kommentare/7
['Emden', 'Wolfsburg', 'Wolfsburg', 'Kassel', 'braunschweig', 'Wolfsburg', 'Wolfsburg', 'Wolfsburg', 'Wolfsburg', 'Wolfsburg']
https://www.kununu.com/de/volkswagen/kommentare/8
['Kassel', 'Wolfsburg', 'Salzgitter', 'Salzgitter', 'Wolfsburg', 'Kassel', 'Wolfsburg', 'Kassel', 'Wolfsburg', 'Wolfsburg']

编辑：

cities = []
with requests.Session() as session:
    session.headers = {
        'x-requested-with': 'XMLHttpRequest'
    }
    page = 1
    while True:
        url = 'https://www.kununu.com/de/volkswagen/kommentare/{}'.format(page)
        print(url)
        response = session.get(url)
        # if response.status_code!=200:
        #      break
        soup = BeautifulSoup(response.text, 'html.parser')
        if soup.select_one("ul.review-info span[itemprop='itemReviewed']").find_next('span').text=="30.Jan. 2008":
             break
        new_comments = [
            cities.find_next_sibling('div').text.strip()
            for cities in soup.select('.review-details.user-content.hidden-xs div.text-uppercase:contains(Stadt)')
        ]
        print(new_comments)
        cities += new_comments
        #print(cities)
        page += 1


print(cities)

【讨论】：

它一直在提取，循环没有结束
在您想从代码中退出多少页之后？以便我可以更新相同的内容。
所有页面，大众汽车只是示例，例如梅赛德斯，我们可以有更多/更少的页面
@codecodecode ：看起来大众汽车在 159 页后只重复最后一页。所以您需要检查最后日期是否到达。尝试编辑代码。