我无法使用 python 循环浏览下一页以进行网络抓取答案

【问题标题】：i am not able to loop through next pages for web scraping using python我无法使用 python 循环浏览下一页以进行网络抓取
【发布时间】：2022-11-30 05:58:43
【问题描述】：

from bs4 import BeautifulSoup

import requests                      
from csv import writer 

def housing_d(page_no):
    
    url = f"https://www.pararius.com/apartments/amsterdam/page-{page}"
    web = requests.get(url)   ## requesting url
    html_code = web.content  ## gives page contents
    
    soup = BeautifulSoup(web.content, 'html.parser')
    # print(soup.prettify)  ## prettify will give html code in indent and proper way
    web_page = soup.find_all('ul', class_="search-list")
     
    with open('housing.csv', 'w', encoding='utf8', newline='') as f:
        thewriter = writer(f)
        header = ['Title', 'Location', 'Price', 'Area']
        thewriter.writerow(header)
        
        for division in web_page:
            lists = division.find_all('li', class_="search-list__item search-list__item--listing")
            for list in lists:
                title = list.find('a', class_="listing-search-item__link--title").text.strip().replace('\n', '')
                location = list.find('div', class_="listing-search-item__location").text.strip().replace('\n', '')
                price = list.find('div', class_="listing-search-item__price").text.strip().replace('\n', '')
                area = list.find('li', class_="illustrated-features__item illustrated-features__item--surface-area").text.strip().replace('\n', '')


                info = [title, location, price, area]
                thewriter.writerow(info)
                               
        return

for page in range(1,10):

    housing_d(page)

【问题讨论】：

我相信您每次调用函数 housing_d 时都在重新编写文件，您可能想编写一次文件，然后附加到它。重写发生在with open('housing.csv', 'w' ... 另外，page 应该是page_no。

标签： python html beautifulsoup export-to-csv

【解决方案1】：

要从不同的页面获取数据，您可以使用以下示例：

import requests
import pandas as pd
from bs4 import BeautifulSoup


url = "https://www.pararius.com/apartments/amsterdam/page-{}"
headers = {"X-Requested-With": "XMLHttpRequest"}

data = []
for page in range(1, 5):        # <-- increase number of pages here
    print("Getting page", page)
    html = requests.get(url.format(page), headers=headers).json()["components"][
        "results"
    ]
    soup = BeautifulSoup(html, "html.parser")

    for li in soup.select("li.search-list__item--listing"):
        title = li.h2.get_text(strip=True)
        loc = li.select_one(".listing-search-item__location").get_text(
            strip=True
        )
        price = li.select_one(".listing-search-item__price").get_text(
            strip=True
        )
        area = li.select_one(".illustrated-features__item--surface-area")
        area = area.get_text(strip=True) if area else "N/A"
        data.append([title, loc, price, area])

df = pd.DataFrame(data, columns=["Title", "Location", "Price", "Area"])
print(df)
df.to_csv("data.csv", index=False)

印刷：


...
120        Apartment Piet Gijzenbrugstraat 30 2           1059 XJ Amsterdam (Hoofddorppleinbuurt)   €2,300 per month   80 m²
121                 House Van Breestraat 139 hs                1071 ZL Amsterdam (Museumkwartier)   €4,950 per month  140 m²
122            Apartment Milovan Djilasplein 26       1102 JZ Amsterdam (Bijlmer Centrum (D,F,H))   €1,850 per month  105 m²
123             Apartment Des Présstraat 4 huis                   1075 NX Amsterdam (Willemspark)   €4,000 per month  138 m²
124   Apartment Pieter de Hoochstraat 23 A + PP                1071 ED Amsterdam (Museumkwartier)   €3,000 per month  114 m²
125                Apartment Rustenburgerstraat                   1072 HD Amsterdam (Nieuwe Pijp)   €2,150 per month   55 m²
126          Apartment Lodewijk Boisotstraat 13                   1057 ZM Amsterdam (Chassébuurt)   €2,600 per month   87 m²
127       Apartment Maarten Jansz. Kosterstraat             1017 VX Amsterdam (De Weteringschans)   €3,150 per month  120 m²

并保存data.csv（来自 LibreOffice 的屏幕截图）：

【讨论】：