您如何提出请求，BeautifulSoup 将链接抓取到 TripAdvisor 的第 2 页？答案

【问题标题】：How do you make requests and BeautifulSoup scrape links past page 2 of TripAdvisor?您如何提出请求，BeautifulSoup 将链接抓取到 TripAdvisor 的第 2 页？
【发布时间】：2022-07-29 13:21:19
【问题描述】：

我目前能够抓取某个位置的酒店的 href 链接，但问题是它只有第一页。第一页显示 32 家酒店，但我希望该位置的所有酒店都超过第一页。有没有办法刮掉给定位置的所有链接？我知道换页时 url 会发生变化，但是有没有办法绕过呢？

from argparse import Action
from calendar import month
from distutils.command.clean import clean
from lib2to3.pgen2 import driver
from os import link
import unittest
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.common.exceptions import ElementNotInteractableException
from selenium.webdriver.common.keys import Keys
from bs4 import BeautifulSoup
from dateutil import relativedelta
from selenium.webdriver.common.action_chains import ActionChains
import time
import datetime
from selenium.common.exceptions import StaleElementReferenceException
from selenium.common.exceptions import NoSuchElementException
import pandas as pd
import requests 
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import WebDriverException

class PythonOrgSearch(unittest.TestCase):

    def setUp(self):
        self.driver = webdriver.Chrome()
    def test_search_in_python_org(self):
        driver = webdriver.Chrome()
        api_url = 'https://www.tripadvisor.com/Hotels-g44535-Kansas_City_Missouri-Hotels.html'
        headers={
        'content-type': 'application/x-www-form-urlencoded',
        'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36',
        'x-puid': '0bac6bde-a8f6-421e-a391-40927b8d7fd5',
        'x-requested-with': 'XMLHttpRequest'
        }
        req = requests.post(api_url,headers=headers,
        )



        soup= BeautifulSoup(req.text,'lxml')
        title_list=[]
        for title in soup.select('.listing_title'):
            title_list.append(title.a.get_text(strip=True))
        
        items = soup.find_all('div', class_="prw_rup prw_meta_hsx_responsive_listing ui_section listItem")
        link_list=[]
        actual_list = []
        for x in items:
            clickable = x.find('a', class_='property_title prominent')
            link3 = clickable['href']
            link_list.append(link3)

        for x in link_list:
            link_text = "https://www.tripadvisor.com" + x
            actual_list.append(link_text)

        print(actual_list)

        dict = {'linklist':actual_list}
        
        # Create the dataframe.
        datafr = pd.DataFrame.from_dict(dict)
        datafr.head(10)
        # Convert dataframe to CSV file.
        datafr.to_csv('maroon1.1.csv', index=False, header=True)
if __name__ == "__main__":
    unittest.main()

【问题讨论】：

标签： beautifulsoup python-requests tripadvisor

【解决方案1】：

在您需要的每个页面的标题中以 30 为增量使用偏移量，然后循环整个页面。等等，就像字段内容类型一样。您还可以检索页面总数或仅使用 for i in range(6)：offset=(i-1)*30

    actual_list = []
    for i in range(6):
        offset=i*30
        headers={
        'content-type': 'application/x-www-form-urlencoded',
        'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36',
        'x-puid': '0bac6bde-a8f6-421e-a391-40927b8d7fd5',
        'x-requested-with': 'XMLHttpRequest',
        'offset':f'{offset}'
        }
        req = requests.post(api_url,headers=headers,
        )



        soup= BeautifulSoup(req.text,'lxml')
        title_list=[]
        for title in soup.select('.listing_title'):
            title_list.append(title.a.get_text(strip=True))
        
        items = soup.find_all('div', class_="prw_rup prw_meta_hsx_responsive_listing ui_section listItem")
        link_list=[]
        
        for x in items:
            clickable = x.find('a', class_='property_title prominent')
            link3 = clickable['href']
            link_list.append(link3)

        for x in link_list:
            link_text = "https://www.tripadvisor.com" + x
            actual_list.append(link_text)

    print(actual_list)

【讨论】：