使用相同的 URL 导航到 Scrapy 中的新页面

Question

我正在编写一个爬虫爬虫来爬取房地产网站 Rightmove。我遇到的问题是，由不同房屋列表的几页组成的房产搜索都位于同一个 URL 下。

这意味着识别“下一页”的 URL 的常规过程不起作用。有什么方法可以使用scrapy而不是selenium（对于此目的不够有效），我可以浏览不同的页面？请看我的代码和相关“下一页”按钮的源代码，如下图所示。

谢谢。

class listingsSpider(scrapy.Spider):
    name = 'listings'

    start_urls = ['https://www.rightmove.co.uk/property-for-sale/find.html?locationIdentifier=STATION%5E1712&maxPrice=500000&radius=0.5&sortType=10&propertyTypes=&mustHave=&dontShow=&furnishTypes=&keywords=']

    def parse(self, response):
        self.logger.info('This my first spider')
        address = response.xpath('//*[@id="property-65695633"]/div/div/div[4]/div[1]/div[2]/a/address')
        listings = response.xpath('//h2[@class="propertyCard-title"]')

        for listing in listings:
            yield{
                'Listing': listing.get()
            }


        nextPage = response.xpath('//*[@id="l-container"]/div[3]/div/div/div/div[3]/button/div/svg/use')
        nextPage = nextPage.get()

        pageTest = response.css('div[class=pagination-button pagination-direction pagination-direction--next] svg a::attr(href)')
        pageTest = pageTest.get()

        if pageTest is not None:
            pageTest = response.urljoin(pageTest)
            yield scrapy.Request(pageTest,callback=self.parse)

```[![enter image description here][1]][1] 


  [1]: https://i.stack.imgur.com/1I1J1.png

Answer 1

实际上，每个页面在网络链接中都有一个唯一的标识符。例如，附加&index = 24，这会将您转到下一页。

您需要弄清楚如何将其包含到请求网址中。有些可能有几页，所以我们每次增加 +24 以进入下一页。但是，我们可以将 +24 递增到无限，因此我们使用页面结果数作为中断的一种方式。第一眼就注意到是相当偷偷摸摸的！但很容易克服。

这是一个可以按要求转到这些下一页的刮板：

import scrapy
from scrapy.item import Field
from itemloaders.processors import TakeFirst
from scrapy.crawler import CrawlerProcess
from scrapy.loader import ItemLoader
import requests
from bs4 import BeautifulSoup

links= []
for i in range(0, 480, 24):
    url = f'https://www.rightmove.co.uk/property-for-sale/find.html?locationIdentifier=STATION%5E1712&maxPrice=500000&radius=0.5&sortType=10&propertyTypes=&mustHave=&dontShow=&index={i}&furnishTypes=&keywords='
    r = requests.get(url)
    soup = BeautifulSoup(r.content, 'lxml')
    ps1 = soup.find_all('span', {'class':'searchHeader-resultCount'})
    for ps in ps1:
        if int(ps.text.strip()) > i:
            links.append(url)
        else:
            break

class ListingsItem(scrapy.Item):
    address = Field(output_processor = TakeFirst())
    listings = Field(output_processor = TakeFirst())


class listingsSpider(scrapy.Spider):
    name = 'listings'
    start_urls = links
    

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(
                url, 
                callback = self.parse
            )

    def parse(self, response):
        container = response.xpath('//div[@class="l-searchResults"]/div')
        for sales in container:
            l = ItemLoader(ListingsItem(), selector = sales)
            l.add_xpath('address', '//address[@class="propertyCard-address"]/meta[@content]')
            l.add_xpath('listings', '//h2[@class="propertyCard-title"]//text()[normalize-space()]')
            yield l.load_item()
        
        #self.logger.info('This my first spider')
        #address = response.xpath('//*[@id="property-65695633"]/div/div/div[4]/div[1]/div[2]/a/address')
        #listings = response.xpath('//h2[@class="propertyCard-title"]')

        #for listing in listings:
        #    yield{
        #        'Listing': listing.get()
        #    }

process = CrawlerProcess(
    settings = {
        'FEED_URI': 'rightmove.jl',
        'FEED_FORMAT': 'jsonlines'
    }

)

process.crawl(listingsSpider)
process.start()