Scrapy Scrape crawlspider 带有输入标签的下一页答案

【问题标题】：Scrapy Scrape crawlspider next page with input tagScrapy Scrape crawlspider 带有输入标签的下一页
【发布时间】：2021-11-10 04:39:44
【问题描述】：

我正在使用 scrapy 和 crawlspinder。我想在这个website 上获得所有帖子。这是代码。

import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule


class ToscrapeSpider(CrawlSpider):
    name = 'toscrape'
    allowed_domains = ['pstrial-2019-12-16.toscrape.com']
    start_urls = ['http://pstrial-2019-12-16.toscrape.com/browse/insunsh']

    rule_articles_page = Rule(LinkExtractor(restrict_xpaths="//div[@id='body']/div[2]/a"), callback='parse_item', follow=False)
    rule_next_page = Rule(LinkExtractor(restrict_xpaths="//form[@class='nav next']/input[1]/@value", tags=('input'), attrs=('value',), process_value='process_value'),
                              follow=True,)

    rules = (
        rule_articles_page,
        rule_next_page,
    )
    def parse_item(self, response):
        yield {
            'Image': response.xpath("//div[@id='body']/img/@src").extract(),
            'Title': response.xpath("//div[@id='content']/h1/text()").extract(),
            'artist': response.xpath("//div[@id='content']/h2/text()").extract(),
            'Description': response.xpath("//div[@class='description']/p/text()").extract(),
            'URL': response.url,
            'Dimention' : response.xpath("//tbody/tr/td[text()='Dimensions']/text()").extract(),

        }

现在的问题是它没有转到下一页。因为下一页按钮是一个表单，而不是一个锚标记。

另外，请帮我获取文章页面上的图像尺寸（如果有的话，以厘米为单位）。

【问题讨论】：

标签： python web-scraping scrapy

【解决方案1】：

这是我创建的基本循环，也许您可以找到更好的循环，但这也可以解决您的问题。

import scrapy

class PagedataSpider(scrapy.Spider):

    name = 'pagedata'
    page=1
    allowed_domains = ['pstrial-2019-12-16.toscrape.com']
    start_urls = ['http://pstrial-2019-12-16.toscrape.com/browse/insunsh?page=1']

    def parse(self, response):

        yield {
            'Title': response.css("div h1::text").getall()
        }

        # next_page=response.css('input[name="page"]::attr(value)').get()
        
        
        if PagedataSpider.page <=114:
            PagedataSpider.page+=1
            nextPage=f'http://pstrial-2019-12-16.toscrape.com/browse/insunsh?page={PagedataSpider.next_page}'

            yield scrapy.Request(nextPage,callback=self.parse)

【讨论】：

【解决方案2】：

要获取尺寸，请从 xpath 中删除“tbody”，并在“key”td 之后搜索下一个兄弟姐妹（以获取“value”td 文本）。
设置'process_value'以正确获取页码。

import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule


class ToscrapeSpider(CrawlSpider):
    name = 'toscrape'
    allowed_domains = ['pstrial-2019-12-16.toscrape.com']
    start_urls = ['http://pstrial-2019-12-16.toscrape.com/browse/insunsh']

    rule_articles_page = Rule(LinkExtractor(restrict_xpaths="//div[@id='body']/div[2]/a"), callback='parse_item', follow=False)
    rule_next_page = Rule(LinkExtractor(restrict_xpaths="//form[@class='nav next']/input[1]", tags=('input'), attrs=('value',),
                                        process_value=(lambda x: f'?page={x[-1]}')),
                          follow=True,)

    rules = (
        rule_articles_page,
        rule_next_page,
    )

    def parse_item(self, response):
        yield {
            'Image': response.xpath("//div[@id='body']/img/@src").get(),
            'Title': response.xpath("//div[@id='content']/h1/text()").get(),
            'artist': response.xpath("//div[@id='content']/h2/text()").get(),
            'Description': response.xpath("//div[@class='description']/p/text()").get(),
            'URL': response.url,
            'Dimension': response.xpath('//tr/td[text()="Dimensions"]/following-sibling::td[@class="value"]/text()').get(),
        }

输出：

[scrapy.core.scraper] DEBUG: Scraped from <200 http://pstrial-2019-12-16.toscrape.com/item/12125/Front_Panel_for_Blouse?back=155>
{'Image': '/content/12125.jpg', 'Title': 'Front Panel for Blouse', 'artist': None, 'Description': 'reverse patchwork of animal designs in blue, yellow, red and orange', 'URL': 'http://pstrial-2019-12-16.toscrape.com/item/12125/Front_Panel_for_Blouse?back=155', 'Dimension': '16 3/4 x 22 1/8in. (42.5 x 56.2cm)'}
...
...
[scrapy.core.engine] DEBUG: Crawled (200) <GET http://pstrial-2019-12-16.toscrape.com/browse/insunsh?page=2> (referer: http://pstrial-2019-12-16.toscrape.com/browse/insunsh?page=1)
[scrapy.core.engine] DEBUG: Crawled (200) <GET http://pstrial-2019-12-16.toscrape.com/item/12712/The_Muses_of_Music_and_Poetry?back=155> (referer: http://pstrial-2019-12-16.toscrape.com/browse/insunsh?page=1)
[scrapy.core.scraper] DEBUG: Scraped from <200 http://pstrial-2019-12-16.toscrape.com/item/12712/The_Muses_of_Music_and_Poetry?back=155>
{'Image': '/content/12712.jpg', 'Title': 'The Muses of Music and Poetry', 'artist': 'Sculptor: Guillaume Coustou the Younger', 'Description': None, 'URL': 'http://pstrial-2019-12-16.toscrape.com/item/12712/The_Muses_of_Music_and_Poetry?back=155', 'Dimension': '24 5/8 x 15 1/2 x 10 1/2 in. (62.55 x 39.37 x 26.67 cm)'}
...
...
[scrapy.core.engine] DEBUG: Crawled (200) <GET http://pstrial-2019-12-16.toscrape.com/browse/insunsh?page=3> (referer: http://pstrial-2019-12-16.toscrape.com/browse/insunsh?page=2)
[scrapy.core.engine] DEBUG: Crawled (200) <GET http://pstrial-2019-12-16.toscrape.com/item/13484/Don_Pascual_y_su_esposa?back=155> (referer: http://pstrial-2019-12-16.toscrape.com/browse/insunsh?page=2)
...
...
...

编辑：

上面的代码只能获取最多 9 个页面，包括。这是修复：

import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule


def get_next_page(value):
    next_page = value.replace('browse/', 'browse/insunsh?page=')
    return next_page


class ToscrapeSpider(CrawlSpider):
    name = 'toscrape'
    allowed_domains = ['pstrial-2019-12-16.toscrape.com']
    start_urls = ['http://pstrial-2019-12-16.toscrape.com/browse/insunsh']

    rule_articles_page = Rule(LinkExtractor(restrict_xpaths="//div[@id='body']/div[2]/a"), callback='parse_item', follow=False)
    rule_next_page = Rule(LinkExtractor(restrict_xpaths="//form[@class='nav next']/input[1]", tags=('input'), attrs=('value',),
                                        process_value=get_next_page),
                          follow=True)

    rules = (
        rule_articles_page,
        rule_next_page,
    )

    def parse_item(self, response):
        yield {
            'Image': response.xpath("//div[@id='body']/img/@src").get(),
            'Title': response.xpath("//div[@id='content']/h1/text()").get(),
            'artist': response.xpath("//div[@id='content']/h2/text()").get(),
            'Description': response.xpath("//div[@class='description']/p/text()").get(),
            'URL': response.url,
            'Dimension': response.xpath('//tr/td[text()="Dimensions"]/following-sibling::td[@class="value"]/text()').get(),
        }

防止抓取空白页面：

使用 CrawlSpider 的解决方案：

import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
from scrapy.exceptions import CloseSpider


def get_next_page(value):
    next_page = value.replace('browse/', 'browse/insunsh?page=')
    return next_page


class ToscrapeSpider(CrawlSpider):
    name = 'toscrape'
    allowed_domains = ['pstrial-2019-12-16.toscrape.com']
    start_urls = ['http://pstrial-2019-12-16.toscrape.com/browse/insunsh']

    rule_articles_page = Rule(LinkExtractor(restrict_xpaths="//div[@id='body']/div[2]/a", deny=('/tarpit/die-scrapers-die')), callback='parse_item', follow=False)
    rule_next_page = Rule(LinkExtractor(restrict_xpaths="//form[@class='nav next']/input[1]", tags=('input'), attrs=('value',),
                                        process_value=get_next_page),
                          follow=True, callback='check_empty_page')

    rules = (
        rule_articles_page,
        rule_next_page,
    )

    def parse_item(self, response):
        yield {
            'Image': response.xpath("//div[@id='body']/img/@src").get(),
            'Title': response.xpath("//div[@id='content']/h1/text()").get(),
            'artist': response.xpath("//div[@id='content']/h2/text()").get(),
            'Description': response.xpath("//div[@class='description']/p/text()").get(),
            'URL': response.url,
            'Dimension': response.xpath('//tr/td[text()="Dimensions"]/following-sibling::td[@class="value"]/text()').get(),
        }


    def check_empty_page(self, response):
        # if the number of urls is less than or equal to 1 then close the spider
        # (The first url is always '/tarpit/die-scrapers-die' so there's always at least one url)
        if len(response.xpath("//div[@id='body']/div[2]/a")) <= 1:
            raise CloseSpider("Finished")

get_next_page 函数说明：Scrapy 返回 'http://pstrial-2019-12-16.toscrape.com/browse/' 的基本 url 并添加页面值，例如第一页的值将是“http://pstrial-2019-12-16.toscrape.com/browse/1”。所以我们将其替换为 'http://pstrial-2019-12-16.toscrape.com/browse/insunsh?page=1' 并返回值。

用scrapy.Spider解决：

import scrapy


class ToscrapeSpider(scrapy.Spider):
    name = 'toscrape'
    allowed_domains = ['pstrial-2019-12-16.toscrape.com']
    start_urls = ['http://pstrial-2019-12-16.toscrape.com/browse/insunsh']

    def parse(self, response, **kwargs):
        # (The first url is always '/tarpit/die-scrapers-die' so there's always at least one url)
        if len(response.xpath("//div[@id='body']/div[2]/a")) > 1:
            for url in response.xpath('//div[@id="body"]/div[2]/a/@href').getall():
                yield response.follow(url=url, callback=self.parse_item)

            next_page = response.xpath('//form[@class="nav next"]/input[1]/@value').get()

            next_page = f'{self.start_urls[0]}?page={next_page}'
            yield scrapy.Request(url=next_page, callback=self.parse)

    def parse_item(self, response):
        yield {
            'Image': response.xpath("//div[@id='body']/img/@src").get(),
            'Title': response.xpath("//div[@id='content']/h1/text()").get(),
            'artist': response.xpath("//div[@id='content']/h2/text()").get(),
            'Description': response.xpath("//div[@class='description']/p/text()").get(),
            'URL': response.url,
            'Dimension': response.xpath('//tr/td[text()="Dimensions"]/following-sibling::td[@class="value"]/text()').get(),
        }

【讨论】：

感谢您的帮助。当我运行此代码时，它完美地转到下一页。但是当它到达第 9 页时，它不会进入下一页，并且扫描的项目总数为 100
@DeepakKumar 我的错。我只取了字符串中的最后一个值。我用固定代码编辑了答案。如果您愿意，也可以使用 lambda 编写它。
再次感谢。你能解释一下下一页代码是如何工作的吗？我是 python 新手。以及爬取完所有文章后如何停止爬取。因为毕竟文章这个网站的空白页面带有下一个和预览输入标签。
@DeepakKumar 我添加了一个解释。我用 Spider 和 CrawlSpider 添加了一个可行的解决方案。如果您对答案感到满意，请接受它。阅读有关 CloseSpider 的 scrapy 文档。