Scrapy 没有正确通过 url答案

【问题标题】：Scrapy not going through urls properlyScrapy 没有正确通过 url
【发布时间】：2021-10-28 00:34:37
【问题描述】：

干杯，我一直在关注一个scrapy tut，并且在编写了与教程中完全相同的代码以scrape网站“引用scrape”之后，脚本并没有遍历它只想到第一页，这是脚本：

import scrapy


class QuotesSpider(scrapy.Spider):
    name = 'quotes'
    allowed_domains = ['quotes.toscrape.com/']
    start_urls = ['http://quotes.toscrape.com//']

    def parse(self, response):

        quotes = response.xpath('//*[@class="quote"]')

        for quote in quotes :
            text = quote.xpath('.//*[@class="text"]/text()').extract_first()
            author = quote.xpath('.//*[@class="author"]/text()').extract_first()
            tags = quote.xpath('.//*[@class="keywords"]/@content').extract_first()

            yield{
                'text':text,
                'author':author,
                'tags':tags}
      
       
            next_page_url = response.xpath('//*[@class="next"]/a/@href').extract_first() 
            absolute_next_page_url = response.urljoin(next_page_url)
            yield scrapy.Request(absolute_next_page_url)

感谢所有帮助并感谢！

【问题讨论】：

你的控制台有什么输出？
它显示第一页的内容：

标签： python web-scraping xpath scrapy yield

【解决方案1】：

这是完整的工作输出。它是 start_urls 中额外的斜线（/），并且分页代码块在 for 循环中。

import scrapy


class QuotesSpider(scrapy.Spider):
    name = 'quotes'
    allowed_domains = ['quotes.toscrape.com']
    start_urls = ['http://quotes.toscrape.com/']

    def parse(self, response):

        quotes = response.xpath('//*[@class="quote"]')

        for quote in quotes :
            text = quote.xpath('.//*[@class="text"]/text()').extract_first()
            author = quote.xpath('.//*[@class="author"]/text()').extract_first()
            tags = quote.xpath('.//*[@class="keywords"]/@content').extract_first()

            yield{
                'text':text,
                'author':author,
                'tags':tags}
      
       
        next_page_url = response.xpath('//*[@class="next"]/a/@href').extract_first() 
        absolute_next_page_url = response.urljoin(next_page_url)
        yield scrapy.Request(absolute_next_page_url)

输出：

{'text': '“You have to write the book that wants to be written. And if the book will be too difficult for grown-ups, then you write it for children.”', 'author': "Madeleine L'Engle", 'tags': 'books,children,difficult,grown-ups,write,writers,writing'}
2021-08-28 18:15:19 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/10/>
{'text': '“Never tell the truth to people who are not worthy of it.”', 'author': 'Mark Twain', 'tags': 'truth'}     
2021-08-28 18:15:19 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/10/>
{'text': "“A person's a person, no matter how small.”", 'author': 'Dr. Seuss', 'tags': 'inspirational'}
2021-08-28 18:15:19 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/10/>
{'text': '“... a mind needs books as a sword needs a whetstone, if it is to keep its edge.”', 'author': 'George R.R. Martin', 'tags': 'books,mind'}
2021-08-28 18:15:19 [scrapy.dupefilters] DEBUG: Filtered duplicate request: <GET http://quotes.toscrape.com/page/10/> - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)
2021-08-28 18:15:19 [scrapy.core.engine] INFO: Closing spider (finished)
2021-08-28 18:15:19 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 3462,
 'downloader/request_count': 10,
 'downloader/request_method_count/GET': 10,
 'downloader/response_bytes': 23058,
 'downloader/response_count': 10,
 'downloader/response_status_count/200': 10,
 'dupefilter/filtered': 1,
 'elapsed_time_seconds': 64.609554,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2021, 8, 28, 12, 15, 19, 176318),
 'httpcompression/response_bytes': 108561,
 'httpcompression/response_count': 10,
 'item_scraped_count': 100,

【讨论】：