Scrapy 错误（不支持的 url 方案 javascript）答案

【问题标题】：Scrapy error (Unsupported url scheme javascript)Scrapy 错误（不支持的 url 方案 javascript）
【发布时间】：2021-05-14 21:44:15
【问题描述】：

我是 Scrapy 的新手。目前使用刮痧 1.8。我能够抓取数据并使用以下命令将其输出到 json 文件中。

scrapy crawl jokes -o data.json

不幸的是，发生了一些错误，如下所示。我该如何解决所有这些错误？

import scrapy

class JokesSpider(scrapy.Spider):
    name = 'jokes'

    start_urls = [
        'http://www.laughfactory.com/jokes/family-jokes'
    ]

    def parse(self, response):
        for joke in response.xpath("//div[@class='jokes']"):
            yield {
                'joke_text': joke.xpath(".//div[@class='joke-text']/p").get()
            }

        next_page = response.xpath("//li[@class='next']/a/@href").get()
        if next_page is not None:
            next_page_link = response.urljoin(next_page)
            yield scrapy.Request(url=next_page_link, callback=self.parse)

错误 1：（可以通过将 ROBOTSTXT_OBEY 设置为 false 来修复，但我认为这不是正确的方法）

ERROR: Error downloading <GET javascript:/robots.txt>: Unsupported URL scheme 'javascript': no handler available for that scheme

错误 2：

ERROR: Error downloading <GET javascript:void(0)>
Traceback (most recent call last):
  File "c:\users\user\desktop\project\scrapycourse\virtual_env\lib\site-packages\twisted\internet\defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "c:\users\user\desktop\project\scrapycourse\virtual_env\lib\site-packages\twisted\python\failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "c:\users\user\desktop\project\scrapycourse\virtual_env\lib\site-packages\scrapy\core\downloader\middleware.py", line 44, in process_request
    defer.returnValue((yield download_func(request=request, spider=spider)))
  File "c:\users\user\desktop\project\scrapycourse\virtual_env\lib\site-packages\scrapy\utils\defer.py", line 45, in mustbe_deferred
    result = f(*args, **kw)
  File "c:\users\user\desktop\project\scrapycourse\virtual_env\lib\site-packages\scrapy\core\downloader\handlers\__init__.py", line 70, in download_request
    (scheme, self._notconfigured[scheme]))
scrapy.exceptions.NotSupported: Unsupported URL scheme 'javascript': no handler available for that scheme

【问题讨论】：

docs.scrapy.org/en/latest/topics/dynamic-content.html

标签： python scrapy

【解决方案1】：

最后一页结果页面（当前 #13）包含此 HTML：

<li class="next"><a href="javascript:void(0)">NEXT</a></li>

如您所见，class="next" 和 a 与 href。我建议稍微重写你的 XPath 表达式（检查是否有 http 部分）：

next_page = response.xpath("//li[@class='next']/a[contains(@href, 'http')]/@href").get()

【讨论】：