如何避免“scrapy.exceptions.NotSupported：不支持的 URL 方案''：该方案没有可用的处理程序”答案

【问题标题】：how to avoid "scrapy.exceptions.NotSupported: Unsupported URL scheme '': no handler available for that scheme"如何避免“scrapy.exceptions.NotSupported：不支持的 URL 方案''：该方案没有可用的处理程序”
【发布时间】：2021-02-06 20:08:10
【问题描述】：

我想点击我编写的 xpath 表达式中的链接，但我收到了这个错误消息。如何避免它并成功点击链接。

import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule


class JobSpider(CrawlSpider):
    name = 'job'
    allowed_domains = ['www.irishjobs.ie']
    user_agent = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36"

    #start_urls = ['http://www.irishjobs.ie/ShowResults.aspx?Keywords=python+&autosuggestEndpoint=%2Fautosuggest&Location=0&Category=&Recruiter=Company&Recruiter=Agency&btnSubmit=Search/']

    def start_requests(self):
        yield scrapy.Request(url = "'http://www.irishjobs.ie/ShowResults.aspx?Keywords=python+&autosuggestEndpoint=%2Fautosuggest&Location=0&Category=&Recruiter=Company&Recruiter=Agency&btnSubmit=Search/", headers= {
            'User-Agent': self.user_agent
        })
    rules = (
        Rule(LinkExtractor(restrict_xpaths= "//div[@class='job-result-title']/h2/a"), callback='parse_item', follow=True),
    )

    def parse_item(self, response):
        print(response.url)

【问题讨论】：

标签： python-3.x scrapy

【解决方案1】：

您的代码中有一个简单的错字。试试这个。

def start_requests(self):
        yield scrapy.Request(url = "http://www.irishjobs.ie/ShowResults.aspx?Keywords=python+&autosuggestEndpoint=%2Fautosuggest&Location=0&Category=&Recruiter=Company&Recruiter=Agency&btnSubmit=Search/", headers= {
            'User-Agent': self.user_agent
        })

url 中有额外的' 引号，删除额外的'。那么蜘蛛就OK了。

yield scrapy.Request(url = "'http://www.irishjobs.ie/ShowResults.aspx?
                            ^^^
Keywords=python+&autosuggestEndpoint=%2Fautosuggest&Location=0&Category=&Recruiter=Company&Recruiter=Agency&btnSubmit=Search/"

【讨论】：