从 Scrapy 中的 csv 文件导入 start_urls

Question

我最近开始使用 scrapy 进行网页抓取，我生成了一个我想从其中抓取的 url 列表到一个 txt 文档中，并用新行分隔。这是我的爬虫代码：

import scrapy
import csv
import sys
from realtor.items import RealtorItem

from scrapy.spider import BaseSpider
#from scrapy.selector import HtmlXPathSelector
#from realtor.items import RealtorItem
class RealtorSpider(scrapy.Spider):
    name = "realtor"
    allowed_domains = ["realtor.com"]

    with open('realtor2.txt') as f:
        start_urls = [url.strip() for url in f.readlines()]


    def parse(self, response):
        #hxs = HtmlXPathSelector(response)
        #sites = hxs.select('//div/li/div/a/@href')
        sites = response.xpath('//a[contains(@href, "/realestateandhomes-detail/")]')
        items =  []
        for site in sites: 
            print(site.extract())
            item = RealtorItem()
            item['link'] = site.xpath('@href').extract()
            items.append(item)
        return items

现在我的目标是从 realtor2.txt 读取链接并开始解析它们，但是我在请求 URL 中得到一个 valueError 缺失方案：

  File "C:\Users\Ash\Anaconda2\lib\site-packages\scrapy\http\request\__init__.py", line 58, in _set_url
    raise ValueError('Missing scheme in request url: %s' % self._url)
ValueError: Missing scheme in request url: 
%FF%FEw%00w%00w%00.%00r%00e%00a%00l%00t%00o%00r%00.%00c%00o%00m%00/%00r%00e%00a%00l%00e%00s%00t%00a%00t%00e%00a%00n%00d%00h%00o%00m%00e%00s%00-%00d%00e%00t%00a%00i%00l%00/%005%000%00-%00M%00e%00n%00o%00r%00e%00s%00-%00A%00v%00e%00-%00A%00p%00t%00-%006%001%000%00_%00C%00o%00r%00a%00l%00-%00G%00a%00b%00l%00e%00s%00_%00F%00L%00_%003%003%001%003%004%00_%00M%005%003%008%000%006%00-%005%008%006%007%007%00%0D%00
2017-06-25 22:28:35 [scrapy.core.engine] INFO: Closing spider (finished)

我认为定义start_urls时可能有问题，但我不知道如何进行，

Answer 1

"ValueError: Missing scheme in request url" 表示您缺少 http。
你可以使用urljoin来避免这个问题。