【发布时间】：2022-01-02 14:45:14
【问题描述】：
我现在正在学习 Scrapy Splash，并通过教程构建一个非常基本的蜘蛛。
但是运行蜘蛛 scrapy crawl quotes -o results_file.csv 它只返回一个空的 csv 文件。
也说爬取了0个网站，所以好像没找到网页？
我使用的是 Windows 10 家庭版
我仔细检查了 xpath 表达式，并且 splash 正在处理 http://localhost:8050/
我知道我在这里遗漏了一些非常简单的东西......你能帮忙吗？
我把这个放到settings.py中
DOWNLOADER_MIDDLEWARES = {
'scrapy_splash.SplashCookiesMiddleware': 723,
'scrapy_splash.SplashMiddleware': 725,
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}
SPLASH_URL = 'http://localhost:8050/'
DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'
HTTPCACHE_STORAGE = 'scrapy_splash.SplashAwareFSCacheStorage'
这是我正在运行的蜘蛛：
import scrapy
from scrapy_splash import SplashRequest
class QuotesSpider(scrapy.Spider):
name = 'quotes'
allowed_domains = ['quotes.toscrape.com']
start_urls = ['http://quotes.toscrape.com/js/']
def start_requests(self):
for url in self.start_urls:
yield SplashRequest(url=url,
callback=self.parse,
endpoint='render.html')
def parse(self, response):
quotes = response.xpath('//*[@class="quote"]')
for quote in quotes:
yield {'author': quote.xpath('.//*[@class="author"]/text()').extract_first(),
'quote': quote.xpath('.//*[@class="text"]/text()').extract_first()}
这是我得到的输出：
2022-01-02 15:32:45 [scrapy.utils.log] INFO: Scrapy 2.4.1 started (bot: quotes_spider)
2022-01-02 15:32:45 [scrapy.utils.log] INFO: Versions: lxml 4.7.1.0, libxml2 2.9.12, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 21.7.0, Python 3.9.7 (default, Sep 16 2021, 16:59:28) [MSC v.1916 64 bit (AMD64)], pyOpenSSL 21.0.0 (OpenSSL 1.1.1f 31 Mar 2020), cryptography 36.0.0, Platform Windows-10-10.0.19042-SP0
2022-01-02 15:32:45 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
2022-01-02 15:32:45 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'quotes_spider',
'DUPEFILTER_CLASS': 'scrapy_splash.SplashAwareDupeFilter',
'HTTPCACHE_STORAGE': 'scrapy_splash.SplashAwareFSCacheStorage',
'NEWSPIDER_MODULE': 'quotes_spider.spiders',
'SPIDER_MODULES': ['quotes_spider.spiders']}
2022-01-02 15:32:45 [scrapy.extensions.telnet] INFO: Telnet Password: 4cc7c27ecac6bf1d
2022-01-02 15:32:45 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.feedexport.FeedExporter',
'scrapy.extensions.logstats.LogStats']
2022-01-02 15:32:45 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy_splash.SplashCookiesMiddleware',
'scrapy_splash.SplashMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2022-01-02 15:32:45 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2022-01-02 15:32:45 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2022-01-02 15:32:45 [scrapy.core.engine] INFO: Spider opened
2022-01-02 15:32:45 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-01-02 15:32:45 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2022-01-02 15:32:46 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/> (referer: None)
2022-01-02 15:32:46 [scrapy.core.engine] INFO: Closing spider (finished)
2022-01-02 15:32:46 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 223,
'downloader/request_count': 1,
'downloader/request_method_count/GET': 1,
'downloader/response_bytes': 2200,
'downloader/response_count': 1,
'downloader/response_status_count/200': 1,
'elapsed_time_seconds': 0.569004,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2022, 1, 2, 14, 32, 46, 398295),
'log_count/DEBUG': 1,
'log_count/INFO': 10,
'response_received_count': 1,
'scheduler/dequeued': 1,
'scheduler/dequeued/memory': 1,
'scheduler/enqueued': 1,
'scheduler/enqueued/memory': 1,
'start_time': datetime.datetime(2022, 1, 2, 14, 32, 45, 829291)}
2022-01-02 15:32:46 [scrapy.core.engine] INFO: Spider closed (finished)
编辑
所以这不是代码本身 - 感谢您清除它！
Docker + Splash 正在运行 @ 0.0.0.0:8050
我需要为 Windows Home 中的 Splash+Scrapy 配置什么我可能错过了它才能正常运行吗？
【问题讨论】：
标签： python scrapy splash-screen scrapy-splash