【发布时间】：2021-12-27 03:50:46
【问题描述】：
在以下代码中，我尝试使用 Scrapy_Splash 将 JavaScript 页面呈现为 html，但在运行我的蜘蛛时收到以下错误（TCP 连接超时 10060）：
2021-12-26 18:57:19 [scrapy.core.scraper] ERROR: Error downloading <GET https://www.tesla.com/en_ca/models/design#overview via h
ttp://172.17.0.1:8050/render.html>
Traceback (most recent call last):
File "C:\Users\Compuester\anaconda3\envs\scrapy\lib\site-packages\scrapy\core\downloader\middleware.py", line 44, in process_r
equest
return (yield download_func(request=request, spider=spider))
twisted.internet.error.TCPTimedOutError: TCP connection timed out: 10060: A connection attempt failed because the connected part
y did not properly respond after a period of time, or established connection failed because connected host has failed to respond
..
蜘蛛抓取（状态码 200），但 Splash 不会在 JavaScript 渲染后将页面渲染回本地主机地址。
蜘蛛的代码：
import scrapy
from scrapy_splash import SplashRequest
class TeslaSpider(scrapy.Spider):
name = 'tesla'
allowed_domains = ['tesla.com']
start_urls = ['https://www.tesla.com/en_ca/models/design#overview',]
def parse(self, response):
for url in self.start_urls:
yield SplashRequest(
url,
endpoint='render.html',
args={
'wait': 2,
'html': 1,
'timeout': 10,
})
print(response.xpath("//title/text()"))
settings.py 文件：
# Scrapy settings for project project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
# https://docs.scrapy.org/en/latest/topics/settings.html
# https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
# https://docs.scrapy.org/en/latest/topics/spider-middleware.html
BOT_NAME = 'project'
SPIDER_MODULES = ['project.spiders']
NEWSPIDER_MODULE = 'project.spiders'
SPLASH_URL = 'http://172.17.0.1:8050/'
# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'project (+http://www.yourdomain.com)'
# Obey robots.txt rules
ROBOTSTXT_OBEY = True
DOWNLOADER_MIDDLEWARES = {
'scrapy_splash.SplashCookiesMiddleware': 723,
'scrapy_splash.SplashMiddleware': 725,
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}
SPIDER_MIDDLEWARES = {
'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,
}
DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'
HTTPCACHE_STORAGE = 'scrapy_splash.SplashAwareFSCacheStorage'
*我没有列出用户代理，因为 Splash 为我们创建了一个
将 Scrapy 与 SplashRequests 一起使用是一个困难的学习过程。 我找不到关于如何将来自 Scrapy_Splash/Splash Requests 的命令集成到 python 中的明确文档，只有来自 Splash 的文档提供了有关 Lua 脚本和 R 语言的详细信息以及一些示例——如果我在这里死胡同，这是我的下一个计划.
我遇到的一个特别困难的障碍是了解到使用来自 scrapy_splash 的 SplashRequests 会将“Yield”语句放入解析函数中，而来自 import 的 Splash meta 类Splash 看到需要在前面的函数中使用 yield 语句，例如
def start_requests(self)
我期待听到有关此问题的想法和想法。谢谢
【问题讨论】：
标签： python scrapy scrapy-splash