我不知道为什么不喜欢错误消息是多行的（它是异常回溯的打印）。无论如何，我们可以自定义 scrapy 日志记录的格式。假设您通过scrapy 命令行运行您的爬虫脚本，例如scrapy crawl 或scrapy runspider。这是一个示例代码（python 3 版本），展示了如何使用您自己的格式化程序。
import logging
import scrapy
class OneLineFormatter(logging.Formatter):
def __init__(self, *args, **kwargs):
super(OneLineFormatter, self).__init__(*args, **kwargs)
def format(self, record):
formatted = super(OneLineFormatter, self).format(record)
return formatted.replace('\n', ' ')
class TestSpider(scrapy.Spider):
name = "test"
start_urls = [
'http://www.somenxdomain.com/robots.txt',
]
def __init__(self, fmt, datefmt, *args, **kwargs):
my_formatter = OneLineFormatter(fmt=fmt, datefmt=datefmt)
root = logging.getLogger()
for h in root.handlers:
h.setFormatter(my_formatter)
super(TestSpider, self).__init__(*args, **kwargs)
@classmethod
def from_crawler(cls, crawler):
settings = crawler.settings
return cls(settings.get('LOG_FORMAT'), settings.get('LOG_DATEFORMAT'))
def parse(self, response):
pass
以下是一些解释。
Python 日志记录工作流程。 scrapy 本身使用 python 内置的日志系统。因此你需要一些python日志的基础知识，尤其是Logger、Handler、Filter和Formatter类之间的关系。我强烈推荐python日志的working flow。
Scrapy 日志记录和设置。如果你的爬虫是通过scrapy 命令行运行的，例如scrapy crawl 或scrapy runspider，那么会调用scrapy 函数[configure_logging](https://docs.python.org/2/howto/logging.html#logging-flow) 来初始化日志记录。 scrapy logging 的说明可以提供一些有关如何自定义日志记录的说明，通过scrapy settings 您可以访问您的设置。
-
示例代码的工作原理。基本工作流程是：
- 首先，您需要定义自己的格式化程序类来自定义日志格式。
- 其次，在您的爬虫中，您需要访问您的格式化设置以初始化您的格式化程序类。
- 最后，在您的爬虫中，您将获得
root 记录器并将您的格式化程序设置为root 的所有处理程序。
如果您编写自己的脚本并使用 scrapy 作为 API，请参阅 [从脚本运行 scrapy] (https://doc.scrapy.org/en/latest/topics/practices.html#run-scrapy-from-a-script)，那么您需要自己配置日志记录。
在蜘蛛初始化之前，上述格式化程序将无法工作。以下是一些印刷品：
2017-10-03 11:59:39 [scrapy.utils.log] INFO: Scrapy 1.3.3 started (bot: scrapybot)
2017-10-03 11:59:39 [scrapy.utils.log] INFO: Overridden settings: {'SPIDER_LOADER_WARN_ONLY': True}
2017-10-03 11:59:39 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.logstats.LogStats']
2017-10-03 11:59:39 [scrapy.middleware] INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2017-10-03 11:59:39 [scrapy.middleware] INFO: Enabled spider middlewares: ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2017-10-03 11:59:39 [scrapy.middleware] INFO: Enabled item pipelines: []
2017-10-03 11:59:39 [scrapy.core.engine] INFO: Spider opened
2017-10-03 11:59:39 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-10-03 11:59:39 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6024
2017-10-03 11:59:39 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://www.somenxdomain.com/robots.txt> (failed 1 times): DNS lookup failed: no results for hostname lookup: www.somenxdomain.com.
2017-10-03 11:59:39 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://www.somenxdomain.com/robots.txt> (failed 2 times): DNS lookup failed: no results for hostname lookup: www.somenxdomain.com.
2017-10-03 11:59:39 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET http://www.somenxdomain.com/robots.txt> (failed 3 times): DNS lookup failed: no results for hostname lookup: www.somenxdomain.com.
2017-10-03 11:59:39 [scrapy.core.scraper] ERROR: Error downloading <GET http://www.somenxdomain.com/robots.txt> Traceback (most recent call last): File "/Users/xxx/anaconda/envs/p3/lib/python3.6/site-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks result = result.throwExceptionIntoGenerator(g) File "/Users/xxx/anaconda/envs/p3/lib/python3.6/site-packages/twisted/python/failure.py", line 393, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) File "/Users/xxx/anaconda/envs/p3/lib/python3.6/site-packages/scrapy/core/downloader/middleware.py", line 43, in process_request defer.returnValue((yield download_func(request=request,spider=spider))) File "/Users/xxx/anaconda/envs/p3/lib/python3.6/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks current.result = callback(current.result, *args, **kw) File "/Users/xxx/anaconda/envs/p3/lib/python3.6/site-packages/twisted/internet/endpoints.py", line 954, in startConnectionAttempts "no results for hostname lookup: {}".format(self._hostStr) twisted.internet.error.DNSLookupError: DNS lookup failed: no results for hostname lookup: www.somenxdomain.com.
2017-10-03 11:59:40 [scrapy.core.engine] INFO: Closing spider (finished)
2017-10-03 11:59:40 [scrapy.statscollectors] INFO: Dumping Scrapy stats: {'downloader/exception_count': 3, 'downloader/exception_type_count/twisted.internet.error.DNSLookupError': 3, 'downloader/request_bytes': 684, 'downloader/request_count': 3, 'downloader/request_method_count/GET': 3, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2017, 10, 3, 15, 59, 40, 46636), 'log_count/DEBUG': 4, 'log_count/ERROR': 1, 'log_count/INFO': 7, 'scheduler/dequeued': 3, 'scheduler/dequeued/memory': 3, 'scheduler/enqueued': 3, 'scheduler/enqueued/memory': 3, 'start_time': datetime.datetime(2017, 10, 3, 15, 59, 39, 793795)}
2017-10-03 11:59:40 [scrapy.core.engine] INFO: Spider closed (finished)
你可以看到蜘蛛运行后，所有的消息都被格式化成一行。 （通过删除'\n'）。
希望这会有所帮助。