【发布时间】：2021-09-23 09:38:50
【问题描述】：
我的 scrapy spider 总是在 Kubernetes pod 中的第 1000 个请求时停止。我找不到任何问题。它只是关闭了我的蜘蛛。
我已经在终端和本地docker中测试过，没有问题。
请帮我处理一下。
2021-09-23 09:36:41 [scrapy.core.engine] INFO: Closing spider (finished)
2021-09-23 09:36:41 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 360643,
'downloader/request_count': 1003,
'downloader/request_method_count/GET': 1000,
'downloader/request_method_count/POST': 3,
'downloader/response_bytes': 2597069,
'downloader/response_count': 1003,
'downloader/response_status_count/200': 1000,
'downloader/response_status_count/404': 3,
'elapsed_time_seconds': 85.16985,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2021, 9, 23, 9, 36, 41, 366720),
'httpcompression/response_bytes': 4896324,
'httpcompression/response_count': 997,
'item_scraped_count': 1000,
'log_count/DEBUG': 5137,
'log_count/INFO': 3016,
'log_count/WARNING': 1,
'memusage/max': 111157248,
'memusage/startup': 92839936,
'request_depth_max': 1,
'response_received_count': 1003,
'scheduler/dequeued': 1006,
'scheduler/dequeued/memory': 1006,
'scheduler/enqueued': 1006,
'scheduler/enqueued/memory': 1006,
'splash/render.html/request_count': 3,
'splash/render.html/response_count/200': 3,
'start_time': datetime.datetime(2021, 9, 23, 9, 35, 16, 196870)}
2021-09-23 09:36:41 [scrapy.core.engine] INFO: Spider closed (finished)
【问题讨论】：
标签： kubernetes scrapy scrapy-splash