scrapy 下载器中间件中的重试请求答案

【问题标题】：Retry request in scrapy downloader middlewarescrapy 下载器中间件中的重试请求
【发布时间】：2020-02-05 17:33:52
【问题描述】：

我使用 scrapoxy，它在 scraping 时实现 IP 轮换。

我有一个BLACKLIST_HTTP_STATUS_CODES 状态码列表，表明当前 IP 已被阻止。

问题：一旦您在BLACKLIST_HTTP_STATUS_CODES scrapoxy 下载器中间件中收到带有状态码的响应，就会引发 IgnoreRequest，然后更改 IP。结果，我的脚本跳过了响应状态码错误的 url。

日志示例：

[scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.some-website.com/profile/190> (referer: None)
[scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.some-website.com/profile/191> (referer: None)
[scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.some-website.com/profile/192> (referer: None)
[spider] DEBUG: Ignoring Blacklisted response https://www.some-website.com/profile/193: HTTP status 429
[urllib3.connectionpool] DEBUG: Starting new HTTP connection (1): 13.33.33.37:8889
[urllib3.connectionpool] DEBUG: http://13.33.33.37:8889 "POST /api/instances/stop HTTP/1.1" 200 11
[spider] DEBUG: Remove: instance removed (1 instances remaining)
[spider] INFO: Sleeping 89 seconds
[scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.some-website.com/profile/194> (referer: None)
[scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.some-website.com/profile/195> (referer: None)
[scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.some-website.com/profile/196> (referer: None)

结果我的脚本跳过了https://www.some-website.com/profile/193。

目标：我想重试请求，其响应的状态码在BLACKLIST_HTTP_STATUS_CODES，直到它不在该列表中。

我的 DownloaderMiddleware 看起来像这样：

class BlacklistDownloaderMiddleware(object):
     def __init__(self, crawler):
         ...
    
     def from_crawler(cls, crawler):
         ...
    
     def process_response(self, request, response, spider):
        """
        Detect blacklisted response and stop the instance if necessary.
        """
        try:
            # self._http_status_codes is actually BLACKLIST_HTTP_STATUS_CODES
            if response.status in self._http_status_codes:
                # I have defined BlacklistErorr
                raise BlacklistError(response, 'HTTP status {}'.format(response.status))
            return response

        # THIS IS HOW ORIGINAL CODE LOOKS
        except BlacklistError as ex:
            # Some logs
            spider.log('Ignoring Blacklisted response {0}: {1}'.format(response.url, ex.message), level=logging.DEBUG)
            # Get the name of proxy that I need to change
            name = response.headers['x-cache-proxyname'].decode('utf-8')
            # Change the proxy
            self._stop_and_sleep(spider, name)
            # drop the url
            raise IgnoreRequest()

            # MY TRY: I have tried this instead of raising IgnoreRequest but
            # it does not work and asks for arguments spider and
            # response for self.process_response
            # return Request(response.url, callback=self.process_response, dont_filter=True)

【问题讨论】：

你为什么被列入黑名单？您是否没有遵守速率限制？
@SuperStew 我想我达到了一些请求限制。不过，它与问题有何关系

标签： python web-scraping proxy scrapy scrapoxy

【解决方案1】：

您应该复制原始请求，如retry = request.copy()，而不是返回一个新的请求对象。你可以看看Scrapy's RetryMiddleware handles retries.

供您参考：

def _retry(self, request):
    ...
    retryreq = request.copy()
    retryreq.dont_filter = True
    ...
    return retryreq

你可以这样称呼它

def process_response(self, request, response, spider):
    try:
        if response.status in self._http_status_codes:
            name = response.headers['x-cache-proxyname'].decode('utf-8')
            self._stop_and_sleep(spider, name)
            return self._retry(request)
        return response

这应该会给你这个想法。

【讨论】：