scrapy采集gb2312网页中文乱码笔记

由于目标采集资源为gb2312发生乱码，采用中间件的解决方式，中间件为DownloaderMiddleware

1     def process_response(self, request, response, spider):
2         # Called with the response returned from the downloader.
3         # Must either;
4         # - return a Response object
5         # - return a Request object
6         response = HtmlResponse(url=response.url, body=response.body, encoding='utf-8')
7         # - or raise IgnoreRequest
8         return response

即在下载网页阶段是将网页转换为utf-8格式，另外需要将中间激活，在配置文件settings.py文件中插入代码，以激活

1 DOWNLOADER_MIDDLEWARES = {'news.middlewares.NewsDownloaderMiddleware': 1000}

至此，爬虫文件中不需要进行额外的转码，即可正常显示中文了