scrapy学习2 爬虫中间件，下载器中间件之添加代理

注意：这些中间件都放在middleware中

下载中间件

 

作用

实例：

 

代理被封，添加代理 

方式一：内置添加代理功能

import os 

    # -*- coding: utf-8 -*-
                import os
                import scrapy
                from scrapy.http import Request

                class ChoutiSpider(scrapy.Spider):
                    name = 'chouti'
                    allowed_domains = ['chouti.com']
                    start_urls = ['https://dig.chouti.com/']

                    def start_requests(self):
                        os.environ['HTTP_PROXY'] = "http://192.168.11.11"

                        for url in self.start_urls:
                            yield Request(url=url,callback=self.parse)

                    def parse(self, response):
                        print(response)

View Code