Scrapy中添加随机User-Agent

方法：使该模块scrapy-fake-useragent

1，pip install scrapy-fake-useragent

2，在setting.py 写：

DOWNLOADER_MIDDLEWARES = {
    'lagoujob.middlewares.RandomUesrAgent': 1,
    'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
}

3，在 middlewares.py 写入：

class RandomUesrAgent(object):
    def process_request(self, request, spider):
        ua = UserAgent()
        request.headers.setdefault("User-Agent", ua.random)

　或者：

然后就可以用啦