Scrapy实战：使用scrapy再再次爬取干货集中营的妹子图片

 需要学习的知识：

1.获取到的json数据如何处理

2.保存到json文件

3.保存到MongoDB数据库

4.下载项目图片(含缩略图)

 

1.创建项目

scrapy startproject gank

2.生成项目爬虫文件

scrapy genspider gank_img gank.io

注意：项目名称gank不能跟项目爬虫文件名gank_img一致

3.gank_img.py文件

import json
import scrapy
from gank.items import GankItem


class GankImgSpider(scrapy.Spider):
    name = \'gank_img\'
    allowed_domains = [\'gank.io\']
    # 开始链接为什么要这样写请参考：https://www.cnblogs.com/sanduzxcvbnm/p/10271493.html
    start_urls = [\'https://gank.io/api/data/福利/700/1\']

    def parse(self, response):
        # 返回的是json字符串，转换成字典，提取出需要的字段
        results = json.loads(response.text)[\'results\']

        for i in results:
            item = GankItem()
            item[\'who\'] = i[\'who\']
            item[\'url\'] = i[\'url\']

            yield item

4.items.py文件

import scrapy

class GankItem(scrapy.Item):
    # define the fields for your item here like:
    who = scrapy.Field()
    url = scrapy.Field()
    # 保存图片,生成图片路径
    image_paths = scrapy.Field()

5.pipelines.py文件

import json
from scrapy.pipelines.images import ImagesPipeline
from scrapy.exceptions import DropItem
import pymongo
import scrapy


# 在settings.py文件中开启该pipeline，则主程序中yield的数据会传输到这边来进行处理

# 保存成json文件
class JsonWriterPipeline(object):

    def open_spider(self, spider):
        self.file = open(\'items.json\', \'w\')

    def close_spider(self, spider):
        self.file.close()

    def process_item(self, item, spider):
        line = json.dumps(dict(item)) + "\n"
        self.file.write(line)
        return item


# 保存到MongoDB数据库
class MongoPipeline(object):
    # 数据表名
    collection_name = \'scrapy_items\'

    def __init__(self, mongo_uri, mongo_db):
        self.mongo_uri = mongo_uri
        self.mongo_db = mongo_db

    @classmethod
    # 从settings.py文件中获取参数
    def from_crawler(cls, crawler):
        return cls(
            mongo_uri=crawler.settings.get(\'MONGO_URI\'),
            mongo_db=crawler.settings.get(\'MONGO_DATABASE\', \'items\') # 数据库名
        )

    def open_spider(self, spider):
        self.client = pymongo.MongoClient(self.mongo_uri)
        self.db = self.client[self.mongo_db]

    def close_spider(self, spider):
        self.client.close()

    def process_item(self, item, spider):
        self.db[self.collection_name].insert_one(dict(item))
        return item

# 下载项目图片
class MyImagesPipeline(ImagesPipeline):
    def get_media_requests(self, item, info):
        # 图片链接是https的转换成http
        if item[\'url\'][0:5] == \'https\':
            item[\'url\'] = item[\'url\'].replace(item[\'url\'][0:5], \'http\')
        # for image_url in item[\'url\']:
        #     print(\'400\',image_url)
        yield scrapy.Request(item[\'url\'])

    def item_completed(self, results, item, info):
        image_paths = [x[\'path\'] for ok, x in results if ok]
        if not image_paths:
            raise DropItem("Item contains no images")
        item[\'image_paths\'] = image_paths
        return item

6.settings.py文件

只修改如下配置，其余保持不变

DEFAULT_REQUEST_HEADERS = {
    \'Accept\': \'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\',
    \'Accept-Language\': \'zh-CN,en-US;q=0.8,zh;q=0.5,en;q=0.3\',
    \'User-Agent\': \'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0\'
}

# MongoDB数据库参数
MONGO_URI = \'127.0.0.1\'
MONGO_DATABASE = \'gank\'

ITEM_PIPELINES = {
    \'gank.pipelines.JsonWriterPipeline\': 300,
    \'gank.pipelines.MyImagesPipeline\': 1,
    \'gank.pipelines.MongoPipeline\': 400,
}
# 图片保存路径
IMAGES_STORE = \'D:\\gank\\images\'

# 90天的图片失效期限
IMAGES_EXPIRES = 90

# 缩略图
IMAGES_THUMBS = {
    \'small\': (50, 50),
    \'big\': (270, 270),
}

7.执行爬虫程序

scrapy crawl gank_img

8.效果

json文件

MongoDB数据库

保存的图片及缩略图

其中full为图片本身大小所存放目录，thubmbs为缩略图存放目录，缩略图有big和small两种尺寸

 scrapy结尾会有相应的统计信息

下载图片561个，无法下载的图片有108个

为什么有的图片无法下载，请参考之前的文章：https://www.cnblogs.com/sanduzxcvbnm/p/10271493.html