Scrapy Image Pipeline不下载图像答案

【问题标题】：Scrapy Image Pipeline not downloading imagesScrapy Image Pipeline不下载图像
【发布时间】：2021-10-25 15:31:00
【问题描述】：

我正在尝试使用 scrapy 来抓取网站以下载图像。当我运行代码时，它运行得很好，但即使我在我的 settings.py 中指定了图像管道 nad 目录，它也不会下载图像

spider.py

import re
import scrapy
import os
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
from ..items import ImagesItem


class ImageSpiderSpider(CrawlSpider):
    name = 'image_spider'
    allowed_domains = ['books.toscrape.com']
    # start_urls = ['http://books.toscrape.com/']

    def start_requests(self):
        url = 'http://books.toscrape.com/'
        yield scrapy.Request(url=url)

    rules = (
        Rule(LinkExtractor(allow=r'catalogue/'), callback='parse_image', follow=True),
    )

    # save_location = os.getcwd()

    custom_settings = {
        "ITEM_PIPELINES": {'scrapy.pipelines.images.ImagesPipeline': 1},
        "IMAGES_STORE": '.images_download/full'
    }

    def parse_image(self, response):
        if response.xpath('//div[@class="item active"]/img').get() is not None:
            img = response.xpath('//div[@class="item active"]/img/@src').get()
            """
            Computing the Absolute path of the image file.
            "image_urls" require absolute path, not relative path
            """
            m = re.match(r"^(?:../../)(.*)$", img).group(1)
            url = "http://books.toscrape.com/"
            img_url = "".join([url, m])
            image = ImagesItem()
            image["image_urls"] = [img_url]  # "image_urls" must be a list
            yield image

items.py

import scrapy


class ImagesItem(scrapy.Item):
    image_urls = scrapy.Field()
    images = scrapy.Field()

settings.py

BOT_NAME = 'images'

SPIDER_MODULES = ['images.spiders']
NEWSPIDER_MODULE = 'images.spiders'

ROBOTSTXT_OBEY = True

ITEM_PIPELINES = {"scrapy.pipelines.images.ImagesPipeline": 1}
IMAGES_STORE = "/Home/PycharmProjects/scrappy/images/images_downloader"

【问题讨论】：

我运行了你的代码，它运行得非常好，它下载了图像。我用 settings.py 和 custom_settings 都试过了，它们都适用。（我使用了 PyCharm 和 scrapy 2.5.0）。
我没有运行它，但首先你必须检查文件夹是否存在，因为它不会自动创建它 - 如果文件夹不存在则它不会下载。第二：您可以使用print() 来查看代码的哪一部分被执行以及变量中有什么——它被称为"print debuging"
我不明白你为什么使用re.match。如果你想创建绝对 URL，那么它有 absolute_url = response.urljoin(relative_url)
你可以用os.getcwd()代替"." - "IMAGES_STORE": "."
标准管道下载到子文件夹full - IMAGES_STORE/full - 所以你应该检查你是否有子文件夹full

标签： python scrapy

【解决方案1】：

我无法将其测试为 project，但我将您的代码测试为 standalone 脚本 - 它对我有用。

我将所有这些代码放在一个文件script.py 并以python script.py 运行它

import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule

class ImagesItem(scrapy.Item):
    image_urls = scrapy.Field()
    images = scrapy.Field()

class ImageSpiderSpider(CrawlSpider):
    name = 'image_spider'

    allowed_domains = ['books.toscrape.com']
    start_urls = ['http://books.toscrape.com/']

    rules = (
        Rule(LinkExtractor(allow=r'catalogue/'), callback='parse_image', follow=True),
    )

    custom_settings = {
        "ITEM_PIPELINES": {'scrapy.pipelines.images.ImagesPipeline': 1},
        "IMAGES_STORE": '.', 
    }
    
    def parse_image(self, response):
        img = response.xpath('//div[@class="item active"]/img/@src').get()
        if img:
            img_url = response.urljoin(img)
            #image = dict()
            image = ImagesItem()
            image["image_urls"] = [img_url]  # "image_urls" must be a list
            yield image


# --- run without project and save in `output.csv` ---

from scrapy.crawler import CrawlerProcess

c = CrawlerProcess({
    #'USER_AGENT': 'Mozilla/5.0',

    # save in file CSV, JSON or XML
    'FEEDS': {'output.csv': {'format': 'csv'}},  # new in 2.1

    #'ITEM_PIPELINES': {'scrapy.pipelines.images.ImagesPipeline': 1},  # used standard ImagePipeline (download to IMAGES_STORE/full)
    #'IMAGES_STORE': '.',                                              # this folder has to exist before downloading
})
c.crawl(ImageSpiderSpider)
c.start()

它会创建子文件夹 full，其中的图像名称类似于 0a007ac89083ad8b68c56ec0f8df5a811e76607c.jpg，因为标准管道使用哈希码作为名称。

它还创建文件output.csv，其中行如下

image_urls,images
http://books.toscrape.com/media/cache/b1/0e/b10eabab1e1c811a6d47969904fd5755.jpg,"[{'url': 'http://books.toscrape.com/media/cache/b1/0e/b10eabab1e1c811a6d47969904fd5755.jpg', 'path': 'full/d78460eb2aa4417e52a8d9850934e35ef6b6117f.jpg', 'checksum': 'e7f8ece4eab2ff898a20ce53b4b50dcb', 'status': 'downloaded'}]"

我也直接在控制台中看到相同的信息

{'image_urls': ['http://books.toscrape.com/media/cache/ee/cf/eecfe998905e455df12064dba399c075.jpg'],
 'images': [{'checksum': '693caff3d97645e73bd28da8e5974946',
             'path': 'full/59d0249d6ae2eeb367e72b04740583bc70f81558.jpg',
             'status': 'downloaded',
             'url': 'http://books.toscrape.com/media/cache/ee/cf/eecfe998905e455df12064dba399c075.jpg'}]}

【讨论】：