无法使用 Scrapy 下载图像

Question

我无法下载图片。我有几个问题（我尝试了很多变化）。这是我的代码（我猜它有很多错误）

目标是抓取起始 URL 并保存所有产品图片并通过 SKU 编号更改其名称。此外，蜘蛛必须单击“下一步按钮”才能在所有页面中执行相同的任务（大约有 24.000 个产品）

我注意到的问题是：

1. 我不知道 Items Pipelines 的确切配置
2. 图片不会下载到 settings.py 的文件夹中
3. 我想按分辨率过滤图像并使用缩略图。推荐的配置是哪一种？
4. 图像位于另一台服务器上。这是个问题？

SETTINGS.PY

BOT_NAME = 'soarimages'

SPIDER_MODULES = ['soarimages.spiders']
NEWSPIDER_MODULE = 'soarimages.spiders'
DEFAULT_ITEM_CLASS = 'soarimages.items'
ITEM_PIPELINES = {'soarimages.pipelines.soarimagesPipeline': 1}
IMAGES_STORE = '/soarimages/images'

ITEMS.PY

import scrapy

class soarimagesItem(scrapy.Item):
    title = scrapy.Field()
    image_urls = scrapy.Field()
    images = scrapy.Field()

PIPELINES.PY

import scrapy
from scrapy.contrib.pipeline.images import ImagesPipeline

class soarimagesPipeline(ImagesPipeline):

def set_filename(self, response):
    #add a regex here to check the title is valid for a filename.
    return 'full/{0}.jpg'.format(response.meta['title'][0])

def get_media_requests(self, item, info):
    for image_url in item['image_urls']:
        yield scrapy.Request(image_url, meta={'title': item['title']})

def get_images(self, response, request, info):
    for key, image, buf in super(soarimagesPipeline, self).get_images(response, request, info):
        key = self.set_filename(response)
    yield key, image, buf

Productphotos.PY（蜘蛛）

# import the necessary packages
import scrapy
from scrapy.spiders import Rule, CrawlSpider
from scrapy.linkextractors import LinkExtractor
from soarimages.items import soarimagesItem

class soarimagesSpider(scrapy.Spider):
name = 'productphotos'
allowed_domains = ['http://sodimac.com.ar','http://sodimacar.scene7.com']
start_urls = ['http://www.sodimac.com.ar/sodimac-ar/search/']
rules = [Rule(LinkExtractor(allow=['http://sodimacar.scene7.com/is/image//SodimacArgentina/.*']), 'parse')]

def parse(self, response):
    SECTION_SELECTOR = '.one-prod'
    for soarimages in response.css(SECTION_SELECTOR):
        image = soarimagesItem()
        image['title'] = response.xpath('.//p[@class="sku"]/text()').re_first(r'SKU:\s*(.*)').strip(),
        rel = response.xpath('//div/a/img/@data-original').extract_first()
        image['image_urls'] = ['http:'+rel[0]]
        yield image

    NEXT_PAGE_SELECTOR = 'a.next ::attr(href)'
    next_page = response.css(NEXT_PAGE_SELECTOR).extract_first()
    if next_page:
        yield scrapy.Request(
            response.urljoin(next_page),
            callback=self.parse
        )

Answer 1

这是我的代码（我猜它有很多错误）

事实上，我至少可以发现一个错误：allowed_domains 应该只列出域。不得包含任何 http:// 前缀：

allowed_domains = ['sodimac.com.ar', 'sodimacar.scene7.com']

您可能想解决这个问题并测试您的蜘蛛。如果出现新问题，请为每个特定问题创建特定问题。这样可以更轻松地帮助您。另见how to ask