下载图像会导致值重复并给出错误

Question

Spider 运行正常并在 mongodb 中保存数据，但突然开始在数据库以及 json 和 csv 中保存重复值，我删除了下载图像的代码，它工作正常，但我需要图像，谁能帮帮我？ 提前致谢。

items.py

导入scrapy

class BucketItem(scrapy.Item):
# define the fields for your item here like:
store_name = scrapy.Field()
category = scrapy.Field()
sub_category = scrapy.Field()
name = scrapy.Field()
unit = scrapy.Field()
price = scrapy.Field()
link = scrapy.Field()
image_urls = scrapy.Field()
images = scrapy.Field()
pass

管道.py

导入 pymongo

类 BucketPipeline(object):

def __init__(self):
    self.conn = pymongo.MongoClient(
        'localhost', 27017
    )
    db = self.conn['bucket']
    self.collection = db['products_tb']

def process_item(self, item, spider):
    self.collection.insert(dict(item))
    return item

蜘蛛.py

        image_url = [product.css('.productVisuals img::attr(src)').extract_first()]

        items = response.meta['items']
        items['name'] = product_name
        items['unit'] = unit
        items['price'] = product_price
        items['link'] = product_link
        items['image_urls'] = image_url
        # print(items['name'])
        yield items

[scrapy.pipelines.files] 警告：文件（代码：400）：从 https://qne.com.pk/../product_images/14768.jpg> 下载文件时出错

Answer 1

不确定您下载的来源，但除了“src”属性之外，您可能会发现“data-src”属性... 如果“src”有一个标准的共同点，您可能想尝试使用 if 语句来捕获“data-src”。

    if product.xpath('.//div..../img[contains(@src, "some_common_text")]'):
     item['image_urls'] = product.xpath('.//div...../img/@data-src').extract()
else:
     item['image_urls'] = product.xpath('.//div...../img/@src').extract()