使用scrapy检查pymongo数据库中是否存在item字段

Question

我正在尝试检查item ['email'] 是否已存在于数据库中，如果不存在则插入到 pymongo 数据库中。

我不希望 pymongo 数据库中有重复的电子邮件。

但是我得到了这个

ValueError: 字典更新序列元素#0 的长度为 17； 2 是 必填

这就是我目前所拥有的

管道.py

class myExporter(object):

    def __init__(self):
        i = 0
        while os.path.exists(SRCFILE % i):
            i += 1
        self.filename = SRCFILE % i
        with open(self.filename, 'w') as output:
            output = csv.writer(output)
            output.writerow(['Email', 'Website', 'Phone Number', 'Location'])
        connection = pymongo.MongoClient(settings['MONGODB_HOST'], settings['MONGODB_PORT'])
        db = connection[settings['MONGODB_DATABASE']]
        self.collection = db[settings['MONGODB_COLLECTION']]

    def process_item(self, item, spider):
        self.email = self.collection.find(dict(item['email']))
        for x in self.email:
            if x not in self.email:               
                self.collection.insert(dict(item))
                log.msg("Item wrote to MongoDB database {}, collection {}, at host {}, port {}".format(
                    settings['MONGODB_DATABASE'],
                    settings['MONGODB_COLLECTION'],
                    settings['MONGODB_HOST'],
                    settings['MONGODB_PORT']))
                with open(self.filename, 'a') as output:
                    output = csv.writer(output)
                    output.writerow([item['email'],
                                     item['website'],
                                     item['phonenumber'],
                                     item['location']])
                folder = os.path.join(DESTINATION_FOLDER, os.path.basename(self.filename))
                shutil.copy(self.filename, folder)
                return item

class MongoPipeline(object):

    collection_name = 'scrapy_items'

    def __init__(self, mongo_uri, mongo_db):
        self.mongo_uri = mongo_uri
        self.mongo_db = mongo_db

    @classmethod
    def from_crawler(cls, crawler):
        return cls(
            mongo_uri=crawler.settings.get('MONGO_URI'),
            mongo_db=crawler.settings.get('MONGO_DATABASE', 'items')
        )

    def open_spider(self, spider):
        self.client = pymongo.MongoClient(self.mongo_uri)
        self.db = self.client[self.mongo_db]

    def close_spider(self, spider):
        self.client.close()

    def process_item(self, item, spider):
        self.db[self.collection_name].insert_one(dict(item))
        return item

Answer 1

dropDups = True 不会保存已添加到数据库的电子邮件

self.collection.ensure_index('email', unique=True, dropDups=True)

完成的代码

def process_item(self, item, spider):
    self.email = self.collection.find({"email": item['email']})
    self.collection.ensure_index('email', unique=True, dropDups=True)
    self.collection.insert(dict(item))
    log.msg("Item wrote to MongoDB database {}, collection {}, at host {}, port {}".format(
        settings['MONGODB_DATABASE'],
        settings['MONGODB_COLLECTION'],
        settings['MONGODB_HOST'],
        settings['MONGODB_PORT']))
    with open(self.filename, 'a') as output:
        output = csv.writer(output)
        output.writerow([item['email'],
                         item['website'],
                         item['phonenumber'],
                         item['location']])
    folder = os.path.join(DESTINATION_FOLDER, os.path.basename(self.filename))
    shutil.copy(self.filename, folder)
    return item

Answer 2

您的错误是在调用 self.collection.find(dict(item['email'])) 时。
item 已经是包含键 email 的字典，所以有 无需用dict() 包装它。使用item['email'] 就可以访问 email 的值了。

然后您应该检查self.email 是否包含所需的结果，然后继续执行其余的函数逻辑。

编辑

您的评论中的新错误表明查找查询中的过滤器需要是字典等。如果您在 Mongo 中寻找 email 字段，请使用 {'email': item['email']}。