Pymongo 低效查询

Question

我目前有以下代码：

houses = self.database[self.database_name][constants.DATABASE_HOUSES_COLLECTION]           

bulk_houses = houses.initialize_unordered_bulk_op()

for house in houses.find().skip(self.from_index).limit(
        constants.MAX_HOUSE_FUNCTION_DOCUMENTS_PER_THREAD):

    house_coords = (house.get("longitude"), house.get("latitude"))

    min = 10000

    for c in self.collection.find({"city": house.get("city")}, {"longtitude": 1, "latitude": 1}):

        collection_coords = (c.get("longitude"), c.get("latitude"))

        distance = geopy.distance.distance(collection_coords, house_coords).km

        if distance < min:
            min = distance

    if min == 10000:
        min = None

    bulk_houses.find({"_id": house.get("_id")}).update(
        {"$set": {f"demography.distanceClosest{translated.get(self.collection.name)}": min}})

bulk_houses.execute()

它的作用是遍历房屋集合中的每个房屋。

对于每个房子，它会遍历已给出的第二个集合，并且只获取经度和纬度。

计算同一城市内的最近距离。

这个函数是多线程的，我这样调用函数：

houses_count = self.houses.count_documents({})

for i in range(len(self.collections)):

    x = 0
    while x < houses_count:
        match_demography_house = MatchDemographyHouse(self.collections[i], self.mongo_db,
                                                        constants.DATABASE_NAME, x,
                                                        x + constants.MAX_HOUSE_FUNCTION_DOCUMENTS_PER_THREAD)
        match_demography_house.add_to_pool(self.match_house_demography_executor)
        x += constants.MAX_HOUSE_FUNCTION_DOCUMENTS_PER_THREAD

而且你可以想象它非常低效。在城市上添加索引可以稍微提高速度，并且仅抓取经度和纬度也会稍微提高速度。

遍历 1000 多所房屋需要 1 分钟，它所遍历的集合有 240 个文档。它目前每个线程有 50 个房子。

Answer 1

试试这个测试工具。在我的机器上，它在不到一秒的时间内运行，没有索引：

import pymongo
import random
import datetime
import geopy.distance

db = pymongo.MongoClient()['testhouses']

db.testhouses.delete_many({})
db.testcollection.delete_many({})

for i in range(1000):
    longitude = random.randint(-89, 89)
    latitude = random.randint(-180, 180)
    city = f'City {i}'
    db.testhouses.insert_one({'city': city, 'longitude': longitude, 'latitude': latitude})
    if i < 240:
        db.testcollection.insert_one({'city': city, 'longitude': longitude, 'latitude': latitude})

start_time = datetime.datetime.now()

bulk_houses = db.testhouses.initialize_unordered_bulk_op()

for house in db.testhouses.find():
    house_coords = (house.get("longitude"), house.get("latitude"))
    minimum = 10000

    for c in db.testcollection.find({"city": house.get("city")}, {"longtitude": 1, "latitude": 1}):
        collection_coords = (c.get("longitude"), c.get("latitude"))
        distance = geopy.distance.distance(collection_coords, house_coords).km

        if distance < minimum: minimum = distance

    if minimum == 10000: minimum = None

    bulk_houses.find({"_id": house.get("_id")}).update({"$set": {f"demography.distanceClosest": minimum}})

result = bulk_houses.execute()
print(f'Bulk updates: {result["nModified"]} updated')
print(f'Time taken: {(datetime.datetime.now() - start_time).microseconds / 1000000} seconds')