Mongoose/Mongodb，更新每个文档查询，很慢

Question

我在猫鼬中有这个更新查询。它有 1600 个帖子，运行大约需要 5 分钟。

瓶颈是什么？我是否使用了错误的方法？

export const getAndStoreLatestKPI = async () => {
  console.log("start kpi");
  try {
    const marketCaps = await getKPI();
    const stocks = await mongoose.model("stock").find().exec();

    for (const stock of stocks) {
      const marketCap = marketCaps.find(
        (marketCap) => marketCap.i === stock.insId
      );

      if (marketCap != null) {
        const marketCapAdjustedVal =
          stock.country === "Finland" ? marketCap.n * 10 : marketCap.n;

        const update = {
          marketCap: marketCapAdjustedVal,
        };
        console.log(marketCapAdjustedVal);
        await mongoose
          .model("stock")
          .findOneAndUpdate({ insId: stock.insId }, { update });
      }
    }
    console.log("done");
    return Promise.resolve();
  } catch (err) {
    return Promise.reject(err);
  }
};

export const getKPI = async (kpiId: number) => {
  try {
    const kpiFetch = await Axios.get(someurl);
    return Promise.resolve(kpiFetch.data.values);
  } catch (err) {
    return Promise.reject(err);
  }
};

Answer 1

所以主要的瓶颈是你的for 循环。对于每个库存商品，您需要执行几个“昂贵”的操作，例如从外部 API 获取数据 + 一次更新，并且您正在一个接一个地执行这些操作。

我建议你做的是一次循环几个项目。类似于multithreading 的想法。 在nodejs 中有几种不同的解决方案，例如nodejs worker threads

不过，我个人使用并推荐使用 bluebird，它为您提供了这种能力以及许多其他开箱即用的能力。

一些示例代码：

import Bluebird = require('bluebird');
const stocks = await mongoose.model("stock").find().exec();

await Bluebird.map(stocks, async (stock) => {
     const marketCap = marketCaps.find(
        (marketCap) => marketCap.i === stock.insId
      );

      if (marketCap != null) {
        const marketCapAdjustedVal =
          stock.country === "Finland" ? marketCap.n * 10 : marketCap.n;

        const update = {
          marketCap: marketCapAdjustedVal,
        };
        console.log(marketCapAdjustedVal);
        await mongoose
          .model("stock")
          .findOneAndUpdate({ insId: stock.insId }, { update });
      }
}, {concurrency: 25})
// concurrency details how many concurrent process run parallel. the heavier they are the less you want concurrent for obvious reasons.