从数据库集合中读取信息，然后批量写入或批量更新

Question

我正在将其他服务中的用户同步到我们的系统中。这些用户被保存到一个名为 TempUser 的集合中，即大约 10k 个文档（这将继续增长）。每当我想更新/创建新用户时，都会执行以下步骤：

1. 从数据库中读取TempUser
2. 确定来自TempUser 的文档是否是数据库中的现有用户。如果他们是现有用户，我需要更新数据库中的User 文档。如果它们不存在于数据库中，我想将它们写为新的User。如果用户是新的User，他们还需要一个名为Profile 的文档，在User 文档中保存为user.profile。

下面概述了我的代码，我相信我可以通过将用户批处理 100+ 来解决这个问题，而不是仅仅循环遍历一个数组并一个一个地创建它们。我可能会做一个bulkWrite 或bulkUpdate，但我很想听到来自社区的反馈。

目前，我正在与用户一一阅读（哈哈，我知道）：

      console.log("Creating accounts in system...");
      const PAGE_LIMIT = 1;
      // After user promise resolve, take tempUser and add them as actual users
      const totalUsers = await TempUser.countDocuments();
      let currentUserCount = 0;
      for (let PAGE_NUM = 1; currentUserCount < totalUsers; PAGE_NUM++) {
        const skips = PAGE_LIMIT * (PAGE_NUM - 1);
        const userList = await TempUser.find()
          .skip(skips)
          .limit(PAGE_LIMIT);
        currentUserCount += userList.length;
        const users = await Promise.all(
          userList.map((student) => {
            return new Promise(async (resolve, reject) => {
              // Find school
              const schoolId = await School.findOne(
                {
                  "cleverData.cleverId": student.school,
                },
                { _id: 1 }
              );

              let userObj = {
                firstName: student.name.first,
                lastName: student.name.last,
                cleverId: student.id,
                schoolId: schoolId._id,
                cleverSchoolId: student.school,
                email: student.email,
                studentNumber: student.student_number,
                stemuliDistrictId: student.stemuliDistrictId,
                districtName: student.districtName,
                districtId: student.district,
     
              };

       
              const user = await userSync(studentObj);
              resolve(students);
            });
          })
        );
      }

userSync 然后获取对象并创建或更新用户，具体取决于它是否存在：

module.exports = exports = ({
  firstName,
  lastName,
  schoolId,
  cleverSchoolId,
  email,
  studentNumber,
 userPrograms,
  program,
  programsEnrolled,
  term,
  password,
  districtName,
  districtId,
  cleverId,
}) => {
  const avatar =
    "https://stemuli.blob.core.windows.net/stemuli/Placeholders/avatar-placeholder.png";
  return new Promise(async (resolve, reject) => {
    School.findOne({
      _id: schoolId,
    })
      .then(async (schoolDoc) => {
        if (schoolDoc) {
          // We need to get the district name, so get a user with it

          // Check if user exists with email
          let checkedUser = null;
          try {
            let update = {
              $set: {
                "district.cleverId": districtId,
                "district.name": districtName,
                schoolCleverId: cleverSchoolId,
                cleverId: cleverId,
                school: schoolId,
              },
            };
            if (typeof studentPrograms !== "undefined") {
              update = {
                ...update,
                $addToSet: { userPrograms: { $each: userPrograms } },
              };
            }

            checkedUser = await User.findOneAndUpdate({ email: email }, update);
          } catch (err) {
            console.error(err);
            return reject({
              code: 500,
              message: "Unexpected error occured",
            });
          }

          if (checkedUser) {
            return resolve("Email already exists");
          }

          let newUser = null;
          try {
            newUser = await new User({
              name: firstName + " " + lastName,
              email: email.toLowerCase(),
              cleverId: cleverId,
              district: {
                cleverId: districtId,
                name: districtName,
              },
              studentInfo: { sis_id: studentNumber },
              account_type: "student",
              profile_type: "StudentProfile",
              program: program,
              school: schoolId,
              schoolCleverId: cleverSchoolId,
              firstName: firstName,
              lastName: lastName,
              password: password,
            }).save();
            let studentProfile = null;
            try {
              userProfile = await new UserProfile({
                user: newUser._id,
                profile_picture: { url: avatar },
              }).save();

              assignProjects(schoolDoc.district, newUser._id);
              newUser.profile = userProfile._id;
              newUser
                .save()
                .then((userDocFinal) => {
                  payload = {
                    id: userDocFinal.id,
                    firstName: userDocFinal.firstName,
                    lastName: userDocFinal.lastName,
                    name: userDocFinal.name,
                    profile_picture: {
                      url: avatar,
                    },
                    hasDailyReport: false,
                    hasSignedOn: false,

                    account_type: userDocFinal.account_type,
                    hasAssessment: userDocFinal.hasAssessment,
                  };

                  resolve(payload);
                })
                .catch((err) => {
                  console.error(err);
                  reject({
                    code: 500,
                    message: "Error saving profile ID to new User profile id",
                  });
                });
            } catch (err) {
              console.error(err);
              reject({
                code: 500,
                message: "Error saving Student Profile",
              });
            }
          } catch (err) {
            console.error("Issue creating user profile");
          }

   
        } else {
          console.log("Could not located school with id");
        }
      })
      .catch((err) => {
        console.error(err);
        reject({
          code: 400,
          message: "School is not registered with system",
        });
      });
  });
};

Answer 1

如果文档尚未插入，您可以使用upsert=true 插入文档。你可以参考这个链接：MongoDB-FindOneAndUpdate

为了尽量减少处理不同文档和集合的时间，您应该使用索引。例如，schoolId 应该是一个索引。

另外，你可以参考这篇关于基于光标的分页的文章：Cursor-Based-Pagination

TLDR;基于光标的分页使用文档 ID 以递增顺序排列的事实，因此您可以使用它们遍历数据（并且它们已被证明比 Skip-Limit 方法更高效）。