for loop is ignoring mongoose await(s)

Question

I build a scraper where the scraped data gets compared with already existing data to avoid duplicates, create new entries and update old entries. I'm doing this with a for loop, which loops over a findOne function where are two awaits in. The problem is that my for loop is ignoring (because it's sync?) my awaits and goes over to a part, where it is important that all of these awaits are done.

async function comparedata(length) {
    console.log("Starting comparing entries in data");
    for (let x = 0; x < length; x++) {
        const model = new dataModel({
            link: dataLinks[x],
            name: dataGames[x].replace('Download', ' '),
            logo: dataLogos[x],
            provider: 'data',

        });
        model.collection.findOne({ "link": dataLinks[x] }, async function (err, found) {
            if (err) throw err;
            if (found == null) {
                    await model.save().then((result) => {
                    console.log(result) // Is not happening because the for loop goes through to the next function and closes the server
                }).catch((err) => { console.log(err) });
            }
            else if (found != null) {
                if (dataGames[x] != found.name) {
                    await model.collection.findOneAndUpdate({ link: dataLinks[x] }, { $set: { name: dataGames[x] } });
                }
            }
        })
    }
    closeServer()//Closes the server is happening before new entries or updates are made. 
}

我的想法是使用 promises，但即使我尝试这样做，它也只是解决得太快并再次关闭服务器。

Answer 1

For the first iteration of for loop, the findOne's callback is put in the callback queue by the event loop and it proceeds with the next iteration not waiting for the awaits, this goes till the last iteration and after the last iteration it immediately calls the closeServer(), after this closeServer() call the tasks put into the callback queue (i.e the findOne's callbacks) are considered and executed by the event loop. Inorder to get an understanding of this, you have to learn about the event loop and how the event loop executes the javascript code. Please check about the event loop working mechanism youtube video here

You can use the promise style execution of findOne and overcome this issue.

根据我的说法，在同一条语句上使用 await 和 then() 功能并不是一个好习惯。

我的建议，

async function comparedata(length) {
   console.log("Starting comparing entries in data");
   for (let x = 0; x < length; x++) {
       const model = new dataModel({
           link: dataLinks[x],
           name: dataGames[x].replace('Download', ' '),
           logo: dataLogos[x],
           provider: 'data',

       });
      //  using .exec() at the end allows us to go with the promise way of dealing things rather than callbacks
      // assuming that 'dataModel' is the Schema, so I directly called findOne on it
       const found = await dataModel.findOne({ "link": dataLinks[x] }).exec();

       if (found == null) {
         // wrap the await in try...catch for catching errors while saving
         try{
            await model.save();
            console.log("Document Saved Successfully !");
         }catch(err) {
            console.log(`ERROR while saving the document. DETAILS: ${model} & ERROR: ${err}`) 
         }
      } else if (found != null) {
         if (dataGames[x] != found.name) {
            await model.collection.findOneAndUpdate({ link: dataLinks[x] }, { $set: { name: dataGames[x] } });
         }
      }

       if (x > length)
           sendErrorMail();
   }
   closeServer()//Closes the server is happening before new entries or updates are made. 
}

注意：请参考 Mongoose official documentation 以获取更新。

Answer 2

我对 mongoose api 不是很熟悉，但是如果你无法使用 promisified 版本，那么你可以使用 Promise 构造函数“跳出”回调：

const found = await new Promise((resolve, reject) => {
  model.collection.findOne({ "link": dataLinks[x] }, function (err, found) {
    if (err) {
      reject(err);
      return;
    };
    resolve(found);
  }
});

这可能会帮助您解决问题。

Answer 3

你应该能够像这样简化你的逻辑：

async function comparedata(length) {
  console.log('Starting comparing entries in data');
  try {
    for (let x = 0; x < length; x++) {
        let found = await model.collection.findOne({ link: dataLinks[x] });
        if (!found) {
            found = await dataModel.create({
                link: dataLinks[x],
                name: dataGames[x].replace('Download', ' '),
                logo: dataLogos[x],
                provider: 'data',
            });
        } else if (found.name !== dataGames[x]) {
            found.name = dataGames[x];
            await found.save();
        }
        console.log(found);
      }
  } catch (e) {
    console.log(e);
  }
  
  closeServer(); 
}