Mongoose insertMany 不适用于大型数组

Question

我一直在尝试使用mongoose + expressjs 将关于 (400-1000) json 对象数组的大数据插入到 mongodb，当我更改关于 (50) 项的数据时，insertMany 效果很好，没有问题。但如果数据超过 100，它会给我一个错误。

Departed.insertMany(results)
  .then(dep => {
    console.log(dep)
    res.sendStatus(201)
  })
  .catch(err => {
    console.log(err)
  })

在摩根控制台中，我得到了以下信息：

creation { active: true,
  _id: 5b73e8af19722d1689d863b0,
  name: 'TEST DATA 241',
  map: '',
  created_at: 2018-08-15T08:47:43.196Z,
  updated_at: 2018-08-15T08:47:43.196Z,
  __v: 0 }
insert read 453
(node:5769) [DEP0079] DeprecationWarning: Custom inspection function on Objects via .inspect() is deprecated

也在客户端（chrome，开发工具网络选项卡）状态得到了

(failed)
net::ERR_EMPTY_RESPONSE

我已经阅读了 mongo 的 insertMany() 有大约 1000 个限制，我使用的是 mongo 4.0 版本。即使我将大 json 分块成几个数组并尝试插入它但仍然得到相同的结果。实际的 sn-ps 是

router.post('/xls', upload.single('file'), async (req, res, next) => {
  try {
    if (req.body && req.file) {
      console.log('req', req.file)
      const segments = req.file.originalname.split('.')
      let exceltojson = segments[segments.length - 1] === 'xlsx' ? xlsx : xls
      exceltojson(
        {
          input: req.file.path,
          output: 'output.json'
        },
        async (err, result) => {
          if (err) console.log(err)
          const section = await Section.create({
            name: req.body.section,
            map: req.body.map
          })
          const results = await result.map(item => {
            return {
              branch: req.body.branch,
              section: String(section._id),
              ...item
            }
          })
          await console.log('creation', section)
          console.log('insert read', results.length)
          if (results.length >= 100) {
            console.log('more than 100')
            const data = _.chunk(results, 100)
            data.forEach(async chunk => {
              console.log('foreach')
              Departed.insertMany(chunk)
                .then(dep => {
                  console.log(dep)
                  res.sendStatus(201)
                })
                .catch(err => {
                  console.log(err)
                })
            })
          }
        }
      )
    }
  } catch (error) {
    next(error)
  }
})

Answer 1

您的问题与任何insertMany 限制无关。您的代码中有一个竞争条件，您无需等待所有块都被插入，然后再发回状态：

data.forEach(async chunk => {
  console.log('foreach')
  Departed.insertMany(chunk)
    .then(dep => { // this will be called as soon as one of the inserts finish
      console.log(dep)
      res.sendStatus(201)
    })
    .catch(err => {
      console.log(err)
    })
})

将其更改为（未经测试）：

Promise.all(data.map(chunk => Departed.insertMany(chunk)))
    .then(dep => { // this will be called when all inserts finish
      console.log(dep)
      res.sendStatus(201)
    })
    .catch(err => {
      console.log(err)
    })
})

Answer 2

另一种选择是使用bulkWrite API，它比发送多个独立操作更快，因为使用bulkWrite() 只有一次往返 MongoDB：

router.post('/xls', upload.single('file'), async (req, res, next) => {
    try {
        if (req.body && req.file) {
            console.log('req', req.file)
            const segments = req.file.originalname.split('.')
            let exceltojson = segments[segments.length - 1] === 'xlsx' ? xlsx : xls
            exceltojson(
                {
                    input: req.file.path,
                    output: 'output.json'
                },
                async (err, result) => {
                    if (err) console.log(err)
                    const section = await Section.create({
                        name: req.body.section,
                        map: req.body.map
                    })

                    let chunk = [];

                    result.forEach(item => {
                        chunk.push({
                            insertOne: {
                                document: {
                                    branch: req.body.branch,
                                    section: String(section._id),
                                    ...item
                                }
                            }
                        });

                        if (chunk.length === 500) {
                            const blkResult = await Departed.bulkWrite(chunk);
                            console.log(blkResult)
                            res.sendStatus(201)
                        } 
                    });

                    if (chunk.length > 0) {
                        const dep = await Departed.bulkWrite(chunk);
                        console.log(dep)
                        res.sendStatus(201)
                    }
                }
            )
        }
    } catch (error) {
        next(error)
    }
})