尝试使用 Nodejs 将数据从 MongoDB 导入 ElasticSearch 时出错答案

【问题标题】：Getting error while trying to Import data from MongoDB to ElasticSearch using Nodejs尝试使用 Nodejs 将数据从 MongoDB 导入 ElasticSearch 时出错
【发布时间】：2023-01-09 00:52:05
【问题描述】：

我正在尝试将小文档从 MongoDB 导入 ElasticSearch 但出现错误

      {
    "index": {
      "_index": "impact-fulltext",
      "_id": "t2oLkoUBwNXsTufYzszL",
      "status": 400,
      "error": {
        "type": "mapper_parsing_exception",
        "reason": "failed to parse field [_id] of type [_id] in document with id \u0027t2oLkoUBwNXsTufYzszL\u0027. Preview of field\u0027s value: \u0027605315a3b4f719d00f69f2d3\u0027",
        "caused_by": {
          "type": "mapper_parsing_exception",
          "reason": "Field [_id] is a metadata field and cannot be added inside a document. Use the index API request parameters."
        }
      }
    }
  }

我一无所知，因为我也定义了 _id 但仍然出现错误。

    db.collection("article_beta")
  .find()
  .limit(100)
  .toArray((err, docs) => {
    if (err) throw err;

    esClient.bulk(
      {
        body: docs.flatMap((doc) => [
          {
            index: {
              _index: "impact-fulltext",
              _id: doc._id.$oid,
            },
          },
          doc,
        ]),
      },
      (err, resp) => {
        if (err) throw err;
        console.log(resp);
        client.close();
      }
    );
  });

【问题讨论】：

标签： node.js mongodb elasticsearch

【解决方案1】：

在elasticsearch中，文档有一些元信息，还有文档的来源。因此，文档结构如下：

{
   "_index": "index-name",
   "_id": "document-id-stored-by-elasticsearch",
   "_source": {
      // documents itself
   }
}

因此，在文档中，您不能使用 _id 字段，因为 _id 字段是保留字段：

POST myindex/_doc
{
  "_id": "123123",
  "field1": "value1"
}

所以这个请求会报错如下：

{
  "error": {
    "root_cause": [
      {
        "type": "mapper_parsing_exception",
        "reason": "Field [_id] is a metadata field and cannot be added inside a document. Use the index API request parameters."
      }
    ],
    "type": "mapper_parsing_exception",
    "reason": "failed to parse field [_id] of type [_id] in document with id '16Y6koUBnxYxC21BP-tS'. Preview of field's value: '123123'",
    "caused_by": {
      "type": "mapper_parsing_exception",
      "reason": "Field [_id] is a metadata field and cannot be added inside a document. Use the index API request parameters."
    }
  },
  "status": 400
}

在您的示例中，来自 Mongodb 的值的 doc 变量有一个字段为 _id。您需要将该字段的名称更改为id。您可以将以下几行放在 flatMap 函数内的匿名函数内。

db.collection("article_beta")
  .find()
  .limit(100)
  .toArray((err, docs) => {
    if (err) throw err;

    esClient.bulk(
      {
        body: docs.flatMap((doc) => {
          doc.id = doc._id  // <---- added
          delete doc._id    // <---- added
          return  [
            {
              index: {
                _index: "impact-fulltext",
                _id: doc.id.$oid,
              },
            },
            doc,
          ]
        }),
      },
      (err, resp) => {
        if (err) throw err;
        console.log(resp);
        client.close();
      }
    );
  });

【讨论】：