没有嵌套字段的弹性搜索不同的父子聚合

Question

在 elasticsearch 6.2 我有一个父子关系：

Document -> NamedEntity

我想通过计算 mention 字段来聚合 NamedEntity，并给出包含每个命名实体的文档数。

我的用例是：

doc1 contains 'NER'(_id=ner11), 'NER'(_id=ner12)
doc2 contains 'NER'(_id=ner2)

父/子关系通过a join field 实现。在Document 我有一个字段：

join: {
  name: "Document"
}

在NamedEntity 孩子们：

join: {
  name: "NamedEntity",
  parent: "parent_id"
}

将_routing 设置为parent_id。

所以我尝试使用术语子聚合：

curl -XPOST elasticsearch:9200/datashare-testjs/_search?pretty -H 'Content-Type: application/json' -d '
{"query":{"term":{"type":"NamedEntity"}},
 "aggs":{
   "mentions":{
     "terms":{
       "field":"mention"
     },
     "aggs":{
       "docs":{
         "terms":{"field":"join"}
       }
     }
   }
 }
}'

我有以下回应：

"aggregations" : {
  "mentions" : {
    "doc_count_error_upper_bound" : 0,
    "sum_other_doc_count" : 0,
    "buckets" : [
      {
        "key" : "NER",
        "doc_count" : 3,
        "docs" : {
          "doc_count_error_upper_bound" : 0,
          "sum_other_doc_count" : 0,
          "buckets" : [
            {
              "key" : "NamedEntity",
              "doc_count" : 3 <-- WRONG ! There are 2 distinct documents
            }
          ]
        }
      }
    ]
  }

我在 mentions.buckets.doc_count 中找到了预期的 3 次出现。但在mentions.buckets.docs.buckets.doc_count 字段中，我希望只有 2 个文档（不是 3 个）。就像select count distinct。

如果我与"terms":{"field":"join.parent"} 聚合，我有：

...
"docs" : {
    "doc_count_error_upper_bound" : 0,
    "sum_other_doc_count" : 0,
    "buckets" : [ ]
}
...

我在join 字段上绑定了cardinality 聚合，我得到一个值1，而join.parent 上的cardinality 聚合返回一个值0。

那么，如何在不使用 reverse nested aggregation 的情况下对父母进行聚合不同的计数？

---

正如@AndreiStefan 所问，这是映射。这是 ES 6 映射中 Document(content) 和 NamedEntity(mention) 之间的简单 1-N 关系（字段在同一级别上定义）：

curl -XPUT elasticsearch:9200/datashare-testjs -H 'Content-Type: application/json' -d '
{
    "mappings": {
    "doc": {
      "properties": {
        "content": {
          "type": "text",
          "index_options": "offsets"
        },
        "type": {
          "type": "keyword"
        },
        "join": {
          "type": "join",
          "relations": {
            "Document": "NamedEntity"
          }
        },
        "mention": {
          "type": "keyword"
        }
      }
    }
}}

以及对最小数据集的请求：

curl -XPUT elasticsearch:9200/datashare-testjs/doc/doc1 -H 'Content-Type: application/json' -d '{"type": "Document", "join": {"name": "Document"}, "content": "a NER document contains 2 NER"}'
curl -XPUT elasticsearch:9200/datashare-testjs/doc/doc2 -H 'Content-Type: application/json' -d '{"type": "Document", "join": {"name": "Document"}, "content": "another NER document"}'
curl -XPUT elasticsearch:9200/datashare-testjs/doc/ner11?routing=doc1 -H 'Content-Type: application/json' -d '{"type": "NamedEntity", "join": {"name": "NamedEntity", "parent": "doc1"}, "mention": "NER"}'
curl -XPUT elasticsearch:9200/datashare-testjs/doc/ner12?routing=doc1 -H 'Content-Type: application/json' -d '{"type": "NamedEntity", "join": {"name": "NamedEntity", "parent": "doc1"}, "mention": "NER"}'
curl -XPUT elasticsearch:9200/datashare-testjs/doc/ner2?routing=doc2 -H 'Content-Type: application/json' -d '{"type": "NamedEntity", "join": {"name": "NamedEntity", "parent": "doc2"}, "mention": "NER"}'

Answer 1

我发布此解决方法以防它可以帮助某人。但如果有人有更清洁的方式来做这件事，我会很感兴趣。

我在子项中添加了一个非规范化字段，其中包含父 id 的副本（join/parent 中已有的值）：

curl -XPUT elasticsearch:9200/datashare-testjs -H 'Content-Type: application/json' -d '
{
    "mappings": {
    "doc": {
      "properties": {
        "content": {
          "type": "text",
          "index_options": "offsets"
        },
        "type": {
          "type": "keyword"
        },
        "join": {
          "type": "join",
          "relations": {
            "Document": "NamedEntity"
          }
        },
        "document_id: {
          "type": "keyword"
        },
        "mention": {
          "type": "keyword"
        }
      }
    }
}}

然后带有这个新字段的基数聚合按预期工作：

curl -XPOST elasticsearch:9200/datashare-testjs/_search?pretty -H 'Content-Type: application/json' -d '
{"query":{"term":{"type":"NamedEntity"}},
 "aggs":{
   "mentions":{
     "terms":{
       "field":"mention"
     },
     "aggs":{
       "docs":{
         "cardinality": {
           "field" : "document_id"
       }
     }
   }
}}}'

它响应：

...
"aggregations" : {
  "mentions" : {
    "doc_count_error_upper_bound" : 0,
    "sum_other_doc_count" : 0,
    "buckets" : [
      {
        "key" : "NER",
        "doc_count" : 3,
        "docs" : {
          "value" : 2
        }
      }
    ]
  }
}

Answer 2

  "aggs": {
    "mentions": {
      "terms": {
        "field": "mention"
      },
      "aggs": {
        "docs": {
          "terms": {
            "field": "join"
          },
          "aggs": {
            "uniques": {
              "cardinality": {
                "field": "join#Document"
              }
            }
          }
        }
      }
    }
  }

或者如果你只是想要计数：

  "aggs": {
    "mentions": {
      "terms": {
        "field": "mention"
      },
      "aggs": {
        "uniques": {
          "cardinality": {
            "field": "join#Document"
          }
        }
      }
    }
  }

如果您需要自定义排序（按唯一计数）：

  "aggs": {
    "mentions": {
      "terms": {
        "field": "mention",
        "order": {
          "uniques": "desc"
        }
      },
      "aggs": {
        "uniques": {
          "cardinality": {
            "field": "join#Document"
          }
        }
      }
    }
  }

Answer 3

我最近在 Elasticsearch 7.1 上遇到了同样的问题，由 elasicsearch 创建的这个附加字段“my_join_field#my_parent”解决了这个问题。我很高兴不必将 parent_id 添加到子文档中。

https://www.elastic.co/guide/en/elasticsearch/reference/current/parent-join.html#_searching_with_parent_join