如何通过 Elasticsearch 中的嵌套字段计算唯一文档的数量？

Question

我正在尝试计算具有唯一嵌套字段值的文档（接下来也是文档本身）。看起来获得独特的文件是可行的。 但是当我尝试执行count 的请求时，我收到如下错误：

抑制：org.elasticsearch.client.ResponseException：方法 [POST]，主机 [http://localhost:9200]，URI [/package/_count?ignore_throttled=true&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true]，状态行 [HTTP/1.1 400 Bad要求] {"error":{"root_cause":[{"type":"parsing_exception","re​​ason":"request 不支持 [collapse]","line":1,"col":216}],"type ":"parsing_exception","re​​ason":"request 不支持 [collapse]","line":1,"col":216},"status":400}

代码：

        BoolQueryBuilder innerTemplNestedBuilder = QueryBuilders.boolQuery();
        NestedQueryBuilder templatesNestedQuery = QueryBuilders.nestedQuery("attachment", innerTemplNestedBuilder, ScoreMode.None);
        BoolQueryBuilder mainQueryBuilder = QueryBuilders.boolQuery().must(templatesNestedQuery);
        if (!isEmpty(templateName)) {
            innerTemplNestedBuilder.filter(QueryBuilders.termQuery("attachment.name", templateName));
        }
        SearchSourceBuilder searchSourceBuilder = SearchSourceBuilder.searchSource()
                    .collapse(new CollapseBuilder("attachment.uuid"))
                    .query(mainQueryBuilder);
    // NEXT LINE CAUSE ERROR
        long count = client.count(new CountRequest("package").source(searchSourceBuilder), RequestOptions.DEFAULT).getCount(); <<<<<<<<<< ERROR HERE
        // THIS WORKS 
        SearchResponse searchResponse = client.search(
                    new SearchRequest(
                            new String[] {"package"},
                            searchSourceBuilder.timeout(new TimeValue(20, TimeUnit.SECONDS)).from(offset).size(limit)
                    ).indices("package").searchType(SearchType.DFS_QUERY_THEN_FETCH),
                    RequestOptions.DEFAULT
        );
        return ....;

该方法的总体意图是获取部分文档和所有此类文档的数量。可能已经存在另一种方法来满足这种需求。如果我尝试使用aggregations 和cardinality 获取count - 我得到的结果为零，并且它看起来不适用于嵌套字段。

计数请求：

{
    "query": {
        "bool": {
            "must": [
                {
                    "nested": {
                        "query": {
                            "bool": {
                                "adjust_pure_negative": true,
                                "boost": 1.0
                            }
                        },
                        "path": "attachment",
                        "ignore_unmapped": false,
                        "score_mode": "none",
                        "boost": 1.0
                    }
                }
            ],
            "adjust_pure_negative": true,
            "boost": 1.0
        }
    },
    "collapse": {
        "field": "attachment.uuid"
    }
}

映射是如何创建的：

curl -X DELETE "localhost:9200/package?pretty"
curl -X PUT    "localhost:9200/package?include_type_name=true&pretty" -H 'Content-Type: application/json' -d '{
    "settings" :  {
        "number_of_shards" : 1,
        "number_of_replicas" : 1
    }}'
curl -X PUT    "localhost:9200/package/_mappings?pretty" -H 'Content-Type: application/json' -d'
{
      "dynamic": false,
      "properties" : {
        "attachment": {
            "type": "nested",
            "properties": {
                "uuid" : { "type" : "keyword" },
                "name" : { "type" : "text" }
            }
        },
        "uuid" : {
          "type" : "keyword"
        }
      }
}
'

代码生成的查询结果应该是这样的：

curl -X POST "localhost:9200/package/_count?&pretty" -H 'Content-Type: application/json' -d' { "query" :
    {
        "bool": {
            "must": [
                {
                    "nested": {
                        "query": {
                            "bool": {
                                "adjust_pure_negative": true,
                                "boost": 1.0
                            }
                        },
                        "path": "attachment",
                        "ignore_unmapped": false,
                        "score_mode": "none",
                        "boost": 1.0
                    }
                }
            ],
            "adjust_pure_negative": true,
            "boost": 1.0
        }
    },
    "collapse": {
        "field": "attachment.uuid"
    }
}'

Answer 1

折叠可以only be used 在_search 上下文中，而不是在_count 中。

其次，您的查询甚至可以做什么？你有很多冗余参数，比如boost:1 等。你不妨说：

POST /package/_count?&pretty
{
  "query": {
    "bool": {
      "must": [
        {
          "nested": {
            "path": "attachment",
            "query": {
              "match_all": {}
            }
          }
        }
      ]
    }
  }
}

这并没有真正做任何事情:)

---

回答您最初的问题“计算具有唯一嵌套字段值的文档”，

假设有 3 个文档，其中 2 个具有相同的 attachment.uuid 值：

[
  {
    "attachment":{
      "uuid":"04144e14-62c3-11ea-bc55-0242ac130003"
    }
  },
  {
    "attachment":{
      "uuid":"04144e14-62c3-11ea-bc55-0242ac130003"
    }
  },
  {
    "attachment":{
      "uuid":"100b9632-62c3-11ea-bc55-0242ac130003"
    }
  }
]

要获取uuids 的terms 细分，请运行

GET package/_search
{
  "size": 0,
  "aggs": {
    "nested_uniques": {
      "nested": {
        "path": "attachment"
      },
      "aggs": {
        "subagg": {
          "terms": {
            "field": "attachment.uuid"
          }
        }
      }
    }
  }
}

产生

...
{
  "aggregations":{
    "nested_uniques":{
      "doc_count":3,
      "subagg":{
        "doc_count_error_upper_bound":0,
        "sum_other_doc_count":0,
        "buckets":[
          {
            "key":"04144e14-62c3-11ea-bc55-0242ac130003",
            "doc_count":2
          },
          {
            "key":"100b9632-62c3-11ea-bc55-0242ac130003",
            "doc_count":1
          }
        ]
      }
    }
  }
}

---

要获得唯一嵌套字段的父文档计数，我们必须稍微聪明一点：

GET package/_search
{
  "size": 0,
  "aggs": {
    "nested_uniques": {
      "nested": {
        "path": "attachment"
      },
      "aggs": {
        "scripted_uniques": {
          "scripted_metric": {
            "init_script": "state.my_map = [:];",
            "map_script": """
              if (doc.containsKey('attachment.uuid')) {
                state.my_map[doc['attachment.uuid'].value.toString()] = 1;
              }
            """,
            "combine_script": """
              def sum = 0;
              for (c in state.my_map.entrySet()) {
                sum += 1
              }
              return sum
            """,
            "reduce_script": """
              def sum = 0;
              for (agg in states) {
                sum += agg;
              }
              return sum;
            """
          }
        }
      }
    }
  }
}

返回

...
{
  "aggregations":{
    "nested_uniques":{
      "doc_count":3,
      "scripted_uniques":{
        "value":2
      }
    }
  }
}

而这个scripted_uniques: 2 正是你所追求的。

---

注意：我使用嵌套的脚本度量 aggs 解决了这个用例，但如果你们中有人知道更简洁的方法，我非常乐意学习它！