Elasticsearch 返回一个字段的唯一值

Question

我正在尝试构建一个仅返回特定字段的唯一值的 Elasticsearch 查询。

我不想返回该字段的所有值，也不想计算它们。

例如，如果该字段当前包含 50 个不同的值，我进行搜索以仅返回 20 个匹配项 (size=20)。我希望 20 个结果中的每一个都有该字段的唯一结果，但我不关心结果中未表示的 30 个其他值。

例如使用以下搜索（伪代码 - 未选中）：

{
    from: 0,
    size: 20,
    query: {
        bool: {
            must: {
                range: { field1: { gte: 50 }},
                term: { field2: 'salt' },

                /**
                * I want to return only unique values for "field3", but I
                * don't want to return all of them or count them.
                *
                * How do I specify this in my query?
                **/
                unique: 'field3',
            },
            mustnot: {
                match: { field4: 'pepper'},
            }
        }
    }
}

Answer 1

您应该可以使用terms aggregation 轻松完成此操作。

这是一个例子。我定义了一个简单的索引，其中包含一个具有"index": "not_analyzed" 的字段，因此我们可以将每个字段的全文作为唯一值获取，而不是通过对其进行标记等生成的术语。

DELETE /test_index

PUT /test_index
{
   "settings": {
      "number_of_shards": 1
   },
   "mappings": {
      "doc": {
         "properties": {
            "title": {
               "type": "string",
               "index": "not_analyzed"
            }
         }
      }
   }
}

然后我添加了一些带有bulk API 的文档。

POST /test_index/_bulk
{"index":{"_index":"test_index","_type":"doc","_id":1}}
{"title":"first doc"}
{"index":{"_index":"test_index","_type":"doc","_id":2}}
{"title":"second doc"}
{"index":{"_index":"test_index","_type":"doc","_id":3}}
{"title":"third doc"}
{"index":{"_index":"test_index","_type":"doc","_id":4}}
{"title":"third doc"}

现在我们可以运行我们的术语聚合：

POST /test_index/_search?search_type=count
{
   "aggs": {
      "unique_vals": {
         "terms": {
            "field": "title"
         }
      }
   }
}
...
{
   "took": 1,
   "timed_out": false,
   "_shards": {
      "total": 1,
      "successful": 1,
      "failed": 0
   },
   "hits": {
      "total": 4,
      "max_score": 0,
      "hits": []
   },
   "aggregations": {
      "unique_vals": {
         "buckets": [
            {
               "key": "third doc",
               "doc_count": 2
            },
            {
               "key": "first doc",
               "doc_count": 1
            },
            {
               "key": "second doc",
               "doc_count": 1
            }
         ]
      }
   }
}

Answer 2

我很惊讶filter aggregation 没有被推荐。它一直追溯到 ES 版本 1.3。

过滤器聚合类似于常规过滤器查询，但可以改为嵌套到聚合链中，以过滤掉不符合特定条件的文档计数，并仅根据符合条件的文档为您提供子聚合结果查询条件。

首先，我们将放置我们的映射。

curl --request PUT \
  --url http://localhost:9200/items \
  --header 'content-type: application/json' \
  --data '{
  "mappings": {
    "item": { 
      "properties": { 
        "field1" :    { "type": "integer"  },
        "field2" :    { "type": "keyword"  },
        "field3" :    { "type": "keyword"  },
        "field4" :    { "type": "keyword"  }
      }
    }
  }
}
'

那么让我们加载一些数据。

curl --request PUT \
  --url http://localhost:9200/items/_bulk \
  --header 'content-type: application/json' \
  --data '{"index":{"_index":"items","_type":"item","_id":1}}
{"field1":50, "field2":["salt", "vinegar"], "field3":["garlic", "onion"], "field4":"paprika"}
{"index":{"_index":"items","_type":"item","_id":2}}
{"field1":40, "field2":["salt", "pepper"], "field3":["onion"]}
{"index":{"_index":"items","_type":"item","_id":3}}
{"field1":100, "field2":["salt", "vinegar"], "field3":["garlic", "chives"], "field4":"pepper"}
{"index":{"_index":"items","_type":"item","_id":4}}
{"field1":90, "field2":["vinegar"], "field3":["chives", "garlic"]}
{"index":{"_index":"items","_type":"item","_id":5}}
{"field1":900, "field2":["salt", "vinegar"], "field3":["garlic", "chives"], "field4":"paprika"}
'

请注意，只有 id 为 1 和 5 的文档才能通过条件，因此我们将在这两个 field3 数组和总共四个值上进行聚合。 ["garlic", "chives"], ["garlic", "onion"]。另请注意，field3 可以是数据中的数组或单个值，但我将它们设为数组以说明计数的工作方式。

curl --request POST \
  --url http://localhost:9200/items/item/_search \
  --header 'content-type: application/json' \
  --data '{
    "size": 0,
    "aggregations": {
        "top_filter_agg" : {
            "filter" : { 
                "bool": { 
                    "must":[
                        {
                            "range" : { "field1" : { "gte":50} }
                        },
                        {
                            "term" : { "field2" : "salt" }
                        }
                    ],
                    "must_not":[
                        {
                            "term" : { "field4" : "pepper" }
                        }
                    ]
                } 

            },
            "aggs" : {
                "field3_terms_agg" : { "terms" : { "field" : "field3" } }
            }
        }
    }
}
'

运行联合过滤器/术语聚合后。我们在 field3 上只有 4 个术语，总共只有 3 个唯一术语。

{
    "took": 46,
    "timed_out": false,
    "_shards": {
        "total": 5,
        "successful": 5,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": 5,
        "max_score": 0.0,
        "hits": []
    },
    "aggregations": {
        "top_filter_agg": {
            "doc_count": 2,
            "field3_terms_agg": {
                "doc_count_error_upper_bound": 0,
                "sum_other_doc_count": 0,
                "buckets": [
                    {
                        "key": "garlic",
                        "doc_count": 2
                    },
                    {
                        "key": "chives",
                        "doc_count": 1
                    },
                    {
                        "key": "onion",
                        "doc_count": 1
                    }
                ]
            }
        }
    }
}