Elasticsearch 为每个用户获取最新的徽章值答案

【问题标题】：Elasticsearch get latest badge value for each userElasticsearch 为每个用户获取最新的徽章值
【发布时间】：2021-04-19 07:04:55
【问题描述】：

我有一个索引“candidate_ranking”，其中包含具有以下文档结构的文档。索引的每个文档都有 1 个候选 ID，并且可以有多个具有相同候选 ID 的文档，但是它们的 created_at 字段会有所不同。这是我的候选排名索引中的文档样本-

    "hits" : [
      {
        "_index" : "candidate_ranking",
        "_type" : "_doc",
        "_id" : "SCa26HgB0zUr7edEvDul",
        "_score" : 1.0,
        "_source" : {
          "id" : 118558,
          "candidate_id" : 29492,
          "created_at" : "2021-03-27T01:34:29.628550+00:00",
          "badge" : "2"
        }
      },
      {
        "_index" : "candidate_ranking",
        "_type" : "_doc",
        "_id" : "SSa26HgB0zUr7edEvDul",
        "_score" : 1.0,
        "_source" : {
          "id" : 133354,
          "candidate_id" : 29492,
          "created_at" : "2021-03-27T02:11:35.811420+00:00",
          "badge" : "2"
        }
      },
      {
        "_index" : "candidate_ranking",
        "_type" : "_doc",
        "_id" : "Sia26HgB0zUr7edEvDul",
        "_score" : 1.0,
        "_source" : {
          "id" : 148136,
          "candidate_id" : 29492,
          "created_at" : "2021-03-29T20:20:36.482066+00:00",
          "badge" : "2"
        }
      },
      {
        "_index" : "candidate_ranking",
        "_type" : "_doc",
        "_id" : "Sya26HgB0zUr7edEvDul",
        "_score" : 1.0,
        "_source" : {
          "id" : 162916,
          "candidate_id" : 29492,
          "created_at" : "2021-03-29T21:05:03.985032+00:00",
          "badge" : null
        }
      },
      {
        "_index" : "candidate_ranking",
        "_type" : "_doc",
        "_id" : "TCa26HgB0zUr7edEvDul",
        "_score" : 1.0,
        "_source" : {
          "id" : 177712,
          "candidate_id" : 29492,
          "created_at" : "2021-03-29T21:33:32.596613+00:00",
          "badge" : null
        }
      },
      {
        "_index" : "candidate_ranking",
        "_type" : "_doc",
        "_id" : "TSa26HgB0zUr7edEvDul",
        "_score" : 1.0,
        "_source" : {
          "id" : 192999,
          "candidate_id" : 29492,
          "created_at" : "2021-03-29T22:20:24.942116+00:00",
          "badge" : null
        }
      },
      {
        "_index" : "candidate_ranking",
        "_type" : "_doc",
        "_id" : "Tia26HgB0zUr7edEvDul",
        "_score" : 1.0,
        "_source" : {
          "id" : 225434,
          "candidate_id" : 29492,
          "created_at" : "2021-03-29T23:13:59.266074+00:00",
          "badge" : null
        }
      },
      {
        "_index" : "candidate_ranking",
        "_type" : "_doc",
        "_id" : "Tya26HgB0zUr7edEvDul",
        "_score" : 1.0,
        "_source" : {
          "id" : 247169,
          "candidate_id" : 29492,
          "created_at" : "2021-03-30T00:16:04.077245+00:00",
          "badge" : null
        }
      },
      {
        "_index" : "candidate_ranking",
        "_type" : "_doc",
        "_id" : "UCa26HgB0zUr7edEvDul",
        "_score" : 1.0,
        "_source" : {
          "id" : 271179,
          "candidate_id" : 29492,
          "created_at" : "2021-03-30T01:19:59.803999+00:00",
          "badge" : null
        }
      },
      {
        "_index" : "candidate_ranking",
        "_type" : "_doc",
        "_id" : "USa26HgB0zUr7edEvDul",
        "_score" : 1.0,
        "_source" : {
          "id" : 295537,
          "candidate_id" : 29492,
          "created_at" : "2021-03-30T02:23:42.077149+00:00",
          "badge" : null
        }
      }
    ]
  }

此徽章值可以是空字符串或“1”或“2”。

我目前正在使用此聚合来获取具有徽章值 1 和 2 的所有用户的计数

GET /candidate_ranking/_search
{
  "aggs": {
      "mega_mogul": {
        "terms": {
          "field": "badge.keyword", 
          "exclude": ["", "2"],
          "size": 500000
        }
      },
      "rising_mogul": {
        "terms": {
          "field": "badge.keyword", 
          "exclude": ["", "1"],
          "size": 500000
        }
      }
  }
}

我的索引包含每个“candidate_id”的多个文档。我只想为每个candidate_id 的最新文档进行徽章聚合。类似于按 created_at 字段降序排序或排序，然后只为每个候选 ID 取最高值。因此将所有拥有最新徽章的候选人计算为 1 或 2。

我试过这样做，但没有用

GET /candidate_ranking/_search
{
  "aggs": {
      "mega_mogul": {
        "terms": {
          "field": "badge.keyword", 
          "exclude": ["", "2"],
          "size": 500000,
          "order": {"created_at": "desc"}, 
          "top_hits": {"size":1}
        }
      },
      "rising_mogul": {
        "terms": {
          "field": "badge.keyword", 
          "exclude": ["", "1"],
          "size": 500000
        }
      }
  }
}

【问题讨论】：

每个文档是否有唯一的“candidate_id”？您能否分享更多示例索引数据和预期的搜索结果？
badge 的值应该是 1 或 2？
可能有多个文档具有相同的candidate_id。徽章值应该是 1。我想分别计算 1 和 2，而不是组合计数

标签： elasticsearch elasticsearch-aggregation

【解决方案1】：

您可以使用terms aggregation 和max aggregation 来实现您所需的用例

要获取桶的数量，您需要使用stats_bucket aggregation

添加一个包含索引数据、映射、搜索查询和搜索结果的工作示例

索引映射：

{
  "mappings": {
    "properties": {
      "created_at": {
        "type": "date",
        "format": "yyyy-MM-dd'T'HH:mm:ss.SSSSSSz"
      }
    }
  }
}

索引数据：

{
  "id": 295537,
  "candidate_id": 29492,
  "created_at": "2021-03-30T02:23:42.077149+00:00",
  "badge": "1"
}
{
  "id": 271179,
  "candidate_id": 29492,
  "created_at": "2021-03-30T01:19:59.803999+00:00",
  "badge": "1"
}
{
  "id": 247169,
  "candidate_id": 29492,
  "created_at": "2021-03-30T00:16:04.077245+00:00",
  "badge": "1"
}
{
  "id": 225434,
  "candidate_id": 29492,
  "created_at": "2021-03-29T23:13:59.266074+00:00",
  "badge": null
}
{
  "id": 192999,
  "candidate_id": 29492,
  "created_at": "2021-03-29T22:20:24.942116+00:00",
  "badge": null
}
{
  "id": 177712,
  "candidate_id": 29492,
  "created_at": "2021-03-29T21:33:32.596613+00:00",
  "badge": null
}
{
  "id": 162916,
  "candidate_id": 29492,
  "created_at": "2021-03-29T21:05:03.985032+00:00",
  "badge": null
}
{
  "id": 148136,
  "candidate_id": 29492,
  "created_at": "2021-03-29T20:20:36.482066+00:00",
  "badge": "2"
}
{
  "id": 118558,
  "candidate_id": 29492,
  "created_at": "2021-03-27T01:34:29.628550+00:00",
  "badge": "2"
}
{
  "id": 133354,
  "candidate_id": 29492,
  "created_at": "2021-03-27T02:11:35.811420+00:00",
  "badge": "2"
}

搜索查询：

    {
  "size": 0,
  "aggs": {
    "badge_1": {
      "terms": {
        "field": "badge.keyword",
        "include": [
          "1"
        ],
        "size": 500000
      },
      "aggs": {
        "unique_id": {
          "terms": {
            "field": "candidate_id",
            "size": 10,
            "order": {
              "latestOrder": "desc"
            }
          },
          "aggs": {
            "top_doc": {
              "top_hits": {
                "size": 1
              }
            },
            "latestOrder": {
              "max": {
                "field": "created_at"
              }
            }
          }
        },
        "stats_1": {
          "stats_bucket": {
            "buckets_path": "unique_id._count"
          }
        }
      }
    },
    "badge_2": {
      "terms": {
        "field": "badge.keyword",
        "include": [
          "2"
        ],
        "size": 500000
      },
      "aggs": {
        "unique_id": {
          "terms": {
            "field": "candidate_id",
            "size": 10,
            "order": {
              "latestOrder": "desc"
            }
          },
          "aggs": {
            "top_doc": {
              "top_hits": {
                "size": 1
              }
            },
            "latestOrder": {
              "max": {
                "field": "created_at"
              }
            }
          }
        },
        "stats_2": {
          "stats_bucket": {
            "buckets_path": "unique_id._count"
          }
        }
      }
    }
  }
}

搜索结果：

    "aggregations": {
    "badge_2": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "2",
          "doc_count": 3,
          "unique_id": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": 29492,
                "doc_count": 3,
                "latestOrder": {
                  "value": 1.617049236482E12,
                  "value_as_string": "2021-03-29T20:20:36.482000Z"
                },
                "top_doc": {
                  "hits": {
                    "total": {
                      "value": 3,
                      "relation": "eq"
                    },
                    "max_score": 1.0,
                    "hits": [
                      {
                        "_index": "67157371",
                        "_type": "_doc",
                        "_id": "2",
                        "_score": 1.0,
                        "_source": {
                          "id": 133354,
                          "candidate_id": 29492,
                          "created_at": "2021-03-27T02:11:35.811420+00:00",
                          "badge": "2"
                        }
                      }
                    ]
                  }
                }
              }
            ]
          },
          "stats_2": {
            "count": 1,      // note this
            "min": 3.0,
            "max": 3.0,
            "avg": 3.0,
            "sum": 3.0
          }
        }
      ]
    },
    "badge_1": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "1",
          "doc_count": 3,
          "unique_id": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": 29492,
                "doc_count": 3,
                "latestOrder": {
                  "value": 1.617071022077E12,
                  "value_as_string": "2021-03-30T02:23:42.077000Z"
                },
                "top_doc": {
                  "hits": {
                    "total": {
                      "value": 3,
                      "relation": "eq"
                    },
                    "max_score": 1.0,
                    "hits": [
                      {
                        "_index": "67157371",
                        "_type": "_doc",
                        "_id": "10",
                        "_score": 1.0,
                        "_source": {
                          "id": 295537,
                          "candidate_id": 29492,
                          "created_at": "2021-03-30T02:23:42.077149+00:00",
                          "badge": "1"
                        }
                      }
                    ]
                  }
                }
              }
            ]
          },
          "stats_1": {
            "count": 1,       // note this
            "min": 3.0,
            "max": 3.0,
            "avg": 3.0,
            "sum": 3.0
          }
        }
      ]
    }
  }

【讨论】：

这仍然给出与以前相同的结果。我希望获得每个候选人 ID 的最新徽章。我确信这不是正确的结果，因为我的数据库中没有候选人的数量，因为它分别在徽章 1 和 2 的上述聚合中恢复。再次为大约 123 个候选 ID。如果该候选人有 5 个文档，那么我打算选择最新的并根据其徽章值增加计数。
@RicheshChouksey 请浏览搜索结果，如上面的答案所示（基于上面答案中给出的索引数据）。让我知道这是否是您期望得到的？
不，答案不正确。对于上述示例案例，我希望 1 作为徽章 1 的计数，因为候选 ID 在所有文档中都是相同的。因此，当汇总徽章 1 时，我们将考虑最新 created_at 提供的候选 ID 并仅计算该特定文档，忽略带有徽章 1 的候选 ID 的其他先前文档，如果存在另一个候选 ID，那么我们也会为徽章 1 获取该条目的最新条目. 徽章 2 也一样
@RicheshChouksey 抱歉回复晚了。请检查更新的搜索结果和查询。现在徽章 1 和徽章 2 的计数为 1