【问题标题】:Elastic Search - Is there a way to ignore certain documents based on condition during aggregation?弹性搜索 - 有没有办法在聚合期间根据条件忽略某些文档?
【发布时间】:2019-08-07 14:53:06
【问题描述】:

我正在记录流的分析。当流程开始时,字段“start”设置为“true”,流程结束时将“true”设置为字段“end”。 少数流可能不包含“结束”字段“真”。我想找到流动到底在哪里停止。

我已尝试使用嵌套聚合,但无法获取未结束流的文档。

这是存储在弹性搜索中的数据

    [
    {
        "date": 1565094409535,
        "start": "true",
        "end": "",
        "message": "Select Option",
        "context": "third",
        "account_id": "123"
    },
    {
        "date": 1565094411152,
        "start": "",
        "end": "",
        "message": "Select Me",
        "context": "third",
        "account_id": "123"
    },
    {
        "date": 1565094409652,
        "start": "true",
        "end": "",
        "message": "Select option",
        "context": "third",
        "account_id": "123"
    },
    {
        "date": 1565094409751,
        "start": "",
        "end": "",
        "message": "Select Me",
        "context": "third",
        "account_id": "123"
    },
    {
        "date": 1565094411187,
        "start": "",
        "end": "true",
        "message": "Bye Bye",
        "context": "third",
        "account_id": "123"
    },
    {
        "date": 1565094411211,
        "start": "true",
        "end": "",
        "message": "Select option",
        "context": "third",
        "account_id": "123"
    },
    {
        "date": 1565094411311,
        "start": "true",
        "end": "",
        "message": "How are you",
        "context": "second",
        "account_id": "123"
    }
]

使用的查询:

 {
"size": 0,
"query": {
    "bool": {
        "must": [{
                "term": {
                    "context.keyword": "third"
                }
            }
        ]
    }
},
"aggs": {
    "sessions": {
        "terms": {
            "field": "account_id.keyword",
            "size": 25000
        },

        "aggs": {
            "top_sessions_hits": {
                "top_hits": {
                    "sort": [{
                        "date": {
                            "order": "asc"
                        }
                    }],
                    "_source": {
                        "includes": ["date", "message", "account_id", "start", "end", "context"]
                    },
                    "size": 10000
                }
            }
        }
    }
}

}

我得到以下输出

    {
      "took": 37,
      "timed_out": false,
      "_shards": {
        "total": 5,
        "successful": 5,
        "failed": 0
      },
      "hits": {
        "total": 4,
        "max_score": 0,
        "hits": []
      },
      "aggregations": {
        "sessions": {
          "doc_count_error_upper_bound": 0,
          "sum_other_doc_count": 0,
          "buckets": [
            {
              "key": "123",
              "doc_count": 6,
              "top_sessions_hits": {
                "hits": {
                  "total": 6,
                  "max_score": null,
                  "hits": [
                    {
                      "_index": "messages",
                      "_type": "doc",
                      "_id": "********",
                      "_score": null,
                      "_source": {
                        "date": 1565094409535,
                        "start": "true",
                        "end": "",
                        "message": "Select Option",
                        "context": "third",
                        "account_id": "123"
                      },
                      "sort": [
                        1565094409535
                      ]
                    },{
                      "_index": "messages",
                      "_type": "doc",
                      "_id": "********",
                      "_score": null,
                      "_source": {
                        "date": 1565094411152,
                        "start": "",
                        "end": "",
                        "message": "Select Me",
                        "context": "third",
                        "account_id": "123"
                      },
                      "sort": [
                        1565094411152
                      ]
                    },
                    {
                      "_index": "messages",
                      "_type": "doc",
                      "_id": "********",
                      "_score": null,
                      "_source": {
                        "date": 1565094409652,
                        "start": "true",
                        "end": "",
                        "message": "Select option",
                        "context": "third",
                        "account_id": "123"
                      },
                      "sort": [
                        1565094409652
                      ]
                    },
                    {
                      "_index": "messages",
                      "_type": "doc",
                      "_id": "********",
                      "_score": null,
                      "_source": {
                        "date": 1565094409751,
                        "start": "",
                        "end": "",
                        "message": "Select Me",
                        "context": "third",
                        "account_id": "123"
                      },
                      "sort": [
                        1565094409751
                      ]
                    },
                    {
                      "_index": "messages",
                      "_type": "doc",
                      "_id": "********",
                      "_score": null,
                      "_source": {
                        "date": 1565094411187,
                        "start": "",
                        "end": "true",
                        "message": "Bye Bye",
                        "context": "third",
                        "account_id": "123"
                      },
                      "sort": [
                        1565094411187
                      ]
                    },
                    {
                      "_index": "messages",
                      "_type": "doc",
                      "_id": "********",
                      "_score": null,
                      "_source": {
                        "date": 1565094411211,
                        "start": "true",
                        "end": "",
                        "message": "Select option",
                        "context": "third",
                        "account_id": "123"
                      },
                      "sort": [
                        1565094411211
                      ]
                    }
                  ]
                }
              }
            }
          ]
        }
      }
    }

但我不想获取文档 #3、#4 和 #5,因为流程已完成。

我对弹性搜索很陌生。由于核心人员休假,我正在尝试解决一些问题。请指导我获取文档 #1、#2 和 #6。

【问题讨论】:

  • 你是怎么告诉这个But I don't want to get the document #3, #4 & #5 since the flow was completed.的?基于什么?
  • 感谢@JBone 的回复。查询结果中有三个流,第一个流和第二个流(doc #1 和 #6)已启动但未结束。第二个流程已启动(文档#3)并成功结束(文档#5)。我只需要那些不是结束的流程。所以在这种情况下,我只需要文档#1,2&6,它们是流开始但不是结束的。

标签: elasticsearch


【解决方案1】:

我了解每个流中有 2 条消息 - 一条带有 "start":true,一条带有 "end":true。为了找到只有开始但没有结束的流,您需要在每个流上都有一个唯一标识符,比如flow-id

如果消息将包含flow-id,您可以在流 ID 上运行术语聚合,以计算每个流存在多少消息,然后根据聚合结果的 _count 对结果进行升序排序 - 第一个agg 结果将使用 count=1,因此只有开始而没有结束的流。

查询应如下所示:

GET /flows_index/_search {
"size": 0,
"aggs": {
    "flow_id_agg": {
        "terms": {
            "field": "flow_id",
            "order": {
                "_count": "asc"
            },
            "aggs": {
                "flow_id_samples": {
                    "top_hits": {
                      "sort": [{
                        "date": {
                            "order": "asc"
                        }
                       }],
                      "_source": {
                         "includes": ["date", "message", "account_id", "start", "end", "context"]
                    },
                    "size": 10000
                   }
                }
            }
          }
       }
    }
}

在这里查看类似的需求:Elasticsearch terms aggregation and querying

【讨论】:

  • 感谢@ziv 的回答。这里的问题是,每个流都没有唯一的 id。肯定会为未来考虑的。但是现在,需要支持现有的数据。
  • @Nagarajks 欢迎。 'message'/'context' 字段是否是唯一的(通常)?如果是这样,您可以使用其中之一作为唯一 ID。另一种选择是同时使用帐户 ID 和消息/上下文,通过为帐户 ID 运行术语聚合和为上下文/消息运行内部聚合,以替换流 ID 想法。
  • 多个键的组合在任何这种情况下都不是唯一的。
猜你喜欢
  • 1970-01-01
  • 2019-12-02
  • 1970-01-01
  • 2014-04-17
  • 2016-08-17
  • 2010-11-19
  • 2020-08-25
  • 2015-09-07
  • 1970-01-01
相关资源
最近更新 更多