【问题标题】:Why document containing all search terms scored lower?为什么包含所有搜索词的文档得分较低?
【发布时间】:2021-01-18 16:24:24
【问题描述】:

我的搜索结果只有 1 个字词出现在与查询中的两个字词匹配的结果上方。以下是我的设置

搜索查询

POST index1/_search

{
    "size": 5,
    "query": {
        "bool": {
            "should": [
                {
                    "match": {
                        "content": {
                            "query": "devtools tutorial"
                        }
                    }
                }
            ]
        }
    }
}

使用的设置和映射:

{
    "mappings": {
        "properties": {
            "content": {
                "type": "text"
            },
            "title": {
                "type": "text"
            }
        }
    }
}

我一直用于测试目的的示例文档。我希望带有 _id : 3 的文档出现在带有 _id:4 的文档上方,因为它同时具有查询中的两个术语:

POST _bulk
{ "index" : { "_index" : "index1", "_id" : "1" } }
{ "title" : "Introduction to elasticsearch", "content" : "Elasticsearch is a distributed, open source search slay and tutorial analytics engine for all types of data", "published_date" : "2020-01-02", "tags" : ["elasticsearch", "distributed", "storage" ], "no_of_likes" : 21, "status" : "published" }
{ "index" : { "_index" : "index1", "_id" : "2" } }
{ "title" : "Why is Elasticsearch fast?", "content" : "It is able to achieve fast search responses because, instead small of tutorial searching the text directly, it searches an index instead", "tags" : ["elasticsearch", "fast", "index" ], "no_of_likes" : 10,"status" : "draft"}
{ "index" : { "_index" : "index1", "_id" : "3" } }
{ "title" : "Introducing the New React DevTools", "content" : "We are excited to announce a new release of accompany the React DevTools tutorial, available today in Chrome, Firefox, and (Chromium) Edge.We are excited to announce a new release of accompany the React tutorial, available today in Chrome, Firefox, and (Chromium) Edge.We are excited to announce a new release of accompany the React DevTools tutorial, available today in Chrome, Firefox, and (Chromium) Edge.We are excited to announce a new release of accompany the React DevTools tutorial, available today in Chrome, Firefox, and (Chromium) Edge.We are excited to announce a new release of accompany the React tutorial, available today in Chrome, Firefox, and (Chromium) Edge.We are excited to announce a new release of accompany the React tutorial, available today in Chrome, Firefox, and (Chromium) Edge.We are excited to announce a new release of accompany the React tutorial, available today in Chrome, Firefox, and (Chromium) EdgeWe are excited to announce a new release of accompany the React  tutorial, available today in Chrome, Firefox, and (Chromium) EdgeWe are excited to announce a new release of accompany the React tutorial, available today in Chrome, Firefox, and (Chromium) EdgeWe are excited to announce a new release of accompany the React DevTools tutorial, available today in Chrome, Firefox, and (Chromium) Edge", "published_date" : "2019-08-25", "tags" : ["react", "devtools" ], "no_of_likes" : 2, "status" : "published"}
{ "index" : { "_index" : "index1", "_id" : "4" } }
{ "title" : "Angular Tools for High Performance", "content" : "devtools", "published_date" : "2014-03-22", "tags" : ["angular", "performance","fast"], "no_of_likes" : 35, "status" : "published"}
{ "index" : { "_index" : "index1", "_id" : "5" } }
{ "title" : "The new features in Java 14", "content" : "Oracle on September 17 said switch expressions tutorial are expected naresh to go final in Java Development Kit 14 (JDK 14). ", "published_date" : "2019-07-20", "tags" : ["java"], "no_of_likes" : 11, "status" : "published"}
{ "index" : { "_index" : "index1", "_id" : "6" } }
{ "title" : "Thread behavior in the JVM", "content" : "Threading refers to the practice of executing programming tutorial processes accompani concurrently to improve application performance.", "tags" : ["java","jvm"], "no_of_likes" : 3, "status" : "draft"}
{ "index" : { "_index" : "index1", "_id" : "7" } }
{ "title" : "Stacks and Queues", "content" : "The main operations of a stack are push, pop, & isEmpty and for queue enqueue, dequeue, & isEmpty., ", "published_date" : "2016-12-12", "tags" : ["stack","queue","datastructures"], "no_of_likes" : 43, "status" : "published"}
{ "index" : { "_index" : "index1", "_id" : "8" } }
{ "title" : "How are big data and ai changing the business world?","content" : "Today’s businesses are ruled by data. Specifically, big data and AI that have gradually been murder  evolving to juvenile day-to-day business murder and playing as the key murder driver in business murder Intelligence decision-making","published_date" : "2020-01-01","tags" :["big data","ai"],"no_of_likes" :120,"status" : "published"}
{ "index" : { "_index" : "index1", "_id" : "9" } }
{ "title" : "Hash Tables", "content" : "A hash table is a data structure used to implement symbol table (associative array), a structure tutorial that can map keys to values", "published_date" : "2017-08-12", "tags" :[ "hash", "datastructures" ], "no_of_likes" :13, "status" : "published" }
{ "index" : { "_index" : "index1", "_id" : "10" } }
{ "title" : "Go vs Python: How to choose", "content" : "Python and Go share a reputation for being convenient tutorial to work with. Both languages have a simple and straightforward syntax and a small and easily remembered feature set", "tags" :[ "go", "python" ], "no_of_likes" :134, "status" : "draft" }
{ "index" : { "_index" : "index1", "_id" : "11" } }
{ "title" : "Android Studio 4.0 backs native UI toolkit", "content" : "Now available in a preview juvenile, the weapon Android murder 4.0 ‘Canary’ upgrade works with the JetPack Compose UI toolkit and improves Java 8 support", "tags" :[ "android", "nativeui" ], "no_of_likes" :113, "status" : "draft" }
{ "index" : { "_index" : "index1", "_id" : "12" } }
{ "title" : "JSON tools you don’t want to miss", "content" : "Developers can choose from many great free and juvenile tools for tutorial JSON formatting, validating, editing, and converting to other formats", "published_date" : "2018-02-13", "tags" :[ "json" ], "no_of_likes" :23, "status" : "published" }
{ "index" : { "_index" : "index1", "_id" : "13" } }
{ "title" : "Get started with method references in Java", "content" : "Use method references to simplify functional programming in Java", "tags" :[ "java", "references" ], "no_of_likes" :102, "status" : "draft" }
{ "index" : { "_index" : "index1", "_id" : "14" } }
{ "title" : "How to choose a database for your application", "content" : "From performance to programmability, the right childlike makes all the difference. Here are 12 key questions to help guide your selection", "published_date" : "2009-02-12", "tags" :[ "database" ], "no_of_likes" :229, "status" : "published" }
{ "index" : { "_index" : "index1", "_id" : "15" } }
{ "title" : "10 reasons to Learn Scala Programming Language", "content" : "One of the questions my reader youthful tutorial ask me is, shall I learn Scala? Does Scala has a better future than Java, or why Java developer should learn Scala and so on", "published_date" : "2009-02-12", "tags" :[ "scala", "language" ], "no_of_likes" :136, "status" : "draft" }
{ "index" : { "_index" : "index1", "_id" : "16" } }
{ "title" : "ways to declare and initialize Two-dimensional (2D) String and Integer Array in Java", "content" : "Declaring a two-dimensional array is very interesting in Java as Java programming youthful provides many ways to declare a 2D array and each one of them has some special things to learn about", "published_date" : "2009-02-12", "tags" :[ "jaava", "datastructure", "array" ], "no_of_likes" :342, "status" : "published" }
{ "index" : { "_index" : "index1", "_id" : "17" } }
{ "title" : "Hibernate Tip: How to customize the association mappings using a composite key", "content" : "Hibernate provides lots of mapping features that allow you to map complex domain and table models. But the availability of these features doesn't mean that you should use them in all of your applications", "tags" :[ "hibernate", "compositekey" ], "no_of_likes" :112, "status" : "draft" }
{ "index" : { "_index" : "index1", "_id" : "18" } }
{ "title" : "Getting started with Python on Spark", "content" : "At my current project I work a lot with Apache Spark juvenile running PySpark jobs on it.", "tags" :[ "python", "spark" ], "no_of_likes" :86, "status" : "draft" }
{ "index" : { "_index" : "index1", "_id" : "19" } }
{ "title" : "Relationship between IOT, big data, and cloud computing", "content" : "Big data analytics is the basis of decision making in an organization. It involves the examination of juvenile a large number of data sets in order to identify the hidden patterns that result in their existence.", "published_date" : "2018-11-10", "tags" :[ "iot", "big data", "cloud computing" ], "no_of_likes" :12, "status" : "published" }
{ "index" : { "_index" : "index1", "_id" : "20" } }
{ "title" : "Get started with juvenile expressions in Java", "content" : "Learn how to use lambda juvenile and tutorial functional programming techniques in your Java programs.", "tags" :[ "java", "lambda", "functional programming" ], "no_of_likes" :128, "status" : "draft" }

请注意,同时具有devtoolstutorial 的doc-3 得分低于仅具有devtools 的doc-4。

【问题讨论】:

  • 您能否提供您的映射、示例文档和预期结果,请提供完整的详细信息,以便我们轻松重现您的问题并提供您的工作解决方案
  • @OpsterElasticsearchNinja 我已经编辑了问题并包含了设置、映射和示例文档
  • @Jam 你能分享你的syn_test.txt 文件吗
  • @Bhavya 更新了问题

标签: elasticsearch tf-idf elasticsearch-query


【解决方案1】:

花了很长时间,找到了根本原因和解决方案,在使用explain=true参数分析搜索输出后,如果你注意到,下面是计算tf分数的公式

"description": "tf, 计算为 freq / (freq + k1 * (1 - b + b * dl / avgdl)) 来自:",

"details": [   {
"value": 1.0,
"description": "freq, occurrences of term within document",
"details": [
  
]   },   {
"value": 1.2,
"description": "k1, term saturation parameter",
"details": [
  
]   },   {
"value": 0.75,
"description": "b, length normalization parameter",
"details": [
  
]   },   {
"value": 2.0,
"description": "dl, length of field",
"details": [
  
]   },   {
"value": 29.545454,
"description": "avgdl, average length of field",
"details": [
  
]   } ]

如果您注意到,它由总共 5 个组件和 dl 组成,即与搜索结果匹配的字段长度在 doc-id 4 的情况下非常少,因为它只包含 devtools 如果您注意到,这dl 是分母的一部分,如果将​​ tf 与所有文档相同的其他组件相乘,则较小的值将增加 tf,最终公式为 score(freq=4.0), computed as boost * idf * tf

这是由于field normalization 而发生的,为了修复它,您必须禁用可搜索字段上的norms 并重试,我再次定义了索引映射,并在@987654334 上禁用了norms @field 并得到你想要的结果。

索引映射

{
    "mappings": {
        "properties": {
            "content": {
                "type": "text",
                "norms": false
            },
            "title": {
                "type": "text"
            }
        }
    }
}

并使用您的 bulk 请求索引文档,然后使用相同的搜索请求,这会产生以下预期结果:

 "hits": [
            {
                "_index": "64180913_1",
                "_type": "_doc",
                "_id": "3",
                "_score": 5.803219,
                "_source": {
                    "title": "Introducing the New React DevTools",
                    "content": "We are excited to announce a new release of accompany the React DevTools tutorial, available today in Chrome, Firefox, and (Chromium) Edge.We are excited to announce a new release of accompany the React tutorial, available today in Chrome, Firefox, and (Chromium) Edge.We are excited to announce a new release of accompany the React DevTools tutorial, available today in Chrome, Firefox, and (Chromium) Edge.We are excited to announce a new release of accompany the React DevTools tutorial, available today in Chrome, Firefox, and (Chromium) Edge.We are excited to announce a new release of accompany the React tutorial, available today in Chrome, Firefox, and (Chromium) Edge.We are excited to announce a new release of accompany the React tutorial, available today in Chrome, Firefox, and (Chromium) Edge.We are excited to announce a new release of accompany the React tutorial, available today in Chrome, Firefox, and (Chromium) EdgeWe are excited to announce a new release of accompany the React  tutorial, available today in Chrome, Firefox, and (Chromium) EdgeWe are excited to announce a new release of accompany the React tutorial, available today in Chrome, Firefox, and (Chromium) EdgeWe are excited to announce a new release of accompany the React DevTools tutorial, available today in Chrome, Firefox, and (Chromium) Edge",
                    "published_date": "2019-08-25",
                    "tags": [
                        "react",
                        "devtools"
                    ],
                    "no_of_likes": 2,
                    "status": "published"
                }
            },
            {
                "_index": "64180913_1",
                "_type": "_doc",
                "_id": "4",
                "_score": 3.5244086,
                "_source": {
                    "title": "Angular Tools for High Performance",
                    "content": "devtools", --> note this its below doc-3
                    "published_date": "2014-03-22",
                    "tags": [
                        "angular",
                        "performance",
                        "fast"
                    ],
                    "no_of_likes": 35,
                    "status": "published"
                }
            },
            {
                "_index": "64180913_1",
                "_type": "_doc",
                "_id": "1",
                "_score": 1.1478703,
                "_source": {
                    "title": "Introduction to elasticsearch",
                    "content": "Elasticsearch is a distributed, open source search slay and tutorial analytics engine for all types of data",
                    "published_date": "2020-01-02",
                    "tags": [
                        "elasticsearch",
                        "distributed",
                        "storage"
                    ],
                    "no_of_likes": 21,
                    "status": "published"
                }
            },
            {
                "_index": "64180913_1",
                "_type": "_doc",
                "_id": "2",
                "_score": 1.1478703,
                "_source": {
                    "title": "Why is Elasticsearch fast?",
                    "content": "It is able to achieve fast search responses because, instead small of tutorial searching the text directly, it searches an index instead",
                    "tags": [
                        "elasticsearch",
                        "fast",
                        "index"
                    ],
                    "no_of_likes": 10,
                    "status": "draft"
                }
            },
            {
                "_index": "64180913_1",
                "_type": "_doc",
                "_id": "5",
                "_score": 1.1478703,
                "_source": {
                    "title": "The new features in Java 14",
                    "content": "Oracle on September 17 said switch expressions tutorial are expected naresh to go final in Java Development Kit 14 (JDK 14). ",
                    "published_date": "2019-07-20",
                    "tags": [
                        "java"
                    ],
                    "no_of_likes": 11,
                    "status": "published"
                }
            }
        ]

P.S:它与synonym 无关,所以我删除了该部分以使您的问题简短明了。

【讨论】:

  • 尝试索引所有文档。结果会以其他方式出现。 doc 4 将出现在 doc 3 之上。此外,我提出这个问题的动机是在我的查询中包含协调因素。
  • @Jam 知道了,现在可以重现该问题,请稍候,我将编辑我的 asnwer
  • @Jam,没必要在这上面花更多时间,等我准备好答案后会回来
  • @Jam,已经有一段时间了,如果您可以投票并接受答案,那就太好了,因为它提供了对您问题的解释和答案。
  • @OpsterElasticsearchNinja 你去。我实际上是在私人房间里寻找答案。谢谢你的回答。会通过
猜你喜欢
  • 1970-01-01
  • 1970-01-01
  • 2017-02-26
  • 1970-01-01
  • 1970-01-01
  • 2011-08-12
  • 2023-03-17
  • 1970-01-01
  • 1970-01-01
相关资源
最近更新 更多