Elasticsearch - 匹配更多术语的文档得分低于匹配更少的文档答案

【问题标题】：Elasticseach - document that match more terms has lower score than one that match fewerElasticsearch - 匹配更多术语的文档得分低于匹配更少的文档
【发布时间】：2017-02-12 19:32:53
【问题描述】：

我有一个查询应该返回具有相似兴趣的个人资料。问题是具有更多匹配词的文档得分较低。

在bool 查询中，我有should 和interests = ['games', 'music', 'sport']

interests = ['games'] 的文档得分为 0.14981213

interests = ['games', 'music'] 的文档得分为 0.11516824。

为什么？我正在使用 AWS elasticsearch，v. 2.3.2。

查询看起来像：

{
    "explain": true,
    "from": 0,
    "query": {
        "bool": {
            "filter": [
                {
                    "bool": {
                        "must_not": [
                            {
                                "term": {
                                    "id": 3918
                                }
                            }
                        ]
                    }
                }
            ],
            "should": [
                {
                    "terms": {
                        "interests": [
                            "games",
                            "music",
                            "sport"
                        ]
                    }
                }
            ]
        }
    },
    "size": 10
}

然后，我得到的结果：

{
    "_shards": {
        "failed": 0,
        "successful": 5,
        "total": 5
    },
    "hits": {
        "hits": [
            {
                "_explanation": {
                    "description": "sum of:",
                    "details": [
                        {
                            "description": "match on required clause, product of:",
                            "details": [
                                {
                                    "description": "# clause",
                                    "details": [],
                                    "value": 0.0
                                },
                                {
                                    "description": "-id:`\b\u0000\u0000\u001eN #*:*, product of:",
                                    "details": [
                                        {
                                            "description": "boost",
                                            "details": [],
                                            "value": 1.0
                                        },
                                        {
                                            "description": "queryNorm",
                                            "details": [],
                                            "value": 0.4494364
                                        }
                                    ],
                                    "value": 0.4494364
                                }
                            ],
                            "value": 0.0
                        },
                        {
                            "description": "product of:",
                            "details": [
                                {
                                    "description": "sum of:",
                                    "details": [
                                        {
                                            "description": "weight(interests:games in 1) [PerFieldSimilarity], result of:",
                                            "details": [
                                                {
                                                    "description": "score(doc=1,freq=1.0), product of:",
                                                    "details": [
                                                        {
                                                            "description": "queryWeight, product of:",
                                                            "details": [
                                                                {
                                                                    "description": "idf(docFreq=2, maxDocs=3)",
                                                                    "details": [],
                                                                    "value": 1.0
                                                                },
                                                                {
                                                                    "description": "queryNorm",
                                                                    "details": [],
                                                                    "value": 0.4494364
                                                                }
                                                            ],
                                                            "value": 0.4494364
                                                        },
                                                        {
                                                            "description": "fieldWeight in 1, product of:",
                                                            "details": [
                                                                {
                                                                    "description": "tf(freq=1.0), with freq of:",
                                                                    "details": [
                                                                        {
                                                                            "description": "termFreq=1.0",
                                                                            "details": [],
                                                                            "value": 1.0
                                                                        }
                                                                    ],
                                                                    "value": 1.0
                                                                },
                                                                {
                                                                    "description": "idf(docFreq=2, maxDocs=3)",
                                                                    "details": [],
                                                                    "value": 1.0
                                                                },
                                                                {
                                                                    "description": "fieldNorm(doc=1)",
                                                                    "details": [],
                                                                    "value": 1.0
                                                                }
                                                            ],
                                                            "value": 1.0
                                                        }
                                                    ],
                                                    "value": 0.4494364
                                                }
                                            ],
                                            "value": 0.4494364
                                        }
                                    ],
                                    "value": 0.4494364
                                },
                                {
                                    "description": "coord(1/3)",
                                    "details": [],
                                    "value": 0.33333334
                                }
                            ],
                            "value": 0.14981213
                        }
                    ],
                    "value": 0.14981213
                },
                "_id": "3917",
                "_index": "test_44024988_profiles",
                "_node": "urWXg5KhREyffYielaa6Rw",
                "_score": 0.14981213,
                "_shard": 2,
                "_source": {
                    "full_name": "Bob Doe",
                    "id": 3916,
                    "interests": [
                        "games"
                    ],
                    "user_id": 3917
                },
                "_type": "profile_document"
            },
            {
                "_explanation": {
                    "description": "sum of:",
                    "details": [
                        {
                            "description": "match on required clause, product of:",
                            "details": [
                                {
                                    "description": "# clause",
                                    "details": [],
                                    "value": 0.0
                                },
                                {
                                    "description": "-id:`\b\u0000\u0000\u001eN #*:*, product of:",
                                    "details": [
                                        {
                                            "description": "boost",
                                            "details": [],
                                            "value": 1.0
                                        },
                                        {
                                            "description": "queryNorm",
                                            "details": [],
                                            "value": 0.9173473
                                        }
                                    ],
                                    "value": 0.9173473
                                }
                            ],
                            "value": 0.0
                        },
                        {
                            "description": "product of:",
                            "details": [
                                {
                                    "description": "sum of:",
                                    "details": [
                                        {
                                            "description": "weight(interests:games in 0) [PerFieldSimilarity], result of:",
                                            "details": [
                                                {
                                                    "description": "score(doc=0,freq=1.0), product of:",
                                                    "details": [
                                                        {
                                                            "description": "queryWeight, product of:",
                                                            "details": [
                                                                {
                                                                    "description": "idf(docFreq=1, maxDocs=1)",
                                                                    "details": [],
                                                                    "value": 0.30685282
                                                                },
                                                                {
                                                                    "description": "queryNorm",
                                                                    "details": [],
                                                                    "value": 0.9173473
                                                                }
                                                            ],
                                                            "value": 0.2814906
                                                        },
                                                        {
                                                            "description": "fieldWeight in 0, product of:",
                                                            "details": [
                                                                {
                                                                    "description": "tf(freq=1.0), with freq of:",
                                                                    "details": [
                                                                        {
                                                                            "description": "termFreq=1.0",
                                                                            "details": [],
                                                                            "value": 1.0
                                                                        }
                                                                    ],
                                                                    "value": 1.0
                                                                },
                                                                {
                                                                    "description": "idf(docFreq=1, maxDocs=1)",
                                                                    "details": [],
                                                                    "value": 0.30685282
                                                                },
                                                                {
                                                                    "description": "fieldNorm(doc=0)",
                                                                    "details": [],
                                                                    "value": 1.0
                                                                }
                                                            ],
                                                            "value": 0.30685282
                                                        }
                                                    ],
                                                    "value": 0.08637618
                                                }
                                            ],
                                            "value": 0.08637618
                                        },
                                        {
                                            "description": "weight(interests:music in 0) [PerFieldSimilarity], result of:",
                                            "details": [
                                                {
                                                    "description": "score(doc=0,freq=1.0), product of:",
                                                    "details": [
                                                        {
                                                            "description": "queryWeight, product of:",
                                                            "details": [
                                                                {
                                                                    "description": "idf(docFreq=1, maxDocs=1)",
                                                                    "details": [],
                                                                    "value": 0.30685282
                                                                },
                                                                {
                                                                    "description": "queryNorm",
                                                                    "details": [],
                                                                    "value": 0.9173473
                                                                }
                                                            ],
                                                            "value": 0.2814906
                                                        },
                                                        {
                                                            "description": "fieldWeight in 0, product of:",
                                                            "details": [
                                                                {
                                                                    "description": "tf(freq=1.0), with freq of:",
                                                                    "details": [
                                                                        {
                                                                            "description": "termFreq=1.0",
                                                                            "details": [],
                                                                            "value": 1.0
                                                                        }
                                                                    ],
                                                                    "value": 1.0
                                                                },
                                                                {
                                                                    "description": "idf(docFreq=1, maxDocs=1)",
                                                                    "details": [],
                                                                    "value": 0.30685282
                                                                },
                                                                {
                                                                    "description": "fieldNorm(doc=0)",
                                                                    "details": [],
                                                                    "value": 1.0
                                                                }
                                                            ],
                                                            "value": 0.30685282
                                                        }
                                                    ],
                                                    "value": 0.08637618
                                                }
                                            ],
                                            "value": 0.08637618
                                        }
                                    ],
                                    "value": 0.17275237
                                },
                                {
                                    "description": "coord(2/3)",
                                    "details": [],
                                    "value": 0.6666667
                                }
                            ],
                            "value": 0.11516824
                        }
                    ],
                    "value": 0.11516824
                },
                "_id": "3918",
                "_index": "test_44024988_profiles",
                "_node": "urWXg5KhREyffYielaa6Rw",
                "_score": 0.11516824,
                "_shard": 4,
                "_source": {
                    "full_name": "Alex Test",
                    "id": 3917,
                    "interests": [
                        "games",
                        "music"
                    ],
                    "user_id": 3918
                },
                "_type": "profile_document"
            },
            ... # not interesting doc
        ],
        "max_score": 0.14981213,
        "total": 3
    },
    "timed_out": false,
    "took": 3
}

我的输入数据：

[{
    "full_name": "Bob Doe",
    "id": 3916,
    "interests": [
        "games"
    ],
    "user_id": 3917
}, {
    "full_name": "Alex Test",
    "id": 3917,
    "interests": [
        "games",
        "music"
    ],
    "user_id": 3918
}, {
    "full_name": "Joe Test",
    "id": 3918,
    "user_id": 3919
}]

【问题讨论】：

标签： elasticsearch

【解决方案1】：

让我们看看 Elasticsearch 中的评分公式。

score(q,d)  =  
            queryNorm(q)  
          · coord(q,d)    
          · ∑ (           
                tf(t in d)   
              · idf(t)²      
              · t.getBoost() 
              · norm(t,d)    
            ) (t in q)

参考是practical scoring formula，如果你不知道的话，你可以在这里得到一些描述。但是您的案例的解释将非常简单，它只是做这些事情的公式，以及所有这些因素的组合（tf、idf、queryNorm 等）。此外，如果您的索引是虚拟的并且仅包含几个文档，则可能会发生这种情况，那么这些值会很奇怪。

我可以深入解释，但主要是一个评分公式。如果你想修复它，那是另一个问题，你可以通过不同的查询来做到这一点

【讨论】：

嘿！感谢您的答复。我理解公式，但现在有一个问题 - 公式是错误的还是我的期望？我认为filter 不应该影响分数，should 作为查询应该可以直接工作。
是的，你是对的，过滤器不会影响评分，这正是你的情况，你只是从条款查询中获得分数。问题是，我们可以手动计算 tf idf，看看公式是否完全相同，相信我会的。 tf idf 很棘手，因为它考虑到了这个词的稀有性
我没有说分数与公式给出的分数不同。让我们同意它考虑到公式是正确的，但我只是想知道考虑到常见的用户期望它是否正确。但也许那只是我。另一件事是它似乎不稳定。围绕这个问题的更多背景是，这是我在单元测试中在 CI 服务器上得到的，在我的本地机器上，分数是“正确的”（符合我的期望）。即使使用相同的 elasticsearch 运行，只是索引名称不同。
您能提供完整的样本数据吗？
我已添加到原始问题的底部。