【发布时间】：2017-09-12 13:49:42
【问题描述】：
我对 ES 比较陌生，我一直在研究 ES 中的评分，以尝试提高搜索结果的质量。我遇到了一种情况，其中queryNorm 函数在分片之间非常不同（5 倍大）。我可以看到查询中的术语对idf 的依赖性，这在各个分片中可能不同。但是，就我而言，我有一个搜索词 + 跨分片的 idf 度量彼此接近（绝对不足以导致 X 5 倍的差异）。我将简要描述我的设置，包括我的查询和解释端点的结果。
设置 我有一个包含约 6500 个文档的索引，这些文档分布在 5 个分片中。我提到下面查询中出现的字段没有索引时间提升。我提到我的设置使用带有“query_then_fetch”的 ES 2.4。我的查询：
{
"query" : {
"bool" : {
"must" : [ {
"bool" : {
"must" : [ ],
"must_not" : [ ],
"should" : [ {
"multi_match" : {
"query" : "pds",
"fields" : [ "field1" ],
"lenient" : true,
"fuzziness" : "0"
}
}, {
"multi_match" : {
"query" : "pds",
"fields" : [ "field2" ],
"lenient" : true,
"fuzziness" : "0",
"boost" : 1000.0
}
}, {
"multi_match" : {
"query" : "pds",
"fields" : [ "field3" ],
"lenient" : true,
"fuzziness" : "0",
"boost" : 500.0
}
}, {
"multi_match" : {
"query" : "pds",
"fields" : [ "field4" ],
"lenient" : true,
"fuzziness" : "0",
"boost": 100.0
}
} ],
"must_not" : [ ],
"should" : [ ],
"filter" : [ ]
}
},
"size" : 1000,
"min_score" : 0.0
}
解释其中 2 个文档的输出（一个的查询规范是另一个的 5 倍）：
{
"_shard" : 4,
"_explanation" : {
"value" : 2.046937,
"description" : "product of:",
"details" : [ {
"value" : 4.093874,
"description" : "sum of:",
"details" : [ {
"value" : 0.112607226,
"description" : "weight(field1:pds in 93) [PerFieldSimilarity], result of:",
"details" : [ {
"value" : 0.112607226,
"description" : "score(doc=93,freq=1.0), product of:",
"details" : [ {
"value" : 0.019996,
"description" : "queryWeight, product of:",
"details" : [ {
"value" : 2.0,
"description" : "boost",
"details" : [ ]
}, {
"value" : 5.6314874,
"description" : "idf(docFreq=11, maxDocs=1232)",
"details" : [ ]
}, {
"value" : 0.0017753748,
"description" : "queryNorm",
"details" : [ ]
} ]
}, {
"value" : 5.6314874,
"description" : "fieldWeight in 93, product of:",
"details" : [ {
"value" : 1.0,
"description" : "tf(freq=1.0), with freq of:",
"details" : [ {
"value" : 1.0,
"description" : "termFreq=1.0",
"details" : [ ]
} ]
}, {
"value" : 5.6314874,
"description" : "idf(docFreq=11, maxDocs=1232)",
"details" : [ ]
}, {
"value" : 1.0,
"description" : "fieldNorm(doc=93)",
"details" : [ ]
} ]
} ]
} ]
}, {
"value" : 3.9812667,
"description" : "weight(field4:pds in 93) [PerFieldSimilarity], result of:",
"details" : [ {
"value" : 3.9812667,
"description" : "score(doc=93,freq=2.0), product of:",
"details" : [ {
"value" : 0.9998001,
"description" : "queryWeight, product of:",
"details" : [ {
"value" : 100.0,
"description" : "boost",
"details" : [ ]
}, {
"value" : 5.6314874,
"description" : "idf(docFreq=11, maxDocs=1232)",
"details" : [ ]
}, {
"value" : 0.0017753748,
"description" : "queryNorm",
"details" : [ ]
} ]
}, {
"value" : 3.9820628,
"description" : "fieldWeight in 93, product of:",
"details" : [ {
"value" : 1.4142135,
"description" : "tf(freq=2.0), with freq of:",
"details" : [ {
"value" : 2.0,
"description" : "termFreq=2.0",
"details" : [ ]
} ]
}, {
"value" : 5.6314874,
"description" : "idf(docFreq=11, maxDocs=1232)",
"details" : [ ]
}, {
"value" : 0.5,
"description" : "fieldNorm(doc=93)",
"details" : [ ]
} ]
} ]
} ]
} ]
}, {
"value" : 0.5,
"description" : "coord(2/4)",
"details" : [ ]
} ]
}
},
{
"_shard" : 2,
"_explanation" : {
"value" : 0.4143453,
"description" : "product of:",
"details" : [ {
"value" : 0.8286906,
"description" : "sum of:",
"details" : [ {
"value" : 0.018336227,
"description" : "weight(field1:pds in 58) [PerFieldSimilarity], result of:",
"details" : [ {
"value" : 0.018336227,
"description" : "score(doc=58,freq=1.0), product of:",
"details" : [ {
"value" : 0.0030464241,
"description" : "queryWeight, product of:",
"details" : [ {
"value" : 2.0,
"description" : "boost",
"details" : [ ]
}, {
"value" : 6.0189342,
"description" : "idf(docFreq=11, maxDocs=1815)",
"details" : [ ]
}, {
"value" : 2.5307006E-4,
"description" : "queryNorm",
"details" : [ ]
} ]
}, {
"value" : 6.0189342,
"description" : "fieldWeight in 58, product of:",
"details" : [ {
"value" : 1.0,
"description" : "tf(freq=1.0), with freq of:",
"details" : [ {
"value" : 1.0,
"description" : "termFreq=1.0",
"details" : [ ]
} ]
}, {
"value" : 6.0189342,
"description" : "idf(docFreq=11, maxDocs=1815)",
"details" : [ ]
}, {
"value" : 1.0,
"description" : "fieldNorm(doc=58)",
"details" : [ ]
} ]
} ]
} ]
}, {
"value" : 0.81035435,
"description" : "weight(field4:pds in 58) [PerFieldSimilarity], result of:",
"details" : [ {
"value" : 0.81035435,
"description" : "score(doc=58,freq=2.0), product of:",
"details" : [ {
"value" : 0.1523212,
"description" : "queryWeight, product of:",
"details" : [ {
"value" : 100.0,
"description" : "boost",
"details" : [ ]
}, {
"value" : 6.0189342,
"description" : "idf(docFreq=11, maxDocs=1815)",
"details" : [ ]
}, {
"value" : 2.5307006E-4,
"description" : "queryNorm",
"details" : [ ]
} ]
}, {
"value" : 5.3200364,
"description" : "fieldWeight in 58, product of:",
"details" : [ {
"value" : 1.4142135,
"description" : "tf(freq=2.0), with freq of:",
"details" : [ {
"value" : 2.0,
"description" : "termFreq=2.0",
"details" : [ ]
} ]
}, {
"value" : 6.0189342,
"description" : "idf(docFreq=11, maxDocs=1815)",
"details" : [ ]
}, {
"value" : 0.625,
"description" : "fieldNorm(doc=58)",
"details" : [ ]
} ]
} ]
} ]
} ]
}, {
"value" : 0.5,
"description" : "coord(2/4)",
"details" : [ ]
} ]
}
}
请注意，分片 4 中文档的 field1 上的 queryNorm 是“0.0017753748”（idf 为 5.6314874），而分片 2 中文档的同一字段的 queryNorm 是“0.0002.5307006”（与 idf 6.0189342）。我尝试使用 http://lucene.apache.org/core/4_0_0/core/org/apache/lucene/search/similarities/TFIDFSimilarity.html 上的公式手动计算 queryNorm ，但未能得到相同的答案。
我没有看到太多关于计算queryNorm 的线程/帖子；我发现有用的一个是http://www.openjems.com/tag/querynorm/（这实际上是 Solr，但由于查询是“query_then_fetch”；Lucene 计算应该是唯一重要的事情，所以我希望它们的行为应该类似）。但是，我无法使用相同的方法得出正确的 queryNorm 值（据我所知，t.getBoost() 在我的情况下应该是 1，因为没有索引时间字段提升 + 没有特殊字段提升上面的查询）。
有人对这里可能发生的事情有任何建议吗？
【问题讨论】：
标签： elasticsearch lucene