java Lucene 最佳匹配不是完全匹配答案

【问题标题】：java Lucene best match is not an exact matchjava Lucene 最佳匹配不是完全匹配
【发布时间】：2015-04-09 14:43:21
【问题描述】：

Lucene 评分似乎完全让我无法理解。

我有一组文件用于以下内容：

Senior Education Recruitment Consultant
Senior IT Recruitment Consultant
Senior Recruitment Consultant

这些已使用EnglishAnalyzer 进行了分析。

搜索查询也是使用EnglishAnalyzer 构建的QueryParser。

当我搜索Senior Recruitment Consultant 时，上述每个文档都以相同的分数返回，其中期望（和预期）的结果将是Senior Recruitment Consultant 作为最高结果。

是否有一种直接的方法可以实现我错过的预期行为？

这是我的调试输出：

4.6491017 = (MATCH) sum of:
  1.1064172 = (MATCH) weight(Title:senior in 22157) [DefaultSimilarity], result of:
    1.1064172 = score(doc=22157,freq=1.0 = termFreq=1.0
), product of:
      0.4878372 = queryWeight, product of:
        4.53601 = idf(docFreq=818, maxDocs=28116)
        0.10754765 = queryNorm
      2.268005 = fieldWeight in 22157, product of:
        1.0 = tf(freq=1.0), with freq of:
          1.0 = termFreq=1.0
        4.53601 = idf(docFreq=818, maxDocs=28116)
        0.5 = fieldNorm(doc=22157)
  2.3421772 = (MATCH) weight(Title:recruit in 22157) [DefaultSimilarity], result of:
    2.3421772 = score(doc=22157,freq=1.0 = termFreq=1.0
), product of:
      0.70978254 = queryWeight, product of:
        6.5997033 = idf(docFreq=103, maxDocs=28116)
        0.10754765 = queryNorm
      3.2998517 = fieldWeight in 22157, product of:
        1.0 = tf(freq=1.0), with freq of:
          1.0 = termFreq=1.0
        6.5997033 = idf(docFreq=103, maxDocs=28116)
        0.5 = fieldNorm(doc=22157)
  1.2005073 = (MATCH) weight(Title:consult in 22157) [DefaultSimilarity], result of:
    1.2005073 = score(doc=22157,freq=1.0 = termFreq=1.0
), product of:
      0.50815696 = queryWeight, product of:
        4.724947 = idf(docFreq=677, maxDocs=28116)
        0.10754765 = queryNorm
      2.3624735 = fieldWeight in 22157, product of:
        1.0 = tf(freq=1.0), with freq of:
          1.0 = termFreq=1.0
        4.724947 = idf(docFreq=677, maxDocs=28116)
        0.5 = fieldNorm(doc=22157)

4.6491017 = (MATCH) sum of:
  1.1064172 = (MATCH) weight(Title:senior in 22292) [DefaultSimilarity], result of:
    1.1064172 = score(doc=22292,freq=1.0 = termFreq=1.0
), product of:
      0.4878372 = queryWeight, product of:
        4.53601 = idf(docFreq=818, maxDocs=28116)
        0.10754765 = queryNorm
      2.268005 = fieldWeight in 22292, product of:
        1.0 = tf(freq=1.0), with freq of:
          1.0 = termFreq=1.0
        4.53601 = idf(docFreq=818, maxDocs=28116)
        0.5 = fieldNorm(doc=22292)
  2.3421772 = (MATCH) weight(Title:recruit in 22292) [DefaultSimilarity], result of:
    2.3421772 = score(doc=22292,freq=1.0 = termFreq=1.0
), product of:
      0.70978254 = queryWeight, product of:
        6.5997033 = idf(docFreq=103, maxDocs=28116)
        0.10754765 = queryNorm
      3.2998517 = fieldWeight in 22292, product of:
        1.0 = tf(freq=1.0), with freq of:
          1.0 = termFreq=1.0
        6.5997033 = idf(docFreq=103, maxDocs=28116)
        0.5 = fieldNorm(doc=22292)
  1.2005073 = (MATCH) weight(Title:consult in 22292) [DefaultSimilarity], result of:
    1.2005073 = score(doc=22292,freq=1.0 = termFreq=1.0
), product of:
      0.50815696 = queryWeight, product of:
        4.724947 = idf(docFreq=677, maxDocs=28116)
        0.10754765 = queryNorm
      2.3624735 = fieldWeight in 22292, product of:
        1.0 = tf(freq=1.0), with freq of:
          1.0 = termFreq=1.0
        4.724947 = idf(docFreq=677, maxDocs=28116)
        0.5 = fieldNorm(doc=22292)

4.6491017 = (MATCH) sum of:
  1.1064172 = (MATCH) weight(Title:senior in 22494) [DefaultSimilarity], result of:
    1.1064172 = score(doc=22494,freq=1.0 = termFreq=1.0
), product of:
      0.4878372 = queryWeight, product of:
        4.53601 = idf(docFreq=818, maxDocs=28116)
        0.10754765 = queryNorm
      2.268005 = fieldWeight in 22494, product of:
        1.0 = tf(freq=1.0), with freq of:
          1.0 = termFreq=1.0
        4.53601 = idf(docFreq=818, maxDocs=28116)
        0.5 = fieldNorm(doc=22494)
  2.3421772 = (MATCH) weight(Title:recruit in 22494) [DefaultSimilarity], result of:
    2.3421772 = score(doc=22494,freq=1.0 = termFreq=1.0
), product of:
      0.70978254 = queryWeight, product of:
        6.5997033 = idf(docFreq=103, maxDocs=28116)
        0.10754765 = queryNorm
      3.2998517 = fieldWeight in 22494, product of:
        1.0 = tf(freq=1.0), with freq of:
          1.0 = termFreq=1.0
        6.5997033 = idf(docFreq=103, maxDocs=28116)
        0.5 = fieldNorm(doc=22494)
  1.2005073 = (MATCH) weight(Title:consult in 22494) [DefaultSimilarity], result of:
    1.2005073 = score(doc=22494,freq=1.0 = termFreq=1.0
), product of:
      0.50815696 = queryWeight, product of:
        4.724947 = idf(docFreq=677, maxDocs=28116)
        0.10754765 = queryNorm
      2.3624735 = fieldWeight in 22494, product of:
        1.0 = tf(freq=1.0), with freq of:
          1.0 = termFreq=1.0
        4.724947 = idf(docFreq=677, maxDocs=28116)
        0.5 = fieldNorm(doc=22494)


Senior Education Recruitment Consultant 4.6491017
Senior IT Recruitment Consultant 4.6491017
Senior Recruitment Consultant 4.6491017

【问题讨论】：

标签： java lucene

【解决方案1】：

您必须依赖的唯一评分元素是长度规范。

Lengthnorm 在索引时与文档一起存储，以及字段的提升。它可以使较短的文档得分更高一些。

那么为什么它不起作用？你有两个问题：

首先：规范以极有损压缩方式存储。它们只占用一个字节，并且具有大约 1 个有效十进制数字的精度。因此，基本上，差异不足以影响分数。

关于这种损失的原因，来自DefaultSimilarity documentation：

...鉴于用户通过查询表达其真实信息需求的困难（和不准确），只有大的差异才重要。

第二：“IT”是英语中的停用词。您的意思是“信息技术”，但分析仪看到的只是常见的英语代词。而且无论您在字段中输入多少停用词，它们都不会影响长度规范。

这是一个显示我得出的一些结果的测试：

Senior Education Recruitment Consultant ::: 0.732527
Senior IT Recruitment Consultant ::: 0.732527
Senior Recruitment Consultant ::: 0.732527
if and but Senior IT IT IT IT IT Recruitment this and that Consultant ::: 0.732527
Senior Education Recruitment Consultant Of Justice ::: 0.64096117
Senior Recruitment Consultant and some other nonsense we don't want to know about ::: 0.3662635

如您所见，通过“司法高级教育招聘顾问”，我们只添加了一个搜索词，而 lengthnorm 开始发挥作用。但是对于“if and but Senior IT IT IT IT IT IT Recruitment this that Consultant”仍然没有区别，因为所有添加的术语都是常见的英语停用词。

解决方案：您可以使用自定义相似性实现来解决规范精度问题，该实现不会那么难编码（复制DefaultSimilarity，并实现无损encodeNormValue 和decodeNormValue）。您还可以使用自定义或空的停用词列表（通过EnglishAnalyzer ctor）设置分析器。

但是，这可能会把婴儿和洗澡水一起扔出去。如果精确匹配的得分更高真的很重要，那么通过使用查询来表达这一点可能会更好地为您服务，如下所示：

\"Senior Recruitment Consultant\" Senior Recruitment Consultant

结果：

Senior Recruitment Consultant ::: 1.465054
Senior Recruitment Consultant and some other nonsense we don't want to know about ::: 0.732527
Senior Education Recruitment Consultant ::: 0.27469763
Senior IT Recruitment Consultant ::: 0.27469763
if and but Senior IT IT IT IT IT Recruitment this and that Consultant ::: 0.27469763
Senior Education Recruitment Consultant Of Justice ::: 0.24036042

【讨论】：

感谢您的简洁回复。您能否详细说明您最后建议的查询？这是整个查询吗？第一部分要编入索引，第二部分是查询吗？干杯。
这就是整个查询，假设您的查询解析器的默认字段可以在这里使用。本质上，它结合了一个短语查询和三个简单的术语查询。你可以认为它是：\"Senior Recruitment Consultant\" OR Senior OR Recruitment OR Consultant。如果有帮助，您可以查看我的测试实现：pastebin.com/YRMCdWaV（使用 Lucene 4.10.2）。

【解决方案2】：

正常的lucene排名是基于频率的，不考虑词之间的距离。

但是，您可以添加邻近搜索词，这需要预先定义距离内的词才能达到目的（但是您需要知道查询中有多少词。

SO上有类似问题的答案 Lucene.Net: Relevancy by distance between words

【讨论】：

我给出的上述示例并不是整个用例，而且很可能不会发生精确的字符串查询。比如本例可以搜索“Senior Recruitment Consultant (Existing Clients) Manchester”，恰好也出现了同样的问题。
首先尝试如果“高级招聘顾问（现有客户）”~10 适用于多个单词，我相信它确实如此，那么您只需将其与单个单词进行 OR，这样它就不会排除不属于的结果紧挨着。在将搜索短语传递给 lucene 之前，您需要对搜索短语进行标记并创建一个新短语。
自己的查询构建器其实写起来也不算太难，但是你需要搜索现有的QueryParser的代码，看看用哪种Query类型来表示“(Existing Clients)”~10搜索，因为我不能轻易地用谷歌搜索它。
我尝试了“高级招聘顾问（现有客户）”~10，在这种情况下结果是一样的。
通过尝试，我的意思是确实可以正确处理多个短语单词。如果你得到搜索结果，它会。使用“Existing Clients”~0 来获得完全匹配（顺便说一下，“”很重要）和 OR 这个短语与分散的一个。