短语查询的 Lucene 评分

Question

我使用 StandardAnalyzer 来索引我的文本。但是，在查询时，我正在进行术语查询和短语查询。对于术语查询和短语查询，我相信 lucene 在计算术语频率和短语频率方面没有问题。然而，这对于像 Dirichlet Similarity 这样的模型来说很好。对于 BM25Similarity 或 TFIDFSimilarity 模型，它需要 IDF(term) 和 IDF(Phrase)。 lucene 如何处理这个问题？

Answer 1

TFIDFSimilarity 短语 IDF 计算为其组成术语的 IDF 之和。即：idf("ab cd") = idf(ab) + idf(cd)

然后将该值乘以短语频率，并将其视为非常像一个术语，用于评分。

要了解整个故事，我认为看一个例子是最有意义的。 IndexSearcher.explain 是一个非常有用理解评分的工具：

索引：

• doc 0：文本 ab 唯一
• 文档 1：文本
• doc 2: text ab cd text ab
• 文档 3：文本

查询："text ab" unique

Explain 第一个（最高得分）命中的输出（doc 0）：

1.3350155 = (MATCH) sum of:
  0.7981777 = (MATCH) weight(content:"text ab" in 0) [DefaultSimilarity], result of:
    0.7981777 = score(doc=0,freq=1.0 = phraseFreq=1.0
), product of:
      0.7732263 = queryWeight, product of:
        2.0645385 = idf(), sum of:
          0.7768564 = idf(docFreq=4, maxDocs=4)
          1.287682 = idf(docFreq=2, maxDocs=4)
        0.37452745 = queryNorm
      1.0322692 = fieldWeight in 0, product of:
        1.0 = tf(freq=1.0), with freq of:
          1.0 = phraseFreq=1.0
        2.0645385 = idf(), sum of:
          0.7768564 = idf(docFreq=4, maxDocs=4)
          1.287682 = idf(docFreq=2, maxDocs=4)
        0.5 = fieldNorm(doc=0)
  0.5368378 = (MATCH) weight(content:unique in 0) [DefaultSimilarity], result of:
    0.5368378 = score(doc=0,freq=1.0 = termFreq=1.0
), product of:
      0.6341301 = queryWeight, product of:
        1.6931472 = idf(docFreq=1, maxDocs=4)
        0.37452745 = queryNorm
      0.8465736 = fieldWeight in 0, product of:
        1.0 = tf(freq=1.0), with freq of:
          1.0 = termFreq=1.0
        1.6931472 = idf(docFreq=1, maxDocs=4)
        0.5 = fieldNorm(doc=0)

请注意，前半部分处理查询的"text ab" 部分的评分与后半部分（评分unique）的算法非常相似，除了为短语 idf 计算添加了求和。

Explain 第二次命中的输出（为了更好的衡量标准）（文档 2）：

0.49384725 = (MATCH) product of:
  0.9876945 = (MATCH) sum of:
    0.9876945 = (MATCH) weight(content:"text ab" in 2) [DefaultSimilarity], result of:
      0.9876945 = score(doc=2,freq=2.0 = phraseFreq=2.0
), product of:
        0.7732263 = queryWeight, product of:
          2.0645385 = idf(), sum of:
            0.7768564 = idf(docFreq=4, maxDocs=4)
            1.287682 = idf(docFreq=2, maxDocs=4)
          0.37452745 = queryNorm
        1.277368 = fieldWeight in 2, product of:
          1.4142135 = tf(freq=2.0), with freq of:
            2.0 = phraseFreq=2.0
          2.0645385 = idf(), sum of:
            0.7768564 = idf(docFreq=4, maxDocs=4)
            1.287682 = idf(docFreq=2, maxDocs=4)
          0.4375 = fieldNorm(doc=2)
  0.5 = coord(1/2)