对 solr 结果中的 score 字段感到困惑

Question

我对 solr 结果有点困惑。我有索引 query 字段

schema.xml 配置

<field name="question" type="text_query" indexed="true" stored="true" multiValued="false"/>


<fieldType name="text_query" class="solr.TextField" positionIncrementGap="100">
      <analyzer type="index">
         <tokenizer class="solr.StandardTokenizerFactory"/>
         <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
        <filter class="solr.LowerCaseFilterFactory"/>
      </analyzer>

      <analyzer type="query">
         <tokenizer class="solr.StandardTokenizerFactory"/>
         <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
         <filter class="solr.EnglishMinimalStemFilterFactory"/>
         <filter class="solr.LowerCaseFilterFactory"/>
      </analyzer>

</fieldType>

我在 solr 中有以下 6 个文档的索引

  [
            {
             "query": "who is gandhi",
             "source": "quora.com",
             "id" : 1
            },
            {
             "query": "who is person know as gandhi",
             "source": "quora.com",
             "id" : 2
            },
            {
             "query": "who is Sachin",
             "source": "quora.com",
             "id" : 3
            },
            {
             "query": "who is mahatma gandhi",
             "source": "quora.com",
             "id" : 4
            },
            {
             "query": "who is gandhis",
             "source": "quora.com",
             "id" : 5
            },
            {
             "query": "who are gandhi brothers",
             "source": "quora.com",
             "id" : 6
            }
]

当我搜索 query:"gandhi" 时，我得到的结果为

来自 solr 的回应

"response": {
    "numFound": 4,
    "start": 0,
    "maxScore": 0.8048013,
    "docs": [
      {
        "query": "who is gandhi btothers",
        "id": "6",
        "source": "quora.com",
        "_version_": 1513810901444067300,
        "score": 0.8048013
      },
      {
        "query": "who is person know as gandhi",
        "id": "2",
        "source": "quora.com",
        "_version_": 1513810901436727300,
        "score": 0.643841
      },
      {
        "query": "who is gandhi",
        "id": "1",
        "source": "quora.com",
        "_version_": 1513810901428338700,
        "score": 0.5945348
      },
      {
        "query": "who is mahatma gandhi",
        "id": "4",
        "source": "quora.com",
        "_version_": 1513810901431484400,
        "score": 0.37158427
      }
    ]
  }

根据我的配置，我认为我应该得到低于 ma​​xscore

的结果

{
    "query": "who is gandhi",
    "id": "1",
    "source": "quora.com",
    "_version_": 1513810901428338700,
    "score": 0.5945348
}

解释调试中的字段

"explain": {
      "1": "\n0.5945348 = (MATCH) weight(query:gandhi in 0) [DefaultSimilarity], result of:\n  0.5945348 = score(doc=0,freq=1.0), product of:\n    0.99999994 = queryWeight, product of:\n      0.5945349 = idf(docFreq=2, maxDocs=2)\n      1.681987 = queryNorm\n    0.5945349 = fieldWeight in 0, product of:\n      1.0 = tf(freq=1.0), with freq of:\n        1.0 = termFreq=1.0\n      0.5945349 = idf(docFreq=2, maxDocs=2)\n      1.0 = fieldNorm(doc=0)\n",
      "2": "\n0.643841 = (MATCH) weight(query:gandhi in 0) [DefaultSimilarity], result of:\n  0.643841 = fieldWeight in 0, product of:\n    1.0 = tf(freq=1.0), with freq of:\n      1.0 = termFreq=1.0\n    1.287682 = idf(docFreq=2, maxDocs=4)\n    0.5 = fieldNorm(doc=0)\n",
      "4": "\n0.37158427 = (MATCH) weight(query:gandhi in 1) [DefaultSimilarity], result of:\n  0.37158427 = score(doc=1,freq=1.0), product of:\n    0.99999994 = queryWeight, product of:\n      0.5945349 = idf(docFreq=2, maxDocs=2)\n      1.681987 = queryNorm\n    0.3715843 = fieldWeight in 1, product of:\n      1.0 = tf(freq=1.0), with freq of:\n        1.0 = termFreq=1.0\n      0.5945349 = idf(docFreq=2, maxDocs=2)\n      0.625 = fieldNorm(doc=1)\n",
      "6": "\n0.8048013 = (MATCH) weight(query:gandhi in 3) [DefaultSimilarity], result of:\n  0.8048013 = fieldWeight in 3, product of:\n    1.0 = tf(freq=1.0), with freq of:\n      1.0 = termFreq=1.0\n    1.287682 = idf(docFreq=2, maxDocs=4)\n    0.625 = fieldNorm(doc=3)\n"
    }

但结果有所不同。为什么会这样？帮助赞赏:)

Answer 1

只是补充一下我认为的实际解释 - 如果您自己进行分片（也称为“传统模式”），则分数是在每个分片上自行计算的。当您有少量文档（或文档不是随机分布在分片中）时，每个分片的分数可能与您在检索到最终结果时所认为的不完全一致。

这不是配置问题，只是响应节点合并之前在每个分片上计算分数的结果。

只要 shard#2 不再比 shard#1 多 50% 的文档（debugQuery 输出中的 maxdoc= 值），差异就会消失。如果您有几百万个文档，则两个文档的差异不会太大，但是当分片与其包含的内容之间存在 50% 的差异时，这将对分数的影响更大。

见“Distributing Documents across Shards”：

在传统的分布式模式中，Solr 不计算通用术语/文档频率。对于大多数大规模实施，Solr 在分片级别计算 TD/IDF 可能并不重要。但是，如果您的集合在服务器之间的分布严重倾斜，您可能会在搜索中发现误导性的相关性结果。一般来说，最好将文档随机分布到您的分片中。

如果您稍后切换到在 SolrCloud 模式下运行，则该部分计算应该以集合方式完成，而不是针对本地每个分片（就像在传统模式下手动分片一样）。

Answer 2

似乎问题出在评分算法上。我的 doc 1 任何 doc 2 都没有相同的算法。不同算法的原因是多个分片。如果我使用单个分片，我会得到写入输出

"response": {
"numFound": 4,
"start": 0,
"maxScore": 1.1823215,
"docs": [
  {
    "query": "who is gandhi",
    "id": "1",
    "source": "quora.com",
    "_version_": 1513817219957522400,
    "score": 1.1823215
  },
  {
    "query": "who is mahatma gandhi",
    "id": "4",
    "source": "quora.com",
    "_version_": 1513817220025680000,
    "score": 0.73895097
  },
  {
    "query": "who are gandhi brothers",
    "id": "6",
    "source": "quora.com",
    "_version_": 1513817220027777000,
    "score": 0.73895097
  },
  {
    "query": "who is person know as gandhi",
    "id": "2",
    "source": "quora.com",
    "_version_": 1513817220023582700,
    "score": 0.5911608
  }
]

} }