Lucene：字段的相似性（BM25）

Question

我想用Lucene的IndexSearcher来计算文档之间的相似度。确切地说，我有一个输入文档，并且想要计算与索引中所有其他文档的相似度。我已经掌握了基本功能，但现在我有一些问题在网上没有找到答案。

• 为什么当我调用is.search(query, Integer.MAX_VALUE) 时IndexSearcher 只返回两个结果？我会期待三个结果。
• 我的方法中是否存在一些我目前没有发现的错误？
• 如何处理多种语言？据我所知，IndexWriter 和QueryParser 应该都具有相同的分析器（在我的示例中为StandardAnalyzer）。如果我使用三种不同的语言，我是否必须创建三个索引？

SSCCE（我使用的是 Lucene 5.3.0）：

public class Main {

    public static void main(String[] args) throws Exception {
        Path path = Paths.get("temp_directoty");

        // create index
        createIndexAndAddDocuments(path);

        // open index reader and create index searcher
        IndexReader ir = DirectoryReader.open(FSDirectory.open(path));
        IndexSearcher is = new IndexSearcher(ir);
        is.setSimilarity(new BM25Similarity());

        // document which is used to create the query
        Document doc = ir.document(1);

        // create query parser
        QueryParser queryParser = new QueryParser("Abstract", new StandardAnalyzer());

        // create query
        Query query = queryParser.parse(doc.get("Abstract"));

        // search
        for (ScoreDoc result : is.search(query, Integer.MAX_VALUE).scoreDocs) {
            System.out.println(result.doc + "\t" + result.score);
        }
    }

    private static void createIndexAndAddDocuments(Path indexPath) throws IOException {
        // create documents
        Document doc1 = new Document();
        doc1.add(new TextField("Title", "Apparatus for manufacturing green bricks for the brick manufacturing industry",
                Store.YES));
        doc1.add(new TextField("Abstract",
                "The invention relates to an apparatus (1) for manufacturing green bricks from clay for the brick manufacturing industry, comprising a circulating conveyor (3) carrying mould containers combined to mould container parts (4), a reservoir (5) for clay arranged above the mould containers, means for carrying clay out of the reservoir (5) into the mould containers, means (9) for pressing and trimming clay in the mould containers, means (11) for supplying and placing take-off plates for the green bricks (13) and means for discharging green bricks released from the mould containers, characterized in that the apparatus further comprises means (22) for moving the mould container parts (4) filled with green bricks such that a protruding edge is formed on at least one side of the green bricks",
                Store.YES));

        Document doc2 = new Document();
        doc2.add(new TextField("Title",
                "Some other title, for example: Apparatus for manufacturing green bricks for the brick manufacturing industry",
                Store.YES));
        doc2.add(new TextField("Abstract",
                "Some other abstract, for example: The invention relates to an apparatus (1) for manufacturing green bricks from clay for the brick manufacturing industry, comprising a circulating conveyor (3) carrying mould containers combined to mould container parts (4), a reservoir (5) for clay arranged above the mould containers, means for carrying clay out of the reservoir (5) into the mould containers, means (9) for pressing and trimming clay in the mould containers, means (11) for supplying and placing take-off plates for the green bricks (13) and means for discharging green bricks released from the mould containers, characterized in that the apparatus further comprises means (22) for moving the mould container parts (4) filled with green bricks such that a protruding edge is formed on at least one side of the green bricks",
                Store.YES));

        Document doc3 = new Document();
        doc3.add(new TextField("Title", "A document with a competely different title", Store.YES));
        doc3.add(new TextField("Abstract",
                "This document also has a completely different abstract which is in no way similar to the abstract of the previous documents.",
                Store.YES));

        IndexWriter iw = new IndexWriter(FSDirectory.open(indexPath), new IndexWriterConfig(new StandardAnalyzer()));
        iw.deleteAll();
        iw.addDocument(doc1);
        iw.addDocument(doc2);
        iw.addDocument(doc3);
        iw.close();
    }
}

Answer 1

我发现您只有 2 个结果的问题。您在createIndexAndAddDocuments 中仅创建了 doc1 和 doc2，之后您在不初始化 doc3 的情况下覆盖了 doc2。 关于我要回答的语言的问题：这取决于您是要单独搜索语句还是全部搜索语句。如果你想区分语言，你需要不同的索引。

希望对你有帮助。