NLP中的文本过滤算法

Question

我正在制作一个问答系统，我的程序在其中从互联网读取数据并进行回复。为此，我制作了一个 java 程序来从 YQL（雅虎查询语言）中获取数据

    String baseUrl = "http://query.yahooapis.com/v1/public/yql?q=";

    String query = "select ChosenAnswer from answers.search where query=\"what is benzene ring\";

    String fullUrlStr = baseUrl + URLEncoder.encode(query, "UTF-8") + "&format=json";
    JSONObject json = readJsonFromUrl(fullUrlStr);

这些是我得到的几个结果：

     "ChosenAnswer": "A benzene ring is a hexagonal (6 sided) ring of 6 carbon atoms.  Each carbon has one single bond and one double bond and one hydrogen so that each carbon has 4 bonds total.  The double bonds can alternate with the single bonds so that the result is a pi electron cloud in a ring shape above and below the plane of the carbon ring.  Benzene rings are very common and stable.  Because they have double bonds, they are not saturated.  If you saturated benzene, you would get cyclohexane."
    }
     "ChosenAnswer": "The IUPAC name for Benzene Ring is Benzene. It forms the basis for other IUPAC-named benzene derivatives like 1,2-dimethylbenzene etc. \n\nBenzene as a substituent group is called the phenyl group. (e.g. phenylethylamine\n\nBenzene is the IUPAC name for an aromatic hydrocarbon with the formula C6H6. It is also called benzol, or cyclohexa-1,3,5-triene. \n  \n\nUses of Benzene - As an industrial solvent for fats and oils, rubber, resins etc. As a starting material for dyes, drugs, perfumes and explosives and polymers For dry-cleaning of woollen cl.."
    },
    {
     "ChosenAnswer": "Benzene rings aren't metallic bonds.  Metallic bonds have the special property of having a \"sea of electrons\" which basically means that the electrons don't really belong to any single atom and just kind of flow around the element.  That is pretty much what makes metals conduct electricity well.  Other materials that don't form metallic bonds can conduct electricity as well, but usually these consists of certain ions so they have positive or negative charges.  A benzene rings doesn't have any charge with it so it's definitely not going to conduct electricity in this way."
    },
    {
     "ChosenAnswer": "You can draw the benzene ring in any orientation you would like, point up or side up, although a point up is more common.  You can also draw the alternating single and double bonds any way you want.  Although you should keep in mind that there really aren't alternating single and double bonds.  Every C-C bond in benzene is exactly alike and the bonds have characteristics that are half-way between a single bond and a double bond.  That is why you will also see benzene with a circle in the middle.  Benzene exhibits delocalized pi bonding that accounts for the many interesting properties of C6H6.\n\n."
    },
    {
     "ChosenAnswer": "Benzene and phenyl seems look the same because they are both aromatic and all aromatic compounds are based on benzene C6H6.  Phenyl or phenyl functional group is a hydrocarbon derived from benzene by removing 1 H, making it a C6H5 then attaching it to something else."
    },
    {
     "ChosenAnswer": "since benzene is an aromatic compound it is highly stable which means it has to be activated.you can activate benzene by adding electron donor groups to his ring(electron donor groups: -OH,-CH3).the electron donor groups stabilize the ring(they help to maintain resonance of the ring by delocalizing electrons into it) while the reaction occurs.benzene undergoes the reactions called electrophilic  aromatic substitution so check that out it will be more clear to you then."
    },
    {
     "ChosenAnswer": "C6H6 is benzene.  It is a 6 membered carbon ring each carbon has one H bonded to it and ONE resonance structure is with double bonds alternating between single bonds.  Ary is a radical meaning it is used to describe the benzene ring portion of a molecule that has some other group attached where one of the H's was.  C6H12 if you are discusing a single ring structure is cyclohexane."
    },
    {
     "ChosenAnswer": "benzene ring is mostly a compound of carbon and hydrogen in a hexagon shape structure"
    },

现在这里的大多数答案都是基于意见的答案（我们都知道雅虎答案，任何人都会回复）。但我必须想办法过滤这些答案。我可以通过使用一些连接等来提高我的查询效率，或者我可以使用一些算法（可能是余弦相似度）来过滤答案并获得最大的前 3 个有效答案。请建议我一些我可以在java中实现的算法以获得相关答案。 例如，在 first 'ChosenAnswer' 上面的例子中是最合适的。 （我知道这是一个很大的话题，我只是想知道一些我可以在这里使用的好算法）

Answer 1

取决于你所说的高效。

您有很多选择，很大程度上取决于查询的类型。例如，如果高效意味着更科学，那么可以尝试将查询与 Wikipedia 概念匹配，然后使用 TFIDF 计算 Yahoo 答案和 Wiki 文档之间的距离。相似度越高，重要性越高，以此类推。

链接：http://en.wikipedia.org/wiki/Vector_space_model

您也可以实施更多的加权方案，然后使用它们的组合来获得答案的最终加权。

Answer 2

我不知道您是如何获得 ChonsenAnswers 列表的，但我认为这里的问题是您将可能的答案与什么进行比较，考虑到该查询只有几个单词，并且您将在大多数情况下获得相似的结果答案。

我会从查询中获取重要的单词（意味着没有停用词），并从 WordNet 中获取它们的同义词集定义。这样，您将有更多的词来构建 tf–idf 的频率表，或构建向量空间模型，或训练朴素贝叶斯。无论您决定使用哪种算法，这都会改善您的结果。