【问题标题】:Testing the NLTK classifier on specific file在特定文件上测试 NLTK 分类器
【发布时间】:2015-05-31 20:32:43
【问题描述】:

以下代码运行朴素贝叶斯电影评论分类器。 该代码生成一个信息量最大的功能列表。

注意:**movie review** 文件夹位于nltk

from itertools import chain
from nltk.corpus import stopwords
from nltk.probability import FreqDist
from nltk.classify import NaiveBayesClassifier
from nltk.corpus import movie_reviews
stop = stopwords.words('english')

documents = [([w for w in movie_reviews.words(i) if w.lower() not in stop and w.lower() not in string.punctuation], i.split('/')[0]) for i in movie_reviews.fileids()]


word_features = FreqDist(chain(*[i for i,j in documents]))
word_features = word_features.keys()[:100]

numtrain = int(len(documents) * 90 / 100)
train_set = [({i:(i in tokens) for i in word_features}, tag) for tokens,tag in documents[:numtrain]]
test_set = [({i:(i in tokens) for i in word_features}, tag) for tokens,tag  in documents[numtrain:]]

classifier = NaiveBayesClassifier.train(train_set)
print nltk.classify.accuracy(classifier, test_set)
classifier.show_most_informative_features(5)

link of code 来自alvas

如何测试特定文件上的分类器?

如果我的问题有歧义或错误,请告诉我。

【问题讨论】:

    标签: python-2.7 nlp classification nltk text-classification


    【解决方案1】:

    您可以使用分类器.classify() 对一个文件进行测试。此方法将字典作为其输入,该字典以特征作为其键,True 或 False 作为其值,具体取决于该特征是否出现在文档中。根据分类器,它为文件输出最可能的标签。然后,您可以将此标签与文件的正确标签进行比较,以查看分类是否正确。

    在您的训练和测试集中,特征字典始终是元组中的第一项,标签是元组中的第二项。

    因此,您可以像这样对测试集中的第一个文档进行分类:

    (my_document, my_label) = test_set[0]
    if classifier.classify(my_document) == my_label:
        print "correct!"
    else:
        print "incorrect!"
    

    【讨论】:

    • 能否请你给我一个完整的例子,如果你的例子可能是根据我的例子。我对 Python 很陌生。你能告诉我你为什么在test_set[0]中写0
    • 这是一个完整的示例:如果您在问题中的代码之后立即粘贴代码,它将起作用。 0 只是获取测试集中的第一个文档(列表中的第一项索引为 0)。
    • 非常感谢。有没有办法在test_set[0] 中写name_of_file 而不是0?我不知道,test_set 确切地指示了哪个文件,因为我们有 2 个文件夹 pos|neg 并且每个文件夹都有它的文件。我问这个是因为most informative 这个词是bad(我的例子的结果)。第一个文件有一百多个‘bad’字。但是程序在输出中显示incorrect。我的错在哪里?
    • 首先,test_set 不包含文件名,因此如果您想使用它来识别文件,一种方法是直接读取文件并将其作为特征传递给分类器我上面描述的字典。其次,您当前的分类器使用二元特征。它只是检查一个词是否出现在文档中,但忽略该词出现的频率。这可能就是为什么它错误地对一个出现很多 bad 的文件进行了分类。
    【解决方案2】:

    首先,请仔细阅读这些答案,它们包含您需要的部分答案,还简要说明了分类器的作用以及它在 NLTK 中的工作原理:


    在带注释的数据上测试分类器

    现在回答你的问题。我们假设您的问题是该问题的后续问题:Using my own corpus instead of movie_reviews corpus for Classification in NLTK

    如果您的测试文本的结构与 movie_review 语料库相同,那么您可以像读取训练数据一样简单地读取测试数据:

    以防万一代码解释不清楚,这里有一个演练:

    traindir = '/home/alvas/my_movie_reviews'
    mr = CategorizedPlaintextCorpusReader(traindir, r'(?!\.).*\.txt', cat_pattern=r'(neg|pos)/.*', encoding='ascii')
    

    上面两行是读取一个目录my_movie_reviews这样的结构:

    \my_movie_reviews
        \pos
            123.txt
            234.txt
        \neg
            456.txt
            789.txt
        README
    

    然后下一行提取带有pos/neg 标记的文档,该标记是目录结构的一部分。

    documents = [([w for w in mr.words(i) if w.lower() not in stop and w not in string.punctuation], i.split('/')[0]) for i in mr.fileids()]
    

    以下是对上述行的解释:

    # This extracts the pos/neg tag
    labels = [i for i.split('/')[0]) for i in mr.fileids()]
    # Reads the words from the corpus through the CategorizedPlaintextCorpusReader object
    words = [w for w in mr.words(i)]
    # Removes the stopwords
    words = [w for w in mr.words(i) if w.lower() not in stop]
    # Removes the punctuation
    words = [w for w in mr.words(i) w not in string.punctuation]
    # Removes the stopwords and punctuations
    words = [w for w in mr.words(i) if w.lower() not in stop and w not in string.punctuation]
    # Removes the stopwords and punctuations and put them in a tuple with the pos/neg labels
    documents = [([w for w in mr.words(i) if w.lower() not in stop and w not in string.punctuation], i.split('/')[0]) for i in mr.fileids()]
    

    读取测试数据时应采用SAME流程!!!

    现在到特征处理:

    以下几行为分类器增加了前 100 个特征:

    # Extract the words features and put them into FreqDist
    # object which records the no. of times each unique word occurs
    word_features = FreqDist(chain(*[i for i,j in documents]))
    # Cuts the FreqDist to the top 100 words in terms of their counts.
    word_features = word_features.keys()[:100]
    

    接下来将文档处理成可分类的格式:

    # Splits the training data into training size and testing size
    numtrain = int(len(documents) * 90 / 100)
    # Process the documents for training data
    train_set = [({i:(i in tokens) for i in word_features}, tag) for tokens,tag in documents[:numtrain]]
    # Process the documents for testing data
    test_set = [({i:(i in tokens) for i in word_features}, tag) for tokens,tag  in documents[numtrain:]]
    

    现在解释一下对train_set 和`test_set 的长列表理解:

    # Take the first `numtrain` no. of documents
    # as training documents
    train_docs = documents[:numtrain]
    # Takes the rest of the documents as test documents.
    test_docs = documents[numtrain:]
    # These extract the feature sets for the classifier
    # please look at the full explanation on https://stackoverflow.com/questions/20827741/nltk-naivebayesclassifier-training-for-sentiment-analysis/
    train_set = [({i:(i in tokens) for i in word_features}, tag) for tokens,tag  in train_docs]
    

    测试文档中的特征提取也需要像上面那样处理文档!!!

    以下是读取测试数据的方法:

    stop = stopwords.words('english')
    
    # Reads the training data.
    traindir = '/home/alvas/my_movie_reviews'
    mr = CategorizedPlaintextCorpusReader(traindir, r'(?!\.).*\.txt', cat_pattern=r'(neg|pos)/.*', encoding='ascii')
    
    # Converts training data into tuples of [(words,label), ...]
    documents = [([w for w in mr.words(i) if w.lower() not in stop and w not in string.punctuation], i.split('/')[0]) for i in mr.fileids()]
    
    # Now do the same for the testing data.
    testdir = '/home/alvas/test_reviews'
    mr_test = CategorizedPlaintextCorpusReader(testdir, r'(?!\.).*\.txt', cat_pattern=r'(neg|pos)/.*', encoding='ascii')
    # Converts testing data into tuples of [(words,label), ...]
    test_documents = [([w for w in mr_test.words(i) if w.lower() not in stop and w not in string.punctuation], i.split('/')[0]) for i in mr_test.fileids()]
    

    然后继续上述处理步骤,只需执行此操作即可获得@yvespeirsman 回答的测试文档的标签:

    #### FOR TRAINING DATA ####
    stop = stopwords.words('english')
    
    # Reads the training data.
    traindir = '/home/alvas/my_movie_reviews'
    mr = CategorizedPlaintextCorpusReader(traindir, r'(?!\.).*\.txt', cat_pattern=r'(neg|pos)/.*', encoding='ascii')
    
    # Converts training data into tuples of [(words,label), ...]
    documents = [([w for w in mr.words(i) if w.lower() not in stop and w not in string.punctuation], i.split('/')[0]) for i in mr.fileids()]
    # Extract training features.
    word_features = FreqDist(chain(*[i for i,j in documents]))
    word_features = word_features.keys()[:100]
    # Assuming that you're using full data set
    # since your test set is different.
    train_set = [({i:(i in tokens) for i in word_features}, tag) for tokens,tag  in documents]
    
    #### TRAINS THE TAGGER ####
    # Train the tagger
    classifier = NaiveBayesClassifier.train(train_set)
    
    #### FOR TESTING DATA ####
    # Now do the same reading and processing for the testing data.
    testdir = '/home/alvas/test_reviews'
    mr_test = CategorizedPlaintextCorpusReader(testdir, r'(?!\.).*\.txt', cat_pattern=r'(neg|pos)/.*', encoding='ascii')
    # Converts testing data into tuples of [(words,label), ...]
    test_documents = [([w for w in mr_test.words(i) if w.lower() not in stop and w not in string.punctuation], i.split('/')[0]) for i in mr_test.fileids()]
    # Reads test data into features:
    test_set = [({i:(i in tokens) for i in word_features}, tag) for tokens,tag  in test_documents]
    
    #### Evaluate the classifier ####
    for doc, gold_label in test_set:
        tagged_label = classifier.classify(doc)
        if tagged_label == gold_label:
            print("Woohoo, correct")
        else:
            print("Boohoo, wrong")
    

    如果上面的代码和解释对您没有意义,那么您必须在继续之前阅读本教程:http://www.nltk.org/howto/classify.html


    现在假设您的测试数据中没有注释,即您的 test.txt 不在像 movie_review 这样的目录结构中,而只是一个纯文本文件:

    \test_movie_reviews
        \1.txt
        \2.txt
    

    那么将其读入分类语料库是没有意义的,您可以简单地阅读并标记文档,即:

    for infile in os.listdir(`test_movie_reviews): 
      for line in open(infile, 'r'):
           tagged_label = classifier.classify(doc)
    

    但是你不能在没有注释的情况下评估结果,所以如果if-else,你不能检查标签,而且你需要标记你的文本如果你'不使用 CategorizedPlaintextCorpusReader。

    如果你只想标记一个纯文本文件test.txt

    import string
    from itertools import chain
    from nltk.corpus import stopwords
    from nltk.probability import FreqDist
    from nltk.classify import NaiveBayesClassifier
    from nltk.corpus import movie_reviews
    from nltk import word_tokenize
    
    stop = stopwords.words('english')
    
    # Extracts the documents.
    documents = [([w for w in movie_reviews.words(i) if w.lower() not in stop and w.lower() not in string.punctuation], i.split('/')[0]) for i in movie_reviews.fileids()]
    # Extract the features.
    word_features = FreqDist(chain(*[i for i,j in documents]))
    word_features = word_features.keys()[:100]
    # Converts documents to features.
    train_set = [({i:(i in tokens) for i in word_features}, tag) for tokens,tag in documents]
    # Train the classifier.
    classifier = NaiveBayesClassifier.train(train_set)
    
    # Tag the test file.
    with open('test.txt', 'r') as fin:
        for test_sentence in fin:
            # Tokenize the line.
            doc = word_tokenize(test_sentence.lower())
            featurized_doc = {i:(i in doc) for i in word_features}
            tagged_label = classifier.classify(featurized_doc)
            print(tagged_label)
    

    再一次,请不要只是复制和粘贴解决方案,并尝试了解它的工作原理和方式。

    【讨论】:

    • 感谢您的完整解释,我尝试理解它们。但我经常遇到错误的结果。我的意思是它应该是pos,但程序显示neg。我不知道原因。
    • 有很多原因,并不完美,可能(i)数据不足,(ii)特征不够好,(iii)分类器选择等。请参加此课程coursera.org/course/ml了解更多信息。如果可以的话,我强烈建议您参加lxmls.it.pt/2015
    • 您通过找出正确的频率来评估输出。分类器学习要注意哪些特征,以及如何将它们结合起来做出决定。没有逻辑规则,都是统计数据和权重。您的文件cv081.txtpos 的形式出现,并带有您的功能集——还有什么要理解的?
    • 通过 coursea 链接上的机器学习课程,您将了解分类器的工作原理和方式。我开始将它们用作黑匣子,一旦您了解它们如何生成注释,就更容易编码并欣赏它们的优雅。
    • 第一种情况是您有注释数据进行测试,第二种情况是您没有。如果您需要我们验证代码的输出,您能否将完整的数据集发布到某个地方以便我们进行测试(当我们有空时)?
    猜你喜欢
    • 2018-09-04
    • 2012-02-05
    • 1970-01-01
    • 2013-10-05
    • 2018-09-11
    • 2017-01-14
    • 2014-01-06
    • 2018-12-21
    相关资源
    最近更新 更多