【问题标题】:Creating a feature dictionary for Python machine learning (naive bayes) algorithm为 Python 机器学习(朴素贝叶斯)算法创建特征字典
【发布时间】:2015-06-08 20:31:30
【问题描述】:

我想预测,例如,使用姓氏的华人与非华人种族。特别是我想从姓氏中提取三个字母的子字符串。例如,姓“gao”将给出一个特征为“gao”,而“chan”将给出两个特征为“cha”和“han”。

在下面的three_split函数中拆分成功。但据我了解,要将其合并为功能集,我需要将输出作为字典返回。关于如何做到这一点的任何提示?对于“Chan”的字典,字典应该返回“cha”和“han”为TRUE。

from nltk.classify import PositiveNaiveBayesClassifier
import re

chinese_names = ['gao', 'chan', 'chen', 'Tsai', 'liu', 'Lee']

nonchinese_names = ['silva', 'anderson', 'kidd', 'bryant', 'Jones', 'harris', 'davis']

def three_split(word):
    word = word.lower()
    word = word.replace(" ", "_")
    split = 3
    return [word[start:start+split] for start in range(0, len(word)-2)]

positive_featuresets = list(map(three_split, chinese_names))
unlabeled_featuresets = list(map(three_split, nonchinese_names))
classifier = PositiveNaiveBayesClassifier.train(positive_featuresets, 
    unlabeled_featuresets)

print three_split("Jim Silva")
print classifier.classify(three_split("Jim Silva"))

【问题讨论】:

    标签: python-2.7 dictionary machine-learning nltk feature-extraction


    【解决方案1】:

    经过反复试验,我想我已经掌握了。谢谢。

    from nltk.classify import PositiveNaiveBayesClassifier
    import re
    
    chinese_names = ['gao', 'chan', 'chen', 'Tsai', 'liu', 'Lee']
    
    nonchinese_names = ['silva', 'anderson', 'kidd', 'bryant', 'Jones', 'harris', 'davis']
    
    def three_split(word):
        word = word.lower()
        word = word.replace(" ", "_")
        split = 3
        return dict(("contains(%s)" % word[start:start+split], True) 
            for start in range(0, len(word)-2))
    
    positive_featuresets = list(map(three_split, chinese_names))
    unlabeled_featuresets = list(map(three_split, nonchinese_names))
    classifier = PositiveNaiveBayesClassifier.train(positive_featuresets, 
        unlabeled_featuresets)
    
    name = "dennis kidd"
    print three_split(name)
    print classifier.classify(three_split(name))
    

    【讨论】:

      【解决方案2】:

      这是一个白盒答案:

      使用您的原始代码,它会输出:

      Traceback (most recent call last):
        File "test.py", line 17, in <module>
          unlabeled_featuresets)
        File "/usr/local/lib/python2.7/dist-packages/nltk/classify/positivenaivebayes.py", line 108, in train
          for fname, fval in featureset.items():
      AttributeError: 'list' object has no attribute 'items'
      

      查看第 17 行:

      classifier = PositiveNaiveBayesClassifier.train(positive_featuresets, 
          unlabeled_featuresets)
      

      似乎PositiveNaiveBayesClassifier 需要一个具有'.items()' 属性的对象,如果NLTK 代码是pythonic,那么直观上它应该是dict。

      查看https://github.com/nltk/nltk/blob/develop/nltk/classify/positivenaivebayes.py#L88,对于positive_featuresets 参数应该包含的内容没有任何明确的解释:

      :param positive_featuresets:被称为特征集的列表 正例(即他们的标签是True)。

      检查文档字符串,我们看到了这个例子:

      Example:
          >>> from nltk.classify import PositiveNaiveBayesClassifier
      Some sentences about sports:
          >>> sports_sentences = [ 'The team dominated the game',
          ...                      'They lost the ball',
          ...                      'The game was intense',
          ...                      'The goalkeeper catched the ball',
          ...                      'The other team controlled the ball' ]
      Mixed topics, including sports:
          >>> various_sentences = [ 'The President did not comment',
          ...                       'I lost the keys',
          ...                       'The team won the game',
          ...                       'Sara has two kids',
          ...                       'The ball went off the court',
          ...                       'They had the ball for the whole game',
          ...                       'The show is over' ]
      The features of a sentence are simply the words it contains:
          >>> def features(sentence):
          ...     words = sentence.lower().split()
          ...     return dict(('contains(%s)' % w, True) for w in words)
      We use the sports sentences as positive examples, the mixed ones ad unlabeled examples:
          >>> positive_featuresets = list(map(features, sports_sentences))
          >>> unlabeled_featuresets = list(map(features, various_sentences))
          >>> classifier = PositiveNaiveBayesClassifier.train(positive_featuresets,
          ...                                                 unlabeled_featuresets)
      

      现在我们找到了feature()函数,它将句子转换为特征并返回

      dict(('contains(%s)' % w, True) for w in words)
      

      基本上这是可以调用.items()的东西。看看字典理解,'contains(%s)' % w 似乎有点多余,除非它是为了人类可读性。所以你可以直接使用dict((w, True) for w in words)。

      此外,用下划线替换空格也可能是多余的,除非以后会用到它。

      最后,切片和有限迭代可以替换为可以提取字符 ngram 的 ngram 函数,例如

      >>> word = 'alexgao'
      >>> split=3
      >>> [word[start:start+split] for start in range(0, len(word)-2)]
      ['ale', 'lex', 'exg', 'xga', 'gao']
      # With ngrams
      >>> from nltk.util import ngrams
      >>> ["".join(ng) for ng in ngrams(word,3)]
      ['ale', 'lex', 'exg', 'xga', 'gao']
      

      您的特征提取功能可以这样简化:

      from nltk.util import ngrams
      def three_split(word):
          return dict(("".join(ng, True) for ng in ngrams(word.lower(),3))
      

      [出]:

      {'im ': True, 'm s': True, 'jim': True, 'ilv': True, ' si': True, 'lva': True, 'sil': True}
      False
      

      事实上,NLTK 分类器用途广泛,您可以使用字符元组作为特征,因此在提取特征时无需修补 ngram,即:

      from nltk.classify import PositiveNaiveBayesClassifier
      import re
      from nltk.util import ngrams
      
      chinese_names = ['gao', 'chan', 'chen', 'Tsai', 'liu', 'Lee']
      
      nonchinese_names = ['silva', 'anderson', 'kidd', 'bryant', 'Jones', 'harris', 'davis']
      
      
      def three_split(word):
          return dict(((ng, True) for ng in ngrams(word.lower(),3))
      
      positive_featuresets = list(map(three_split, chinese_names))
      unlabeled_featuresets = list(map(three_split, nonchinese_names))
      
      classifier = PositiveNaiveBayesClassifier.train(positive_featuresets, 
          unlabeled_featuresets)
      
      print three_split("Jim Silva")
      print classifier.classify(three_split("Jim Silva"))
      

      [出]:

      {('m', ' ', 's'): True, ('j', 'i', 'm'): True, ('s', 'i', 'l'): True, ('i', 'l', 'v'): True, (' ', 's', 'i'): True, ('l', 'v', 'a'): True, ('i', 'm', ' '): True}
      

      【讨论】:

        猜你喜欢
        • 2016-05-01
        • 2016-12-11
        • 2016-06-10
        • 2016-03-10
        • 2012-04-28
        • 2019-03-01
        • 2021-01-13
        • 2019-03-15
        • 2019-04-29
        相关资源
        最近更新 更多