如何从 NLTK 树中获取令牌？答案

【问题标题】：How to get the tokens from an NLTK Tree?如何从 NLTK 树中获取令牌？
【发布时间】：2016-11-22 10:30:58
【问题描述】：

所以我把这棵树还给了我

Tree('S', [('This', 'DT'), ('is', 'VBZ'), ('a', 'DT'), ('test', 'NN'), (',', ','), Tree('PERSON', [('Stackoverflow', 'NNP'), ('Users', 'NNP')]), ('.', '.')])

我可以像这样把它变成一个漂亮的 python 列表

sentence = "This is a test, Stackoverflow Users."
tokens = nltk.word_tokenize(sentence)
tagged = nltk.pos_tag(tokens)
entities = nltk.chunk.ne_chunk(tagged)
tree = repr(entities) # THIS VARIABLE IS THE TREE THAT IS RETURNED TO ME
# below this point it's about turning the tree into a python list
tree = (("[" + tree[5:-1] + "]")).replace("Tree", "").replace(")", "]").replace("(", "[")
tree = ast.literal_eval(tree) #you'll need to import ast (included with python)

现在，树变量是这样的：

['S', [['This', 'DT'], ['is', 'VBZ'], ['a', 'DT'], ['test', 'NN'], [',', ','], ['ORGANIZATION', [['Stackoverflow', 'NNP']]], ['users', 'NNS'], ['.', '.']]]

当我尝试遍历并获取句子的字符串时，我得到了

"This is a test, ORGANIZATION."

而不是想要的

"This is a test, Stackoverflow users."

我不能简单地使用句子变量，我需要能够从这个列表列表中取回句子。任何代码 sn-ps 或建议将不胜感激。

【问题讨论】：

我必须安装 nltk 并使用 nltk.download() 下载一些包。现在我被困住了。什么是ast？它没有在您的代码中定义。你能编辑你的问题吗？
对不起！尝试在代码顶部添加“import ast”，它包含在 python 中。

标签： python list python-2.7 nlp nltk

【解决方案1】：

>>> from nltk import Tree
>>> yourtree = Tree('S', [('This', 'DT'), ('is', 'VBZ'), ('a', 'DT'), ('test', 'NN'), (',', ','), Tree('PERSON', [('Stackoverflow', 'NNP'), ('Users', 'NNP')]), ('.', '.')])
>>> yourtree.leaves()
[('This', 'DT'), ('is', 'VBZ'), ('a', 'DT'), ('test', 'NN'), (',', ','), ('Stackoverflow', 'NNP'), ('Users', 'NNP'), ('.', '.')]
>>> tokens, pos = zip(*yourtree.leaves())
>>> tokens
('This', 'is', 'a', 'test', ',', 'Stackoverflow', 'Users', '.')
>>> pos
('DT', 'VBZ', 'DT', 'NN', ',', 'NNP', 'NNP', '.')

另请参阅：How to Traverse an NLTK Tree object?

【讨论】：