从文本列表中删除单词答案

【问题标题】：Deleting words from text list从文本列表中删除单词
【发布时间】：2016-01-19 08:18:18
【问题描述】：

我正在尝试从文本字符串列表中删除某些单词（除了使用停用词），但由于某种原因它不起作用

documents = ["Human machine interface for lab abc computer applications",
         "A survey of user opinion of computer system response time",
         "The EPS user interface management system",
         "System and human system engineering testing of EPS",
         "Relation of user perceived response time to error measurement",
         "The generation of random binary unordered trees",
         "The intersection graph of paths in trees",
         "Graph minors IV Widths of trees and well quasi ordering",
         "Graph minors A survey"]

exclude = ['am', 'there','here', 'for', 'of', 'user']

new_doc = [word for word in documents if word not in exclude]

print new_doc

输出

['Human machine interface for lab abc computer applications', 'A survey of user opinion of computer system response time', 'The EPS user interface management system', 'System and human system engineering testing of EPS', 'Relation of user perceived response time to error measurement', 'The generation of random binary unordered trees', 'The intersection graph of paths in trees', 'Graph minors IV Widths of trees and well quasi ordering', 'Graph minors A survey']

如您所见，EXCLUDE 中的任何词都不会从 DOCUMENTS 中删除（例如，“for”就是一个主要示例）

它与这个运算符一起工作：

new_doc = [word for word in str(documents).split() if word not in exclude]

但是我如何取回 DOCUMENTS 中的初始元素（尽管是“清理过的元素”）？

非常感谢您的帮助！

【问题讨论】：

word 不是一个词，而是一整行（例如"Human machine interface for lab abc computer applications"），因此永远不会出现在exclude 中。
@jonrsharpe - 刚刚添加了一个更正，但问题仍然存在（有点不同）

标签： python text stop-words

【解决方案1】：

您正在循环遍历句子而不是单词。为此，您需要拆分句子并使用嵌套循环遍历单词并过滤它们，然后加入结果。

>>> new_doc = [' '.join([word for word in sent.split() if word not in exclude]) for sent in documents]
>>> 
>>> new_doc
['Human machine interface lab abc computer applications', 'A survey opinion computer system response time', 'The EPS interface management system', 'System and human system engineering testing EPS', 'Relation perceived response time to error measurement', 'The generation random binary unordered trees', 'The intersection graph paths in trees', 'Graph minors IV Widths trees and well quasi ordering', 'Graph minors A survey']
>>>

您还可以使用regex 将exclude 单词替换为具有re.sub 函数的空字符串，而不是嵌套列表理解、拆分和过滤：

>>> import re
>>> 
>>> new_doc = [re.sub(r'|'.join(exclude),'',sent) for sent in documents]
>>> new_doc
['Human machine interface  lab abc computer applications', 'A survey   opinion  computer system response time', 'The EPS  interface management system', 'System and human system engineering testing  EPS', 'Relation   perceived response time to error measurement', 'The generation  random binary unordered trees', 'The intersection graph  paths in trees', 'Graph minors IV Widths  trees and well quasi ordering', 'Graph minors A survey']
>>>

r'|'.join(exclude) 会将单词与 pip 连接起来（表示正则表达式中的逻辑 OR）。

【讨论】：

太棒了！对于大文本，您认为哪种方法更有效？
@Toly 是的，我想是的。
你会对大文本文件使用正则表达式还是嵌套理解？
@TolyUsing 正则表达式将比拆分、循环和过滤性能更好。
@Toly 您可以使用timeit 模块在这两种方法上运行基准测试。

【解决方案2】：

您应该在过滤它们之前将行拆分为单词：

new_doc = [' '.join([word for word in line.split() if word not in exclude]) for line in documents]

【讨论】：

刚刚添加了一个更正，但问题仍然存在（有点不同）。明白了！谢谢！！