【发布时间】:2014-04-03 22:06:30
【问题描述】:
我正在着手编写代码以对文本进行语言分析。我使用 Python 和 NLTK 库。问题是单词的实际计数占用了我的 CPU(iCore5、8GB RAM、macbook air 2014)的近 100%,并且在我关闭进程之前运行了 14 小时。如何加快循环和计数?
我从三个瑞典语 UTF-8 格式、制表符分隔的文件 Swe_Newspapers.txt、Swe_Blogs.txt、Swe_Twitter.txt 中创建了一个 NLTK 语料库。它工作正常:
import nltk
my_corpus = nltk.corpus.CategorizedPlaintextCorpusReader(".", r"Swe_.*", cat_pattern=r"Swe_(\w+)\.txt")
然后我将一个每行一个单词的文本文件加载到 NLTK 中。这也很好用。
my_wordlist = nltk.corpus.WordListCorpusReader("/Users/mos/Documents/", "wordlist.txt")
我要分析的文本文件 (Swe_Blogs.txt) 具有这种结构,并且可以很好地解析:
Wordpress.com 2010/12/08 3 1,4,11 osv osv osv …
bloggagratis.se 2010/02/02 3 0 Jag är utled på plogade vägar, matte är lika utled hon.
wordpress.com 2010/03/10 3 0 1 kruka Sallad, riven
编辑:生成计数器的建议如下,不起作用,但可以修复:
counter = collections.Counter(word for word in my_corpus.words(categories=["Blogs"]) if word in my_wordlist)
这会产生错误:
IOError Traceback (most recent call last)
<ipython-input-41-1868952ba9b1> in <module>()
----> 1 counter = collections.Counter(word for word in my_corpus.words("Blogs") if word in my_wordlist)
/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nltk/corpus/reader/plaintext.pyc in words(self, fileids, categories)
182 def words(self, fileids=None, categories=None):
183 return PlaintextCorpusReader.words(
--> 184 self, self._resolve(fileids, categories))
185 def sents(self, fileids=None, categories=None):
186 return PlaintextCorpusReader.sents(
/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site- packages/nltk/corpus/reader/plaintext.pyc in words(self, fileids, sourced)
89 encoding=enc)
90 for (path, enc, fileid)
---> 91 in self.abspaths(fileids, True, True)])
92
93
/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nltk/corpus/reader/api.pyc in abspaths(self, fileids, include_encoding, include_fileid)
165 fileids = [fileids]
166
--> 167 paths = [self._root.join(f) for f in fileids]
168
169 if include_encoding and include_fileid:
/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/ lib/python2.7/site-packages/nltk/data.pyc in join(self, fileid)
174 def join(self, fileid):
175 path = os.path.join(self._path, *fileid.split('/'))
--> 176 return FileSystemPathPointer(path)
177
178 def __repr__(self):
/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/ lib/python2.7/site-packages/nltk/data.pyc in __init__(self, path)
152 path = os.path.abspath(path)
153 if not os.path.exists(path):
--> 154 raise IOError('No such file or directory: %r' % path)
155 self._path = path
IOError: No such file or directory: '/Users/mos/Documents/Blogs'
解决方法是将 my_corpus(categories=["Blogs"] 分配给一个变量:
blogs_text = my_corpus.words(categories=["Blogs"])
当我尝试在语料库 (115,7 MB) 的博客中的单词列表中计算每个单词(大约 20K 单词)的所有出现次数时,我的计算机有点累了。如何加快以下代码的速度?它似乎工作,没有错误消息,但它需要> 14小时才能执行。
import collections
counter = collections.Counter()
for word in my_corpus.words(categories="Blogs"):
for token in my_wordlist.words():
if token == word:
counter[token]+=1
else:
continue
非常感谢任何有助于提高我的编码技能的帮助!
【问题讨论】:
标签: python-2.7 optimization nlp nltk