使用线程获取文件中每个单词的计数

Question

我目前正在尝试使用线程以并行方式获取文件中每个单词的计数，但目前当我添加一个额外的线程时，我的代码会变得更慢。我觉得随着线程的增加，时间应该会减少，直到我的 CPU 出现瓶颈，然后我的时间应该会再次变慢。我不明白为什么它不并行。

这里是代码

import thread
import threading
import time
import sys
class CountWords(threading.Thread):
    def __init__(self,lock,tuple):
        threading.Thread.__init__(self)
        self.lock = lock
        self.list = tuple[1]
        self.dit = tuple[0]
    def run(self):
        for word in self.list:
            #self.lock.acquire()
            if word in self.dit.keys():
                self.dit[word] = self.dit[word] + 1
            else:
                self.dit[word] = 1
            #self.lock.release()


def getWordsFromFile(numThreads, fileName):
    lists = []
    for i in range(int(numThreads)):
        k = []
        lists.append(k)
    print len(lists)
    file = open(fileName, "r")  # uses .read().splitlines() instead of readLines() to get rid of "\n"s
    all_words = map(lambda l: l.split(" "), file.read().splitlines()) 
    all_words = make1d(all_words)
    cur = 0
    for word in all_words:
        lists[cur].append(word)
        if cur == len(lists) - 1:
            cur = 0
        else:
            cur = cur + 1
    return lists

def make1d(list):
    newList = []
    for x in list:
        newList += x
    return newList

def printDict(dit):# prints the dictionary nicely
    for key in sorted(dit.keys()):
        print key, ":", dit[key]  



if __name__=="__main__":
    print "Starting now"
    start = int(round(time.time() * 1000))
    lock=threading.Lock()
    ditList=[]
    threadList = []
    args = sys.argv
    numThreads = args[1]
    fileName = "" + args[2]
    for i in range(int(numThreads)):
        ditList.append({})
    wordLists = getWordsFromFile(numThreads, fileName)
    zipped = zip(ditList,wordLists)
    print "got words from file"
    for tuple in zipped:
        threadList.append(CountWords(lock,tuple))
    for t in threadList:
        t.start()
    for t in threadList:
        if t.isAlive():
            t.join()
    fin = int(round(time.time() * 1000)) - start
    print "with", numThreads, "threads", "counting the words took :", fin, "ms"
    #printDict(dit)

Answer 1

您可以使用 itertools 来计算文件中的单词。下面是简单的示例代码。探索 itertools.groupby 并根据您的逻辑修改代码。

import itertools

tweets = ["I am a cat", "cat", "Who is a good cat"]

words = sorted(list(itertools.chain.from_iterable(x.split() for x in tweets)))
count = {k:len(list(v)) for k,v in itertools.groupby(words)}

Answer 2

由于 GIL (What is a global interpreter lock (GIL)?)，Python 无法并行运行线程（利用多个内核）。

向此任务添加线程只会增加代码的开销，使其变慢。

我可以说两种可以使用线程的情况：

• 当您有大量 I/O 时：线程可以使您的代码并发运行（不是并行运行https://blog.golang.org/concurrency-is-not-parallelism），因此您的代码可以在等待响应得到很好的加速时做很多事情。
• 您不希望大量计算阻塞您的代码：您使用线程与其他任务同时运行此计算。

如果您想利用所有内核，您需要使用多处理模块 (https://docs.python.org/3.6/library/multiprocessing.html)。