如何将文本从文本文件转换为具有词频值的库键？

Question

我正在尝试从具有多个关键字的四个不同文本文件中提取信息。我想提取这些关键字并将词频附加到关键字上。文本文件如下所示：

test1 = apple banana lemon
test2 = apple banana
test3 = lemon apple lemon
test4 =  apple lemon grape

我认为粗体代码（第二段）有问题，我不确定应该如何构造初始字典。

test1= [line.rstrip('\n') for line in open("test1.txt")]
test2= [line.rstrip('\n') for line in open("test2.txt")]
test3= [line.rstrip('\n') for line in open("test3.txt")]
test4= [line.rstrip('\n') for line in open("test4.txt")]

**
text_file = test1, test2, test3, test4
word_frequencies = 0
text_collection = {}
**

def dictionary(text):
    keywords = re.split(r'\W', text)
    print(text)
    word_frequencies = dict()
    for word in keyword:
        if word in word_frequences:
            word_frequences[word] += 1
        else:
            word_frequencies[word] = 1
    return word_frequencies

for all in text_file:
    file = open(all)
    text = file.read()
    print(file)
    text_collection[all] = dictionary(text)
print(text_collection)

期望的输出：

{'test1.txt': {'apple': 1, 'banana': 1, 'lemon': 1},
'test2.txt': {'apple': 1, 'banana': 1},
'test3.txt': {'apple': 1, 'lemon': 2},
'test4.txt': {'apple': 1, 'lemon': 1, 'grape': 1}}

我宁愿不使用导入的库作为答案。这段代码更多的是为了练习而不是效率:)

Answer 1

重复使用来自Efficiently count word frequencies in python 的代码并稍作修改

from collections import Counter
from itertools import chain
import pprint

def file_word_counts(filename):
    " Word count of file "
    # Use intertools.Counter to count words
    # Convert counter result to regular dict (i.e. dict(Counter(..))
    with open(filename) as f:
        return dict(Counter(chain.from_iterable(map(str.split, f))))

def file_counts(files):
  " Aggregate word count of muiltiple files into dictionary "
  return {filename:file_word_counts(filename) for filename in files}

# Show Results
pp = pprint.PrettyPrinter(indent=4)

pp.pprint(file_counts(['test1.txt', 'test2.txt', 'test3.txt', 'test4.txt']))

输出

{   'test1.txt': {'apple': 1, 'banana': 1, 'lemon': 1},    
    'test2.txt': {'apple': 1, 'banana': 1},    
    'test3.txt': {'apple': 1, 'lemon': 2},
    'test4.txt': {'apple': 1, 'grape': 1, 'lemon': 1}}

替代方案

在不使用额外模块的情况下产生相同的效果

def file_counts(files):
  " Aggregate word count of muiltiple files into dictionary "
  return {filename:file_word_counts(filename) for filename in files}

def file_word_counts(filename):
    " Word count of file "
    count_ = {}
    with open(filename) as f:
      for line in f:
        for i in line.rstrip().split():
          count_.setdefault(i, 0)
          count_[i] += 1
      return count_

def file_counts(files):
  " Aggregate word count of muiltiple files into dictionary "
  return {filename:file_word_counts(filename) for filename in files}

print(file_counts(['test1.txt', 'test2.txt', 'test3.txt', 'test4.txt']))