Python 词性标注：训练后返回一个词最常见的词性

Question

我正在尝试编写一个代码，该代码将创建一个包含键（单词）、值（单词出现的 POS 标记 + 相应计数）的字典。 最终目标是了解给定单词最常用的 POS 标签

一个例子： most_common({"NOUN": 2, "DET": 5, "ADP": 1 }) returns "DET" 因为给定的单词作为限定词出现频率最高。

首先，我想在一个带注释的小型语料库上训练我的代码。这是我目前所拥有的：

import pprint

trainfile = open("small_train.connlu")
list_of_lists = []
for line in trainfile:
    stripped_line = line.strip()
    line_list = stripped_line.split()
    list_of_lists.append(line_list)

list_keys = [] #a list that will contain all the keys (including duplicates)
list_values = [] #a list that will contain all the values

for line in list_of_lists:
    if line == []:
        pass
    elif line != []:
        list_keys.append(line[1]) # second column of the file contains all words
        list_values.append(line[3]) # fourth column of the file contains all POS tags (see below)

list_keys = [key.lower() for key in list_keys] #lowercase all keys - 'The' and 'the' should be assigned the same POS

我被困在这一点上。我现在需要创建一个字典，其中包含出现在语料库中的所有单词，然后是它们出现的相应 POS 标签（以及每个单词与某个 POS 标签出现的次数）。这是我得到的最接近的：

dict = {}

for key in range(len(list_keys)):
    dict[list_keys[key]] = list_values[key]

pprint.pprint(dict)

这会返回带有正确 POS 标签的键，但是，我不知道如何实现计数。我尝试过的任何事情都会导致错误。

这是训练数据的格式 (small_train.connlu)

1   The       _   DET   _ _ _ _ _ _
2   hottest   _   ADJ   _ _ _ _ _ _
3   item      _   NOUN  _ _ _ _ _ _
4   on        _   ADP   _ _ _ _ _ _
5   Christmas _   PROPN _ _ _ _ _ _
6   wish      _   NOUN  _ _ _ _ _ _
7   lists     _   NOUN  _ _ _ _ _ _
8   this      _   DET   _ _ _ _ _ _
9   year      _   NOUN  _ _ _ _ _ _
10  is        _   AUX   _ _ _ _ _ _
11  nuclear   _   ADJ   _ _ _ _ _ _
12  weapons   _   NOUN  _ _ _ _ _ _
13  .         _   PUNCT _ _ _ _ _ _

1   I         _   PRON  _ _ _ _ _ _
2   wish      _   VERB  _ _ _ _ _ _
3   you       _   PRON  _ _ _ _ _ _
4   all       _   DET   _ _ _ _ _ _
5   of        _   ADP   _ _ _ _ _ _
6   the       _   DET   _ _ _ _ _ _
7   best      _   ADJ   _ _ _ _ _ _

如果有人能提供帮助，我将不胜感激。非常感谢:)

Answer 1

假设您的 list_of_lists 格式如下：

list_of_lists=[["","THE","","DET"],["","hottest","","ADJ"],["","the","","NOUN"],["","the","","DET"]]

您可以创建一个字典，其中键是单词，值是另一个字典，其中键是 pos 标签，值是对应单词的频率。

# A dictionary of dictionaries
word_map = {}

# Iterate through the list of lists that has been prepared after parsing the file.
for line in list_of_lists:
        # Convert the word to lowercase
        word = line[1].lower()
        pos_tag = line[3]
        
        # If we have seen the word before.
        if word in word_map:
            
            # Get the inner dictionary for this word which contains the frequency of the pos tags for this word
            pos_tags_for_current_word = word_map[word]
             
            # If this pos tag has already been seen for this word then increment the frequency by 1
            if pos_tag in pos_tags_for_current_word:
                pos_tags_for_current_word[pos_tag] += 1
            # If the pos tag has been seen for the first time then add a new entry to the dictionary with a  frequency 1
            else:
                pos_tags_for_current_word[pos_tag] = 1
        
        # If the word has been seen for the first time then add a new entry for the word with value as a dictionary and the current pos tag with frequency 1.
        else:
            word_map[word] = {pos_tag:1}

打印 word_map 给我们

{'hottest': {'ADJ': 1}, 'the': {'DET': 2, 'NOUN': 1}}

现在遍历这个字典，找到每个单词出现频率最高的 pos 标签。

for word in word_map:
        print(word, max(word_map[word].items(), key = lambda entry : entry[1]))

这是上面提到的for循环的输出：

the ('DET', 2)
hottest ('ADJ', 1)

这只是在任何字典中找出具有最大值的条目的一种简写方式。

max(word_map[word].items(), key = lambda entry : entry[1])

Answer 2

这是一种完整的、更实用的方法，可以将文件解析为一种格式，您可以在（dict 列表）上执行其他操作并查找最常见的单词类型：

def parse_connlu(lines: list) -> list:
    # return a list of dict of the connlu file
    items = [[item for item in line.split()
              if item != '']
             for line in lines
             if line != '']
    return [{'pos': int(item[0]),
             'word': item[1].lower(),
             'type': item[3]}
            for item in items]

def read_connlu(path: str) -> list:
    # read the connlu file (will return list[dict])
    with open(path, 'r') as file:
        lines = file.read().splitlines()
        return parse_connlu(lines)

def count(items: list) -> dict:
    # count unique elements in list
    unique = set(items)
    return {i: items.count(i) for i in unique}

def most_common(items: list):
    # get the most commen item
    counts = count(items)
    # {'NOUN': 5, 'PROPN': 1, 'VERB': 1, 'ADP': 2, 'DET': 4, 'PRON': 2, 'AUX': 1, 'PUNCT': 1, 'ADJ': 3}
    return max(counts, key=counts.get)

words = read_connlu('./small_train.connlu')
# gives us:
# [{'pos': 1, 'type': 'DET', 'word': 'the'},
#  {'pos': 2, 'type': 'ADJ', 'word': 'hottest'},
# ...

answer = most_common([i['type'] for i in words])
print(answer)
# gives us: NOUN