使用 nltk 从文本文档中构建带有标签的自定义语料库答案

【问题标题】：build custom corpus with labels from text documents using nltk使用 nltk 从文本文档中构建带有标签的自定义语料库
【发布时间】：2020-09-03 21:02:32
【问题描述】：

我正在按照教程使用 ntlk 构建自定义语料库。这是我的代码示例：

import pandas as pd
import numpy as np
import re
import nltk
import matplotlib.pyplot as plt

pd.options.display.max_colwidth = 200
%matplotlib inline

corpus = ['t_3_0 v_0_17 v_1_20 v_2_78 u_0_1 u_0_2 u_1_2',
          't_3_1 v_0_144 v_1_17 v_2_20 u_0_1 u_0_2 u_1_2',
          't_3_2 v_0_143 v_1_233 v_2_238 u_0_1 u_0_2 u_1_2',
          't_3_3 v_0_20 v_1_253 v_2_275 u_0_1 u_0_2 u_1_2',
          't_3_4 v_0_144 v_1_209 v_2_90 u_0_1 u_0_2 u_1_2',
          't_3_59 v_0_233 v_1_222 v_2_51 v_3_52 u_0_1 u_0_2 u_0_3',
          't_3_60 v_0_238 v_1_11 v_2_137 v_3_143 u_0_1 u_0_2 u_0_3',
          't_3_61 v_0_238 v_1_111 v_2_214 v_3_94 u_0_1 u_0_2 u_0_3',
          't_3_62 v_0_238 v_1_111 v_2_214 v_3_97 u_0_1 u_0_2 u_0_3',
          't_3_63 v_0_238 v_1_137 v_2_214 v_3_51 u_0_1 u_0_2 u_0_3'
             
]
labels = ['block_1', 'block_1', 'block_1', 'block_1', 'block_1',
          'block_2', 'block_2', 'block_2', 'block_2', 'block_2']

corpus = np.array(corpus)
corpus_df = pd.DataFrame({'Document': corpus, 
                          'Category': labels})
corpus_df = corpus_df[['Document', 'Category']]
corpus_df

输出如下：

    Document                                                Category
0   t_3_0 v_0_17 v_1_20 v_2_78 u_0_1 u_0_2 u_1_2            block_1
1   t_3_1 v_0_144 v_1_17 v_2_20 u_0_1 u_0_2 u_1_2           block_1
2   t_3_2 v_0_143 v_1_233 v_2_238 u_0_1 u_0_2 u_1_2         block_1
3   t_3_3 v_0_20 v_1_253 v_2_275 u_0_1 u_0_2 u_1_2          block_1
4   t_3_4 v_0_144 v_1_209 v_2_90 u_0_1 u_0_2 u_1_2          block_1
5   t_3_59 v_0_233 v_1_222 v_2_51 v_3_52 u_0_1 u_0_2 u_0_3  block_2
6   t_3_60 v_0_238 v_1_11 v_2_137 v_3_143 u_0_1 u_0_2 u_0_3 block_2
7   t_3_61 v_0_238 v_1_111 v_2_214 v_3_94 u_0_1 u_0_2 u_0_3 block_2
8   t_3_62 v_0_238 v_1_111 v_2_214 v_3_97 u_0_1 u_0_2 u_0_3 block_2
9   t_3_63 v_0_238 v_1_137 v_2_214 v_3_51 u_0_1 u_0_2 u_0_3 block_2

我想做的不是硬编码语料库和标签，而是从 txt 文档中读取每一行并使用文件名自动分配标签。例如：

corpus = ['block_1.txt', 'block_2.txt', 'block_3.txt', 'block_4.txt']

labels = ['block_1', 'block_2', 'block_3', 'block_4']

corpus = np.array(corpus)
corpus_df = pd.DataFrame({'Document': corpus, 
                          'Category': labels})
corpus_df = corpus_df[['Document', 'Category']]
corpus_df

理想输出的示例如下所示：

    Document                                                Category
0   t_3_0 v_0_17 v_1_20 v_2_78 u_0_1 u_0_2 u_1_2            block_1
1   t_3_1 v_0_144 v_1_17 v_2_20 u_0_1 u_0_2 u_1_2           block_1
2   t_3_2 v_0_143 v_1_233 v_2_238 u_0_1 u_0_2 u_1_2         block_1
...

如果有任何建议和帮助，我将不胜感激！

谢谢！奥尔哈

【问题讨论】：

标签： python nlp nltk

【解决方案1】：

我想出了答案。基本上，我创建了一个元组列表，然后对每一行进行分类。

transactions = []
with open('block_1.txt', 'r') as block1, open('block_2.txt', 'r') as block2, open('block_3.txt', 'r') as block3, open('t_block_4.txt', 'r') as block4:
    transactions = ([(transaction, 'block_1') for transaction in block1.readlines()] + [(transaction, 'block_2') for transaction in block2.readlines()] + [(transaction, 'block_3') for transaction in block3.readlines()] + [(transaction, 'block_4') for transaction in block4.readlines()])

#transactions

corpus = np.array(transactions)
corpus_df = pd.DataFrame({'Document': [i[0] for i in transactions], 
                          'Category': [i[1] for i in transactions]})
corpus_df = corpus_df[['Document', 'Category']]
corpus_df

【讨论】：