提取具有周围上下文的多词短语

Question

我编写了一些代码，用于查找某些关键字前后的三个单词，这些关键字包含在多个列表中，这些列表随后加入一个更大的列表（代码中的列表“单词”）。它将输出写入 .csv 文件，适用于单个单词，但不适用于短语。为了说明，有一系列包含诸如

之类的词的列表

approx = ["approximately"]
could = ["could"]

以及诸如

之类的短语

can_be = ["can be"]
shouldbeso = ["should be so"]

这些被连接成一个更大的列表，我们称之为“单词”。程序可以找到“大约”、“可以”或类似词左右的所有 3 个词，但完全漏掉了短语。

有没有办法修改代码来检测特定短语前后的三个单词？我的代码如下。

path = 'D:/Testing'

context_d = {}
for filename in glob.glob(os.path.join(path, '*.txt')):
    if filename.endswith('.txt'):
        f = open(filename)
        file = f.read()
       # txt = file.lower()
        txt = file.split()
        txt = [item.replace('May', '') for item in txt]  # locate and replace all months of May before lowering
        # txt = list([[word.lower() for word in line.split()] for line in txt])
        txt = (list(map(lambda x: x.lower(), txt)))
        for j in range(len(txt)):
          if (j + 3) < len(txt):
           if txt[j] in words:
            if txt[j] in context_d:
             context_d[txt[j]] += txt[(j - 3):j]
             context_d[txt[j]] += txt[(j + 1):(j + 3)]
            else:
             context_d[txt[j]] = txt[(j - 3):j]
             context_d[txt[j]] += txt[(j + 1):(j + 3)]

Answer 1

用 uno.txt 试过：

gallina sparviero Mulo tonno  ASINO maiale foca cavallo boa struzzo May May
peperone cipolla aglio cavolo oca gallo bue pippo mucca

due.txt：

pizza pasta tagliolini acciughe peperone burro oca cavolo baccala
pollo pizza pasta gamberi peperone uva cipolla grano
pesce gatto mulo ASINO verdura broccoli pippo 'hamburger formaggio' panna

tre.txt：

pinguino leone mummia 'oca gallo' levriero bassotto pecora capra

和prova.py：

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Created on Mon Dec 28 16:11:33 2020

@author: Pietro
"""

import glob
import os
import shlex


path = './'

words = ['pippo' ,'papera', 'asino', 'ciuchino' , 'peperone' ,'oca gallo' , 'hamburger formaggio']


print('\n\n keywords to search for : ',words,'\n\n')

context_d = {}
for filename in glob.glob(os.path.join(path, '*.txt')):
    if filename.endswith('.txt'):
        f = open(filename)
        file = f.read()
#        print('\nprinting file :  \n\n'+file)
       # txt = file.lower()
#        print('++++++++hhhhhhhh++++++\n\n',file)
#        txt = file.split()
        txt = shlex.split(file)
#        print('+++++++++++++++++++++++++++\n\n',txt)
        txt = [item.replace('May', '') for item in txt]  # locate and replace all months of May before lowering
       #txt = list([[word.lower() for word in line.split()] for line in txt])
        txt = (list(map(lambda x: x.lower(), txt)))
        print('\n\n printing txt : \n\n',txt,'\n\n')
        for j in range(len(txt)):
#          if (j + 3) < len(txt):
           if txt[j] in words:
            if txt[j] in context_d:
             context_d[txt[j]] += txt[(j - 3):j]
             context_d[txt[j]] += txt[(j + 1):(j + 4)]
            else:
             context_d[txt[j]] = txt[(j - 3):j]
             context_d[txt[j]] += txt[(j + 1):(j + 4)]

#    print('\n\n printing context_d : \n\n ', context_d,'\n\n')
    

for i in context_d:
    context_d[i] = list(set(context_d[i]))
    
for i in context_d:
    print(i , context_d[i],'\n\n')

当我运行时：

 keywords to search for :  ['pippo', 'papera', 'asino', 'ciuchino', 'peperone', 'oca gallo', 'hamburger formaggio'] 


asino ['cavallo', 'pesce', 'tonno', 'foca', 'sparviero', 'broccoli', 'maiale', 'verdura', 'mulo', 'gatto', 'pippo'] 


peperone ['', 'struzzo', 'uva', 'burro', 'grano', 'acciughe', 'pizza', 'cipolla', 'gamberi', 'cavolo', 'tagliolini', 'pasta', 'oca', 'aglio'] 


pippo ['panna', 'hamburger formaggio', 'bue', 'mucca', 'broccoli', 'gallo', 'verdura', 'asino', 'oca'] 


oca gallo ['pecora', 'pinguino', 'leone', 'mummia', 'bassotto', 'levriero'] 


hamburger formaggio ['broccoli', 'verdura', 'panna', 'pippo'] 

这是正确的行为吗？

如果是这样我需要改变：

 if txt[j] in context_d:
             context_d[txt[j]] += txt[(j - 3):j]
             context_d[txt[j]] += txt[(j + 1):(j + 3)]
            else:
             context_d[txt[j]] = txt[(j - 3):j]
             context_d[txt[j]] += txt[(j + 1):(j + 3)]

与

 if txt[j] in context_d:
             context_d[txt[j]] += txt[(j - 3):j]
             context_d[txt[j]] += txt[(j + 1):(j + 4)]
            else:
             context_d[txt[j]] = txt[(j - 3):j]
             context_d[txt[j]] += txt[(j + 1):(j + 4)]

和：

txt = file.split()

与

txt = shlex.split(file)

让我知道它是否正确（肯定会有更好更快更pythonic的方式）。

Answer 2

如果没有要测试的示例文本，很难知道会发生什么 - 例如，我们是否可以假设空格是正则化的，标点符号总是与前面的单词相邻，并通过空格与后面的单词隔开？但这至少应该让您开始使用正则表达式解决方案。

import re

words = set(approx + could)         # add more
phrases = set(can_be + shouldbeso)  # add more

expr = r'((?:\S+\s+){0,3}(?:%s)[.,:;?!]*(?:\s+\S+){0,3})' % '|'.join(words.union(phrases))
regex = re.compile(expr, flags=re.IGNORECASE)

for filename in glob.glob(os.path.join(path, '*.txt')):
    #  if filename.endswith('.txt'): -- useless, we already globbed for only files with that extension
    # prefer a with statement
    with open(filename) as f:
        for line in f:
            line = line.replace('May', '')
            m = regex.findall(line)
            if m:
                print(m)

这有一个明显的缺点，即匹配多个搜索短语且分隔它们的单词少于六个的行将在每个匹配项周围具有较少的上下文。比如输入行

the diameter can be no longer than approximately 3in

将报告“直径不能超过”作为第一个匹配项，而将“大约 3 英寸”作为第二个匹配项，因为第一个匹配项已经“吃掉”了它后面的单词；和

the diameter can be approximately 3in

将简单地找到“可以”并完全消耗“大约”作为捕获的周围上下文的一部分。

如果这是个问题，也许可以为每一行循环一组正则表达式，并从中提取任何匹配项。

regexes = [re.compile(r'((?:\S+\s+){0,3}(?:%s)[.,:;?!]*(?:\s+\S+){0,3})' % phrase, flags=re.IGNORECASE) for phrase in words.union(phrases)]

for filename in glob.glob(os.path.join(path, '*.txt')):
    with open(filename) as f:
        for line in f:
            line = line.replace('May', '')
            for regex in regexes:
                m = regex.findall(line)
                if m:
                    print(m)

您会注意到可选组[.,:;?!]*，它允许在匹配的单词之后立即使用无限的标点符号。也许您想排除它，或者 - 对于更多真实世界的文本示例 - 可能将其扩展为还允许引用、破折号（如本句前面的嵌入状语）、括号（如此处）等。

通常，您会预处理输入以丢弃或规范化标点符号以及可能的大小写。使用 only 正则表达式对于任何现实世界的输入都是痛苦的。

Answer 3

试图改进第一个答案，不需要在输入文件中突出显示短语（如果输入文件中存在任何单引号则不起作用）：

用 uno.txt 试过：

gallina sparviero Mulo tonno  ASINO maiale foca cavallo boa struzzo May May
peperone cipolla aglio cavolo oca gallo bue pippo mucca

due.txt：

pizza pasta tagliolini acciughe peperone burro oca cavolo baccala
pollo pizza pasta gamberi peperone uva cipolla grano
pesce gatto mulo ASINO verdura broccoli pippo hamburger formaggio panna

tre.txt：

pinguino leone mummia oca gallo levriero bassotto pecora capra

和prova.py：

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Created on Mon Dec 28 16:11:33 2020

@author: Pietro
"""

import glob

import os

import shlex

path = './'

words = ['pippo' ,'papera', 'asino', 'ciuchino' , 'peperone' ]

phrases =['oca gallo' , 'hamburger formaggio'  ]

wordphrases = words + phrases

print('\n\nkeywords to search for : ',words)

print('\n\nphrases to search for : ',phrases,'\n\n')


print('\n\nword + phrases to search for : ',wordphrases,'\n\n')

context_d = {}
for filename in glob.glob(os.path.join(path, '*.txt')):
    if filename.endswith('.txt'):
        f = open(filename)
        file = f.read()
        
        for k in range(len(phrases)):
            if phrases[k] in file:
                print('frase : ',phrases[k], ' in ' , filename)
                newfile  = file.replace(phrases[k], ('\''+phrases[k]+'\''))
                print(newfile)
#        print('\nprinting file :  \n\n'+file)
       # txt = file.lower()
#        print('++++++++hhhhhhhh++++++\n\n',file)
#        txt = file.split()
                txt = shlex.split(newfile)
#        print('+++++++++++++++++++++++++++\n\n',txt)
                txt = [item.replace('May', '') for item in txt]  # locate and replace all months of May before lowering
       #txt = list([[word.lower() for word in line.split()] for line in txt])
                txt = (list(map(lambda x: x.lower(), txt)))
#        print('\n\n printing txt : \n\n',txt,'\n\n')
            for j in range(len(txt)):
#          if (j + 3) < len(txt):
                if txt[j] in wordphrases:
                    if txt[j] in context_d:
                        context_d[txt[j]] += txt[(j - 3):j]
                        context_d[txt[j]] += txt[(j + 1):(j + 4)]
                    else:
                        context_d[txt[j]] = txt[(j - 3):j]
                        context_d[txt[j]] += txt[(j + 1):(j + 4)]


    

for i in context_d:
    context_d[i] = list(set(context_d[i]))
    
for i in context_d:
    print('\nRISULTATO per ',i ,' : ','\n      ', context_d[i],'\n\n')

当我运行时：

keywords to search for :  ['pippo', 'papera', 'asino', 'ciuchino', 'peperone']


phrases to search for :  ['oca gallo', 'hamburger formaggio'] 




word + phrases to search for :  ['pippo', 'papera', 'asino', 'ciuchino', 'peperone', 'oca gallo', 'hamburger formaggio'] 


frase :  oca gallo  in  ./uno.txt
gallina sparviero Mulo tonno  ASINO maiale foca cavallo boa struzzo May May

peperone cipolla aglio cavolo 'oca gallo' bue pippo mucca

frase :  oca gallo  in  ./tre.txt
pinguino leone mummia 'oca gallo' levriero bassotto pecora capra

frase :  hamburger formaggio  in  ./due.txt
pizza pasta tagliolini acciughe peperone burro oca cavolo baccala

pollo pizza pasta gamberi peperone uva cipolla grano

pesce gatto mulo ASINO verdura broccoli pippo 'hamburger formaggio' panna


RISULTATO per  asino  :  
       ['cavallo', 'pesce', 'tonno', 'foca', 'sparviero', 'broccoli', 'maiale', 'verdura', 'mulo', 'gatto', 'pippo'] 



RISULTATO per  peperone  :  
       ['', 'struzzo', 'uva', 'burro', 'grano', 'acciughe', 'pizza', 'cipolla', 'gamberi', 'cavolo', 'tagliolini', 'pasta', 'oca', 'aglio'] 



RISULTATO per  oca gallo  :  
       ['pecora', 'pinguino', 'leone', 'bue', 'mucca', 'mummia', 'cipolla', 'cavolo', 'pippo', 'bassotto', 'levriero', 'aglio'] 



RISULTATO per  pippo  :  
       ['panna', 'hamburger formaggio', 'bue', 'mucca', 'broccoli', 'oca gallo', 'verdura', 'cavolo', 'asino'] 



RISULTATO per  hamburger formaggio  :  
       ['broccoli', 'verdura', 'panna', 'pippo'] 

这是正确的行为吗？

这一次，当一个短语紧邻关键字时，它会被标记为要检索

但算作“1”位置，因此您在 a 旁边获得的不仅仅是 -3 +3 个字

关键字

让我知道它是否正确（肯定会有更好更快更pythonic的方式）。