有没有办法概括 Spacy 的 retokenizer.split 的论点中的正交？

Question

我正在尝试从文本文件中修复错误合并的西班牙语单词，并且我正在使用 Spacy 的 retokenizer.split，但是，我想在 retokenizer.split 中概括 orth 的参数。我有下一个代码

doc= nlp("the wordsare wronly merged and weneed split them") #example
words = ["wordsare"] # Example: words to be split
matcher = PhraseMatcher(nlp.vocab)
patterns = [nlp.make_doc(text) for text in words]
matcher.add("Terminology", None, *patterns)
matches = matcher(doc)
with doc.retokenize() as retokenizer:
    for match_id, start, end in matches:
        heads = [(doc[start],1), doc[start]]
        attrs = {"POS": ["PROPN", "PROPN"], "DEP": ["pobj", "compound"]}
        orths= [str(doc[start]),str(doc[end])]
    retokenizer.split(doc[start], orths=orths, heads=heads, attrs=attrs)
token_split=[token.text for token in doc]
print(token_split) 

但是当我将 orths 放在 orths= [str(doc[start]),str(doc[end])] 而不是 ["words","are"] 时，我得到了这个错误：

ValueError: [E117] 新拆分的标记必须与原始标记的文本匹配。新的 orths：wordsarewrongly。旧文本：wordsare。

我想要一些关于概括的帮助，因为我希望代码不仅仅是修复单词 wordsare 还要修复单词 weneed 和其他文件本来可以的。

Answer 1

在你的例子中我会改变的是：

1. words = ["wordsare"] 到 words = ["wordsare","weneed"] 那是拼写错误的单词列表。

2. 将该映射的拆分规则添加到第一个列表：splits = {"wordsare":["words","are"], "weneed":["we","need"]}

3. orths= [str(doc[start]),str(doc[end])] 到 orths= splits[doc[start:end].text] 这是用于替换找到的匹配项的拆分列表。你原来的[str(doc[start]),str(doc[end])] 没有太大意义。

4. 将retokenizer.split 移动到循环中。

5. 考虑为attrs添加另一个字典

一旦你有了它，你就有了一个工作和概括的例子：

import spacy
from spacy.matcher import PhraseMatcher
nlp = spacy.load("en_core_web_sm")

doc= nlp("the wordsare wronly merged and weneed split them") #example
words = ["wordsare","weneed"] # Example: words to be split
splits = {"wordsare":["words","are"], "weneed":["we","need"]}
matcher = PhraseMatcher(nlp.vocab)
patterns = [nlp.make_doc(text) for text in words]
matcher.add("Terminology", None, *patterns)
matches = matcher(doc)

with doc.retokenize() as retokenizer:
    for match_id, start, end in matches:
        heads = [(doc[start],1), doc[start]]
        attrs = {"POS": ["PROPN", "PROPN"], "DEP": ["pobj", "compound"]}
        orths= splits[doc[start:end].text]           
        retokenizer.split(doc[start], orths=orths, heads=heads, attrs=attrs)
token_split=[token.text for token in doc]
print(token_split) 
['the', 'words' ,'are', 'wronly', 'merged', 'and', 'we', 'need', 'split', 'them']

注意，如果您只关心标记化，还有更简单、或许更快的方法来做同样的事情：

[splits[tok.text] if tok.text in words else tok.text for tok in doc]
['the', 'words', 'are', 'wronly', 'merged', 'and', 'we', 'need', 'split', 'them']

另外请注意，在第一个示例中，attrs 在某些情况下是固定的并且分配错误。您可以通过制作另一个字典来解决这个问题，但要拥有一个功能齐全的管道，更简单明了的方法是重新定义标记器，让spacy 为您完成剩下的工作：

from spacy.tokens import Doc
nlp.make_doc = lambda txt: Doc(nlp.vocab, [i for l in [splits[tok.text] if tok.text in words else [tok.text] for tok in nlp.tokenizer(txt)] for i in l])
doc2 = nlp("the wordsare wronly merged and weneed split them")
for tok in doc2:
    print(f"{tok.text:<10}", f"{tok.pos_:<10}", f"{tok.dep_:<10}")
the        DET        det       
words      NOUN       nsubjpass 
are        AUX        auxpass   
wronly     ADV        advmod    
merged     VERB       ROOT      
and        CCONJ      cc        
we         PRON       nsubj     
need       VERB       aux       
split      VERB       conj      
them       PRON       dobj