在拼写检查器中如何获取 3 个编辑后的单词（norvig）

Question

我一直在尝试对我的数据库表使用拼写更正器来更正一个表中的地址，为此我使用了http://norvig.com/spell-correct.html 的引用 使用 Address_mast 表作为字符串集合，我正在尝试更正和更新“customer_master”中更正后的字符串

地址桅杆

ID        Address
1    sonal plaza,harley road,sw-309012
2    rose apartment,kell road, juniper, la-293889
3    plot 16, queen's tower, subbden - 399081
4    cognizant plaza, abs road, ziggar - 500234

现在从参考代码中，它只对那些“远离单词的两次编辑”的单词进行了处理。但我试图在 3 或 4 之前执行此操作，同时尝试更新那些更正的单词到其他表。这里是包含拼写错误单词的表，将用更正的单词进行更新

Customer_master

Address_1

josely apartmt,kell road, juneeper, la-293889
zoonal plaza, harli road,sw-309012
plot 16, queen's tower, subbden - 399081
cognejantt pluza, abs road, triggar - 500234

这是我尝试过的

import re
import pyodbc
import numpy as np
from collections import Counter

cnxn = pyodbc.connect('DRIVER={SQLServer};SERVER=localhost;DATABASE=DBM;UID=ADMIN;PWD=s@123;autocommit=True')
cursor = cnxn.cursor()
cursor.execute("select address as data  from Address_mast")
data=[]
for row in cursor.fetchall():

    data.append(row[0]) 

data = np.array(data)

def words(text): return re.findall(r'\w+', text.lower())

WORDS = Counter(words(open('data').read()))
def P(word, N=sum(WORDS.values())): 
    "Probability of `word`."
    return WORDS[word] / N

def correction(word): 
    "Most probable spelling correction for word."
    return max(candidates(word), key=P)

def candidates(word): 
    "Generate possible spelling corrections for word."
    return (known([word]) or known(edits1(word)) or known(edits2(word)) or known(edits3(word)) or known(edits4(word)) or [word])

def known(words): 
    "The subset of `words` that appear in the dictionary of WORDS."
    return set(w for w in words if w in WORDS)

def edits1(word):
    "All edits that are one edit away from `word`."
    letters    = 'abcdefghijklmnopqrstuvwxyz'
    splits     = [(word[:i], word[i:])    for i in range(len(word) + 1)]
    deletes    = [L + R[1:]               for L, R in splits if R]
    transposes = [L + R[1] + R[0] + R[2:] for L, R in splits if len(R)>1]
    replaces   = [L + c + R[1:]           for L, R in splits if R for c in letters]
    inserts    = [L + c + R               for L, R in splits for c in letters]
    return set(deletes + transposes + replaces + inserts)

def edits2(word): 
    "All edits that are two edits away from `word`."
    return (e2 for e1 in edits1(word) for e2 in edits1(e1))

def edits3(word): 

    return (e3 for e2 in edits2(word) for e3 in edits1(e2))

def edits4(word): 

    return (e4 for e3 in edits3(word) for e4 in edits1(e3))


sqlstr = ""
j=0
k=0
for i in data:
    sqlstr=" update customer_master set Address='"+correction(data)+"' where data="+correction(data)
    cursor.execute(sqlstr)

    j=j+1
    k=k+cursor.rowcount
cnxn.commit()
cursor.close()
cnxn.close()
print(str(k) +" Records Completed")

从此我无法获得正确的输出，任何关于应该进行哪些更改的建议..在此先感谢

Answer 1

我们可以使用现有的 1 个编辑列表并为该列表中的成员进行 1 个编辑

算法： One_Edit_Words = 编辑 1（字） 对于 One_Edit_Words 中的每个 进行编辑1（每个）

def edit2(word): new = edits1(word) # Get list of all the one edits for i in edits1(word): # Iterate through all the objects in one edit list new.update(edits1(i)) # make one edit for each object and add in list return new # Return list with all the edits

同样，我们可以使用相同的方法来获得任意数量的编辑，下面的函数将帮助您获得 3 次编辑

def edit3(word): new = edit2(word)
for i in edit2my(word): new.update(edits1(i)) return new 警告 ： 即使是小计算也需要太多时间（时间复杂度高）

Answer 2

上面的答案是可以的，但是有一个比检查编辑距离为 k 的指数增加的字符串集更快的解决方案。假设我们有一个数据结构，它将所有单词的集合存储在一个树形结构中。这很有用，因为我们知道，例如，我们不需要搜索没有单词的路径。这既节省内存，又节省计算。

假设我们有一个词汇表存储在一个集合、dict，或者理想情况下是一个 collections.Counter 对象，那么我们可以如下设置数据结构：

class VocabTreeNode:
    def __init__(self):
        self.children = {}
        self.word = None
        
    def build(self, vocab):
        for w in vocab:
            self.insert(w)

    def insert( self, word):
        node = self
        for letter in word:
            if letter not in node.children: 
                node.children[letter] = VocabTreeNode()
            node = node.children[letter]
        node.word = word

要仅搜索与单词的编辑距离为 k 的元素集，我们可以赋予此结构递归搜索。

    def search(self, word, maxCost):
        currentRow = range( len(word) + 1 )    
        results = []
        for letter in self.children:
            self.searchRecursive(self.children[letter], letter, 
                                 word, currentRow, results, 
                                 maxCost)   
        return results
            
    def searchRecursive(self, node, letter, word, previousRow, 
                        results, maxCost):
        columns = len( word ) + 1
        currentRow = [ previousRow[0] + 1 ]
        for column in range( 1, columns ):
            insertCost = currentRow[column - 1] + 1
            deleteCost = previousRow[column] + 1
            if word[column - 1] != letter:
                replaceCost = previousRow[ column - 1 ] + 1
            else:                
                replaceCost = previousRow[ column - 1 ]
            currentRow.append( min( insertCost, deleteCost, replaceCost ) )
    
        if currentRow[-1] <= maxCost and node.word != None:
            results.append( (node.word, currentRow[-1] ) )
        if min( currentRow ) <= maxCost:
            for next_letter in node.children:
                self.searchRecursive( node.children[next_letter], next_letter, word,
                                      currentRow, results, maxCost)

只有一个问题我不知道如何克服；换位作为路径无效，所以我不确定如何将换位合并为编辑距离 1 而不需要一些复杂的 hack。

我的词库是 97722（几乎所有 linux 发行版中的词集）。

sleep(1)
start = time()

for i in range(100):
    x = V.search('elephant',3)
    
print(time()- start)

>>> 17.5 

这相当于每 0.175 秒编辑此单词的距离 3 计算。编辑距离 4 可以在 0.377 秒内完成，而使用 edits1 的连续编辑距离会很快导致您的系统内存不足。

需要注意的是不容易处理转置，这是一种快速有效地实现用于高编辑距离的 Norvig 类型算法的方法。