Python Untokenize 一个句子

Question

关于如何标记句子的指南有很多，但我没有找到任何相反的方法。

 import nltk
 words = nltk.word_tokenize("I've found a medicine for my disease.")
 result I get is: ['I', "'ve", 'found', 'a', 'medicine', 'for', 'my', 'disease', '.']

是否有任何功能可以将标记化的句子恢复到原始状态。 tokenize.untokenize() 函数由于某种原因不起作用。

编辑：

我知道我可以这样做，这可能会解决问题，但我很好奇是否有一个集成功能：

result = ' '.join(sentence).replace(' , ',',').replace(' .','.').replace(' !','!')
result = result.replace(' ?','?').replace(' : ',': ').replace(' \'', '\'')   

Answer 1

您可以使用“treebank detokenizer” - TreebankWordDetokenizer:

from nltk.tokenize.treebank import TreebankWordDetokenizer
TreebankWordDetokenizer().detokenize(['the', 'quick', 'brown'])
# 'The quick brown'

---

还有MosesDetokenizer，它在nltk 中，但由于licensing issues 而被删除，但它可以作为Sacremoses standalone package 使用。

Answer 2

from nltk.tokenize.treebank import TreebankWordDetokenizer
TreebankWordDetokenizer().detokenize(['the', 'quick', 'brown'])
# 'The quick brown'

Answer 3

没有简单答案的原因是您实际上需要字符串中原始标记的跨度位置。如果您没有那个，并且您没有对原始标记化进行逆向工程，那么您重新组装的字符串是基于对所使用的标记化规则的猜测。如果你的分词器没有给你跨度，如果你有三件事，你仍然可以这样做：

1) 原始字符串

2) 原始令牌

3) 修改后的令牌（我假设您已经以某种方式更改了令牌，因为如果您已经拥有 #1，那是我能想到的唯一应用程序）

使用原始标记集来识别跨度（如果标记器这样做不是很好吗？）并从后到前修改字符串，这样跨度就不会随着你去改变。

这里我使用的是 TweetTokenizer，但只要您使用的标记器不会更改标记的值，因此它们实际上不在原始字符串中，这无关紧要。

tokenizer=nltk.tokenize.casual.TweetTokenizer()
string="One morning, when Gregor Samsa woke from troubled dreams, he found himself transformed in his bed into a horrible vermin."
tokens=tokenizer.tokenize(string)
replacement_tokens=list(tokens)
replacement_tokens[-3]="cute"

def detokenize(string,tokens,replacement_tokens):
    spans=[]
    cursor=0
    for token in tokens:
        while not string[cursor:cursor+len(token)]==token and cursor<len(string):
            cursor+=1        
        if cursor==len(string):break
        newcursor=cursor+len(token)
        spans.append((cursor,newcursor))
        cursor=newcursor
    i=len(tokens)-1
    for start,end in spans[::-1]:
        string=string[:start]+replacement_tokens[i]+string[end:]
        i-=1
    return string

>>> detokenize(string,tokens,replacement_tokens)
'One morning, when Gregor Samsa woke from troubled dreams, he found himself transformed in his bed into a cute vermin.'

Answer 4

对我来说，当我安装 python nltk 3.2.5 时，它就起作用了，

pip install -U nltk

那么，

import nltk
nltk.download('perluniprops')

from nltk.tokenize.moses import MosesDetokenizer

如果你使用的是 insides pandas 数据框，那么

df['detoken']=df['token_column'].apply(lambda x: detokenizer.detokenize(x, return_str=True))

Answer 5

我正在使用以下代码，但没有任何主要的库函数用于去噪目的。我正在对某些特定令牌使用去令牌化

_SPLITTER_ = r"([-.,/:!?\";)(])"

def basic_detokenizer(sentence):
""" This is the basic detokenizer helps us to resolves the issues we created by  our tokenizer"""
detokenize_sentence =[]
words = sentence.split(' ')
pos = 0
while( pos < len(words)):
    if words[pos] in '-/.' and pos > 0 and pos < len(words) - 1:
        left = detokenize_sentence.pop()
        detokenize_sentence.append(left +''.join(words[pos:pos + 2]))
        pos +=1
    elif  words[pos] in '[(' and pos < len(words) - 1:
        detokenize_sentence.append(''.join(words[pos:pos + 2]))   
        pos +=1        
    elif  words[pos] in ']).,:!?;' and pos > 0:
        left  = detokenize_sentence.pop()
        detokenize_sentence.append(left + ''.join(words[pos:pos + 1]))            
    else:
        detokenize_sentence.append(words[pos])
    pos +=1
return ' '.join(detokenize_sentence)

Answer 6

我建议在标记化中保留偏移量：(token, offset)。 我认为，这些信息对于处理原始句子很有用。

import re
from nltk.tokenize import word_tokenize

def offset_tokenize(text):
    tail = text
    accum = 0
    tokens = self.tokenize(text)
    info_tokens = []
    for tok in tokens:
        scaped_tok = re.escape(tok)
        m = re.search(scaped_tok, tail)
        start, end = m.span()
        # global offsets
        gs = accum + start
        ge = accum + end
        accum += end
        # keep searching in the rest
        tail = tail[end:]
        info_tokens.append((tok, (gs, ge)))
    return info_token

sent = '''I've found a medicine for my disease.

This is line:3.'''

toks_offsets = offset_tokenize(sent)

for t in toks_offsets:
(tok, offset) = t
print (tok == sent[offset[0]:offset[1]]), tok, sent[offset[0]:offset[1]]

给予：

True I I
True 've 've
True found found
True a a
True medicine medicine
True for for
True my my
True disease disease
True . .
True This This
True is is
True line:3 line:3
True . .

Answer 7

从here 使用token_utils.untokenize

import re
def untokenize(words):
    """
    Untokenizing a text undoes the tokenizing operation, restoring
    punctuation and spaces to the places that people expect them to be.
    Ideally, `untokenize(tokenize(text))` should be identical to `text`,
    except for line breaks.
    """
    text = ' '.join(words)
    step1 = text.replace("`` ", '"').replace(" ''", '"').replace('. . .',  '...')
    step2 = step1.replace(" ( ", " (").replace(" ) ", ") ")
    step3 = re.sub(r' ([.,:;?!%]+)([ \'"`])', r"\1\2", step2)
    step4 = re.sub(r' ([.,:;?!%]+)$', r"\1", step3)
    step5 = step4.replace(" '", "'").replace(" n't", "n't").replace(
         "can not", "cannot")
    step6 = step5.replace(" ` ", " '")
    return step6.strip()

 tokenized = ['I', "'ve", 'found', 'a', 'medicine', 'for', 'my','disease', '.']
 untokenize(tokenized)
 "I've found a medicine for my disease."

Answer 8

要从nltk 反转word_tokenize，我建议查看http://www.nltk.org/_modules/nltk/tokenize/punkt.html#PunktLanguageVars.word_tokenize 并进行一些逆向工程。

没有在 nltk 上做疯狂的 hack，你可以试试这个：

>>> import nltk
>>> import string
>>> nltk.word_tokenize("I've found a medicine for my disease.")
['I', "'ve", 'found', 'a', 'medicine', 'for', 'my', 'disease', '.']
>>> tokens = nltk.word_tokenize("I've found a medicine for my disease.")
>>> "".join([" "+i if not i.startswith("'") and i not in string.punctuation else i for i in tokens]).strip()
"I've found a medicine for my disease."

Answer 9

tokenize.untokenize 不起作用的原因是它需要更多信息而不仅仅是单词。这是一个使用tokenize.untokenize的示例程序：

from StringIO import StringIO
import tokenize

sentence = "I've found a medicine for my disease.\n"
tokens = tokenize.generate_tokens(StringIO(sentence).readline)
print tokenize.untokenize(tokens)

附加帮助： Tokenize - Python Docs | Potential Problem

Answer 10

使用join函数：

你可以只做一个' '.join(words) 来取回原始字符串。