Python：根据短语长度标记文本[关闭]

Question

我想根据短语长度对文本进行标记。

例如，构建一个函数process_text("Some text to be tokenized please", n = 3)，其中n是短语长度，结果应该是这样的["Some text to","be tokenized please"]。

我该如何实现？

谢谢！

编辑：

好吧，也许我想出了一些可行的方法

from nltk import ngrams

def process_text(text, n = 1):
    text= list(ngrams(text.split(), n))
    tokenised=[" ".join(i) for i in text]
            
    return tokenised

process_text("Some text to be tokenized please", n = 3)

Answer 1

这是使用列表推导的另一种方式：

def tokenize(text):
    words = text.split(" ")
    return [' '.join(words[i:i+3]) for i in range(0, len(words), 3)]

print(tokenize("Some text to be tokenized please"))
# ['Some text to', 'be tokenized please']

然而，这并不完美，即

>>> tokenize("Some text to be tokenized please")
['Some text to', 'be tokenized please']
>>> tokenize("Some text to be tokenized please ")
['Some text to', 'be tokenized please', '']
>>> tokenize(" Some text to be tokenized please ")
[' Some text', 'to be tokenized', 'please ']
>>> tokenize(" Some text  to be tokenized please ")
[' Some text', ' to be', 'tokenized please ']
>>> tokenize(" Some text  to be   tokenized please ")
[' Some text', ' to be', '  tokenized', 'please ']

但您可以根据您的用例进行调整。

Answer 2

这是我想出的一种方法：

def process_text(text, n):
    text = text.split(' ')  # create a list of words the text entered when calling the function
    new_text_1 = ''
    new_text_2 = ''
    x = 0
    for word in text:  # iterate through the words in the provided text
        if x < n:  # if the word is before the split
            new_text_1 = new_text_1 + word + ' '
        elif x < len(text) - 1:  # if the word is after the split but not the last word
            new_text_2 = new_text_2 + word + ' '
        else:  # if this is the last word
            new_text_2 = new_text_2 + word
        x += 1
    return [new_text_1, new_text_2]


print(process_text('Some text to be tokenized please', 3))

您可能可以稍微改进一下，但它可以完成工作。