AttributeError：“NoneType”对象没有属性“tokenize”

Question

我正在尝试通过转换器使用 XLNET。但是我不断收到问题“AttributeError：'NoneType'对象没有属性'tokenize'”。我不确定如何进行。如果有人能指出我正确的方向，将不胜感激。

tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased', do_lower_case=True)

print(' Original: ', X_train[1])

# Print the tweet split into tokens.
print('Tokenized: ', tokenizer.tokenize(X_train[1]))

# Print the tweet mapped to token ids.
print('Token IDs: ', tokenizer.convert_tokens_to_ids(tokenizer.tokenize(X_train[1])))




Original:  hey angel duh sexy really thanks haha
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-67-2b1b432b3e15> in <module>()
      2 
      3 # Print the tweet split into tokens.
----> 4 print('Tokenized: ', tokenizer.tokenize(X_train[2]))
      5 
      6 # Print the tweet mapped to token ids.

AttributeError: 'NoneType' object has no attribute 'tokenize'

Answer 1

我假设：

from transformers import XLNetTokenizerFast
tokenizer = XLNetTokenizerFast.from_pretrained('xlnet-base-cased', do_lower_case=True)

有效吗？ 在这种情况下，您只是缺少 sentencepiece 包：

pip install sentencepiece

Answer 2

如果安装了 SenencePience 仍然有错误

当然，@cronoik 的答案是正确的。毫无疑问。但是如果你已经安装了SenencePience 包还是有错误，那就restart the runtime environment and it will work。