【问题标题】:IndexError: list index out of range, NLP BERT TensorflowIndexError:列表索引超出范围,NLP BERT Tensorflow
【发布时间】:2021-05-17 21:27:36
【问题描述】:

所以我使用Bert模型对其进行训练并将其保存为hdf5文件,但是当我尝试预测时,它显示了这个错误:

IndexError: 列表索引超出范围

这里是代码

import os.path
import numpy as np
import tensorflow as tf
import ktrain
from ktrain import text

"""## Part 1: Data Preprocessing

### Loading the IMDB dataset
"""

dataset = tf.keras.utils.get_file(fname="aclImdb_v1.tar.gz",
                              
origin="http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz",
                                  extract=True)
IMDB_DATADIR = os.path.join(os.path.dirname(dataset), 'aclImdb')

print(os.path.dirname(dataset))
print(IMDB_DATADIR)

"""### Creating the training and test sets"""

(x_train, y_train), (x_test, y_test), preproc = text.texts_from_folder(datadir=IMDB_DATADIR,
                                                                   classes=['pos','neg'],
                                                                   maxlen=500,
                                                                   train_test_names=['train','test'],
                                                                   preprocess_mode='bert')

"""## Part 2: Building the BERT model"""

model = text.text_classifier(name='bert',
                         train_data=(x_train, y_train),
                         preproc=preproc)

"""## Part 3: Training the BERT model"""

learner = ktrain.get_learner(model=model,
                            train_data=(x_train, y_train),
                            val_data=(x_test, y_test),
                            batch_size=6)

learner.fit_onecycle(lr=2e-5,
                    epochs=1)

tf.keras.models.save_model(model, 'NLP_model.hdf5')

from keras_bert import get_custom_objects

model = tf.keras.models.load_model('NLP_model.hdf5', custom_objects=get_custom_objects())

model.predict('This movie is not the scariest of all time, but it is a great example of a campy 
eighties horror flick -- low budget, no stars, lots of inventive death scenes, and enough nudity to 
keep the teenagers in their seats. The premise is interesting and fun and the three evil kids play 
their parts well. A nice starting point for "Just Say" Julie Brown exposing her talents early in her 
career. This film wont be seen by many, but for fans of 80s horror its a must.ense love would be more 
believable.n.')

我正在尝试预测测试集上的一个句子。

picture of the full code error

非常感谢您的帮助,您

编辑:

【问题讨论】:

    标签: tensorflow machine-learning keras deep-learning nlp


    【解决方案1】:

    如 ktrain 教程和示例笔记本(如 this one)所示,您需要使用 Predictor 实例对原始文本输入进行预测:

    # create a Predictor instance
    predictor = ktrain.get_predictor(learner.model, preproc)
    
    # make prediction
    output = predictor.predict('I loved this movie!')
    print(output)
    
    # save Predictor to disk
    predictor.save('/tmp/mypredictor')
    
    # reload Predictor from disk
    reloaded_predictor = ktrain.load_predictor('/tmp/mypredictor')
    
    # make another prediction
    output = reloaded_predictor.predict('I loved this movie!')
    print(output)
    

    【讨论】:

    • 解决了问题!非常感谢!
    • 顺便说一句,我想知道为什么我在 training_set 上的准确度得分为 83%,但在 val_set 上的准确度得分为 93%,而与您链接的相比(您可以在帖子中看到输出屏幕截图)
    猜你喜欢
    • 2021-02-11
    • 2019-01-22
    • 2011-10-31
    • 2015-06-26
    • 1970-01-01
    • 1970-01-01
    相关资源
    最近更新 更多