如何将 pandas DataFrame 转换为 tensorflow.python.data.ops.dataset_ops.PrefetchDataset

Question

鉴于我有以下 TensorFlow 数据集：

import tensorflow_datasets as tfds
(raw_train_ds, raw_val_ds, raw_test_ds), info = tfds.load('ag_news_subset',
                                                          split=['train[:90%]',
                                                                 'train[-90%:]',
                                                                 'test'],
                                                          with_info=True)

raw_train_ds 的类型是 tensorflow.python.data.ops.dataset_ops.PrefetchDataset

我需要将以下remove_stop_words() 方法应用于数据集的description 功能，因此我应该将其转换为DataFrame，并且可以使用以下代码进行转换：

train_sample_df = \
    tfds.as_dataframe(raw_train_ds.shuffle(batch_size),
                      ds_info=info)[['description', 'label']]

我必须将remove_stop_words() 应用于此数据框，如下所示：

def remove_stop_words(tweet):
    tweet = tweet.decode("utf-8")
    #print(tweet," ",type(tweet))
    stopwords = ["a", "about", "above", "after", "again", "against", "all", "am", "an", "and", "any", "are", "as", "at",
                 "be", "because", "been", "before", "being", "below", "between", "both", "but", "by", "could", "did",
                 "do", "does", "doing", "down", "during", "each", "few", "for", "from", "further", "had", "has", "have",
                 "having", "he", "he'd", "he'll", "he's", "her", "here", "here's", "hers", "herself", "him", "himself",
                 "his", "how", "how's", "i", "i'd", "i'll", "i'm", "i've", "if", "in", "into", "is", "it", "it's",
                 "its", "itself", "let's", "me", "more", "most", "my", "myself", "nor", "of", "on", "once", "only",
                 "or", "other", "ought", "our", "ours", "ourselves", "out", "over", "own", "same", "she", "she'd",
                 "she'll", "she's", "should", "so", "some", "such", "than", "that", "that's", "the", "their", "theirs",
                 "them", "themselves", "then", "there", "there's", "these", "they", "they'd", "they'll", "they're",
                 "they've", "this", "those", "through", "to", "too", "under", "until", "up", "very", "was", "we",
                 "we'd", "we'll", "we're", "we've", "were", "what", "what's", "when", "when's", "where", "where's",
                 "which", "while", "who", "who's", "whom", "why", "why's", "with", "would", "you", "you'd", "you'll",
                 "you're", "you've", "your", "yours", "yourself", "yourselves"]
    tweet = tweet.lower()
    words = tweet.split(' ')
    non_stop_words = [w for w in words if w not in stopwords]
    return (" ").join(non_stop_words)

train_sample_df['description'] = train_sample_df['description'].apply(lambda tweet: remove_stop_words(tweet) if tweet is not np.nan else tweet)

最后我需要再次将train_sample_df 转换为tensorflow.python.data.ops.dataset_ops.PrefetchDataset，但我不知道该怎么做。

任何想法 ？

Answer 1

尝试使用 tf.data.Dataset.from_tensor_slices 然后做你想做的事：

import tensorflow as tf

dataset = tf.data.Dataset.from_tensor_slices((train_sample_df['description'], train_sample_df['label'])).prefetch(10) # call batch, shuffle etc.

我不确定您是否知道自己在做什么，但您可以尝试：

import tensorflow as tf

dataset = tf.data.Dataset.from_tensor_slices((train_sample_df['description'], train_sample_df['label'])).prefetch(10) 
dataset = dataset.map(lambda x, y: {'description': x, 'label': y})

def convert_ds_to_tuple(sample):
    return sample['description'], sample['label']

dataset = dataset.map(convert_ds_to_tuple).batch(32)