TensorFlow：导入 gzip mnist-dataset

Question

我目前正在学习机器学习并完成了这个 link

一切都很好，很酷，但是 mnist-import 会抛出警告并说使用的方法已弃用。

我不知道我是否应该“更新”这个，所以这是我的第一个问题，但我也想稍后导入另一个数据集（再次是 train-images.gz 等）

所以，我需要一种从文件夹中读取 .gz-Datasets 并导入它们的方法。我读过关于tf.data.Dataset 的文章，但我认为我并没有真正理解它，或者它不是我需要的。

Answer 1

我明白你的意思，我今天早上确实尝试过（但失败了），但是我又试了一次：这似乎有效（这该死的丑陋，需要解决方法；但它有效）

def _read32(bytestream):
  dt = np.dtype(np.uint32).newbyteorder('>')
  return np.frombuffer(bytestream.read(4), dtype=dt)[0]

def extract_images(f):
    print('Extracting', f.name)
    with gzip.GzipFile(fileobj=f) as bytestream:
        magic = _read32(bytestream)
        if magic != 2051:
            raise ValueError('Invalid magic number %d in MNIST image file: %s' %
                           (magic, f.name))
        num_images = _read32(bytestream)
        rows = _read32(bytestream)
        cols = _read32(bytestream)
        buf = bytestream.read(rows * cols * num_images)
        data = np.frombuffer(buf, dtype=np.uint8)
        data = data.reshape(num_images, rows, cols, 1)
        assert data.shape[3] == 1
        data = data.reshape(data.shape[0],data.shape[1] * data.shape[2])
        data = data.astype(np.float32)
        data = np.multiply(data, 1.0 / 255.0)
        return data
def extract_labels(f):
    with gzip.GzipFile(fileobj=f) as bytestream:
        magic = _read32(bytestream)
        if magic != 2049:
            raise ValueError('Invalid magic number %d in MNIST label file: %s' %
                           (magic, f.name))
        num_items = _read32(bytestream)
        buf = bytestream.read(num_items)
        labels = np.frombuffer(buf, dtype=np.uint8)
        return labels

with gfile.Open("MNIST_data/train-images-idx3-ubyte.gz", "rb") as f:
    train_images = extract_images(f)
with gfile.Open("MNIST_data/train-labels-idx1-ubyte.gz", "rb") as f:
    train_labels = extract_labels(f)
with gfile.Open("MNIST_data/t10k-images-idx3-ubyte.gz", "rb") as f:
    test_images = extract_images(f)
with gfile.Open("MNIST_data/t10k-labels-idx1-ubyte.gz", "rb") as f:   
    test_labels = extract_labels(f)

Answer 2

根据https://www.tensorflow.org/tutorials 提供的适用于 TFv1.9 的文档，使用 Tensorflow 导入 MNIST（作为 numpy 数组）并避免弃用警告的规范方法是：

 mnist = tf.keras.datasets.mnist
 (x_train, y_train),(x_test, y_test) = mnist.load_data()
 x_train, x_test = x_train / 255.0, x_test / 255.0

因此，从现在开始，您应该避免以下情况：

• mnist = tf.contrib.learn.datasets.load_dataset("mnist")
• from tensorflow.examples.tutorials.mnist import input_data mnist = input_data.read_data_sets('MNIST_data', one_hot=True)

无论如何，如果tf.data 不是一个选项，那么调整下面的函数可能会起作用：

def extract_images(f):
"""Extract the images into a 4D uint8 numpy array [index, y, x, depth].
Args:
   f: A file object that can be passed into a gzip reader.
Returns:
   data: A 4D uint8 numpy array [index, y, x, depth].
Raises:
   ValueError: If the bytestream does not start with 2051.
"""
print('Extracting', f.name)
with gzip.GzipFile(fileobj=f) as bytestream:
   magic = _read32(bytestream)
   num_images = _read32(bytestream)
   rows = _read32(bytestream)
   cols = _read32(bytestream)
   buf = bytestream.read(rows * cols * num_images)
   data = numpy.frombuffer(buf, dtype=numpy.uint8)
   data = data.reshape(num_images, rows, cols, 1)
return data