映射一个 tensorflow 数据集并改变 tf.train.Feature，它是一个字节字符串列表

Question

我有一个功能是字节字符串列表，例如

data = [b"lksjdflksdjfdlk", b"owiueroiewuroi.skjdf", b"oweiureoiwlkapq"]

这里是创建、写出和读回 + 解析 tfrecord 的示例代码。

>>> data = [b"lksjdflksdjfdlk", b"owiueroiewuroi.skjdf", b"oweiureoiwlkapq"]
>>> feature = tf.train.Feature(bytes_list=tf.train.BytesList(value=data))
>>> feature
{'raws': bytes_list {
   value: "lksjdflksdjfdlk"
   value: "owiueroiewuroi.skjdf"
   value: "oweiureoiwlkapq"
 }}
>>> example = tf.train.Example(features=features).SerializeToString()
>>> with tf.io.TFRecordWriter("/tmp/out.tfrecord") as writer:
        writer.write(example)
>>> # Now read it back in and parse thee example
>>> feature_desc = {'raws': tf.io.FixedLenFeature([], tf.string)}
>>> def _parse(example):
        return tf.io.parse_single_example(example, feature_desc)
>>> ds = tf.data.TFRecordDataset(["/tmp/out.tfrecord"])
>>> parsed = ds.map(_parse)
>>> @tf.function
    def upper(x):
        x['raws'] = [s.upper() for s in x['raws']]
>>> parsed.map(upper)

这会导致以下错误：

OperatorNotAllowedInGraphError: in user code:

    <ipython-input-33-be19a774366f>:3 upper  *
        x['raws'] = [s.upper() for s in x['raws']]
    /data/jkyle/venv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py:503 __iter__
        self._disallow_iteration()
    /data/jkyle/venv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py:496 _disallow_iteration
        self._disallow_when_autograph_enabled("iterating over `tf.Tensor`")
    /data/jkyle/venv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py:474 _disallow_when_autograph_enabled
        " indicate you are trying to use an unsupported feature.".format(task))

    OperatorNotAllowedInGraphError: iterating over `tf.Tensor` is not allowed: AutoGraph did convert this function. This might indicate you are trying to use an unsupported feature.

对于完整的上下文，列表是原生不支持的原始图像格式的字节字符串。每个原始图像都是一个帧。我需要遍历列表，转换为 jpeg，然后将它们堆叠成一个三维数组。转换需要由 OpenCV 完成。所以 raw -> jpeg -> numpy 矩阵，例如

输入：[b'raw1', b'raw2', b'raw3'] 输出：形状为(1920,1080,3)的图像数组

但是，当然，在我弄清楚如何迭代列表之前，不能执行任何这些操作。

Answer 1

是的，正如错误所暗示的那样，对张量进行迭代不是受支持的恐惧。这是相当笼统的，可能无法回答您的具体问题，但您可以使用以下方法对其进行迭代： tf.unstack

它将秩-R张量的给定维度解包成秩-(R-1)张量。

所以给每个张量加 1 看起来像：

import tensorflow as tf
x = tf.placeholder(tf.float32, shape=(None, 10))
x_unpacked = tf.unstack(x) # defaults to axis 0, returns a list of tensors

processed = [] # this will be the list of processed tensors
for t in x_unpacked:
    # do whatever
    result_tensor = t + 1
    processed.append(result_tensor)

output = tf.concat(processed, 0)

with tf.Session() as sess:
    print(sess.run([output], feed_dict={x: np.zeros((5, 10))}))

显然，您可以进一步从列表中解压缩每个张量以对其进行处理，直至处理单个元素。不过，为了避免大量嵌套拆包，您可以先尝试使用tf.reshape(x, [-1]) 将 x 展平，然后像这样循环遍历它

flattened_unpacked = tf.unstack(tf.reshape(x, [-1])
for elem in flattened_unpacked:
    process(elem)

在这种情况下，elem 是一个标量。