NumPy - 显示为 2 维的 4 维数组

Question

我正在研究共指解析模型，并试图将大量数据矩阵输入到我的 CNN 输入层。出于说明目的，我截断了我的数据以使用更易于管理的数字。

格式化数据函数

EMBEDDING_DIM = 400

...

@staticmethod
def get_train_data(data: DefaultDict[ClusteredDictKey, PreCoCoreferenceDatapoint], embedding_model) -> Tuple[List[Tensor], List[Tensor]]:
    """
    (n_samples, n_words, n_attributes (word embedding, pos, etc))
    [ [ [ word_embedding, pos ] ] ]

    xtrain[sentence_sample][word_position][attribute]
    xtrain[0][0] -> first word's attributes in first sentence
    xtrain[37][5] -> sixth word's attributes in 38th sentence
    xtrain[0][0][0] -> word_embedding
    xtrain[0][0][1] -> pos one-hot encoding
    """
    xtrain = []
    ytrain = []
    pos_onehot = PreCoParser.get_pos_onehot() # dictionary mapping POS to one-hot encoding

    for key, value in data.items():
        training_data = []

        sentence_embeddings = PreCoParser.get_embedding_for_sent(key.sentence, embedding_model) # Returns tensor (ndarray) of shape: (tokens_in_sent, EMBEDDING_DIM)
        pos = PreCoParser.get_pos_onehot_for_sent(key.sentence, pos_onehot) # Returns tensor (ndarray) of shape: (45,)

        assert sentence_embeddings.shape == (len(key.sentence), EMBEDDING_DIM)
        assert pos.shape == (len(key.sentence), 45)

        for i, embedding in enumerate(sentence_embeddings):
            training_data.append(np.asarray([embedding, np.asarray(pos[i])]))

        cluster_indices = list(sum([cluster.indices for cluster in value], ()))
        # Delete every third element to remove sentence index
        del cluster_indices[0::3]

        if len(training_data) > 0:
            xtrain.append(np.asarray(training_data))
            ytrain.append(np.asarray(cluster_indices) / len(key.sentence)) # normalize output data

    gc.collect()
    return (np.asarray(xtrain), np.asarray(ytrain))

缩写问题

简而言之，我有一个能够成功运行以下断言的 NumPy 数组：

assert self.xtrain[0][0][0].shape == (EMBEDDING_DIM,)

至少对我而言，这意味着该数组包含 4 个维度，最终向量包含 EMBEDDING_DIM 个元素（在我的情况下为 400 个）。

但是，运行以下代码会产生奇怪的结果：

>>> self.xtrain.shape
(500,) 
>>> self.xtrain[0].shape # on sentence with 11 words
(11,2)
>>> self.xtrain[0][0].shape # two attributes
(2,)
>>> self.xtrain[0][0][0].shape
(400,)
>>> self.xtrain[0][0][1].shape
(45,)

500 指的是我截断的样本数（并且所有输出都符合我的预期）。此外，当通过一个简单的 Keras Conv2D 输入层输入这些数据时，我会遇到以下错误：

    self.model.fit(self.xtrain, self.ytrain, epochs=1)
  File "/usr/local/lib/python3.7/site-packages/keras/engine/training.py", line 1154, in fit
    batch_size=batch_size)
  File "/usr/local/lib/python3.7/site-packages/keras/engine/training.py", line 579, in _standardize_user_data
    exception_prefix='input')
  File "/usr/local/lib/python3.7/site-packages/keras/engine/training_utils.py", line 135, in standardize_input_data
    'with shape ' + str(data_shape))
ValueError: Error when checking input: expected conv2d_1_input to have 4 dimensions, but got array with shape (499, 1)

我很乐意发布更多代码，需要，但目前的任何帮助将不胜感激！

Answer 1

我想留下这个作为评论，但因为我的声誉不是>50，它不会让我:(

我对错误的唯一猜测是，对于 model.fit()，您将 ytrain 与 xtrain 错位，因为对于 ytrain，我可以想象输入形状 (499, 1 ）。恐怕我需要更多代码来为模型提供输入数据和标签。

Answer 2

对于将来可能会发现此问题的任何其他人，hpaulj 已部分正确地指出了该问题。我面临的另一个问题是我的数据解析。我最终将输入数据转换为形状为(n_training_samples, INPUT_MAXLEN, 2, EMBEDDING_DIM) 的 NumPy 数组。一旦我用dtype='object' 标准化了矩阵的结构并删除了数组，一切正常。

我参考了这个网站来有效地初始化一个空的 NumPy 数组：http://akuederle.com/create-numpy-array-with-for-loop

最终代码：

EMBEDDING_DIM = 400

...
@staticmethod
def get_train_data(data: DefaultDict[ClusteredDictKey, PreCoCoreferenceDatapoint], inputmaxlen: int, embedding_model) -> Tuple[List[Tensor], List[Tensor]]:
    """
    (n_samples, n_words, n_attributes (word embedding, pos, etc))
    [ [ [ word_embedding, pos ] ] ]

    xtrain[sentence_sample][word_position][attribute]
    xtrain[37][5] -> sixth word's attributes in 38th sentence (np.ndarray containing two np.ndarrays)
    xtrain[0][0][0] -> word_embedding (np.ndarray)
    xtrain[0][0][1] -> pos one-hot encoding (np.ndarray)
    """
    xtrain = np.empty((len(data), inputmaxlen, 2, EMBEDDING_DIM))
    ytrain = []
    pos_onehot = PreCoParser.get_pos_onehot()

    for i, (key, value) in enumerate(data.items()):
        training_data = []

        sentence_embeddings = PreCoParser.get_embedding_for_sent(key.sentence, embedding_model)
        pos = PreCoParser.get_pos_onehot_for_sent(key.sentence, pos_onehot)
        assert sentence_embeddings.shape == (len(key.sentence), EMBEDDING_DIM)
        assert pos.shape == (len(key.sentence), 45)

        for j, word_embeddings in enumerate(sentence_embeddings):
            pos_embeddings = sequence.pad_sequences([pos[j]], maxlen=EMBEDDING_DIM, dtype='float32', padding='post')[0]
            xtrain[i][j][0] = word_embeddings
            xtrain[i][j][1] = pos_embeddings

        cluster_indices = list(sum([cluster.indices for cluster in value], ()))
        # Delete every third element to remove sentence index
        del cluster_indices[0::3]

        ytrain.append(np.asarray(cluster_indices) / len(key.sentence))

    gc.collect()
    return (np.asarray(xtrain, dtype='float32'), np.asarray(ytrain, dtype='float32'))