加快从列表中创建 numpy 数组答案

【问题标题】：Speed up creation of numpy array from list加快从列表中创建 numpy 数组
【发布时间】：2017-02-19 04:12:09
【问题描述】：

我有一个 33620x160 pandas DataFrame，其中有一列包含数字列表。 DataFrame 中的每个列表条目包含 30 个元素。

df['dlrs_col']

0        [0.048142470608688, 0.047021138711858, 0.04573...
1        [0.048142470608688, 0.047021138711858, 0.04573...
2        [0.048142470608688, 0.047021138711858, 0.04573...
3        [0.048142470608688, 0.047021138711858, 0.04573...
4        [0.048142470608688, 0.047021138711858, 0.04573...
5        [0.048142470608688, 0.047021138711858, 0.04573...
6        [0.048142470608688, 0.047021138711858, 0.04573...
7        [0.048142470608688, 0.047021138711858, 0.04573...
8        [0.048142470608688, 0.047021138711858, 0.04573...
9        [0.048142470608688, 0.047021138711858, 0.04573...
10       [0.048142470608688, 0.047021138711858, 0.04573...

我正在创建一个 33620x30 数组，其条目是来自该单个 DataFrame 列的未列出值。我目前正在这样做：

np.array(df['dlrs_col'].tolist(), dtype = 'float64')

这很好用，但需要大量时间，尤其是考虑到我对另外 6 列列表进行类似计算时。关于如何加快速度的任何想法？

【问题讨论】：

标签： python arrays performance pandas numpy

【解决方案1】：

你可以这样做：

In [140]: df
Out[140]:
                                          dlrs_col
0  [0.048142470608688, 0.047021138711858, 0.04573]
1  [0.048142470608688, 0.047021138711858, 0.04573]
2  [0.048142470608688, 0.047021138711858, 0.04573]
3  [0.048142470608688, 0.047021138711858, 0.04573]
4  [0.048142470608688, 0.047021138711858, 0.04573]
5  [0.048142470608688, 0.047021138711858, 0.04573]
6  [0.048142470608688, 0.047021138711858, 0.04573]
7  [0.048142470608688, 0.047021138711858, 0.04573]
8  [0.048142470608688, 0.047021138711858, 0.04573]
9  [0.048142470608688, 0.047021138711858, 0.04573]

In [141]: df.dlrs_col.apply(pd.Series)
Out[141]:
          0         1        2
0  0.048142  0.047021  0.04573
1  0.048142  0.047021  0.04573
2  0.048142  0.047021  0.04573
3  0.048142  0.047021  0.04573
4  0.048142  0.047021  0.04573
5  0.048142  0.047021  0.04573
6  0.048142  0.047021  0.04573
7  0.048142  0.047021  0.04573
8  0.048142  0.047021  0.04573
9  0.048142  0.047021  0.04573

In [142]: df.dlrs_col.apply(pd.Series).values
Out[142]:
array([[ 0.04814247,  0.04702114,  0.04573   ],
       [ 0.04814247,  0.04702114,  0.04573   ],
       [ 0.04814247,  0.04702114,  0.04573   ],
       [ 0.04814247,  0.04702114,  0.04573   ],
       [ 0.04814247,  0.04702114,  0.04573   ],
       [ 0.04814247,  0.04702114,  0.04573   ],
       [ 0.04814247,  0.04702114,  0.04573   ],
       [ 0.04814247,  0.04702114,  0.04573   ],
       [ 0.04814247,  0.04702114,  0.04573   ],
       [ 0.04814247,  0.04702114,  0.04573   ]])

【讨论】：

我很欣赏您的回复，但在我的快速测试中，这实际上是我之前方法的两倍。

【解决方案2】：

你可以先通过values转换成numpy array：

df = pd.DataFrame({'dlrs_col':[
[0.048142470608688, 0.047021138711858, 0.04573],
[0.048142470608688, 0.047021138711858, 0.04573],
[0.048142470608688, 0.047021138711858, 0.04573],
[0.048142470608688, 0.047021138711858, 0.04573],
[0.048142470608688, 0.047021138711858, 0.04573],
[0.048142470608688, 0.047021138711858, 0.04573],
[0.048142470608688, 0.047021138711858, 0.04573],
[0.048142470608688, 0.047021138711858, 0.04573]]})

print (df)
                                          dlrs_col
0  [0.048142470608688, 0.047021138711858, 0.04573]
1  [0.048142470608688, 0.047021138711858, 0.04573]
2  [0.048142470608688, 0.047021138711858, 0.04573]
3  [0.048142470608688, 0.047021138711858, 0.04573]
4  [0.048142470608688, 0.047021138711858, 0.04573]
5  [0.048142470608688, 0.047021138711858, 0.04573]
6  [0.048142470608688, 0.047021138711858, 0.04573]
7  [0.048142470608688, 0.047021138711858, 0.04573]

print (np.array(df['dlrs_col'].values.tolist(), dtype = 'float64'))
[[ 0.04814247  0.04702114  0.04573   ]
 [ 0.04814247  0.04702114  0.04573   ]
 [ 0.04814247  0.04702114  0.04573   ]
 [ 0.04814247  0.04702114  0.04573   ]
 [ 0.04814247  0.04702114  0.04573   ]
 [ 0.04814247  0.04702114  0.04573   ]
 [ 0.04814247  0.04702114  0.04573   ]
 [ 0.04814247  0.04702114  0.04573   ]]

时间安排：

In [56]: %timeit (np.array(df['dlrs_col'].values.tolist(), dtype = 'float64'))
The slowest run took 9.76 times longer than the fastest. This could mean that an intermediate result is being cached.
100000 loops, best of 3: 14.1 µs per loop

In [57]: %timeit (np.array(df['dlrs_col'].tolist(), dtype = 'float64'))
The slowest run took 9.33 times longer than the fastest. This could mean that an intermediate result is being cached.
10000 loops, best of 3: 28.4 µs per loop

【讨论】：