python中具有巨大矩阵的矩阵运算[重复]

Question

有人知道如何在 python 中处理巨大的矩阵吗？我必须使用形状 (10^6,10^6) 的邻接矩阵并执行包括加法、缩放和点积在内的操作。使用 numpy 数组我遇到了 ram 问题。

Answer 1

这样的事情怎么样...

import numpy as np

# Create large arrays x and y.
# Note they are 1e4 not 1e6 b/c of memory issues creating random numpy matrices (CookieOfFortune) 
# However, the same principles apply to larger arrays
x = np.random.randn(10000, 10000)
y = np.random.randn(10000, 10000)

# Create memory maps for x and y arrays
xmap = np.memmap('xfile.dat', dtype='float32', mode='w+', shape=x.shape)
ymap = np.memmap('yfile.dat', dtype='float32', mode='w+', shape=y.shape)

# Fill memory maps with data
xmap[:] = x[:]
ymap[:] = y[:]

# Create memory map for out of core dot product result
prodmap = np.memmap('prodfile.dat', dtype='float32', mode='w+', shape=x.shape)

# Due out of core dot product and write data
prodmap[:] = np.memmap.dot(xmap, ymap)

# Create memory map for out of core addition result
addmap = np.memmap('addfile.dat', dtype='float32', mode='w+', shape=x.shape)

# Due out of core addition and write data
addmap[:] = xmap + ymap

# Create memory map for out of core scaling result
scalemap = np.memmap('scalefile.dat', dtype='float32', mode='w+', shape=x.shape)

# Define scaling constant
scale = 1.3

# Do out of core  scaling and write data
scalemap[:] = scale * xmap

此代码将创建包含二进制格式数组的文件 xfile.dat、yfile.dat 等。要稍后访问它们，您只需执行np.memmap(filename)。 np.memmap 的其他参数是可选的，但建议使用（dtype、shape 等参数）。