【发布时间】：2022-06-14 12:46:58
【问题描述】：
我使用的是 Windows 10。我的系统配置是 RAM = 256 GB，硬盘 = 2 TB。我正在使用 PyCharm 社区版进行编码。我正在使用python 版本如下所示：
Python 3.9.7 [MSC v.1929 64 bit (AMD64)] on win32
我的数据大小为 120 万行，大约 7000 列。当我尝试将处理后的数据拟合到 Scikit-learn Random Forrest 模型中时，出现以下错误：
joblib.externals.loky.process_executor._RemoteTraceback:
"""
Traceback (most recent call last):
File "C:\Program Files\Python39\lib\site-packages\joblib\externals\loky\process_executor.py", line 431, in _process_worker
r = call_item()
File "C:\Program Files\Python39\lib\site-packages\joblib\externals\loky\process_executor.py", line 285, in __call__
return self.fn(*self.args, **self.kwargs)
File "C:\Program Files\Python39\lib\site-packages\joblib\_parallel_backends.py", line 595, in __call__
return self.func(*args, **kwargs)
File "C:\Program Files\Python39\lib\site-packages\joblib\parallel.py", line 262, in __call__
return [func(*args, **kwargs)
File "C:\Program Files\Python39\lib\site-packages\joblib\parallel.py", line 262, in <listcomp>
return [func(*args, **kwargs)
File "C:\Program Files\Python39\lib\site-packages\sklearn\utils\fixes.py", line 209, in __call__
return self.function(*args, **kwargs)
File "C:\Program Files\Python39\lib\site-packages\sklearn\model_selection\_validation.py", line 674, in _fit_and_score
X_test, y_test = _safe_split(estimator, X, y, test, train)
File "C:\Program Files\Python39\lib\site-packages\sklearn\utils\metaestimators.py", line 286, in _safe_split
X_subset = _safe_indexing(X, indices)
File "C:\Program Files\Python39\lib\site-packages\sklearn\utils\__init__.py", line 377, in _safe_indexing
return _array_indexing(X, indices, indices_dtype, axis=axis)
File "C:\Program Files\Python39\lib\site-packages\sklearn\utils\__init__.py", line 201, in _array_indexing
return array[key] if axis == 0 else array[:, key]
File "C:\Program Files\Python39\lib\site-packages\numpy\core\memmap.py", line 331, in __getitem__
res = super(memmap, self).__getitem__(index)
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 721. MiB for an array with shape (120000, 7000) and data type uint8
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\...\main.py", line 45, in <module>
model.fit(X, y)
File "C:\Program Files\Python39\lib\site-packages\sklearn\model_selection\_search.py", line 891, in fit
self._run_search(evaluate_candidates)
File "C:\Program Files\Python39\lib\site-packages\sklearn\model_selection\_search.py", line 1766, in _run_search
evaluate_candidates(
File "C:\Program Files\Python39\lib\site-packages\sklearn\model_selection\_search.py", line 838, in evaluate_candidates
out = parallel(
File "C:\Program Files\Python39\lib\site-packages\joblib\parallel.py", line 1054, in __call__
self.retrieve()
File "C:\Program Files\Python39\lib\site-packages\joblib\parallel.py", line 933, in retrieve
self._output.extend(job.get(timeout=self.timeout))
File "C:\Program Files\Python39\lib\site-packages\joblib\_parallel_backends.py", line 542, in wrap_future_result
return future.result(timeout=timeout)
File "C:\Program Files\Python39\lib\concurrent\futures\_base.py", line 445, in result
return self.__get_result()
File "C:\Program Files\Python39\lib\concurrent\futures\_base.py", line 390, in __get_result
raise self._exception
numpy.core._exceptions.MemoryError: Unable to allocate 721. MiB for an array with shape (120000, 7000) and data type uint8
我在这里尝试了一些解决方案：
- 增加 PyCharm 的控制台内存。 Increase output buffer when running or debugging in PyCharm
- 增加分页文件大小Unable to allocate array with shape and data type
但是这些都不起作用。当我在 RAM 和硬盘驱动器中有足够的内存时，如何解决这个问题？我用 PyCharm、Jupyter 和 google colab 进行了尝试。但它显示相同的错误。
【问题讨论】：
-
也许stackoverflow.com/a/20952691/16744221 可以帮助你。我尚未对其进行测试，但您似乎无法立即使用生成器来训练您的随机森林。
-
那个数组看起来没那么大。我怀疑还有更多的数组会占用内存。
-
@hpaulj 我在
n_jobs=-1参数中使用了所有 64 个内核。可能是这个原因？ -
我没有用过大型多核计算机，所以帮不上忙。
标签： python python-3.x windows numpy memory