【发布时间】：2021-07-20 14:51:39
【问题描述】：
我在运行节点应用程序时遇到错误，该应用程序处理一个非常大的文件（7600 万条记录）被读入数组。
Suse 服务器有 64GB 的 RAM，我也有 100GB 的交换文件。但是堆转储发生在大约 30GB 的内存上。读取了大约 3100 万条记录。
节点版本为 12.22.1
这是一个分析应用程序，它是初始数据提取的一次性工作负载。应用程序一次读取 10 万条记录的文件。
我已经尝试了这些参数的多种排列，这是最后一次运行的。
--max-old-space-size=196608
--max-semi-space-size=256
--semi-space-growth-factor=4
我之前在节点 4 上运行过此类大型数据集，仅使用 --max-old-space-size 没有问题，但根据我所阅读的内容，内存管理与节点 12 之后的不同。
感谢您提供的任何指导或建议。
感谢和问候，
迪兰克
^[[0;34mINFO ^[[0m 2021-04-27 08:10:50 ^[[0;1mDSParser^[[0m Read 31700000 lines
^[[0;4mtable^[[0m: "ekpo"
^[[0;4mstep^[[0m: 0
^[[0;4mmemory^[[0m: {"rss":21905813504,"heapTotal":17302638592,"heapUsed":17253034680,"external":4473602959,"arrayBuffers":4472358538}
^[[0;34mINFO ^[[0m 2021-04-27 08:10:56 ^[[0;1mDSParser^[[0m Read 31800000 lines
^[[0;4mtable^[[0m: "ekpo"
^[[0;4mstep^[[0m: 0
^[[0;4mmemory^[[0m: {"rss":22110760960,"heapTotal":17492168704,"heapUsed":17435101184,"external":4473635727,"arrayBuffers":4472391306}
<--- Last few GCs --->
[11070:0x4c9f3c0] 1092787 ms: Scavenge 16733.7 (16784.8) -> 16733.6 (16784.8) MB, 84.5 / 0.0 ms (average mu = 0.988, current mu = 0.990) allocation failure
[11070:0x4c9f3c0] 1092869 ms: Scavenge 16733.7 (16784.8) -> 16733.6 (16784.8) MB, 80.9 / 0.0 ms (average mu = 0.988, current mu = 0.990) allocation failure
[11070:0x4c9f3c0] 1092953 ms: Scavenge 16733.7 (16784.8) -> 16733.7 (16784.8) MB, 83.9 / 0.0 ms (average mu = 0.988, current mu = 0.990) allocation failure
<--- JS stacktrace --->
Cannot get stack trace in GC.
> FATAL ERROR: Scavenger: semi-space copy Allocation failed -
> JavaScript heap out of memory 1: 0xa1a640 node::Abort()
> [/home/forestpin/data/retail/bin/node/node/bin/node] 2: 0xa1aa4c
> node::OnFatalError(char const*, char const*)
> [/home/forestpin/data/retail/bin/node/node/bin/node] 3: 0xb9a68e
> v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool)
> [/home/forestpin/data/retail/bin/node/node/bin/node] 4: 0xb9aa09
> v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char
> const*, bool) [/home/forestpin/data/retail/bin/node/node/bin/node] 5:
> 0xd57c85 [/home/forestpin/data/retail/bin/node/node/bin/node] 6:
> 0xdba162 [/home/forestpin/data/retail/bin/node/node/bin/node] 7:
> 0xdbbfb6 void
> v8::internal::BodyDescriptorBase::IteratePointers<v8::internal::IterateAndScavengePromotedObjectsVisitor>(v8::internal::HeapObject,
> int, int, v8::internal::IterateAndScavengePromotedObjectsVisitor*)
> [/home/forestpin/data/retail/bin/node/node/bin/node] 8: 0xdbc680 void
> v8::internal::BodyDescriptorBase::IterateJSObjectBodyImpl<v8::internal::IterateAndScavengePromotedObjectsVisitor>(v8::internal::Map,
> v8::internal::HeapObject, int, int,
> v8::internal::IterateAndScavengePromotedObjectsVisitor*)
> [/home/forestpin/data/retail/bin/node/node/bin/node] 9: 0xdbcfd1 void
> v8::internal::BodyDescriptorApply<v8::internal::CallIterateBody, void,
> v8::internal::Map, v8::internal::HeapObject, int,
> v8::internal::IterateAndScavengePromotedObjectsVisitor*>(v8::internal::InstanceType,
> v8::internal::Map, v8::internal::HeapObject, int,
> v8::internal::IterateAndScavengePromotedObjectsVisitor*)
> [/home/forestpin/data/retail/bin/node/node/bin/node] 10: 0xdbd87e
> v8::internal::Scavenger::IterateAndScavengePromotedObject(v8::internal::HeapObject,
> v8::internal::Map, int)
> [/home/forestpin/data/retail/bin/node/node/bin/node] 11: 0xdbdb8b
> v8::internal::Scavenger::Process(v8::internal::OneshotBarrier*)
> [/home/forestpin/data/retail/bin/node/node/bin/node] 12: 0xdc15bd
> v8::internal::ScavengingTask::RunInParallel(v8::internal::ItemParallelJob::Task::Runner)
> [/home/forestpin/data/retail/bin/node/node/bin/node] 13: 0xd75885
> v8::internal::ItemParallelJob::Task::RunInternal()
> [/home/forestpin/data/retail/bin/node/node/bin/node] 14: 0xd75c94
> v8::internal::ItemParallelJob::Run()
> [/home/forestpin/data/retail/bin/node/node/bin/node] 15: 0xdbf65a
> v8::internal::ScavengerCollector::CollectGarbage()
> [/home/forestpin/data/retail/bin/node/node/bin/node] 16: 0xd50741
> v8::internal::Heap::Scavenge()
> [/home/forestpin/data/retail/bin/node/node/bin/node] 17: 0xd64f1b
> v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector,
> v8::GCCallbackFlags)
> [/home/forestpin/data/retail/bin/node/node/bin/node] 18: 0xd65a85
> v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace,
> v8::internal::GarbageCollectionReason, v8::GCCallbackFlags)
> [/home/forestpin/data/retail/bin/node/node/bin/node] 19: 0xd6853c
> v8::internal::Heap::AllocateRawWithRetryOrFail(int,
> v8::internal::AllocationType, v8::internal::AllocationOrigin,
> v8::internal::AllocationAlignment)
> [/home/forestpin/data/retail/bin/node/node/bin/node] 20: 0xd2ef5b
> v8::internal::Factory::NewFillerObject(int, bool,
> v8::internal::AllocationType, v8::internal::AllocationOrigin)
> [/home/forestpin/data/retail/bin/node/node/bin/node] 21: 0x107158e
> v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*,
> v8::internal::Isolate*)
> [/home/forestpin/data/retail/bin/node/node/bin/node] 22: 0x140de99
> [/home/forestpin/data/retail/bin/node/node/bin/node]
【问题讨论】：
标签： javascript node.js arrays out-of-memory