有没有办法优化这个 Promise 循环，这样我就不会出现 FATAL JavaScript heap out of memory 错误了？

Question

我已经调用了高达 2048(mb?) 的节点，但没有任何成功，所以在这一点上，我认为继续提高内存限制是没有意义的，尤其是在我的代码开始时效率低下的情况下。这是来自FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory in ionic 3 的答案。

参考：node --max-old-space-size=2048

由于不够简洁：

<--- Last few GCs --->
io[5481:0x5693440]   286694 ms: Mark-sweep 2048.0 (2051.1) -> 2047.3 (2051.3) MB, 1268.3 / 0.0 ms  (+ 0.0 ms in 13 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 1272 ms) (average mu = 0.087, current mu = 0.003) allocatio[5481:0x
5693440]   290098 ms: Mark-sweep 2048.2 (2051.3) -> 2047.5 (2051.3) MB, 3398.6 / 0.0 ms  (+ 0.0 ms in 13 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 3404 ms) (average mu = 0.026, current mu = 0.002) allocatio

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xa2b020 node::Abort() [node]
 2: 0x97a467 node::FatalError(char const*, char const*) [node]
 3: 0xb9e0ee v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xb9e467 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xd3e875  [node]
 6: 0xd3f21b v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]
 7: 0xd4d012 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
 8: 0xd4de65 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xd5082c v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
10: 0xd1fecb v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
11: 0x10501ef v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
12: 0x13a9ed9  [node]
Aborted
npm ERR! code ELIFECYCLE

这是我有问题的功能：

    public async upsertToDb(courses: Record<string, Record<string, any>>): Promise<string> {
        this.courseTransformationUtility.transformToFlatStructure(courses);
        const flatCourses: Array<Course> = this.courseTransformationUtility.getFlatCourses();
        const flatClasses: Array<Class> = this.courseTransformationUtility.getFlatClasses();

        console.info(`Courses exploded into ${flatCourses.length} rows.`)
        console.info(`Classes exploded into ${flatClasses.length} rows.`)

        await this._deleteTable("Course");
        await this._deleteTable("Class");

        for(let i = 0; i < flatCourses.length; i+=100) {
            Backendless.Data.of(Course).bulkCreate(flatCourses.slice(i, i + 100))
                .then(() => {
                    process.stdout.write(".");
                })
                .catch((e: Error) => console.info(e));
        }


        for(let i = 0; flatClasses.length; i+=100) {
            Backendless.Data.of(Class).bulkCreate(flatClasses.slice(i, i + 100))
                .then(() => {
                    process.stdout.write(".");
                })
                .catch((e: Error) => console.info(e));
        }

        return "";
    }

如果我注释掉第二个循环节点，则在默认内存设置上运行它，没有任何问题。如果我不得不猜测这个问题与异步有关，但我真的不能确定。如果这是问题所在，我们可以完全取消异步吗？抱歉，这里有很多问题我没有深度回答。

编辑：代码更新

Promise.all 不是解决方案，至少对我来说是这样。

        for(let i = 0; i < flatCourses.length; i+=100) {
            let promise: Promise<Array<string>> = Backendless.Data.of(Course).bulkCreate(flatCourses.slice(i, i + 100));
            promise.then(() => process.stdout.write('.'));
            coursePromises.push(promise);
        }

        await Promise.all(coursePromises.slice(0, Math.floor(coursePromises.length/2))).then(() => console.info("1/4"));
        await Promise.all(coursePromises.slice(Math.floor(coursePromises.length/2))).then(() => console.info("2/4"));

        for(let i = 0; flatClasses.length; i+=100) {
            let promise: Promise<Array<string>> = Backendless.Data.of(Class).bulkCreate(flatClasses.slice(i, i + 100));
            promise.then(() => process.stdout.write('.'));
            classPromises.push(promise);
        }

        await Promise.all(classPromises.slice(0, Math.floor(classPromises.length/2))).then(() => console.info("3/4"));
        await Promise.all(classPromises.slice(Math.floor(classPromises.length/2))).then(() => console.info("4/4"));

记录

Using term 202008...
Publishing 2113 courses...
Courses exploded into 11630 rows.
Classes exploded into 10986 rows.
....................................................................................................................1/4
.2/4

<--- Last few GCs --->
io[5628:0x5a47390]   310095 ms: Mark-sweep 2047.4 (2050.7) -> 2046.7 (2051.0) MB, 1935.6 / 0.0 ms  (+ 0.0 ms in 14 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 1941 ms) (average mu = 0.117, current mu = 0.003) allocatio[5628:0x
5a47390]   311960 ms: Mark-sweep 2047.6 (2051.0) -> 2046.9 (2051.2) MB, 1860.4 / 0.0 ms  (+ 0.0 ms in 13 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 1865 ms) (average mu = 0.063, current mu = 0.003) allocatio

<--- JS stacktrace --->

Answer 1

您可以一次执行每个异步操作，或者一次执行一个课程和课程。

一次一个。

for (let i = 0; i < flatCourses.length; i+=100) {
    try {
        await Backendless.Data.of(Course).bulkCreate(flatCourses.slice(i, i + 100))
        process.stdout.write(".");
    } catch (e) {
        console.info(e)
    }
}
// Do the same for flatClasses

每次一门课程和课程

const promises = []

for (let i = 0; i < flatCourses.length; i+=100) {
    const promise = Backendless.Data.of(Course).bulkCreate(flatCourses.slice(i, i + 100))
    promise.then(() => {
        process.stdout.write(".");
    })
    .catch((e: Error) => console.info(e));

    promises.push(promise)
}

await Promise.all(promises)
// Do the same for flatClasses

更复杂的方法涉及一次执行 N 次操作，但如果这些简单的方法能解决您的问题，我不会走那么远。

Answer 2

您不会阻塞任何异步代码，因此所有异步代码都会立即启动并扼杀您的记忆。一个临时解决方案是使用 await 而不是 then。更全面的解决方案将涉及批处理调用并使用 Promie.all 来解析批处理。