使用 Spark Streaming Application 的 Sparklens 进行性能分析

Question

我正在尝试使用 sparklens 对 spark 流应用程序 进行性能分析。它给出了这样的结果

Executor count     1  ( 80%) estimated time 01m 29s and estimated cluster utilization 18.55%
Executor count     2  (100%) estimated time 01m 28s and estimated cluster utilization 9.36%
Executor count     3  (150%) estimated time 01m 28s and estimated cluster utilization 6.26%
Executor count     4  (200%) estimated time 01m 28s and estimated cluster utilization 4.69%
Executor count     6  (300%) estimated time 01m 28s and estimated cluster utilization 3.13%
Executor count     8  (400%) estimated time 01m 28s and estimated cluster utilization 2.35%
Executor count    10  (500%) estimated time 01m 28s and estimated cluster utilization 1.88%

即使如果我增加执行者，估计的时间也是一样的。

这些建议是否正确？

Answer 1

你为什么不试试？您可能正在运行单个任务。单个任务不会受益于额外的内核，除非您想办法将其分解为并行任务。