异步不保序,但大作业执行时间后移。
.set("spark.scheduler.mode", "FAIR")
公平调度,充分使用集群资源。

Spark Doc

def
collectAsync(): FutureAction[Seq[T]]
Returns a future for retrieving all elements of this RDD.

def
countAsync(): FutureAction[Long]
Returns a future for counting the number of elements in the RDD.

def
foreachAsync(f: (T) ⇒ Unit): FutureAction[Unit]
Applies a function f to all elements of this RDD.

def
foreachPartitionAsync(f: (Iterator[T]) ⇒ Unit): FutureAction[Unit]
Applies a function f to each partition of this RDD.

def
takeAsync(num: Int): FutureAction[Seq[T]]
Permalink
Returns a future for retrieving the first num elements of the RDD.

相关文章:

  • 2022-12-23
  • 2022-12-23
  • 2021-12-21
  • 2022-12-23
  • 2022-12-23
  • 2021-11-07
  • 2022-12-23
  • 2022-12-23
猜你喜欢
  • 2022-12-23
  • 2022-12-23
  • 2022-12-23
  • 2021-09-14
  • 2021-12-16
  • 2022-12-23
  • 2022-12-23
相关资源
相似解决方案