用scala写spark堆空间问题解决

用scala写spark堆空间问题解决

19/05/27 17:23:00 INFO SecurityManager: Changing modify acls groups to: 
19/05/27 17:23:00 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(Administrator); groups with view permissions: Set(); users  with modify permissions: Set(Administrator); groups with modify permissions: Set()
19/05/27 17:23:01 INFO Utils: Successfully started service 'sparkDriver' on port 62158.
19/05/27 17:23:01 INFO SparkEnv: Registering MapOutputTracker
19/05/27 17:23:01 ERROR SparkContext: Error initializing SparkContext.
java.lang.IllegalArgumentException: System memory 259522560 must be at least 471859200. Please increase heap size using the --driver-memory option or spark.driver.memory in Spark configuration.
    at org.apache.spark.memory.UnifiedMemoryManager$.getMaxMemory(UnifiedMemoryManager.scala:216)
    at org.apache.spark.memory.UnifiedMemoryManager$.apply(UnifiedMemoryManager.scala:198)
    at org.apache.spark.SparkEnv$.create(SparkEnv.scala:330)
    at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:174)
    at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:257)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:432)
    at Spark1$.main(Spark1.scala:6)
    at Spark1.main(Spark1.scala)
19/05/27 17:23:01 INFO SparkContext: Successfully stopped SparkContext
Exception in thread "main" java.lang.IllegalArgumentException: System memory 259522560 must be at least 471859200. Please increase heap size using the --driver-memory option or spark.driver.memory in Spark configuration.
    at org.apache.spark.memory.UnifiedMemoryManager$.getMaxMemory(UnifiedMemoryManager.scala:216)
    at org.apache.spark.memory.UnifiedMemoryManager$.apply(UnifiedMemoryManager.scala:198)
    at org.apache.spark.SparkEnv$.create(SparkEnv.scala:330)
    at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:174)
    at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:257)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:432)
    at Spark1$.main(Spark1.scala:6)
    at Spark1.main(Spark1.scala)

 

解决方法:搞定。

用scala写spark堆空间问题解决

用scala写spark堆空间问题解决

相关文章:

  • 2022-12-23
  • 2022-12-23
  • 2022-12-23
  • 2022-12-23
  • 2021-07-05
  • 2021-11-26
  • 2022-12-23
  • 2021-12-07
猜你喜欢
  • 2022-12-23
  • 2022-12-23
  • 2022-12-23
  • 2021-10-16
  • 2021-10-19
  • 2021-07-24
相关资源
相似解决方案