举个例子:

val logFile = "file:///home/soyo/桌面/6.txt" 
    val conf = new SparkConf().setAppName("Simple Application")
    val sc = new SparkContext(conf)
val RDD =sc.textFile(logFile)
    RDD.saveAsTextFile("file:////home/soyo/桌面/667.txt")   
    val repartitionedRDD=context.repartition(8) //控制RDD分区的个数
    repartitionedRDD.saveAsTextFile("file:////home/soyo/桌面/668.txt")
 

667.txt:                                        

part-00000    _SUCCESS

668.txt:

part-00000   part-00001    part-00002    part-00003     part-00004    part-00005   part-00006    part-00007     _SUCCESS

如果设置.setMaster("local[2]")默认最少是2个分区,别的类同

相关文章:

  • 2021-08-02
  • 2021-07-28
  • 2022-12-23
  • 2021-06-22
  • 2021-08-07
  • 2022-12-23
  • 2021-10-12
猜你喜欢
  • 2021-08-19
  • 2021-12-01
  • 2021-10-11
  • 2021-04-28
  • 2021-12-21
  • 2021-04-09
  • 2022-01-15
相关资源
相似解决方案