spark开发实战技巧

1. table转map

        //事件权重
        val eventWeightQuery = "select eventId,eventCode,eventName,weight from dwd.dwd_event_weight_dim";
        val eventWeightMap = spark.sql(eventWeightQuery).distinct().rdd
            .map(row => (row.getAs[String]("eventCode"),row.getAs[Double]("weight")))
            .collect().toMap[String,Double]

         println(eventWeightMap)

打印结果：

Map(goodsRate -> 2, goodsFavorite -> 6, goodsCart -> 8, goodsView -> 2, orderTrade -> 10, search -> 4)

 

2.