如何使用scala检查RDD [Map [Int，String]]中所有地图中的特定值？答案

【问题标题】：How to check a particular value in all Maps in an RDD[Map[Int,String]] at a strech using scala?如何使用scala检查RDD [Map [Int，String]]中所有地图中的特定值？
【发布时间】：2014-11-28 13:33:20
【问题描述】：

我想使用 scala 来检查 RDD[Map[Int,String]] 中所有地图中的特定值。我的 csv 文件是，

Map(0 -> sunny, 1 -> hot, 2 -> high, 3 -> false, 4 -> no)
Map(0 -> sunny, 1 -> hot, 2 -> high, 3 -> true, 4 -> no)
Map(0 -> overcast, 1 -> hot, 2 -> high, 3 -> false, 4 -> yes)
Map(0 -> rainy, 1 -> mild, 2 -> high, 3 -> false, 4 -> yes)
Map(0 -> rainy, 1 -> cool, 2 -> normal, 3 -> false, 4 -> yes)

这里我要检查每张地图中的所有最后一个值，即 no,no,yes,yes,yes 具有特定值 check(yes/no)一口气。

【问题讨论】：

last value 是什么意思？具有最大键的值？
@SergeyLagutin，这里最后一个值表示每个地图都包含 Map(0->bb, 1->mnj,....)。这个 0,1,..etc 代表实际数据的列数。我们需要取最后一列的值。因此，从每个映射中，最后一个值，这里，4 -> no，4 -> no，4 -> yes,.....，所以从这对中取 no，no，yes... 并检查字符串值。

标签： scala collections mapreduce

【解决方案1】：

scala> val a = List(Map(0 -> "sunny", 1 -> "hot", 2 -> "high", 3 -> "false", 4 -> "no"),
     |   Map(0 -> "sunny", 1 -> "hot", 2 -> "high", 3 -> "true", 4 -> "no"),
     |   Map(0 -> "overcast", 1 -> "hot", 2 -> "high", 3 -> "false", 4 -> "yes"),
     |   Map(0 -> "rainy", 1 -> "mild", 2 -> "high", 3 -> "false", 4 -> "yes"),
     |   Map(0 -> "rainy", 1 -> "cool", 2 -> "normal", 3 -> "false", 4 -> "yes"))
a: List[scala.collection.immutable.Map[Int,String]] = List(Map(0 -> sunny, 1 -> hot, 2 -> high, 3 -> false, 4 -> no), Map(0 -> sunny, 1 -> hot, 2 -> high, 3 -> true, 4 -> no), Map(0 -> overcast, 1 -> hot, 2 -> high, 3 -> false, 4 -> yes), Map(0 -> rainy, 1 -> mild, 2 -> high, 3 -> false, 4 -> yes), Map(0 -> rainy, 1 -> cool, 2 -> normal, 3 -> false, 4 -> yes))

scala> sc.parallelize(a)
res0: org.apache.spark.rdd.RDD[scala.collection.immutable.Map[Int,String]] = ParallelCollectionRDD[0] at parallelize at <console>:15

scala> val l = sc.parallelize(a)
l: org.apache.spark.rdd.RDD[scala.collection.immutable.Map[Int,String]] = ParallelCollectionRDD[1] at parallelize at <console>:14

scala> def check( s : String) : Boolean = if (s.equals("yes")) true else false
check: (s: String)Boolean

scala> val res = l.map{ x => check(x(4)) }
res: org.apache.spark.rdd.RDD[Boolean] = MappedRDD[4] at map at <console>:18


14/11/28 00:18:47 INFO DAGScheduler: Stage 5 (take at <console>:21) finished in 0.020 s
14/11/28 00:18:47 INFO TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool 
14/11/28 00:18:47 INFO DAGScheduler: Job 5 finished: take at <console>:21, took 0.026501 s
false
false
true
true
true

更新仅当所有值都是true 时，以下将是true，否则它将是false。

scala> res.reduce( _ && _ )

【讨论】：

Not List[Map[Int,String]]，它是 RDD[Map[Int,String]]....，我需要检查所有值是 yes 还是 no。不一一取。如果我的检查字符串是yes，检查所有的值是yes，如果是，做function1 else function2
它只是一个 RDD[Map[Int,String]]。请参阅org.apache.spark.rdd.RDD[scala.collection.immutable.Map[Int,String]] 我已经更新了结果以满足您的最终要求。