应用 K-Means 算法后如何查看添加到集群的数据点？答案

【问题标题】：How do I view the datapoints that are added to a cluster after applying K-Means algorithm?应用 K-Means 算法后如何查看添加到集群的数据点？
【发布时间】：2016-08-21 15:15:55
【问题描述】：

我在 scala 中实现了 k-means 算法，如下所示。

def clustering(clustnum:Int,iternum:Int,parsedData: RDD[org.apache.spark.mllib.linalg.Vector]): Unit= {
val clusters = KMeans.train(parsedData, clustnum, iternum)

println("The Cluster centers of each column for "+clustnum+" clusters and "+iternum+" iterations are:- ")


clusters.clusterCenters.foreach(println) 

val predictions= clusters.predict(parsedData)

 predictions.collect()

}

我知道如何打印每个集群的集群中心，但是 scala 中是否有一个函数可以打印哪些行已添加到哪个集群？

我正在处理的数据包含多行浮点值，每行都有一个 ID。它有大约 34 列和大约 200 行。我正在scala中研究火花。

我需要能够看到结果。就像 Id_1 在集群 1 左右一样。

编辑：我能够做到这一点

println(clustnum+" clusters and "+iternum+" iterations ")

val vectorsAndClusterIdx = parsedData.map{ point => 
val prediction = clusters.predict(point) 
(point.toString, prediction) 
} 

vectorsAndClusterIdx.collect().foreach(println)

它打印集群 ID 和添加到集群的行

行显示为字符串，簇ID是后面打印的

([1.0,1998.0,1.0,1.0,1.0,1.0,14305.0,39567.0,1998.0,23.499,25.7,27.961,29.04,28.061,26.171,24.44,24.619,24.529,24.497,23.838,22.322,1998.0,0.0,0.007,0.007,96.042,118.634,61.738,216.787,262.074,148.697,216.564,49.515,28.098],4)
([2.0,1998.0,1.0,1.0,2.0,1.0,185.0,2514.0,1998.0,23.499,25.7,27.961,29.04,28.061,26.171,24.44,24.619,24.529,24.497,23.838,22.322,1998.0,0.0,0.007,0.007,96.042,118.634,61.738,216.787,262.074,148.697,216.564,49.515,28.098],0)
([3.0,1998.0,1.0,1.0,2.0,2.0,27.0,272.0,1998.0,23.499,25.7,27.961,29.04,28.061,26.171,24.44,24.619,24.529,24.497,23.838,22.322,1998.0,0.0,0.007,0.007,96.042,118.634,61.738,216.787,262.074,148.697,216.564,49.515,28.098],0)

但是有没有办法只打印行 ID 和集群 ID？

在这里使用数据框对我有帮助吗？

【问题讨论】：

标签： scala apache-spark cluster-analysis ibm-cloud k-means

【解决方案1】：

您可以使用KMeansModel 的predict() 功能。

查看文档：http://spark.apache.org/docs/1.6.0/api/scala/index.html#org.apache.spark.mllib.clustering.KMeansModel

在您的代码中：

KMeans.train(parsedData, clustnum, iternum)

返回一个KMeansModel 对象。

所以，你可以这样做：

val predictions = clusters.predict(parsedData)

并得到一个MapPartitionsRDD 作为结果。

predictions.collect()

为您提供带有集群索引分配的Array。

【讨论】：

我尝试了预测功能，但它似乎没有打印任何内容？我已经编辑了问题以显示输出和我拥有的完整功能。你能看一下吗？
predictions.collect() 单独不输出任何东西？我会仔细看看它，因为它工作正常。关于输出，1s和0s来自predictions.collect()这一行，对吧？
predictions.collect().foreach(println) 似乎有效。非常感谢。 :)

【解决方案2】：

println(clustnum+" clusters and "+iternum+" iterations ")

val vectorsAndClusterIdx = parsedData.map{ point => 
val prediction = clusters.predict(point) 
(point.toString, prediction) 
} 

vectorsAndClusterIdx.collect().foreach(println)

似乎解决了我的问题。它打印集群 ID 和添加到集群的行

行显示为字符串，簇ID是后面打印的

([1.0,1998.0,1.0,1.0,1.0,1.0,14305.0,39567.0,1998.0,23.499,25.7,27.961,29.04,28.061,26.171,24.44,24.619,24.529,24.497,23.838,22.322,1998.0,0.0,0.007,0.007,96.042,118.634,61.738,216.787,262.074,148.697,216.564,49.515,28.098],4)
([2.0,1998.0,1.0,1.0,2.0,1.0,185.0,2514.0,1998.0,23.499,25.7,27.961,29.04,28.061,26.171,24.44,24.619,24.529,24.497,23.838,22.322,1998.0,0.0,0.007,0.007,96.042,118.634,61.738,216.787,262.074,148.697,216.564,49.515,28.098],0)
([3.0,1998.0,1.0,1.0,2.0,2.0,27.0,272.0,1998.0,23.499,25.7,27.961,29.04,28.061,26.171,24.44,24.619,24.529,24.497,23.838,22.322,1998.0,0.0,0.007,0.007,96.042,118.634,61.738,216.787,262.074,148.697,216.564,49.515,28.098],0)

【讨论】：