Spark Scala FPGrowth 没有任何结果？答案

【问题标题】：Spark Scala FPGrowth without any results?Spark Scala FPGrowth 没有任何结果？
【发布时间】：2018-09-02 22:54:35
【问题描述】：

我正在尝试使用 Scala 从 Spark MLLib 中获取一些频繁项集和关联规则。但实际上我什么也没得到，甚至没有错误。 code（spark/databricks notebook）和data input file可以在here找到。

算法没有找到任何频繁项集和/或关联规则，但有一些证据表明这是错误的。我对 KNIME（一个非编程分析平台）做了同样的事情，但使用 Borgelt 算法进行关联规则学习。在那里，我得到了带有升力和所有其他所需比率的前因、后因映射。但是在使用 Scala 的 Spark 中，我什么也得不到。

%scala

import org.apache.spark.rdd.RDD
import org.apache.spark.mllib.fpm.AssociationRules
import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset

// loading data
val data = sc.textFile("/FileStore/tables/onlinePurchasedProducts.txt")
val onlineTrx: RDD[Array[String]] = data.map(s => s.trim.split(' '))
println("Read: " + onlineTrx.count() + " online baskets")

// checking how transactions look like
val dataframe = onlineTrx.toDF()

println("Schema of transactions looks like: ")
dataframe.printSchema()

println("Content of transactions looks like: ")
dataframe.show()

val fpg = new FPGrowth()
val model = fpg
    .setMinSupport(0.2)
    .setNumPartitions(1)
    .run(onlineTrx)

model.freqItemsets.collect().foreach { itemset =>
     println(itemset.items.mkString("[", ",", "]") + ", " + itemset.freq)
}

model.generateAssociationRules(0.4).collect().foreach { rule =>
     println(s"${rule.antecedent.mkString("[", ",", "]")}=> " +
       s"${rule.consequent .mkString("[", ",", "]")},${rule.confidence}")
}

这段代码的输出是：

Read: 42897 online baskets
Schema of transactions looks like: 
root
    |-- value: array (nullable = true)
    |    |-- element: string (containsNull = true)
Content of transactions looks like: 
e+--------------------+
|               value|
+--------------------+
|      [34502, 70312]|
|             [44247]|
|             [45127]|
|             [79560]|
|             [74801]|
|             [15500]|
|             [74801]|
|      [31149, 78707]|
|             [74801]|
|             [40774]|
|             [76675]|
|[26507, 26638, 33...|
|             [74801]|
|             [78707]|
|             [74801]|
|             [21253]|
|             [74801]|
|[75729, 10899, 26...|
|             [24834]|
|             [74801]|
+--------------------+
only showing top 20 rows

import org.apache.spark.rdd.RDD
import org.apache.spark.mllib.fpm.AssociationRules
import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
data: org.apache.spark.rdd.RDD[String]=
/FileStore/tables/onlinePurchasedProducts.txt MapPartitionsRDD[150] at
textFile at command-4263745371438753:8

onlineTrx: org.apache.spark.rdd.RDD[Array[String]] = MapPartitionsRDD[151] at map at command-4263745371438753:9
dataframe: org.apache.spark.sql.DataFrame = [value: array<string>]
fpg: org.apache.spark.mllib.fpm.FPGrowth = org.apache.spark.mllib.fpm.FPGrowth@23fd0c4
model: org.apache.spark.mllib.fpm.FPGrowthModel[String] = org.apache.spark.mllib.fpm.FPGrowthModel@41278271

任何想法都将不胜感激。

【问题讨论】：

标签： scala apache-spark apache-spark-mllib databricks fpgrowth

【解决方案1】：

贴出的代码完美运行，没有结果的原因是作为最低支持传递的参数。如果将最低支持设置为显着较低的级别，则代码可以工作并显示结果。

部分显示的结果是：

[70423,70422], 123
[70423,70422,70800], 106
[70423,70800], 138
[45005], 400
[37991], 56
[33759], 73
[22024], 57
[34420], 46
[45132], 69
[78515], 53
[11407], 51
[54431], 60
[54432], 55
[35431], 58
[17488], 54
[82885], 45
[99678], 47
[70312], 791
[22087], 44
[70424,70425]=> [70800],0.825
[70425,70422]=> [70800],0.8533333333333334
[52570]=> [52577],0.6129032258064516
[70423,70800]=> [70422],0.7681159420289855
[70423,70422]=> [70800],0.8617886178861789
[26634]=> [26633],0.4909090909090909

【讨论】：