【问题标题】:H2O implicit conversion throws compilation errorH2O 隐式转换引发编译错误
【发布时间】:2018-03-19 13:13:19
【问题描述】:

下面的代码在分配 H2OFrame 时抛出错误,很可能是隐式转换有问题。错误是:

类型不匹配;发现:org.apache.spark.h2o.RDD[Int](扩展 to) org.apache.spark.rdd.RDD[Int] 需要: org.apache.spark.h2o.H2OFrame(扩展为)water.fvec.H2OFrame

和代码:

import org.apache.spark.h2o._

import org.apache.spark._
import org.apache.spark.SparkContext._

object App1 extends App{

         val conf = new SparkConf()
         conf.setAppName("Test")
         conf.setMaster("local[1]")
         conf.set("spark.executor.memory","1g");

         val sc = new SparkContext(conf)

         val rawData = sc.textFile("c:\\spark\\data.csv")        
         val data = rawData.map(line => line.split(',').map(_.toDouble))    
         val response: RDD[Int] = data.map(row => row(0).toInt)

         val h2oResponse: H2OFrame = response   // <-- this line throws the error
         sc.stop

}

【问题讨论】:

标签: scala apache-spark h2o


【解决方案1】:

你所缺少的只是 h2oContext 的隐含

import h2oContext.implicits._
val h2oResponse: H2OFrame = response.toDF()

【讨论】:

  • 我收到以下错误:value toDF is not a member of org.apache.spark.h2o.RDD[Int]
  • 你也需要import sqlContext.implicits._
  • 为什么是sqlContext.implicits?另外,我没有变量sqlContext
  • .toDF() 在 sqlContext.implicits._ 中可用,您可以在创建 sparkContext 时创建 sqlContext
  • 现在它可以编译了,但是当我运行我得到When using the Sparkling Water as Spark package via --packages option, the 'no.priv.garshol.duke:duke:1.2' dependency has to be specified explicitly due to a bug in Spark dependency resolution.
猜你喜欢
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 2018-12-28
相关资源
最近更新 更多