SparkR 窗口函数：错误“任务不可序列化”答案

【问题标题】：SparkR window function : Error "Task not serializable"SparkR 窗口函数：错误“任务不可序列化”
【发布时间】：2016-01-23 02:11:43
【问题描述】：

感谢 SparkR 的 Spark SQL 模块，我尝试测试 Window function。我使用 Spark 1.6，并尝试在两种不同的部署模式（local 和 yarn-client）中复制 zero323 提供的示例。

set.seed(1)

hc <- sparkRHive.init(sc)
sdf <- createDataFrame(hc, data.frame(x=1:12, y=1:3, z=rnorm(12)))
registerTempTable(sdf, "sdf")

query <- sql(hc, "SELECT x, y, z, LAG(z) OVER (PARTITION BY y ORDER BY x) FROM sdf") 
head(query)

##    x y          z        _c3
## 1  1 1 -0.6264538         NA
## 2  4 1  1.5952808 -0.6264538
## 3  7 1  0.4874291  1.5952808
## 4 10 1 -0.3053884  0.4874291
## 5  2 2  0.1836433         NA
## 6  5 2  0.3295078  0.1836433

但是对于这两种部署模式，我在执行 Spark Action head(query) 时遇到相同的错误：

16/01/21 18:03:17 ERROR r.RBackendHandler: dfToCols on     org.apache.spark.sql.api.r.SQLUtils failed
Error in invokeJava(isStatic = TRUE, className, methodName, ...) : 
org.apache.spark.SparkException: Task not serializable
at    org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:304)
   at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:294)
at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:122)
at org.apache.spark.SparkContext.clean(SparkContext.scala:2055)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1.apply(RDD.scala:707)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1.apply(RDD.scala:706)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
at org.apache.spark.rdd.RDD.mapPartitions(RDD.scala:706)
at org.apache.spark.sql.execution.Window.doExecute(Window.scala:245)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:132)
at org.

我直接在 HIVE 中尝试了这个 HQL 查询并且工作正常。像classical_query <- sql(hc, "SELECT * FROM sdf") head(classical_query) 这样的“正常”查询也可以正常工作。

谢谢

【问题讨论】：

标签： r hive apache-spark-sql sparkr

【解决方案1】：

我解决了我的问题。这只是 Spark 配置问题。

我刚刚从spark-defaults.confconfiguration 文件中的spark.driver.extraClassPath 变量中删除了/usr/hdp/current/hive-client/lib/hive-exec.jar JAR。

【讨论】：