无法序列化类 mongodb spark答案

【问题标题】：can't serialize class mongodb spark无法序列化类 mongodb spark
【发布时间】：2015-07-23 08:33:32
【问题描述】：

在我的应用程序中，有一个模型类

import java.io.Serializable;
import java.util.Date;

public class Observation implements Serializable{

private static final long serialVersionUID = 1L;
...
}

我通过实现 Serializable 接口对这个类进行了序列化。这个类是 MongoDB 集合的模型。将 MongoDB 记录映射到 Observation 对象一切正常。

当我运行我的应用程序时，Spark 作业会执行 map-reduce 作业。在 map reduce 之后，我得到了这个异常。我添加了堆栈跟踪：

ERROR Executor: Exception in task ID 133
java.lang.IllegalArgumentException: can't serialize class    com.mongodb.spark.demo.Observation
at org.bson.BasicBSONEncoder._putObjectField(BasicBSONEncoder.java:284)
at org.bson.BasicBSONEncoder.putObject(BasicBSONEncoder.java:185)
at org.bson.BasicBSONEncoder.putObject(BasicBSONEncoder.java:131)
at com.mongodb.DefaultDBEncoder.writeObject(DefaultDBEncoder.java:33)
at com.mongodb.BSONBinaryWriter.encodeDocument(BSONBinaryWriter.java:339)
at com.mongodb.InsertCommandMessage.writeTheWrites(InsertCommandMessage.java:45)
at com.mongodb.InsertCommandMessage.writeTheWrites(InsertCommandMessage.java:23)
at com.mongodb.BaseWriteCommandMessage.encodeMessageBody(BaseWriteCommandMessage.java:69)
at com.mongodb.BaseWriteCommandMessage.encodeMessageBody(BaseWriteCommandMessage.java:23)
at com.mongodb.RequestMessage.encode(RequestMessage.java:66)
at com.mongodb.BaseWriteCommandMessage.encode(BaseWriteCommandMessage.java:53)
at com.mongodb.DBCollectionImpl.sendWriteCommandMessage(DBCollectionImpl.java:473)
at com.mongodb.DBCollectionImpl.writeWithCommandProtocol(DBCollectionImpl.java:427)
at com.mongodb.DBCollectionImpl.insertWithCommandProtocol(DBCollectionImpl.java:387)
at com.mongodb.DBCollectionImpl.insert(DBCollectionImpl.java:186)
at com.mongodb.DBCollectionImpl.insert(DBCollectionImpl.java:165)
at com.mongodb.DBCollection.insert(DBCollection.java:161)
at com.mongodb.DBCollection.insert(DBCollection.java:107)
at com.mongodb.DBCollection.save(DBCollection.java:966)
at com.mongodb.DBCollection.save(DBCollection.java:934)
at com.mongodb.hadoop.output.MongoRecordWriter.write(MongoRecordWriter.java:93)
at org.apache.spark.rdd.PairRDDFunctions.org$apache$spark$rdd$PairRDDFunctions$$writeShard$1(PairRDDFunctions.scala:716)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:730)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:730)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)
at org.apache.spark.scheduler.Task.run(Task.scala:51)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

那么，为什么我会收到这个异常？谁能帮我找到解决办法？

【问题讨论】：

标签： java mongodb serialization apache-spark

【解决方案1】：

也许，问题是因为你没有默认构造函数

public Observation () {
}

【讨论】：

我的问题是关于序列化的。当我通过 BasicDBObject 类扩展 Observation 类时，问题就解决了。谢谢。

【解决方案2】：

当我通过 BasicDBObject 扩展我的类时，问题就解决了。

因此，下面给出了最新且正确的课程版本

public class Observation extends BasicDBObject implements Serializable{

private static final long serialVersionUID = 1L;
...
}

【讨论】：