为什么创建 HiveContext 失败并显示“java.lang.OutOfMemoryError: PermGen space”？

Question

我正在使用 Spark SQL 解析 JSON，它运行良好，它找到了架构，我正在使用它进行查询。

现在我需要“扁平化” JSON，并且我在论坛中读到最好的方法是使用 Hive（横向视图）爆炸，所以我尝试对它做同样的事情。但我什至无法创建上下文... Spark 给了我一个错误，我找不到解决方法。

正如我所说，在这一点上我只是想创建de context：

println ("Create Spark Context:")
val sc = new SparkContext( "local", "Simple", "$SPARK_HOME")
println ("Create Hive context:")
val hiveContext = new HiveContext(sc)

它给了我这个错误：

Create Spark Context:
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/12/26 15:13:44 INFO Remoting: Starting remoting
15/12/26 15:13:45 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@192.168.80.136:40624]

Create Hive context:
15/12/26 15:13:50 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
15/12/26 15:13:50 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
15/12/26 15:13:56 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
15/12/26 15:13:56 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
15/12/26 15:13:58 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
15/12/26 15:13:58 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
15/12/26 15:13:59 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
15/12/26 15:14:01 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
15/12/26 15:14:01 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
Exception in thread "main" java.lang.reflect.InvocationTargetException
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
  at org.apache.spark.sql.hive.client.IsolatedClientLoader.liftedTree1$1(IsolatedClientLoader.scala:183)
  at org.apache.spark.sql.hive.client.IsolatedClientLoader.<init>(IsolatedClientLoader.scala:179)
  at org.apache.spark.sql.hive.HiveContext.metadataHive$lzycompute(HiveContext.scala:226)
  at org.apache.spark.sql.hive.HiveContext.metadataHive(HiveContext.scala:185)
  at org.apache.spark.sql.hive.HiveContext.setConf(HiveContext.scala:392)
  at org.apache.spark.sql.hive.HiveContext.defaultOverrides(HiveContext.scala:174)
  at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:177)
  at pebd.emb.Bicing$.main(Bicing.scala:73)
  at pebd.emb.Bicing.main(Bicing.scala)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:601)
  at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
Caused by: java.lang.OutOfMemoryError: PermGen space

Process finished with exit code 1

我知道这是一个非常简单的问题，但我真的不知道该错误的原因。 提前谢谢大家。

Answer 1

这是异常的相关部分：

Caused by: java.lang.OutOfMemoryError: PermGen space

您需要增加提供给 JVM 的 PermGen 内存量。默认情况下 (SPARK-1879)，Spark 自己的启动脚本会将其增加到 128 MB，因此我认为您必须在 IntelliJ 运行配置中执行类似的操作。尝试将-XX:MaxPermSize=128m 添加到“VM 选项”列表中。