Oozie Spark Action (Containing HiveContext) Giving java.lang.OutOfMemoryError: PermGen space

Question

我正在尝试在 Oozie 中运行 spark-scala 独立应用程序。 请注意，我正在使用具有 20G RAM 的 CDH5.13 Quickstart VM（包含 Cloudera Manager、HUE ...，并且我将 Java 从 7 升级到 8）。

代码几乎什么都不做，它只是创建 HiveContext 然后创建一个 Hive 表：

import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.rdd.RDD

object ThirdApp {
def main(args: Array[String]) {
val conf = new SparkConf().setAppName("Third Application")
val sc = new SparkContext(conf)

val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
import sqlContext.implicits._
sqlContext.sql("CREATE TABLE IF NOT EXISTS default.src (key INT, value STRING)")
}
}

sbt 文件：

name := "Third Project"
version := "1.0"
scalaVersion := "2.10.5"
libraryDependencies ++= Seq("org.apache.spark" %% "spark-core" % "1.6.0",
 "org.apache.spark" %% "spark-hive"  % "1.6.0")

当我提交应用程序时（在 shell 中），应用程序运行良好，并且 Hive 表已创建。 但是当我在 oozie 中运行同一个应用程序时，它会出现内存问题。

请注意，我习惯于在 oozie 中运行 spark 应用程序，它们工作正常，除了这个包含 hiveContext 的用例。

这是 workflow.xml：

<workflow-app name="spark-scala" xmlns="uri:oozie:workflow:0.5">
    <start to="spark-5a6a"/>
    <kill name="Kill">
        <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <action name="spark-5a6a">
        <spark xmlns="uri:oozie:spark-action:0.2">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <master>local</master>
            <mode>client</mode>
            <name>MySpark</name>
              <class>ThirdApp</class>
            <jar>third-project_2.10-1.0.jar</jar>
            <file>/user/cloudera/oozie-spark/third-project_2.10-1.0.jar#third-project_2.10-1.0.jar</file>
        </spark>
        <ok to="End"/>
        <error to="Kill"/>
    </action>
    <end name="End"/>
</workflow-app>

这里是 job.properties：

oozie.use.system.libpath=True
send_email=False
dryrun=False
nameNode=hdfs://quickstart.cloudera:8020
jobTracker=quickstart.cloudera:8032
security_enabled=False

请注意，我从 Cloudera Manager > Category > Security > Superuser group 添加了 spark 超级用户组，以避免权限问题：

Adding spark to superuser group (Cloudera Manager View)

hive-site.xml view

标准输出日志：

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], exception invoking main(), PermGen space

ERROR org.apache.hadoop.mapred.YarnChild  - Error running child : java.lang.OutOfMemoryError: PermGen space

WARN  org.apache.hadoop.ipc.Client  - Unexpected error reading responses on connection Thread[IPC Client (1722336150) connection to /127.0.0.1:59738 from job_1547905343759_0002,5,main]

java.lang.OutOfMemoryError: PermGen space

INFO  org.apache.hadoop.mapred.Task  - Communication exception: java.io.IOException: The client is stopped

ERROR org.apache.hadoop.yarn.YarnUncaughtExceptionHandler  - Thread Thread[main,5,main] threw an Error.

stderr 日志：

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], exception invoking main(), PermGen space
Halting due to Out Of Memory Error...

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "main"

系统日志：

INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started
NFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens:
INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1547905343759_0002, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@3a06520)
INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: RM_DELEGATION_TOKEN, Service: 127.0.0.1:8032, Ident: (RM_DELEGATION_TOKEN owner=cloudera, renewer=oozie mr token, realUser=oozie, issueDate=1547907649379, maxDate=1548512449379, sequenceNumber=6, masterKeyId=2)
INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now.
INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /yarn/nm/usercache/cloudera/appcache/application_1547905343759_0002
INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
INFO [main] org.apache.hadoop.mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: org.apache.oozie.action.hadoop.OozieLauncherInputFormat$EmptySplit@1ab7aa29
NFO [main] org.apache.hadoop.mapred.MapTask: numReduceTasks: 0
INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
NFO [main] org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at quickstart.cloudera/127.0.0.1:8032
INFO [main] org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at quickstart.cloudera/127.0.0.1:8032

我还在 Cloudera Manager > Logs > ERROR 中查找了日志：

Exception in doCheckpoint
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.RetriableException): NameNode still not started
...(more)

Error starting JobHistoryServer
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Error creating done directory: [hdfs://quickstart.cloudera:8020/user/history/done]
...
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.RetriableException): NameNode still not started
...(more)

SERVER[quickstart.cloudera] USER[-] GROUP[-] TOKEN[] APP[-] JOB[0000001-190120120522295-oozie-oozi-W] ACTION[0000001-190120120522295-oozie-oozi-W@spark-5a6a] XException, 
org.apache.oozie.command.CommandException: E0800: Action it is not running its in [KILLED] state, action [0000001-190120120522295-oozie-oozi-W@spark-5a6a]
    at org.apache.oozie.command.wf.CompletedActionXCommand.eagerVerifyPrecondition(CompletedActionXCommand.java:92)
    at org.apache.oozie.command.XCommand.call(XCommand.java:257)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

getting attribute DatanodeNetworkCounts of Hadoop:service=DataNode,name=DataNodeInfo threw an exception
javax.management.RuntimeMBeanException: java.lang.NullPointerException
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:852)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:651)
    at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
    at org.apache.hadoop.jmx.JMXJsonServlet.writeAttribute(JMXJsonServlet.java:342)
...More

这是日志的（近似）完整视图：

/var/log/spark/...log

/var/log/hadoop-hdfs/...log.out

我尝试通过以下方式解决这些问题：

在 mapred-site.xml 中为 map/reduce 增加内存：

  <property>
    <name>mapreduce.map.memory.mb</name>
    <value>2128</value>
  </property>
  <property>
  <property>
    <name>mapreduce.reduce.memory.mb</name>
    <value>2128</value>
  </property>
  <property>
  <property>
    <name>yarn.app.mapreduce.am.resource.mb</name>
    <value>2128</value>
  </property>

Global View of mapred-site.xml

我也尝试增加 Java Heap： View of Java Heap in Cloudera Manager

我也尝试设置网关默认组： View of Client Java Configuration Options

我尝试在工作流程中添加选项列表：--driver-memory 5G

但它总是给出同样的错误。 能否请您帮忙！

Answer 1

我不确定内存问题 - 但我看到了“权限被拒绝”问题 由于某些原因，文件夹'/user/spark/applicationHistory/local-1547821006998'归用户'cloudera'所有，而不是spark，因此spark无法写入。 要解决它，请登录到 VM 并将组 supergroup 添加到用户 spark： “usermod -G 超级组 saprk” 干杯，多伦