Livy 创建会话已死

Question

我在 spark 配置中添加了一个包（在 spark-default.conf 中），但是当我使用 创建一个新会话时>livy 它给我带来了问题（请参阅下面的错误）以及会话和死亡。 ps：当我删除这个包时一切正常。

20/05/04 00:17:35 WARN RSCClient: Error stopping RPC.                                                                                                                   
io.netty.util.concurrent.BlockingOperationException: DefaultChannelPromise@6d493840(uncancellable)                                                                      
        at io.netty.util.concurrent.DefaultPromise.checkDeadLock(DefaultPromise.java:394)                                                                               
        at io.netty.channel.DefaultChannelPromise.checkDeadLock(DefaultChannelPromise.java:157)                                                                         
   ...........
Exception in thread "Thread-32" java.io.IOException: Stream closed                                                                                                      
        at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170)                                                                                       
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:283)                                                                                              
        ........                                                                                            
        at org.apache.livy.utils.LineBufferedStream$$anon$1.run(LineBufferedStream.scala:46)                                                                            
20/05/04 00:17:36 WARN ContextLauncher: Child process exited with code 143.                                                                                             
20/05/04 00:17:36 ERROR SparkProcApp: job was killed by user                                                                                                            
20/05/04 00:17:36 INFO InteractiveSession: Stopped InteractiveSession 0.                                                                                                
20/05/04 00:28:17 INFO InteractiveSessionManager: Deleting InteractiveSession 0 because it was inactive for more than 3600000.0 ms.                                     
20/05/04 00:28:17 INFO InteractiveSessionManager: Deleting session 0                                                                                                    
20/05/04 00:28:17 INFO InteractiveSession: Stopping InteractiveSession 0...                                                                                             
20/05/04 00:28:17 INFO InteractiveSession: Stopped InteractiveSession 0.                                                                                                
20/05/04 00:28:17 INFO InteractiveSessionManager: Deleted session 0  

我使用： cloudera hdp2.6.5：

• 火花2.3
• livy 0.7.0
• Hadoop 2.7
• lib 无监督 (https://github.com/unsupervise/spark-tss)

步骤：

• livy conf => livy.spark.master 纱线集群
• spark-default conf => spark.jars.repositories https://dl.bintray.com/unsupervise/maven/
• spark-defaultconf => spark.jars.packages com.github.unsupervise:spark-tss:0.1.1

Answer 1

请尝试添加到 spark 默认值

spark.jars.repositories https://dl.bintray.com/unsupervise/maven/

或者（只是为了确保您对 spark-default.conf 文件没有任何问题）请尝试在提交 Spark 作业时将这些配置包含到 Livy 请求正文中（请参阅 Livy API）。