【问题标题】:Flume - HiveSink: java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/streaming/RecordWriterFlume - HiveSink: java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/streaming/RecordWriter
【发布时间】:2016-01-15 14:06:45
【问题描述】:

我正在尝试在我的水槽配置中创建一个蜂巢接收器,当我运行水槽-ng 时,我遇到了一些依赖问题,如下所示。你能告诉我如何克服这种依赖吗?看起来有一些运行时依赖。我已经正确安装了 hive 并进行了所需的环境变量设置以指向 HIVE_HOME。任何帮助都将不胜感激。谢谢。

2016-01-15 14:41:37,757 (conf-file-poller-0) 
[INFO -org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:42)]  
Creating instance of sink: hiveSink, type: hive
2016-01-15 14:41:37,763 (conf-file-poller-0) 
[ERROR -      org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunn    able.run(PollingPropertiesFileConfigurationProvider.java:145)] 
Failed to  start  agent because dependencies were not found in classpath. Error follows.

java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/streaming/RecordWriter
at org.apache.flume.sink.hive.HiveSink.createSerializer(HiveSink.java:223)
at org.apache.flume.sink.hive.HiveSink.configure(HiveSink.java:203)
at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:413)
at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:98)
at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.apache.hive.hcatalog.streaming.RecordWriter
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 13 more

【问题讨论】:

    标签: hive flume flume-ng


    【解决方案1】:

    我正在学习有关 CDH 5.7 发行版的 Flume。我确实遇到了同样的问题,flume-env.sh 脚本似乎对我不起作用,所以我使用 --classpath 命令参数来引用与 hive 和 hive- 相关的所有库- h目录。遵循试验和错误方法来添加多个 lib 文件夹引用。没有找到很多关于这个论点的文档。

    例如flume-ng agent --conf /home/cloudera/flume/ --conf-file /home/cloudera/flume/netcat_memchannel_hivesink.conf --name agent1 --classpath "/usr/lib/hive-hcatalog/share/hcatalog/*":"/usr/lib/hive/lib/*"

    【讨论】:

      【解决方案2】:

      您应该通过 flume-env.sh 设置 HIVE_HOME 和 HCAT_HOME 环境变量,或者将它们提供给用户配置文件。

      【讨论】:

        猜你喜欢
        • 1970-01-01
        • 1970-01-01
        • 2019-06-07
        • 1970-01-01
        • 1970-01-01
        • 2012-10-31
        • 2015-03-25
        • 2015-08-20
        相关资源
        最近更新 更多