【问题标题】:Loading file to Hbase using PIG使用 PIG 将文件加载到 Hbase
【发布时间】:2012-03-13 05:36:06
【问题描述】:

文件内容:

one,1
two,2
three,3

文件位置:hdfs:/hbasetest.txt

Hbase 中的表:

create 'mydata', 'mycf'

猪脚本:

A = LOAD '/hbasetest.txt' USING PigStorage(',') as (strdata:chararray, intdata:long);
STORE A INTO 'hbase://mydata'
        USING org.apache.pig.backend.hadoop.hbase.HBaseStorage(
              'mycf:intdata');

我收到以下错误:

在控制台

2012-03-13 16:26:22,170 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN
2012-03-13 16:26:22,170 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - pig.usenewlogicalplan is set to true. New logical plan will be used.
2012-03-13 16:26:22,204 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2998: Unhandled internal error. org/apache/zookeeper/KeeperException

在日志文件中:

Pig Stack Trace
---------------
ERROR 2998: Unhandled internal error. org/apache/zookeeper/KeeperException

java.lang.NoClassDefFoundError: org/apache/zookeeper/KeeperException
    at org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:198)
    at org.apache.pig.backend.hadoop.hbase.HBaseStorage.getOutputFormat(HBaseStorage.java:389)
    at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:87)
    at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:76)
    at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64)
    at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
    at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
    at org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53)
    at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
    at org.apache.pig.newplan.logical.rules.InputOutputFileValidator.validate(InputOutputFileValidator.java:52)
    at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:292)
    at org.apache.pig.PigServer.compilePp(PigServer.java:1365)
    at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1207)
    at org.apache.pig.PigServer.execute(PigServer.java:1201)
    at org.apache.pig.PigServer.access$100(PigServer.java:129)
    at org.apache.pig.PigServer$Graph.execute(PigServer.java:1528)
    at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1575)
    at org.apache.pig.PigServer.registerQuery(PigServer.java:534)
    at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:871)
    at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:388)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:168)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:144)
    at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:76)
    at org.apache.pig.Main.run(Main.java:455)
    at org.apache.pig.Main.main(Main.java:107)
Caused by: java.lang.ClassNotFoundException: org.apache.zookeeper.KeeperException
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    ... 25 more
============================================

如果我使用 DUMP 命令,它会在 grunt 上显示文件内容。 如何解决这个问题??

【问题讨论】:

  • 如果您先为之前的一些问题标记答案,您可能会获得更多帮助
  • 不熟悉 hdfs/pig,但这看起来像是一个类路径问题。你的类路径中有 zookeeper jar 吗?
  • 是的,我在类路径中添加了所有必需的 jars

标签: hadoop hbase hdfs apache-pig


【解决方案1】:

hbase-*.jarzookeeper-3.3.2.jar 添加到PIG_HOMEPIG_HOME/lib.. 并将PIG_CLASSPATH=/HADOOP_HOME/confHADOOP_CONF_DIR=/HADOOP_HOME/confHADOOP_HOME 设置到您的hadoop 安装目录。

【讨论】:

    【解决方案2】:

    在 .bashrc 中添加以下行

    export PIG_CLASSPATH=$PIG_HOME/pig-x.x.x-withouthadoop.jar:$HBASE_HOME/hbase-x.x.x.jar:$HBASE_HOME/lib/*:$HADOOP_HOME/lib/*:$PIG_CLASSPATH
    

    【讨论】:

      猜你喜欢
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 2011-12-27
      • 2011-10-25
      • 1970-01-01
      相关资源
      最近更新 更多