来自storm-hive的HiveTopology问题答案

【问题标题】：Issue with HiveTopology from storm-hive来自storm-hive的HiveTopology问题
【发布时间】：2015-12-23 19:04:45
【问题描述】：

我正在尝试从我的本地机器上的storm-hive 项目运行HiveTopology，但连接到在Linux 服务器上运行的远程hive。我已按照使用 ORC 格式创建表的说明进行操作，并按照here

完成所有设置

我遇到了错误

com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: Unable to instantiate org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) ~[guava-18.0.jar:na]
    at com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[guava-18.0.jar:na]
    at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739) ~[guava-18.0.jar:na]
    at org.apache.hive.hcatalog.common.HiveClientCache.getOrCreate(HiveClientCache.java:227) ~[hive-hcatalog-core-1.2.1.jar:1.2.1]
    at org.apache.hive.hcatalog.common.HiveClientCache.get(HiveClientCache.java:202) ~[hive-hcatalog-core-1.2.1.jar:1.2.1]
    at org.apache.hive.hcatalog.common.HCatUtil.getHiveMetastoreClient(HCatUtil.java:558) ~[hive-hcatalog-core-1.2.1.jar:1.2.1]
    at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.getMetaStoreClient(HiveEndPoint.java:448) ~[hive-hcatalog-streaming-1.2.1.jar:1.2.1]
    at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:274) ~[hive-hcatalog-streaming-1.2.1.jar:1.2.1]
    at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:243) ~[hive-hcatalog-streaming-1.2.1.jar:1.2.1]
    at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnectionImpl(HiveEndPoint.java:180) ~[hive-hcatalog-streaming-1.2.1.jar:1.2.1]
    at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnection(HiveEndPoint.java:157) ~[hive-hcatalog-streaming-1.2.1.jar:1.2.1]
    at org.apache.storm.hive.common.HiveWriter$5.call(HiveWriter.java:229) ~[storm-hive-0.10.0.jar:0.10.0]
    at org.apache.storm.hive.common.HiveWriter$5.call(HiveWriter.java:226) ~[storm-hive-0.10.0.jar:0.10.0]
    at org.apache.storm.hive.common.HiveWriter$9.call(HiveWriter.java:332) ~[storm-hive-0.10.0.jar:0.10.0]
    at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.8.0_45]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:1.8.0_45]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:1.8.0_45]
    at java.lang.Thread.run(Unknown Source) [na:1.8.0_45]
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523) ~[hive-metastore-1.2.1.jar:1.2.1]
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:86) ~[hive-metastore-1.2.1.jar:1.2.1]
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132) ~[hive-metastore-1.2.1.jar:1.2.1]
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:118) ~[hive-metastore-1.2.1.jar:1.2.1]
    at org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:230) ~[hive-hcatalog-core-1.2.1.jar:1.2.1]
    at org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:227) ~[hive-hcatalog-core-1.2.1.jar:1.2.1]
    at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742) ~[guava-18.0.jar:na]
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) ~[guava-18.0.jar:na]
    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) ~[guava-18.0.jar:na]
    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) ~[guava-18.0.jar:na]
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) ~[guava-18.0.jar:na]
    ... 17 common frames omitted
Caused by: java.lang.reflect.InvocationTargetException: null
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_45]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) ~[na:1.8.0_45]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) ~[na:1.8.0_45]
    at java.lang.reflect.Constructor.newInstance(Unknown Source) ~[na:1.8.0_45]
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521) ~[hive-metastore-1.2.1.jar:1.2.1]
    ... 27 common frames omitted
Caused by: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.NullPointerException
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) ~[guava-18.0.jar:na]
    at com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[guava-18.0.jar:na]
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) ~[guava-18.0.jar:na]
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) ~[guava-18.0.jar:na]
    at org.apache.hadoop.security.Groups.getGroups(Groups.java:182) ~[hadoop-common-2.7.1.jar:na]
    at org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:1518) ~[hadoop-common-2.7.1.jar:na]
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:436) ~[hive-metastore-1.2.1.jar:1.2.1]
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:236) ~[hive-metastore-1.2.1.jar:1.2.1]
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:181) ~[hive-metastore-1.2.1.jar:1.2.1]
    at org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient.<init>(HiveClientCache.java:330) ~[hive-hcatalog-core-1.2.1.jar:1.2.1]
    ... 32 common frames omitted

请指出正确的方向来解决问题。我在用 Hadoop：2.7.1 Hive：1.2.1 带有事务属性。

【问题讨论】：

标签： hive apache-storm

【解决方案1】：

我在本地运行storm-hive bolt时遇到了同样的问题。根本原因如下所示：

Caused by: java.lang.NullPointerException: null
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012) ~[na:1.8.0_20]
at org.apache.hadoop.util.Shell.runCommand(Shell.java:483) ~[hadoop-common-2.7.1.2.3.0.0-2557.jar:na]
at org.apache.hadoop.util.Shell.run(Shell.java:456) ~[hadoop-common-2.7.1.2.3.0.0-2557.jar:na]
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722) ~[hadoop-common-2.7.1.2.3.0.0-2557.jar:na]
at org.apache.hadoop.util.Shell.execCommand(Shell.java:815) ~[hadoop-common-2.7.1.2.3.0.0-2557.jar:na]
at org.apache.hadoop.util.Shell.execCommand(Shell.java:798) ~[hadoop-common-2.7.1.2.3.0.0-2557.jar:na]
at org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:84) ~[hadoop-common-2.7.1.2.3.0.0-2557.jar:na]
at org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:52) ~[hadoop-common-2.7.1.2.3.0.0-2557.jar:na]
at org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:51) ~[hadoop-common-2.7.1.2.3.0.0-2557.jar:na]
at org.apache.hadoop.security.Groups$GroupCacheLoader.fetchGroupList(Groups.java:239) ~[hadoop-common-2.7.1.2.3.0.0-2557.jar:na]
at org.apache.hadoop.security.Groups$GroupCacheLoader.load(Groups.java:220) ~[hadoop-common-2.7.1.2.3.0.0-2557.jar:na]
at org.apache.hadoop.security.Groups$GroupCacheLoader.load(Groups.java:208) ~[hadoop-common-2.7.1.2.3.0.0-2557.jar:na]
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599) ~[guava-14.0.1.jar:na]
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2379) ~[guava-14.0.1.jar:na]
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342) ~[guava-14.0.1.jar:na]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2257) ~[guava-14.0.1.jar:na]

org.apache.hadoop.util.Shell 类中的问题。如果您在 Windows 上运行并且找不到 winutils.exe，方法 getGroupsForUserCommand 返回错误的命令。

public static String[] getGroupsForUserCommand(final String user) {
//'groups username' command return is non-consistent across different unixes
return (WINDOWS)? new String[] { WINUTILS, "groups", "-F", "\"" + user + "\""}
                : new String [] {"bash", "-c", "id -gn " + user
                                 + "&& id -Gn " + user};}

此代码将返回包含 4 个元素的字符串数组，第一个元素将是 null。

要解决此问题，请设置 HADOOP_HOME 环境变量，并且在 dir %HADOOP_HOME%/bin 中必须是 winutils.exe（下载它here）。

【讨论】：