在测试环境中使用 Hadoop 和 Zookeeper 请求 Apache Accumulo 错误帮助答案

【问题标题】：Request Apache Accumulo error help using Hadoop and Zookeeper in a test environment在测试环境中使用 Hadoop 和 Zookeeper 请求 Apache Accumulo 错误帮助
【发布时间】：2012-03-07 23:10:13
【问题描述】：

我正在使用 apache accumulo 版本 1.4.0、hadoop 版本 0.20.2 和 zookeeper 版本 3.3.3 设置测试环境。请参阅下面的问题。

Hadoop 和 Zookeeper 可以很好地协同工作，但是当我使用 apache 孵化器上的程序开始 accumulo 时，我得到以下 zookeeper 信息流和警告：

2011-12-08 20:13:56,601 - INFO  [main:QuorumPeerConfig@90] - Reading configuration from: /home/hadoop/zookeeper-3.3.3/bin/../conf/zoo.cfg
2011-12-08 20:13:56,603 - WARN  [main:QuorumPeerMain@105] - Either no config or no quorum defined in config, running  in standalone mode
2011-12-08 20:13:56,616 - INFO  [main:QuorumPeerConfig@90] - Reading configuration from: /home/hadoop/zookeeper-3.3.3/bin/../conf/zoo.cfg
2011-12-08 20:13:56,617 - INFO  [main:ZooKeeperServerMain@94] - Starting server
2011-12-08 20:13:56,626 - INFO  [main:Environment@97] - Server environment:zookeeper.version=3.3.3-1073969, built on 02/23/2011 22:27 GMT
2011-12-08 20:13:56,627 - INFO  [main:Environment@97] - Server environment:host.name.paz
2011-12-08 20:13:56,627 - INFO  [main:Environment@97] - Server environment:java.version=1.6.0_26
2011-12-08 20:13:56,628 - INFO  [main:Environment@97] - Server environment:java.vendor=Sun Microsystems Inc.
2011-12-08 20:13:56,629 - INFO  [main:Environment@97] - Server environment:java.home=/usr/lib/jvm/java-6-sun-1.6.0.26/jre
2011-12-08 20:13:56,629 - INFO  [main:Environment@97] - Server environment:java.class.path=/home/hadoop/zookeeper-3.3.3/bin/../build/classes:/home/hadoop/zookeeper-3.3.
3/bin/../build/lib/*.jar:/home/hadoop/zookeeper-3.3.3/bin/../zookeeper-3.3.3.jar:/home/hadoop/zookeeper-3.3.3/bin/../lib/log4j-1.2.15.jar:/home/hadoop/zookeeper-3.3.3/b
in/../lib/jline-0.9.94.jar:/home/hadoop/zookeeper-3.3.3/bin/../src/java/lib/*.jar:/home/hadoop/zookeeper-3.3.3/bin/../conf:
2011-12-08 20:13:56,630 - INFO  [main:Environment@97] - Server environment:java.library.path=/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/i386/server:/usr/lib/jvm/java-6-su
n-1.6.0.26/jre/lib/i386:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/../lib/i386:/usr/java/packages/lib/i386:/lib:/usr/lib
2011-12-08 20:13:56,630 - INFO  [main:Environment@97] - Server environment:java.io.tmpdir=/tmp
2011-12-08 20:13:56,631 - INFO  [main:Environment@97] - Server environment:java.compiler=<NA>
2011-12-08 20:13:56,631 - INFO  [main:Environment@97] - Server environment:os.name=Linux
2011-12-08 20:13:56,632 - INFO  [main:Environment@97] - Server environment:os.arch=i386
2011-12-08 20:13:56,633 - INFO  [main:Environment@97] - Server environment:os.version=3.0.0-13-generic
2011-12-08 20:13:56,633 - INFO  [main:Environment@97] - Server environment:user.name=hadoop
2011-12-08 20:13:56,634 - INFO  [main:Environment@97] - Server environment:user.home=/home/hadoop
2011-12-08 20:13:56,634 - INFO  [main:Environment@97] - Server environment:user.dir=/home/hadoop
2011-12-08 20:13:56,641 - INFO  [main:ZooKeeperServer@663] - tickTime set to 2000
2011-12-08 20:13:56,641 - INFO  [main:ZooKeeperServer@672] - minSessionTimeout set to -1
2011-12-08 20:13:56,642 - INFO  [main:ZooKeeperServer@681] - maxSessionTimeout set to -1
2011-12-08 20:13:56,661 - INFO  [main:NIOServerCnxn$Factory@143] - binding to port 0.0.0.0/0.0.0.0:2181
2011-12-08 20:13:56,691 - INFO  [main:FileSnap@82] - Reading snapshot /home/hadoop/zoo/dataDir/version-2/snapshot.0
2011-12-08 20:13:56,708 - INFO  [main:FileTxnSnapLog@208] - Snapshotting: 4e
2011-12-08 20:14:52,147 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /0:0:0:0:0:0:0:1:40694
2011-12-08 20:14:52,153 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@777] - Client attempting to establish new session at /0:0:0:0:0:0:0:1:40694
2011-12-08 20:14:52,154 - INFO  [SyncThread:0:FileTxnLog@197] - Creating new log file: log.4f
2011-12-08 20:14:52,410 - INFO  [SyncThread:0:NIOServerCnxn@1580] - Established session 0x13420623ee70000 with negotiated timeout 30000 for client /0:0:0:0:0:0:0:1:4069
4
2011-12-08 20:14:52,959 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:38446
2011-12-08 20:14:52,962 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@777] - Client attempting to establish new session at /127.0.0.1:38446
2011-12-08 20:14:53,007 - INFO  [SyncThread:0:NIOServerCnxn@1580] - Established session 0x13420623ee70001 with negotiated timeout 30000 for client /127.0.0.1:38446
2011-12-08 20:14:59,932 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@634] - EndOfStreamException: Unable to read additional data from client session
id 0x13420623ee70000, likely client has closed socket

当我启动 accumulo shell 时出现以下错误：

18 12:44:38,746 [impl.ServerClient] WARN : Failed to find an available server in the list of servers: []
18 12:44:38,846 [impl.ServerClient] WARN : Failed to find an available server in the list of servers: []
18 12:44:38,947 [impl.ServerClient] WARN : Failed to find an available server in the list of servers: []
18 12:44:39,048 [impl.ServerClient] WARN : Failed to find an available server in the list of servers: []
18 12:44:39,148 [impl.ServerClient] WARN : Failed to find an available server in the list of servers: []
18 12:44:39,249 [impl.ServerClient] WARN : Failed to find an available server in the list of servers: []
18 12:44:39,350 [impl.ServerClient] WARN : Failed to find an available server in the list of servers: []
18 12:44:39,450 [impl.ServerClient] WARN : Failed to find an available server in the list of servers: []

【问题讨论】：

标签： hadoop apache-zookeeper

【解决方案1】：

已更正 tserver 内存设置，使其不超过 JVM 允许的 waht。 Tserver 不会崩溃并且错误已解决。答案来自 accumulo-incubator 用户列表，并在底部重新发布。基本上，当进入并修改内存设置以在笔记本电脑上以伪分布式模式运行时，我对有关平板电脑服务器内存使用的 accumulo-site.xml 和 accumulo-env.sh 文件进行了不正确的修改。在 /home/hadoop/accumulo/logs/tserver*.log 文件中找到了错误的线索：

20 18:20:00,951 [tabletserver.NativeMap] ERROR: Failed to load native
map library
/home/hadoop/accumulo/lib/native/map/libNativeMap-Linux-i386-32.so
java.lang.UnsatisfiedLinkError: Can't load library:
/home/hadoop/accumulo/lib/native/map/libNativeMap-Linux-i386-32.so
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1706)
at java.lang.Runtime.load0(Runtime.java:770)
at java.lang.System.load(System.java:1003)
at
org.apache.accumulo.server.tabletserver.NativeMap.loadNativeLib(NativeMap.java:144)
at
org.apache.accumulo.server.tabletserver.NativeMap.<clinit>(NativeMap.java:156)
at
org.apache.accumulo.server.tabletserver.TabletServerResourceManager.<init>(TabletServerResourceManager.java:123)
at
org.apache.accumulo.server.tabletserver.TabletServer.config(TabletServer.java:2959)
at
org.apache.accumulo.server.tabletserver.TabletServer.main(TabletServer.java:3085)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.accumulo.start.Main$1.run(Main.java:89)
at java.lang.Thread.run(Thread.java:662)
20 18:20:00,999 [tabletserver.TabletServer] ERROR: Uncaught exception in
TabletServer.main, exiting
java.lang.IllegalArgumentException: Maximum tablet server map memory
134,217,728 and block cache sizes 186,646,528 is too large for this JVM
configuration 132,579,328
at
org.apache.accumulo.server.tabletserver.TabletServerResourceManager.<init>(TabletServerResourceManager.java:134)
at
org.apache.accumulo.server.tabletserver.TabletServer.config(TabletServer.java:2959)
at
org.apache.accumulo.server.tabletserver.TabletServer.main(TabletServer.java:3085)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.accumulo.start.Main$1.run(Main.java:89)
at java.lang.Thread.run(Thread.java:662)

具体帮助文本： “本机库未加载，这正在内存中移动映射到 java 工作区。再加上您的块缓存大小，以及您的内存使用规范高于 JVM 将被允许分配。平板服务器抱怨并退出。你应该看到这些 accumulo 监控网页上的投诉。

您可能会发现重建原生地图库的好处，这将将内存映射的分配移到 JVM 之外。这是不需要。

任何专用于缓存的内存大小必须小于 JVM，它必须为 RPC 调用提供大量工作空间并随着时间的推移进行垃圾收集。”

【讨论】：

【解决方案2】：

如果您运行 Zookeeper 客户端 (/home/hadoop/zookeeper-3.3.3/bin/zkCli.sh) 并执行 ls of

/accumulo/<instance uuid>/tservers

我假设您不会看到列出的任何服务器。如果 Accumulo 已正确初始化，您应该会看到列出的一台或多台平板电脑服务器。您确定按照说明在 accumulo-site.xml 中设置 Zookeeper 服务器后运行了Accumulo init script？

【讨论】：

感谢您的回复。你是对的 - 没有列出 tservers。我已经在几个不同的 linux 机器上构建了几次：Ubuntu 和 Centos，并仔细按照说明进行操作。初始化后在HDFS中创建accumulo目录，可从localhost:50095获取accumulo网页，但tablet server计数始终保持为零。任何建议。
我建议发布您的问题并将您的 accumulo-site.xml 和日志输出包含到 user mailing list。开发人员非常乐于助人且反应迅速。
谢谢贾里德。我会试试 accumulo 列表...我认为这是 zookeeper 配置错误，但可能是我在 accumulo 设置中没有做对的事情。

【解决方案3】：

确保在 accumulo-site.xml 文件中将“instance.zookeeper.host”设置为 Zookeeper 节点的位置。

此外，检查您的 tserver 和记录器的日志。如果您有其他配置问题，它们将无法上线，这将导致 master 报告难以找到 tserver。

【讨论】：

【解决方案4】：

进入 $ACCUMULO_HOME/conf 。结束文件 masters、slaves 和 tracers 以包含一行读取“localhost”（我假设您正在执行单个节点）

【讨论】：