apache风暴喷口不发出元组答案

【问题标题】：apache storm spout not emitting tuplesapache风暴喷口不发出元组
【发布时间】：2015-12-08 03:09:34
【问题描述】：

我已经使用三台机器和this 指南在 aws 集群上设置了一个测试环境。

我在本地模式下测试了我的代码，并使用 wirbelsturm 创建了一个本地 vagrant 集群，这两种方法都可以得到预期的结果。

当我现在将我的代码提交到网络服务器时，我的 spouts 和我所有的 bolts 都是静默的。我的 spout 从 csv 读取，我已将其复制到 nimbus 和我的主管。 Storm UI 将拓扑显示为活动状态并显示所有螺栓和我的喷口，但计数器不可见。主管没有使用过的工人。防火墙配置为让 nimbus 和 supervisor 接受来自 supervisor 和 nimbus 的端口 6700-6703。 zookeeper 会在这些端口上说话吗？

我似乎也无法在我的机器上找到我的输出日志。我在 nimbus 和 slave 的 /usr/local/storm/logs 中找到了 ui 和 nimbus 日志，但除此之外，我似乎没有收到错误，甚至没有收到 spouts/bolts 的日志。 vagrant 机器显示 worker-xxxx.log，但我的 aws 服务器没有。

那是因为我的代码因某些错误而崩溃还是因为我做错了配置？

更新：我使用storm-starter 示例验证了我的拓扑，这些似乎也不起作用。我使用 mvn 包构建了一个 uberjar。

更新2：

我包含了我的主管的日志，没有显示任何错误，但其中可能有一些东西......

 2015-12-08 13:42:55.168 b.s.u.Utils [INFO] Using defaults.yaml from resources
2015-12-08 13:42:55.297 b.s.u.Utils [INFO] Using storm.yaml from resources
2015-12-08 13:42:57.434 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2015-12-08 13:42:57.435 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:host.name=ip-172-31-26-239.us-west-2.compute.internal
2015-12-08 13:42:57.435 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.version=1.7.0_91
2015-12-08 13:42:57.435 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.vendor=Oracle Corporation
2015-12-08 13:42:57.435 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91.x86_64/jre
2015-12-08 13:42:57.435 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.class.path=/usr/local/apache-storm-0.10.0/lib/clojure-1.6.0.jar:/usr/local/apache-storm-0.10.0/lib/log4j-core-2.1.jar:/usr/local/apache-storm-0.10.0/lib/log4j-api-2.1.jar:/usr/local/apache-sto$
2015-12-08 13:42:57.435 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
2015-12-08 13:42:57.435 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.io.tmpdir=/tmp
2015-12-08 13:42:57.435 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.compiler=<NA>
2015-12-08 13:42:57.436 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:os.name=Linux
2015-12-08 13:42:57.436 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:os.arch=amd64
2015-12-08 13:42:57.436 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:os.version=2.6.32-504.8.1.el6.x86_64
2015-12-08 13:42:57.436 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:user.name=storm
2015-12-08 13:42:57.436 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:user.home=/app/home/storm
2015-12-08 13:42:57.436 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:user.dir=/
2015-12-08 13:42:57.459 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2015-12-08 13:42:57.459 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:host.name=ip-172-31-26-239.us-west-2.compute.internal
2015-12-08 13:42:57.459 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:java.version=1.7.0_91
2015-12-08 13:42:57.459 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:java.vendor=Oracle Corporation
2015-12-08 13:42:57.459 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91.x86_64/jre
2015-12-08 13:42:57.460 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:java.class.path=/usr/local/apache-storm-0.10.0/lib/clojure-1.6.0.jar:/usr/local/apache-storm-0.10.0/lib/log4j-core-2.1.jar:/usr/local/apache-storm-0.10.0/lib/log4j-api-2.1.jar:/usr/local/ap$
2015-12-08 13:42:57.460 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:java.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
2015-12-08 13:42:57.460 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:java.io.tmpdir=/tmp
2015-12-08 13:42:57.460 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:java.compiler=<NA>
2015-12-08 13:42:57.460 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:os.name=Linux
2015-12-08 13:42:57.460 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:os.arch=amd64
2015-12-08 13:42:57.460 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:os.version=2.6.32-504.8.1.el6.x86_64
2015-12-08 13:42:57.460 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:user.name=storm
2015-12-08 13:42:57.460 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:user.home=/app/home/storm
2015-12-08 13:42:57.460 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:user.dir=/
2015-12-08 13:42:57.774 b.s.u.Utils [INFO] Using defaults.yaml from resources
2015-12-08 13:42:57.803 b.s.u.Utils [INFO] Using storm.yaml from resources
2015-12-08 13:42:57.939 b.s.d.supervisor [INFO] Starting Supervisor with conf {"topology.builtin.metrics.bucket.size.secs" 60, "nimbus.childopts" "-Xmx1024m -Djava.net.preferIPv4Stack=true", "ui.filter.params" nil, "storm.cluster.mode" "distributed", "storm.messaging.net$
2015-12-08 13:42:57.963 b.s.u.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5]
2015-12-08 13:42:58.063 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting
2015-12-08 13:42:58.066 o.a.s.s.o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=zkserver1:2181 sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@502016b8
2015-12-08 13:42:58.081 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server zkServer1/xx.xx.xx.xx:2181. Will not attempt to authenticate using SASL (unknown error)
2015-12-08 13:42:58.089 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection established to zkServer1/xx.xx.xx.xx:2181, initiating session
2015-12-08 13:42:58.094 o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment complete on server zkServer1/xx.xx.xx.xx:2181, sessionid = 0x15182c7ba25000d, negotiated timeout = 20000
2015-12-08 13:42:58.096 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2015-12-08 13:42:58.097 b.s.zookeeper [INFO] Zookeeper state update: :connected:none
2015-12-08 13:42:59.109 o.a.s.s.o.a.z.ClientCnxn [INFO] EventThread shut down
2015-12-08 13:42:59.110 o.a.s.s.o.a.z.ZooKeeper [INFO] Session: 0x15182c7ba25000d closed
2015-12-08 13:42:59.111 b.s.u.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5]
2015-12-08 13:42:59.116 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting
2015-12-08 13:42:59.116 o.a.s.s.o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=zkserver1:2181/storm sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@5edfa0aa
2015-12-08 13:42:59.121 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server zkServer1/xx.xx.xx.xx:2181. Will not attempt to authenticate using SASL (unknown error)
2015-12-08 13:42:59.122 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection established to zkServer1/xx.xx.xx.xx:2181, initiating session
2015-12-08 13:42:59.124 o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment complete on server zkServer1/xx.xx.xx.xx:2181, sessionid = 0x15182c7ba25000e, negotiated timeout = 20000
2015-12-08 13:42:59.124 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2015-12-08 13:42:59.169 b.s.d.supervisor [INFO] Starting supervisor with id cc5e1723-cc06-4bc1-a1bf-192a1d7f5bf6 at host xxxxxxx.us-west-2.compute.internal
2015-12-08 13:43:06.059 b.s.d.supervisor [INFO] Downloading code for storm id production-topology-4-1449599549 from /app/storm/nimbus/stormdist/production-topology-4-1449599549
2015-12-08 13:43:06.075 b.s.u.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs [2000] the maxSleepTimeMs [60000] the maxRetries [5]

有什么想法吗？

更新2：

所以我确实找到了这个：

java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection timed out
        at backtype.storm.security.auth.TBackoffConnect.retryNext(TBackoffConnect.java:59) ~[storm-core-0.10.0.jar:0.10.0]
        at backtype.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:51) ~[storm-core-0.10.0.jar:0.10.0]
        at backtype.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:103) ~[storm-core-0.10.0.jar:0.10.0]
        at backtype.storm.security.auth.ThriftClient.<init>(ThriftClient.java:72) ~[storm-core-0.10.0.jar:0.10.0]
        at backtype.storm.utils.NimbusClient.<init>(NimbusClient.java:74) ~[storm-core-0.10.0.jar:0.10.0]
        at backtype.storm.utils.NimbusClient.getConfiguredClient(NimbusClient.java:37) ~[storm-core-0.10.0.jar:0.10.0]
        at backtype.storm.utils.Utils.downloadFromMaster(Utils.java:361) ~[storm-core-0.10.0.jar:0.10.0]
        at backtype.storm.daemon.supervisor$fn__7720.invoke(supervisor.clj:581) ~[storm-core-0.10.0.jar:0.10.0]
        at clojure.lang.MultiFn.invoke(MultiFn.java:241) ~[clojure-1.6.0.jar:?]
        at backtype.storm.daemon.supervisor$mk_synchronize_supervisor$this__7638.invoke(supervisor.clj:465) ~[storm-core-0.10.0.jar:0.10.0]
        at backtype.storm.event$event_manager$fn__7258.invoke(event.clj:40) [storm-core-0.10.0.jar:0.10.0]
        at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
        at java.lang.Thread.run(Thread.java:745) [?:1.7.0_91]
Caused by: org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection timed out
        at org.apache.thrift7.transport.TSocket.open(TSocket.java:187) ~[storm-core-0.10.0.jar:0.10.0]
        at org.apache.thrift7.transport.TFramedTransport.open(TFramedTransport.java:81) ~[storm-core-0.10.0.jar:0.10.0]
        at backtype.storm.security.auth.SimpleTransportPlugin.connect(SimpleTransportPlugin.java:103) ~[storm-core-0.10.0.jar:0.10.0]
        at backtype.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:48) ~[storm-core-0.10.0.jar:0.10.0]
        ... 11 more
Caused by: java.net.ConnectException: Connection timed out
        at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.7.0_91]
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) ~[?:1.7.0_91]
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) ~[?:1.7.0_91]
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) ~[?:1.7.0_91]
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.7.0_91]
        at java.net.Socket.connect(Socket.java:579) ~[?:1.7.0_91]
        at org.apache.thrift7.transport.TSocket.open(TSocket.java:182) ~[storm-core-0.10.0.jar:0.10.0]
        at org.apache.thrift7.transport.TFramedTransport.open(TFramedTransport.java:81) ~[storm-core-0.10.0.jar:0.10.0]
        at backtype.storm.security.auth.SimpleTransportPlugin.connect(SimpleTransportPlugin.java:103) ~[storm-core-0.10.0.jar:0.10.0]
        at backtype.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:48) ~[storm-core-0.10.0.jar:0.10.0]
        ... 11 more
2015-12-08 14:26:41.028 b.s.util [ERROR] Halting process: ("Error when processing an event")
java.lang.RuntimeException: ("Error when processing an event")
        at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:336) [storm-core-0.10.0.jar:0.10.0]
        at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.6.0.jar:?]
        at backtype.storm.event$event_manager$fn__7258.invoke(event.clj:48) [storm-core-0.10.0.jar:0.10.0]
        at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
        at java.lang.Thread.run(Thread.java:745) [?:1.7.0_91]

【问题讨论】：

您说“主管没有使用过的工人”。并且您看不到任何工作人员日志。这听起来好像工作 JVM 无法由主管启动。主管日志中是否有任何错误？
supervisor.log 似乎没有说明为什么应该有任何故障的任何原因。会不会是由于错误的端口设置，主管似乎没有得到启动工作人员的“命令”？
我需要打开 DRPC 端口吗？
如果没有错误信息，我不知道。对不起。 ://
还是添加了我的日志，也许你能找到一些东西......

标签： amazon-web-services apache-storm

【解决方案1】：

我遵循与您相同的指南并遇到了同样的问题。是什么为我解决了这个问题：

用同样的方法编辑你的三台机器（zookeeper、nimbus和slave1）的3个/etc/hosts文件

首先删除以::1开头的IPv6行，Apache Storm不支持。
在包含本地别名的文件的第一行中，将本地机器的公共主机名（集群的其他节点知道的主机名）放在127.0.0.1 之后。我想这是别名风暴会考虑的。
最后，如指南中所述，列出所有其他机器和风暴已知的主机名

最后我的/etc/hosts 看起来像这样（用于灵气）

127.0.0.1 vm-matthias-02 localhost.localdomain localhost
192.168.200.48  vm-matthias-01
192.168.200.49  vm-matthias-02
192.168.200.50  vm-matthias-03

编辑配置文件时请注意使用与机器相同的名称。

【讨论】：