【发布时间】:2014-04-27 23:21:08
【问题描述】:
我已经使用多个来源设置了一个 Storm 开发集群,例如 Storm Real-time Processing Cookbook、P. Taylor Goetz 在 OSX 上设置 Storm,当然还有来自 Nathan Marz 的所有内容.目前,我使用 Puppet 和 Vagrant 在一台机器上设置了一个集群,该机器具有用于 Nimbus、Zookeeper 和两个 Supervisor 的 VM。本地机器本身就是主人。我似乎可以从storm-starter https://github.com/nathanmarz/storm-starter 提交示例拓扑并让它们运行,但我在通过Storm 运行各种工具或验证集群是否实际工作时遇到了一些问题!
问题: 每当我从它的 bin 中运行任何 Storm 工具时,例如storm ui 或storm logviewer,控制台都会立即吐出以下内容并且通常会挂起:
Running: java -server -Dstorm.options= -Dstorm.home=/Users/jhogan/Development/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /Users/jhogan/Development/storm/lib/asm-4.0.jar:/Users/jhogan/Development/storm/lib/carbonite-1.3.2.jar:/Users/jhogan/Development/storm/lib/clj-stacktrace-0.2.4.jar:/Users/jhogan/Development/storm/lib/clj-time-0.4.1.jar:/Users/jhogan/Development/storm/lib/clojure-1.4.0.jar:/Users/jhogan/Development/storm/lib/clout-1.0.1.jar:/Users/jhogan/Development/storm/lib/commons-codec-1.4.jar:/Users/jhogan/Development/storm/lib/commons-exec-1.1.jar:/Users/jhogan/Development/storm/lib/commons-fileupload-1.2.1.jar:/Users/jhogan/Development/storm/lib/commons-io-1.4.jar:/Users/jhogan/Development/storm/lib/commons-lang-2.5.jar:/Users/jhogan/Development/storm/lib/commons-logging-1.1.1.jar:/Users/jhogan/Development/storm/lib/compojure-1.1.3.jar:/Users/jhogan/Development/storm/lib/core.incubator-0.1.0.jar:/Users/jhogan/Development/storm/lib/curator-client-1.0.1.jar:/Users/jhogan/Development/storm/lib/curator-framework-1.0.1.jar:/Users/jhogan/Development/storm/lib/disruptor-2.10.1.jar:/Users/jhogan/Development/storm/lib/guava-13.0.jar:/Users/jhogan/Development/storm/lib/hiccup-0.3.6.jar:/Users/jhogan/Development/storm/lib/httpclient-4.1.1.jar:/Users/jhogan/Development/storm/lib/httpcore-4.1.jar:/Users/jhogan/Development/storm/lib/jetty-6.1.26.jar:/Users/jhogan/Development/storm/lib/jetty-util-6.1.26.jar:/Users/jhogan/Development/storm/lib/jgrapht-core-0.9.0.jar:/Users/jhogan/Development/storm/lib/jline-2.11.jar:/Users/jhogan/Development/storm/lib/joda-time-2.0.jar:/Users/jhogan/Development/storm/lib/json-simple-1.1.jar:/Users/jhogan/Development/storm/lib/junit-3.8.1.jar:/Users/jhogan/Development/storm/lib/kryo-2.17.jar:/Users/jhogan/Development/storm/lib/log4j-over-slf4j-1.6.6.jar:/Users/jhogan/Development/storm/lib/logback-classic-1.0.6.jar:/Users/jhogan/Development/storm/lib/logback-core-1.0.6.jar:/Users/jhogan/Development/storm/lib/math.numeric-tower-0.0.1.jar:/Users/jhogan/Development/storm/lib/meat-locker-0.3.1.jar:/Users/jhogan/Development/storm/lib/minlog-1.2.jar:/Users/jhogan/Development/storm/lib/netty-3.6.3.Final.jar:/Users/jhogan/Development/storm/lib/objenesis-1.2.jar:/Users/jhogan/Development/storm/lib/reflectasm-1.07-shaded.jar:/Users/jhogan/Development/storm/lib/ring-core-1.1.5.jar:/Users/jhogan/Development/storm/lib/ring-devel-0.3.11.jar:/Users/jhogan/Development/storm/lib/ring-jetty-adapter-0.3.11.jar:/Users/jhogan/Development/storm/lib/ring-servlet-0.3.11.jar:/Users/jhogan/Development/storm/lib/servlet-api-2.5-20081211.jar:/Users/jhogan/Development/storm/lib/servlet-api-2.5.jar:/Users/jhogan/Development/storm/lib/slf4j-api-1.6.5.jar:/Users/jhogan/Development/storm/lib/snakeyaml-1.11.jar:/Users/jhogan/Development/storm/lib/storm-core-0.9.1-incubating.jar:/Users/jhogan/Development/storm/lib/tools.cli-0.2.2.jar:/Users/jhogan/Development/storm/lib/tools.logging-0.2.3.jar:/Users/jhogan/Development/storm/lib/tools.macro-0.1.0.jar:/Users/jhogan/Development/storm/lib/zookeeper-3.3.3.jar:/Users/jhogan/Development/storm:/Users/jhogan/Development/storm/conf -Xmx768m -Dlogfile.name=ui.log -Dlogback.configurationFile=/Users/jhogan/Development/storm/logback/cluster.xml backtype.storm.ui.core
在这种特殊情况下,我想启动storm ui。 ui登录storm(从master机器):
2014-03-20 12:26:24 o.m.log [INFO] Logging to Logger[org.mortbay.log] via org.mortbay.log.Slf4jLog
2014-03-20 12:26:24 o.m.log [INFO] jetty-6.1.26
2014-03-20 12:26:24 o.m.log [INFO] Started SocketConnector@0.0.0.0:8080
.. 没有什么意义,因为我假设我需要连接到从 Nimbus 机器生成的 UI:
流浪文件:
boxes = [
{ :name => :nimbus, :ip => '10.200.1.100', :cpus =>2, :memory => 512, :instance => 'm1.small' },
{ :name => :supervisor1, :ip => '10.200.1.101', :cpus =>4, :memory => 1024, :instance => 'm1.medium' },
{ :name => :supervisor2, :ip => '10.200.1.102', :cpus =>4, :memory => 1024, :instance => 'm1.medium' },
{ :name => :zookeeper1, :ip => '10.200.1.201', :cpus =>1, :memory => 1024, :instance => 'm1.small' },
]
在任何一种情况下,我都会收到页面的 404 错误。
运行“风暴名单”:
`1941 [main] INFO backtype.storm.thrift - Connecting to Nimbus at 10.200.1.100:6627
Exception in thread "main" org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused
at org.apache.thrift7.transport.TSocket.open(TSocket.java:183)
at org.apache.thrift7.transport.TFramedTransport.open(TFramedTransport.java:81)
at backtype.storm.thrift$nimbus_client_and_conn.invoke(thrift.clj:71)
at backtype.storm.command.list$_main.invoke(list.clj:22)
at clojure.lang.AFn.applyToHelper(AFn.java:159)
at clojure.lang.AFn.applyTo(AFn.java:151)
at backtype.storm.command.list.main(Unknown Source)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.thrift7.transport.TSocket.open(TSocket.java:178)
... 6 more`
每台机器似乎都已正确设置和运行,我可以使用 Vagrant ssh ssh 进入其中的任何一台。我可以提交拓扑:
`storm jar storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar storm.starter.WordCountTopology`
并让它们看似运行,但我不确定如何检查系统是否实际运行正确。在哪里查看拓扑进度的日志?为什么我无法连接到文档中指定的 10.200.1.100:8080 上的storm ui?
我对这个系统很陌生,所以如果我没有发布正确的数据,我深表歉意。让我知道我需要发布 Puppet、Vagrant 或 Storm 的哪些部分来协助解决此问题。
谢谢!
【问题讨论】:
标签: vagrant puppet apache-storm