一脸懵逼学习Storm的搭建--（一个开源的分布式实时计算系统）

1:集群部署的基本流程(基本套路)： 集群部署的流程：下载安装包、解压安装包、修改配置文件、分发安装包、启动集群;

#指定storm使用的zk集群 storm.zookeeper.servers: - "zk01" - "zk02" - "zk03" #指定storm集群中的nimbus节点所在的服务器 nimbus.host: "storm01" #指定nimbus启动JVM最大可用内存大小 nimbus.childopts: "-Xmx1024m" #指定supervisor启动JVM最大可用内存大小 supervisor.childopts: "-Xmx1024m" #指定supervisor节点上，每个worker启动JVM最大可用内存大小 worker.childopts: "-Xmx768m" #指定ui启动JVM最大可用内存大小，ui服务一般与nimbus同在一个节点上。 ui.childopts: "-Xmx768m" #指定supervisor节点上，启动worker时对应的端口号，每个端口对应槽，每个槽位对应一个worker supervisor.slots.ports: - 6700 - 6701 - 6702 - 6703

#所使用的zookeeper集群主机 storm.zookeeper.servers: - "master" - "slaver1" - "slaver2" #nimbus所在的主机名 nimbus.host: "master"

supervisor.slots.ports
-6701
-6702
-6703
-6704
-6705

启动storm 在nimbus主机上，在nimbus.host所属的机器上启动 nimbus服务： nohup ./storm nimbus 1>/dev/null 2>&1 & 或者使用命令：nohup ./storm nimbus &
在nimbus.host所属的机器上启动ui服务： nohup ./storm ui 1>/dev/null 2>&1 & 或者使用命令：nohup ./storm ui & 在supervisor主机上，在其它个点击上启动supervisor服务： nohup ./storm supervisor 1>/dev/null 2>&1 & 或者使用命令：nohup ./storm supervisor &

1：有许多简单且有用的命令可以用来管理拓扑，它们可以提交、杀死、禁用、再平衡拓扑。 提交任务命令格式：storm jar 【jar路径】 【拓扑包名.拓扑类名】 【拓扑名称】 2：bin/storm jar examples/storm-starter/storm-starter-topologies-0.9.6.jar storm.starter.WordCountTopology wordcount 杀死任务命令格式：storm kill 【拓扑名称】 -w 10（执行kill命令时可以通过-w [等待秒数]指定拓扑停用以后的等待时间） 3：storm kill topology-name -w 10 停用任务命令格式：storm deactivte 【拓扑名称】 4：storm deactivte topology-name 5：我们能够挂起或停用运行中的拓扑。当停用拓扑时，所有已分发的元组都会得到处理，但是spouts的nextTuple方法不会被调用。销毁一个拓扑，可以使用kill命令。它会以一种安全的方式销毁一个拓扑，首先停用拓扑，在等待拓扑消息的时间段内允许拓扑完成当前的数据流。 启用任务命令格式：storm activate【拓扑名称】 storm activate topology-name 重新部署任务命令格式：storm rebalance 【拓扑名称】 storm rebalance topology-name 再平衡使你重分配集群任务。这是个很强大的命令。比如，你向一个运行中的集群增加了节点。再平衡命令将会停用拓扑，然后在相应超时时间之后重分配工人，并重启拓扑。

 1 [root@master storm]# bin/storm jar examples/storm-starter/storm-starter-topologies-0.9.2-incubating.jar storm.starter.WordCountTopology wordcount  2 Running: /home/hadoop/soft/jdk1.7.0_65/bin/java -client -Dstorm.options= -Dstorm.home=/home/hadoop/soft/apache-storm-0.9.2-incubating -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /home/hadoop/soft/apache-storm-0.9.2-incubating/lib/commons-codec-1.6.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/hiccup-0.3.6.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/curator-client-2.4.0.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/clout-1.0.1.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/json-simple-1.1.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/reflectasm-1.07-shaded.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/httpclient-4.3.3.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/jgrapht-core-0.9.0.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/jline-2.11.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/ring-servlet-0.3.11.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/asm-4.0.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/clojure-1.5.1.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/joda-time-2.0.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/minlog-1.2.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/logback-classic-1.0.6.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/kryo-2.21.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/netty-3.6.3.Final.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/log4j-over-slf4j-1.6.6.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/commons-lang-2.5.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/tools.logging-0.2.3.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/commons-logging-1.1.3.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/compojure-1.1.3.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/logback-core-1.0.6.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/math.numeric-tower-0.0.1.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/ring-devel-0.3.11.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/servlet-api-2.5.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/commons-fileupload-1.2.1.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/guava-13.0.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/snakeyaml-1.11.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/jetty-6.1.26.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/disruptor-2.10.1.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/clj-time-0.4.1.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/jetty-util-6.1.26.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/httpcore-4.3.2.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/core.incubator-0.1.0.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/carbonite-1.4.0.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/commons-io-2.4.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/clj-stacktrace-0.2.4.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/slf4j-api-1.6.5.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/curator-framework-2.4.0.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/netty-3.2.2.Final.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/ring-core-1.1.5.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/chill-java-0.3.5.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/commons-exec-1.1.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/tools.macro-0.1.0.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/storm-core-0.9.2-incubating.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/ring-jetty-adapter-0.3.11.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/zookeeper-3.4.5.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/servlet-api-2.5-20081211.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/objenesis-1.2.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/lib/tools.cli-0.2.4.jar:examples/storm-starter/storm-starter-topologies-0.9.2-incubating.jar:/home/hadoop/soft/apache-storm-0.9.2-incubating/conf:/home/hadoop/soft/apache-storm-0.9.2-incubating/bin -Dstorm.jar=examples/storm-starter/storm-starter-topologies-0.9.2-incubating.jar storm.starter.WordCountTopology wordcount  3 712 [main] INFO backtype.storm.StormSubmitter - Jar not uploaded to master yet. Submitting jar...  4 772 [main] INFO backtype.storm.StormSubmitter - Uploading topology jar examples/storm-starter/storm-starter-topologies-0.9.2-incubating.jar to assigned location: storm-local/nimbus/inbox/stormjar-76fccf41-491e-4d61-8a98-4092c8630161.jar  5 Start uploading file 'examples/storm-starter/storm-starter-topologies-0.9.2-incubating.jar' to 'storm-local/nimbus/inbox/stormjar-76fccf41-491e-4d61-8a98-4092c8630161.jar' (2927299 bytes)  6 [==================================================] 2927299 / 2927299  7 File 'examples/storm-starter/storm-starter-topologies-0.9.2-incubating.jar' uploaded to 'storm-local/nimbus/inbox/stormjar-76fccf41-491e-4d61-8a98-4092c8630161.jar' (2927299 bytes)  8 941 [main] INFO backtype.storm.StormSubmitter - Successfully uploaded topology jar to assigned location: storm-local/nimbus/inbox/stormjar-76fccf41-491e-4d61-8a98-4092c8630161.jar  9 941 [main] INFO backtype.storm.StormSubmitter - Submitting topology wordcount in distributed mode with conf {"topology.workers":3,"topology.debug":true} 10 1431 [main] INFO backtype.storm.StormSubmitter - Finished submitting topology: wordcount 11 [root@master storm]#

 1 [root@slaver1 bin]# ./storm supervisor  2 Exception in thread "main" java.lang.ExceptionInInitializerError  3  at java.lang.Class.forName0(Native Method)  4 at java.lang.Class.forName(Class.java:190)  5 at backtype.storm.config$loading__4910__auto__.invoke(config.clj:17)  6  at backtype.storm.config__init.load(Unknown Source)  7 at backtype.storm.config__init.<clinit>(Unknown Source)  8  at java.lang.Class.forName0(Native Method)  9 at java.lang.Class.forName(Class.java:270)  10 at clojure.lang.RT.loadClassForName(RT.java:2098)  11 at clojure.lang.RT.load(RT.java:430)  12 at clojure.lang.RT.load(RT.java:411)  13 at clojure.core$load$fn__5018.invoke(core.clj:5530)  14 at clojure.core$load.doInvoke(core.clj:5529)  15 at clojure.lang.RestFn.invoke(RestFn.java:408)  16 at clojure.core$load_one.invoke(core.clj:5336)  17 at clojure.core$load_lib$fn__4967.invoke(core.clj:5375)  18 at clojure.core$load_lib.doInvoke(core.clj:5374)  19 at clojure.lang.RestFn.applyTo(RestFn.java:142)  20 at clojure.core$apply.invoke(core.clj:619)  21 at clojure.core$load_libs.doInvoke(core.clj:5417)  22 at clojure.lang.RestFn.applyTo(RestFn.java:137)  23 at clojure.core$apply.invoke(core.clj:621)  24 at clojure.core$use.doInvoke(core.clj:5507)  25 at clojure.lang.RestFn.invoke(RestFn.java:408)  26 at backtype.storm.command.config_value$loading__4910__auto__.invoke(config_value.clj:16)  27  at backtype.storm.command.config_value__init.load(Unknown Source)  28 at backtype.storm.command.config_value__init.<clinit>(Unknown Source)  29  at java.lang.Class.forName0(Native Method)  30 at java.lang.Class.forName(Class.java:270)  31 at clojure.lang.RT.loadClassForName(RT.java:2098)  32 at clojure.lang.RT.load(RT.java:430)  33 at clojure.lang.RT.load(RT.java:411)  34 at clojure.core$load$fn__5018.invoke(core.clj:5530)  35 at clojure.core$load.doInvoke(core.clj:5529)  36 at clojure.lang.RestFn.invoke(RestFn.java:408)  37 at clojure.lang.Var.invoke(Var.java:415)  38 at backtype.storm.command.config_value.<clinit>(Unknown Source)  39 Caused by: while parsing a block mapping  40 in 'reader', line 18, column 1:  41  storm.zookeeper.servers:  42 ^  43 expected <block end>, but found BlockMappingStart  44 in 'reader', line 23, column 2:  45 nimbus.host: "master"  46 ^  47  48 at org.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingKey.produce(ParserImpl.java:570)  49 at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:158)  50 at org.yaml.snakeyaml.parser.ParserImpl.checkEvent(ParserImpl.java:143)  51 at org.yaml.snakeyaml.composer.Composer.composeMappingNode(Composer.java:230)  52 at org.yaml.snakeyaml.composer.Composer.composeNode(Composer.java:159)  53 at org.yaml.snakeyaml.composer.Composer.composeDocument(Composer.java:122)  54 at org.yaml.snakeyaml.composer.Composer.getSingleNode(Composer.java:105)  55 at org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:120)  56 at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:481)  57 at org.yaml.snakeyaml.Yaml.load(Yaml.java:424)  58 at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:141)  59 at backtype.storm.utils.Utils.readStormConfig(Utils.java:188)  60 at backtype.storm.utils.Utils.<clinit>(Utils.java:71)  61 ... 36 more  62 Exception in thread "main" java.lang.ExceptionInInitializerError  63  at java.lang.Class.forName0(Native Method)  64 at java.lang.Class.forName(Class.java:190)  65 at backtype.storm.config$loading__4910__auto__.invoke(config.clj:17)  66  at backtype.storm.config__init.load(Unknown Source)  67 at backtype.storm.config__init.<clinit>(Unknown Source)  68  at java.lang.Class.forName0(Native Method)  69 at java.lang.Class.forName(Class.java:270)  70 at clojure.lang.RT.loadClassForName(RT.java:2098)  71 at clojure.lang.RT.load(RT.java:430)  72 at clojure.lang.RT.load(RT.java:411)  73 at clojure.core$load$fn__5018.invoke(core.clj:5530)  74 at clojure.core$load.doInvoke(core.clj:5529)  75 at clojure.lang.RestFn.invoke(RestFn.java:408)  76 at clojure.core$load_one.invoke(core.clj:5336)  77 at clojure.core$load_lib$fn__4967.invoke(core.clj:5375)  78 at clojure.core$load_lib.doInvoke(core.clj:5374)  79 at clojure.lang.RestFn.applyTo(RestFn.java:142)  80 at clojure.core$apply.invoke(core.clj:619)  81 at clojure.core$load_libs.doInvoke(core.clj:5417)  82 at clojure.lang.RestFn.applyTo(RestFn.java:137)  83 at clojure.core$apply.invoke(core.clj:621)  84 at clojure.core$use.doInvoke(core.clj:5507)  85 at clojure.lang.RestFn.invoke(RestFn.java:408)  86 at backtype.storm.command.config_value$loading__4910__auto__.invoke(config_value.clj:16)  87  at backtype.storm.command.config_value__init.load(Unknown Source)  88 at backtype.storm.command.config_value__init.<clinit>(Unknown Source)  89  at java.lang.Class.forName0(Native Method)  90 at java.lang.Class.forName(Class.java:270)  91 at clojure.lang.RT.loadClassForName(RT.java:2098)  92 at clojure.lang.RT.load(RT.java:430)  93 at clojure.lang.RT.load(RT.java:411)  94 at clojure.core$load$fn__5018.invoke(core.clj:5530)  95 at clojure.core$load.doInvoke(core.clj:5529)  96 at clojure.lang.RestFn.invoke(RestFn.java:408)  97 at clojure.lang.Var.invoke(Var.java:415)  98 at backtype.storm.command.config_value.<clinit>(Unknown Source)  99 Caused by: while parsing a block mapping 100 in 'reader', line 18, column 1: 101  storm.zookeeper.servers: 102 ^ 103 expected <block end>, but found BlockMappingStart 104 in 'reader', line 23, column 2: 105 nimbus.host: "master" 106 ^ 107 108 at org.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingKey.produce(ParserImpl.java:570) 109 at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:158) 110 at org.yaml.snakeyaml.parser.ParserImpl.checkEvent(ParserImpl.java:143) 111 at org.yaml.snakeyaml.composer.Composer.composeMappingNode(Composer.java:230) 112 at org.yaml.snakeyaml.composer.Composer.composeNode(Composer.java:159) 113 at org.yaml.snakeyaml.composer.Composer.composeDocument(Composer.java:122) 114 at org.yaml.snakeyaml.composer.Composer.getSingleNode(Composer.java:105) 115 at org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:120) 116 at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:481) 117 at org.yaml.snakeyaml.Yaml.load(Yaml.java:424) 118 at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:141) 119 at backtype.storm.utils.Utils.readStormConfig(Utils.java:188) 120 at backtype.storm.utils.Utils.<clinit>(Utils.java:71) 121 ... 36 more 122 Exception in thread "main" java.lang.ExceptionInInitializerError 123  at java.lang.Class.forName0(Native Method) 124 at java.lang.Class.forName(Class.java:190) 125 at backtype.storm.config$loading__4910__auto__.invoke(config.clj:17) 126  at backtype.storm.config__init.load(Unknown Source) 127 at backtype.storm.config__init.<clinit>(Unknown Source) 128  at java.lang.Class.forName0(Native Method) 129 at java.lang.Class.forName(Class.java:270) 130 at clojure.lang.RT.loadClassForName(RT.java:2098) 131 at clojure.lang.RT.load(RT.java:430) 132 at clojure.lang.RT.load(RT.java:411) 133 at clojure.core$load$fn__5018.invoke(core.clj:5530) 134 at clojure.core$load.doInvoke(core.clj:5529) 135 at clojure.lang.RestFn.invoke(RestFn.java:408) 136 at clojure.core$load_one.invoke(core.clj:5336) 137 at clojure.core$load_lib$fn__4967.invoke(core.clj:5375) 138 at clojure.core$load_lib.doInvoke(core.clj:5374) 139 at clojure.lang.RestFn.applyTo(RestFn.java:142) 140 at clojure.core$apply.invoke(core.clj:619) 141 at clojure.core$load_libs.doInvoke(core.clj:5417) 142 at clojure.lang.RestFn.applyTo(RestFn.java:137) 143 at clojure.core$apply.invoke(core.clj:621) 144 at clojure.core$use.doInvoke(core.clj:5507) 145 at clojure.lang.RestFn.invoke(RestFn.java:408) 146 at backtype.storm.command.config_value$loading__4910__auto__.invoke(config_value.clj:16) 147  at backtype.storm.command.config_value__init.load(Unknown Source) 148 at backtype.storm.command.config_value__init.<clinit>(Unknown Source) 149  at java.lang.Class.forName0(Native Method) 150 at java.lang.Class.forName(Class.java:270) 151 at clojure.lang.RT.loadClassForName(RT.java:2098) 152 at clojure.lang.RT.load(RT.java:430) 153 at clojure.lang.RT.load(RT.java:411) 154 at clojure.core$load$fn__5018.invoke(core.clj:5530) 155 at clojure.core$load.doInvoke(core.clj:5529) 156 at clojure.lang.RestFn.invoke(RestFn.java:408) 157 at clojure.lang.Var.invoke(Var.java:415) 158 at backtype.storm.command.config_value.<clinit>(Unknown Source) 159 Caused by: while parsing a block mapping 160 in 'reader', line 18, column 1: 161  storm.zookeeper.servers: 162 ^ 163 expected <block end>, but found BlockMappingStart 164 in 'reader', line 23, column 2: 165 nimbus.host: "master" 166 ^ 167 168 at org.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingKey.produce(ParserImpl.java:570) 169 at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:158) 170 at org.yaml.snakeyaml.parser.ParserImpl.checkEvent(ParserImpl.java:143) 171 at org.yaml.snakeyaml.composer.Composer.composeMappingNode(Composer.java:230) 172 at org.yaml.snakeyaml.composer.Composer.composeNode(Composer.java:159) 173 at org.yaml.snakeyaml.composer.Composer.composeDocument(Composer.java:122) 174 at org.yaml.snakeyaml.composer.Composer.getSingleNode(Composer.java:105) 175 at org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:120) 176 at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:481) 177 at org.yaml.snakeyaml.Yaml.load(Yaml.java:424) 178 at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:141) 179 at backtype.storm.utils.Utils.readStormConfig(Utils.java:188) 180 at backtype.storm.utils.Utils.<clinit>(Utils.java:71) 181 ... 36 more