Pulsar 集群安装
0x1 摘要
本文记录Pulsar 2.2.0版本安装步骤，单机模式（standalone）比较简单，直接参考官网：http://pulsar.apache.org/docs/en/standalone/
按步骤执行就行，主要讲解集群模式安装，以及过程中遇到问题的解决。
0x2 环境要求
- Linux
- Java 8 及以上
- 3 台ZooKeeper集群
0x3 安装顺序
- 安装ZooKeeper集群
- 初始化集群元数据信息
- 安装BookKeeper集群
- 安装Pulsar brokers
下面针对每一步进行详细介绍。
0x4 安装ZooKeeper集群
由于我本地环境已经有安装好的ZK集群，可以直接使用，此步省略。
0x5 初始化集群元数据信息
初始化元数据信息非常简单，只需一条命令就可以，具体参数的意义看官网更好理解：
bin/pulsar initialize-cluster-metadata \
--cluster pulsar-cluster-1 \
--zookeeper zk1.us-west.example.com:2181 \
--configuration-store zk1.us-west.example.com:2181 \
--web-service-url http://pulsar.us-west.example.com:8080 \
--web-service-url-tls https://pulsar.us-west.example.com:8443 \
--broker-service-url pulsar://pulsar.us-west.example.com:6650 \
--broker-service-url-tls pulsar+ssl://pulsar.us-west.example.com:6651初始化成功会看到以下日志信息：
10:36:09.876 [main] INFO org.apache.bookkeeper.discover.ZKRegistrationManager - Successfully formatted BookKeeper metadata
10:36:09.880 [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x16734464b360002 closed
10:36:09.880 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x16734464b360002
10:36:10.033 [main] INFO org.apache.pulsar.PulsarClusterMetadataSetup - Cluster metadata for 'pulsar-cluster-1' setup correctly并且通过ZK客户端登录查看会看到以下节点信息：
[zk: localhost:2186(CONNECTED) 2] ls /
[zookeeper, counters, bookies, ledgers, managed-ledgers, schemas, namespace, admin, loadbalance]此步非常重要，我在安装过程中忽略此步后启动BookKeeper直接报错，错误信息如下：
启动命令bin/bookkeeper bookie
10:26:17.185 [main] INFO org.apache.bookkeeper.proto.BookieNettyServer - Shutting down BookieNettyServer
10:26:17.196 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server
org.apache.bookkeeper.bookie.BookieException$MetadataStoreException: Failed to get cluster instance id
at org.apache.bookkeeper.discover.ZKRegistrationManager.getClusterInstanceId(ZKRegistrationManager.java:387) ~[org.apache.bookkeeper-bookkeeper-server-4.7.2.jar:4.7.2]
at org.apache.bookkeeper.bookie.Bookie.checkEnvironmentWithStorageExpansion(Bookie.java:412) ~[org.apache.bookkeeper-bookkeeper-server-4.7.2.jar:4.7.2]
at org.apache.bookkeeper.bookie.Bookie.checkEnvironment(Bookie.java:256) ~[org.apache.bookkeeper-bookkeeper-server-4.7.2.jar:4.7.2]
at org.apache.bookkeeper.bookie.Bookie.<init>(Bookie.java:640) ~[org.apache.bookkeeper-bookkeeper-server-4.7.2.jar:4.7.2]
at org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:131) ~[org.apache.bookkeeper-bookkeeper-server-4.7.2.jar:4.7.2]
at org.apache.bookkeeper.proto.BookieServer.<init>(BookieServer.java:100) ~[org.apache.bookkeeper-bookkeeper-server-4.7.2.jar:4.7.2]
at org.apache.bookkeeper.server.service.BookieService.<init>(BookieService.java:43) ~[org.apache.bookkeeper-bookkeeper-server-4.7.2.jar:4.7.2]
at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:299) ~[org.apache.bookkeeper-bookkeeper-server-4.7.2.jar:4.7.2]
at org.apache.bookkeeper.server.Main.doMain(Main.java:219) [org.apache.bookkeeper-bookkeeper-server-4.7.2.jar:4.7.2]
at org.apache.bookkeeper.server.Main.main(Main.java:201) [org.apache.bookkeeper-bookkeeper-server-4.7.2.jar:4.7.2]
at org.apache.bookkeeper.proto.BookieServer.main(BookieServer.java:280) [org.apache.bookkeeper-bookkeeper-server-4.7.2.jar:4.7.2]
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for BookKeeper metadata
at org.apache.bookkeeper.discover.ZKRegistrationManager.getClusterInstanceId(ZKRegistrationManager.java:377) ~[org.apache.bookkeeper-bookkeeper-server-4.7.2.jar:4.7.2]
... 10 more验证BookKeeper时报以下错误：
验证命令bin/bookkeeper shell bookiesanity
09:49:14.430 [main] INFO org.apache.bookkeeper.client.BookKeeper - Weighted ledger placement is not enabled
09:49:14.489 [main] ERROR org.apache.bookkeeper.client.BookieWatcher - Failed to get bookie list :
org.apache.bookkeeper.client.BKException$ZKException: Error while using ZooKeeper
at org.apache.bookkeeper.discover.ZKRegistrationClient.lambda$getChildren$0(ZKRegistrationClient.java:212) ~[org.apache.bookkeeper-bookkeeper-server-4.7.2.jar:4.7.2]
at org.apache.bookkeeper.zookeeper.ZooKeeperClient$25$1.processResult(ZooKeeperClient.java:1174) ~[org.apache.bookkeeper-bookkeeper-server-4.7.2.jar:4.7.2]
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent_aroundBody0(ClientCnxn.java:604) ~[org.apache.pulsar-pulsar-broker-2.2.0.jar:2.2.0]
at org.apache.zookeeper.ClientCnxn$EventThread$AjcClosure1.run(ClientCnxn.java:1) ~[org.apache.pulsar-pulsar-broker-2.2.0.jar:2.2.0]
at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) ~[org.aspectj-aspectjrt-1.9.1.jar:?]
at org.apache.pulsar.broker.zookeeper.aspectj.ClientCnxnAspect.timedProcessEvent(ClientCnxnAspect.java:72) ~[org.apache.pulsar-pulsar-broker-2.2.0.jar:2.2.0]
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:528) ~[org.apache.pulsar-pulsar-broker-2.2.0.jar:2.2.0]
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:508) ~[org.apache.pulsar-pulsar-broker-2.2.0.jar:2.2.0]
Exception in thread "main" org.apache.bookkeeper.client.BKException$ZKException: Error while using ZooKeeper
at org.apache.bookkeeper.discover.ZKRegistrationClient.lambda$getChildren$0(ZKRegistrationClient.java:212)
at org.apache.bookkeeper.zookeeper.ZooKeeperClient$25$1.processResult(ZooKeeperClient.java:1174)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent_aroundBody0(ClientCnxn.java:604)
at org.apache.zookeeper.ClientCnxn$EventThread$AjcClosure1.run(ClientCnxn.java:1)
at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
at org.apache.pulsar.broker.zookeeper.aspectj.ClientCnxnAspect.timedProcessEvent(ClientCnxnAspect.java:72)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:528)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:508)0x6 安装BookKeeper集群
直接使用Pulsar内嵌的BK，单独安装未尝试。
修改$PULSAR_HOME/conf/bookkeeper.conf配置文件中的zkServers属性，如：
zkServers=zk1.us-west.example.com:2181,zk2.us-west.example.com:2181,zk3.us-west.example.com:2181修改完成后直接命令启动，启动命令bin/pulsar-daemon start bookie
启动完成后使用命令bin/bookkeeper shell bookiesanity验证是否成功，看到以下信息说明启动成功
10:44:49.735 [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x46734464bc90004 closed
10:44:49.735 [main] INFO org.apache.bookkeeper.bookie.BookieShell - Bookie sanity test succeeded
10:44:49.735 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x46734464bc900040x7 安装Pulsar brokers
修改$PULSAR_HOME/conf/broker.conf配置文件
zookeeperServers=zk1.us-west.example.com:2181,zk2.us-west.example.com:2181,zk3.us-west.example.com:2181
configurationStoreServers=zk1.us-west.example.com:2181,zk2.us-west.example.com:2181,zk3.us-west.example.com:2181
clusterName=pulsar-cluster-1修改完成后使用命令bin/pulsar-daemon start broker启动broker
启动完成后可以通过bin/pulsar-client produce命令发送消息测试，我自己通过Java Client代码测试。
Producer代码：
PulsarClient client = PulsarClient.builder().serviceUrl("pulsar://pulsar.us-west.example.com:6650").build();
Producer<String> producer = client.newProducer(Schema.STRING).topic("my-topic").create();
producer.send("aaaaaaaaaaaaaa");
producer.close();Consumer代码：
PulsarClient client = PulsarClient.builder().serviceUrl("pulsar://pulsar.us-west.example.com:6650").build();
Consumer<String> consumer = client.newConsumer(Schema.STRING)
.topic("my-topic")
.subscriptionName("my-topic-group-1")
.subscribe();
do {
Message<String> receive = consumer.receive();
System.out.println(receive.getValue());
} while (true);
