新的 3.5 Zookeeper 节点似乎没有动态添加到我的集群中答案

【问题标题】：New 3.5 Zookeeper nodes do not appear to be dynamically added to my cluster新的 3.5 Zookeeper 节点似乎没有动态添加到我的集群中
【发布时间】：2017-11-09 12:16:38
【问题描述】：

我使用的是 Zookeeper 版本：3.5.3-beta，并在标准的 3 节点配置中进行了配置。我现在尝试按照文档添加 3 个新节点，但是新节点似乎没有被动态添加为观察者（请参阅https://zookeeper.apache.org/doc/trunk/zookeeperReconfig.html#ch_reconfig_dyn）。

首先，我尝试使用以下设置添加第一个节点："ZOOKEEPER_IP_LIST=10.144.81.81 10.144.81.83 10.144.81.86 10.144.46.54"，但它没有加入集群。当我启动新节点时，我仍然看到以下内容，仅显示集群中的 3 个初始节点。我什至尝试重新启动 3 个初始节点以包含新节点，但它仍然反映了旧配置。

clientPort=2181
secureClientPort=-1
dataDir=/dbpath/data/version-2
dataDirSize=134254305
dataLogDir=/dbpath/data/version-2
dataLogSize=134254305
tickTime=2000
maxClientCnxns=30
minSessionTimeout=4000
maxSessionTimeout=40000
serverId=3
initLimit=60
syncLimit=5
electionAlg=3
electionPort=3888
quorumPort=2888
peerType=0
membership: 
server.0=10.144.81.81:2888:3888:participant
server.1=10.144.81.83:2888:3888:participant
server.2=10.144.81.86:2888:3888:participant

当这不起作用时，我决定先在 leader 上尝试 zkCli.sh 命令，看看上面尝试的配置是否与 REST 接口的结果匹配。

$ sudo bin/zkCli.sh -server 10.144.81.83:2181 config
...
2017-11-09 12:13:38,172 [myid:10.144.81.83:2181] - INFO  [main-SendThread(10.144.81.83:2181):ClientCnxn$SendThread@1381] - Session establishment complete on server 10.144.81.83/10.144.81.83:2181, sessionid = 0x100301c01db000a, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
server.0=10.144.81.81:2888:3888:participant
server.1=10.144.81.83:2888:3888:participant
server.2=10.144.81.86:2888:3888:participant
version=100000000

然后我尝试使用 zkCli.sh 重新配置和添加一个新节点，但它引发了身份验证问题，这令人困惑，因为我没有配置 ACL 或我知道的任何身份验证都会导致此错误。

$ sudo bin/zkCli.sh -server 10.144.81.83:2181 reconfig -file ../newconfig.cfg.dynamic
/usr/bin/java
Connecting to 10.144.81.83:2181
2017-11-09 12:03:32,729 [myid:] - INFO  [main:Environment@109] - Client environment:zookeeper.version=3.5.3-beta-8ce24f9e675cbefffb8f21a47e06b42864475a60, built on 04/03/2017 16:19 GMT
2017-11-09 12:03:32,735 [myid:] - INFO  [main:Environment@109] - Client environment:host.name=DockerNode-master-cloudeng-20170925-1652-Jy1.localdomain
2017-11-09 12:03:32,739 [myid:] - INFO  [main:Environment@109] - Client environment:java.version=1.7.0
2017-11-09 12:03:32,739 [myid:] - INFO  [main:Environment@109] - Client environment:java.vendor=IBM Corporation
2017-11-09 12:03:32,739 [myid:] - INFO  [main:Environment@109] - Client environment:java.home=/opt/ibm/ibm-java-x86_64-70/jre
2017-11-09 12:03:32,739 [myid:] - INFO  [main:Environment@109] - Client environment:java.class.path=/home/ibmadmin/zookeeper-3.5.3-beta/bin/../build/classes:/home/ibmadmin/zookeeper-3.5.3-beta/bin/../build/lib/*.jar:/home/ibmadmin/zookeeper-3.5.3-beta/bin/../lib/slf4j-log4j12-1.7.5.jar:/home/ibmadmin/zookeeper-3.5.3-beta/bin/../lib/slf4j-api-1.7.5.jar:/home/ibmadmin/zookeeper-3.5.3-beta/bin/../lib/netty-3.10.5.Final.jar:/home/ibmadmin/zookeeper-3.5.3-beta/bin/../lib/log4j-1.2.17.jar:/home/ibmadmin/zookeeper-3.5.3-beta/bin/../lib/jline-2.11.jar:/home/ibmadmin/zookeeper-3.5.3-beta/bin/../lib/jetty-util-9.2.18.v20160721.jar:/home/ibmadmin/zookeeper-3.5.3-beta/bin/../lib/jetty-servlet-9.2.18.v20160721.jar:/home/ibmadmin/zookeeper-3.5.3-beta/bin/../lib/jetty-server-9.2.18.v20160721.jar:/home/ibmadmin/zookeeper-3.5.3-beta/bin/../lib/jetty-security-9.2.18.v20160721.jar:/home/ibmadmin/zookeeper-3.5.3-beta/bin/../lib/jetty-io-9.2.18.v20160721.jar:/home/ibmadmin/zookeeper-3.5.3-beta/bin/../lib/jetty-http-9.2.18.v20160721.jar:/home/ibmadmin/zookeeper-3.5.3-beta/bin/../lib/javax.servlet-api-3.1.0.jar:/home/ibmadmin/zookeeper-3.5.3-beta/bin/../lib/jackson-mapper-asl-1.9.11.jar:/home/ibmadmin/zookeeper-3.5.3-beta/bin/../lib/jackson-core-asl-1.9.11.jar:/home/ibmadmin/zookeeper-3.5.3-beta/bin/../lib/commons-cli-1.2.jar:/home/ibmadmin/zookeeper-3.5.3-beta/bin/../zookeeper-3.5.3-beta.jar:/home/ibmadmin/zookeeper-3.5.3-beta/bin/../src/java/lib/*.jar:/home/ibmadmin/zookeeper-3.5.3-beta/bin/../conf:
2017-11-09 12:03:32,740 [myid:] - INFO  [main:Environment@109] - Client environment:java.library.path=/opt/ibm/ibm-java-x86_64-70/jre/lib/amd64/compressedrefs:/opt/ibm/ibm-java-x86_64-70/jre/lib/amd64:/usr/lib64:/usr/lib
2017-11-09 12:03:32,740 [myid:] - INFO  [main:Environment@109] - Client environment:java.io.tmpdir=/tmp
2017-11-09 12:03:32,740 [myid:] - INFO  [main:Environment@109] - Client environment:java.compiler=j9jit26
2017-11-09 12:03:32,740 [myid:] - INFO  [main:Environment@109] - Client environment:os.name=Linux
2017-11-09 12:03:32,740 [myid:] - INFO  [main:Environment@109] - Client environment:os.arch=amd64
2017-11-09 12:03:32,740 [myid:] - INFO  [main:Environment@109] - Client environment:os.version=4.4.0-89-generic
2017-11-09 12:03:32,740 [myid:] - INFO  [main:Environment@109] - Client environment:user.name=root
2017-11-09 12:03:32,741 [myid:] - INFO  [main:Environment@109] - Client environment:user.home=/root
2017-11-09 12:03:32,741 [myid:] - INFO  [main:Environment@109] - Client environment:user.dir=/home/ibmadmin/zookeeper-3.5.3-beta
2017-11-09 12:03:32,743 [myid:] - INFO  [main:Environment@109] - Client environment:os.memory.free=1MB
2017-11-09 12:03:32,744 [myid:] - INFO  [main:Environment@109] - Client environment:os.memory.max=256MB
2017-11-09 12:03:32,744 [myid:] - INFO  [main:Environment@109] - Client environment:os.memory.total=4MB
2017-11-09 12:03:32,749 [myid:] - INFO  [main:ZooKeeper@865] - Initiating client connection, connectString=10.144.81.83:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@f9b15d67
2017-11-09 12:03:32,765 [myid:] - INFO  [main:ClientCnxnSocket@236] - jute.maxbuffer value is 4194304 Bytes
2017-11-09 12:03:32,785 [myid:10.144.81.83:2181] - INFO  [main-SendThread(10.144.81.83:2181):ClientCnxn$SendThread@1113] - Opening socket connection to server 10.144.81.83/10.144.81.83:2181. Will not attempt to authenticate using SASL (unknown error)
2017-11-09 12:03:32,802 [myid:10.144.81.83:2181] - INFO  [main-SendThread(10.144.81.83:2181):ClientCnxn$SendThread@948] - Socket connection established, initiating session, client: /10.144.81.83:59114, server: 10.144.81.83/10.144.81.83:2181
2017-11-09 12:03:32,812 [myid:10.144.81.83:2181] - INFO  [main-SendThread(10.144.81.83:2181):ClientCnxn$SendThread@1381] - Session establishment complete on server 10.144.81.83/10.144.81.83:2181, sessionid = 0x100301c01db0008, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
Authentication is not valid : 

$ cat ../newconfig.cfg.dynamic
server.0=10.144.81.81:2888:3888:participant
server.1=10.144.81.83:2888:3888:participant
server.2=10.144.81.86:2888:3888:participant
server.3=10.144.46.54:2888:3888:observer

文档对我的印象是对还是错，我可以只指定新的拓扑更改，而无需更改任何特定的节点配置。我宁愿不要使用 zkCli，因为它只会使我的部署过程复杂化，但如果我必须这样做的话。

【问题讨论】：

标签： apache-zookeeper mesos

【解决方案1】：

发生的情况是，不允许新的观察者进入 zookeeper 保存配置信息的路径。最初所有节点都将自己添加到静态配置中，但不允许任何动态新节点，因为它没有路径权限（/config）

您可以尝试使用此 jvm 选项重新启动节点：

-Dzookeeper.skipACL=true

或者你可以成为超级用户：

 $ export ZK_CLASSPATH=/etc/zookeeper/conf/:/usr/hdp/current/zookeeper-server/lib/*:/usr/hdp/current/zookeeper-server/*
 $ java -cp $ZK_CLASSPATH org.apache.zookeeper.server.auth.DigestAuthenticationProvider super:super123

输出：

super:super123->super:UdxDQl4f9v5oITwcAsO9bmWgHSI=

然后以root身份：

$ export SERVER_JVMFLAGS=-Dzookeeper.DigestAuthenticationProvider.superDigest=super:UdxDQl4f9v5oITwcAsO9bmWgHSI=
$ bin/zkCli.sh -server 10.144.81.83:2181 reconfig -file ../newconfig.cfg.dynamic

希望对你有帮助。

【讨论】：

我实际上尝试了-Dzookeeper.skipACL=true，并强制将一堆其他 SASL 设置为 false 以禁用它们，但没有任何效果。起作用的是手动创建新的动态配置文件，然后更新主 zoo.cfg 以指向它并重新启动服务。但恕我直言，我认为这完全违背了动态配置的整个想法。在这一点上，我很困惑这一切应该如何工作，我必须承认文档不够详细，无法彻底解释。
可能与 zookeeper 如何保持状态和仲裁有关。有时你需要完全从头开始