【问题标题】:Setting up multinode hadoop cluster Blockpool ID mismatch设置多节点 hadoop 集群 Blockpool ID 不匹配
【发布时间】:2019-04-13 00:59:11
【问题描述】:

在设置多节点 hadoop 集群时,我遇到了几个问题。 通过不同的门户网站进行正确设置。出现了一些基本问题
我正在使用 Hadoop 2.8.5 在主从配置中设置 2 节点集群。
在第一个机器格式上,namenode 使用hdfs namenode format

clusterIDBlockpoolID 分配如下:

#Fri Mar 29 11:14:41 IST 2019
namespaceID=576041649
clusterID=CID-98480e8d-f7a9-4e1a-8997-400a7aa150c3
cTime=1553838281164
storageType=NAME_NODE
blockpoolID=BP-954411427-x.x.x.y-1553838281164
layoutVersion=-63

现在在第二台机器上,我运行了命令hdfs namenode format -clusterId CID-98480e8d-f7a9-4e1a-8997-400a7aa150c3

#Fri Mar 29 11:15:38 IST 2019
namespaceID=304822257
clusterID=CID-98480e8d-f7a9-4e1a-8997-400a7aa150c3
cTime=1553838338130
storageType=NAME_NODE
blockpoolID=BP-1421744029-x.x.x.x-1553838338130
layoutVersion=-63

考虑到slave和master应该有相同的clusterID,如果我错了,请纠正我。 配置似乎工作正常,但在logs/hadoop-cassandra-datanode-localnosql1.loglogs/hadoop-cassandra-datanode-localnosql2.log 的日志中出现错误

2019-03-29 11:25:44,009 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool BP-954411427-x.x.x.y-1553838281164 (Datanode Uuid 4b90bebb-3c34-43d5-8285-6ec6dfefc0a7) service to localnosql1/x.x.x.x:8020 Blockpool ID mismatch: previously connected to Blockpool ID BP-954411427-x.x.x.y-1553838281164 but now connected to Blockpool ID BP-1421744029-x.x.x.x-1553838338130  
2019-03-29 11:25:49,010 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool BP-954411427-x.x.x.y-1553838281164 (Datanode Uuid 4b90bebb-3c34-43d5-8285-6ec6dfefc0a7) service to localnosql1/x.x.x.x:8020 Blockpool ID mismatch: previously connected to Blockpool ID BP-954411427-x.x.x.y-1553838281164 but now connected to Blockpool ID BP-1421744029-x.x.x.x-1553838338130  
2019-03-29 11:25:54,012 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool BP-954411427-x.x.x.y-1553838281164 (Datanode Uuid 4b90bebb-3c34-43d5-8285-6ec6dfefc0a7) service to localnosql1/x.x.x.x:8020 Blockpool ID mismatch: previously connected to Blockpool ID BP-954411427-x.x.x.y-1553838281164 but now connected to Blockpool ID BP-1421744029-x.x.x.x-1553838338130  
2019-03-29 11:25:59,013 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool BP-954411427-x.x.x.y-1553838281164 (Datanode Uuid 4b90bebb-3c34-43d5-8285-6ec6dfefc0a7) service to localnosql1/x.x.x.x:8020 Blockpool ID mismatch: previously connected to Blockpool ID BP-954411427-x.x.x.y-1553838281164 but now connected to Blockpool ID BP-1421744029-x.x.x.x-1553838338130  
2019-03-29 11:26:04,014 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool BP-954411427-x.x.x.y-1553838281164 (Datanode Uuid 4b90bebb-3c34-43d5-8285-6ec6dfefc0a7) service to localnosql1/x.x.x.x:8020 Blockpool ID mismatch: previously connected to Blockpool ID BP-954411427-x.x.x.y-1553838281164 but now connected to Blockpool ID BP-1421744029-x.x.x.x-1553838338130  

这些错误日志表明了什么?

所有主从节点上的blockpool ID是否需要和clusterId一样,如果是怎么做?

【问题讨论】:

    标签: hadoop hadoop2


    【解决方案1】:

    你为什么要格式化 namenode 两次?理想情况下,在多节点配置中,有一个名称节点和多个数据节点。 第一次设置时,通过“hdfs namenode -format”初始化namenode,然后启动datanodes,它工作正常。

    如果您正在尝试多主机配置(同时运行多个名称节点),我不确定这是否可行。

    如果您正在尝试对 namenode 进行主备配置,您可以尝试以下步骤

    Hadoop Namenode HA setup

    【讨论】:

    • 感谢您的链接。它大大增加了我的理解。
    猜你喜欢
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 2016-07-12
    • 2013-07-06
    • 1970-01-01
    • 2019-07-22
    相关资源
    最近更新 更多