hadoop集群启动遇到的问题及解决方案

1、启动hadoop，发现50070页面无法访问，jps后发现namenode未启动，到hadoop的logs日志中查看发现报错
java.io.IOException: Premature EOF from inputStream
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:201)
at org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:221)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:924)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:908)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:727)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:663)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:281)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1058)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:762)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:582)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:641)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:808)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:792)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1484)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1550)
2016-08-12 08:52:00,113 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Encountered exception loading fsimage
java.io.IOException: Failed to load an FSImage file!
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:674)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:281)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1058)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:762)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:582)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:641)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:808)
查不到合适的解决方案，就采取格式化namenode ：hadoop namenode -format 格式化之前要删除根目录下hdfs下的文件，并删除hadoop的logs文件，可能还要删别的，我只删除了这俩。

再次启动hadoop ，jps后发现namenode正常启动，50070正常访问，结果datanode无法启动，到hadoop的日志文件中查看发现报错
2、datanode无法启动
2016-08-12 10:06:14,316 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2016-08-12 10:06:14,317 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 50020: starting
2016-08-12 10:06:14,592 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /data1/cdh/in_use.lock acquired by nodename 15432@is-nn-01
2016-08-12 10:06:14,594 WARN org.apache.hadoop.hdfs.server.common.Storage: java.io.IOException: Incompatible clusterIDs in /data1/cdh: namenode clusterID = CID-c61b3952-cf9d-47d4-97df-ee86aa63cdf4; datanode clusterID = CID-b0cf57dd-646c-4c7a-9654-9568305f2ac1
2016-08-12 10:06:14,607 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /data2/cdh/in_use.lock acquired by nodename 15432@is-nn-01
2016-08-12 10:06:14,668 WARN org.apache.hadoop.hdfs.server.common.Storage: java.io.IOException: Incompatible clusterIDs in /data2/cdh: namenode clusterID = CID-c61b3952-cf9d-47d4-97df-ee86aa63cdf4; datanode clusterID = CID-b0cf57dd-646c-4c7a-9654-9568305f2ac1
2016-08-12 10:06:14,670 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /data3/cdh/in_use.lock acquired by nodename 15432@is-nn-01
2016-08-12 10:06:14,670 WARN org.apache.hadoop.hdfs.server.common.Storage: java.io.IOException: Incompatible clusterIDs in /data3/cdh: namenode clusterID = CID-c61b3952-cf9d-47d4-97df-ee86aa63cdf4; datanode clusterID = CID-b0cf57dd-646c-4c7a-9654-9568305f2ac1
2016-08-12 10:06:14,671 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool <registering> (Datanode Uuid unassigned) service to is-nn-01/192.168.248.128:8020. Exiting.
java.io.IOException: All specified directories are failed to load.
at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:479)
at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1397)
at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1362)
at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:316)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:227)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:839)
at java.lang.Thread.run(Thread.java:745)
2016-08-12 10:06:14,673 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool <registering> (Datanode Uuid unassigned) service to is-nn-01/192.168.248.128:8020
2016-08-12 10:06:14,675 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Removed Block pool <registering> (Datanode Uuid unassigned)
2016-08-12 10:06:16,675 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Exiting Datanode
2016-08-12 10:06:16,676 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 0
2016-08-12 10:06:16,678 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:

从日志上看是因为 datanode的clusterID 和 namenode的clusterID 不匹配。

解决：
根据日志中的路径所报的路径，像我的是clusterIDs in /data1/cdh，到根目录下的data1中的cdh中（还有一个current文件夹下面有个VERSION的文件中）将日志信息中提示的namenode 地址copy 替换各个data文件夹中的clusterID即可，其他节点也要根据各个hadoop日志中报错信息相应替换ID 即可

重新启动 jps 全部都有 开心。

引用：
出现该问题的原因：在第一次格式化dfs后，启动并使用了hadoop，后来又重新执行了格式化命令（hdfs namenode -format)，这时namenode的clusterID会重新生成，而datanode的clusterID 保持不变。