方法一、使用namespaceID

    1、在namenode节点上，将dfs.name.dir指定的目录中(这里是name目录)的内容情况，以此来模拟故障发生。

1[hadoop@node1 name]$ ls2currentimage in_use.lock3[hadoop@node1 name]$ rm-rf*

    2、将集群关闭后，再重启我们看到namenode守护进程消失。

1[hadoop@node1 name]$ stop-all.sh 2stopping jobtracker 3192.168.1.152: stopping tasktracker 4192.168.1.153: stopping tasktracker 5stopping namenode 6192.168.1.152: stopping datanode 7192.168.1.153: stopping datanode 8192.168.1.152: stopping secondarynamenode 9[hadoop@node1 name]$ start-all.sh 10starting namenode, loggingto/app/hadoop/bin/../logs/hadoop-hadoop-namenode-node1.out11192.168.1.152: starting datanode, loggingto/app/hadoop/bin/../logs/hadoop-hadoop-datanode-node2.out12192.168.1.153: starting datanode, loggingto/app/hadoop/bin/../logs/hadoop-hadoop-datanode-node3.out13192.168.1.152: starting secondarynamenode, loggingto/app/hadoop/bin/../logs/hadoop-hadoop-secondarynamenode-node2.out14starting jobtracker, loggingto/app/hadoop/bin/../logs/hadoop-hadoop-jobtracker-node1.out15192.168.1.152: starting tasktracker, loggingto/app/hadoop/bin/../logs/hadoop-hadoop-tasktracker-node2.out16192.168.1.153: starting tasktracker, loggingto/app/hadoop/bin/../logs/hadoop-hadoop-tasktracker-node3.out17[hadoop@node1 name]$ jps1831942 Jps1931872JobTracker

    而且namenode的日志中有报错：

12013-11-1406:19:59,172 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG: 2/************************************************************ 3STARTUP_MSG: Starting NameNode 4STARTUP_MSG: host = node1/192.168.1.151 5STARTUP_MSG: args = [] 6STARTUP_MSG: version = 0.20.2 7STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010 8************************************************************/ 92013-11-1406:19:59,395INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC MetricswithhostName=NameNode, port=9000102013-11-1406:19:59,400INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Namenode up at: node1.com/192.168.1.151:9000112013-11-1406:19:59,403INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM MetricswithprocessName=NameNode, sessionId=null122013-11-1406:19:59,407 INFO org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics: Initializing NameNodeMeterics using context object:org.apache.hadoop.metrics.spi.NullContext132013-11-1406:19:59,557INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=hadoop,hadoop142013-11-1406:19:59,558INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=supergroup152013-11-1406:19:59,558INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled=true162013-11-1406:19:59,568 INFO org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics: Initializing FSNamesystemMetrics using context object:org.apache.hadoop.metrics.spi.NullContext172013-11-1406:19:59,569 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered FSNamesystemStatusMBean182013-11-1406:19:59,654 ERROR org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem initialization failed.19java.io.IOException: NameNodeisnot formatted.20at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:317)21at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87)22at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:311)23at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:292)24at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:201)25at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279)26at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956)27at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)282013-11-1406:19:59,658INFO org.apache.hadoop.ipc.Server: Stopping serveron9000292013-11-1406:19:59,663ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException: NameNodeisnot formatted.30at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:317)31at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87)32at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:311)33at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:292)34at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:201)35at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279)36at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956)37at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)38392013-11-1406:19:59,664 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:40/************************************************************41SHUTDOWN_MSG: Shutting down NameNode at node1/192.168.1.15142************************************************************/

    3、查看HDFS的文件失败：

1[hadoop@node1 name]$ hadoop dfs-ls/user/hive/warehouse213/11/1406:21:06INFO ipc.Client: Retrying connecttoserver: node1/192.168.1.151:9000. Already tried0 time(s).313/11/1406:21:07INFO ipc.Client: Retrying connecttoserver: node1/192.168.1.151:9000. Already tried1 time(s).413/11/1406:21:08INFO ipc.Client: Retrying connecttoserver: node1/192.168.1.151:9000. Already tried2 time(s).513/11/1406:21:09INFO ipc.Client: Retrying connecttoserver: node1/192.168.1.151:9000. Already tried3time(s).

    4、关闭集群，格式化namenode：

1[hadoop@node1 name]$ stop-all.sh 2stopping jobtracker 3192.168.1.152: stopping tasktracker 4192.168.1.153: stopping tasktracker 5no namenodeto stop 6192.168.1.152: stopping datanode 7192.168.1.153: stopping datanode 8192.168.1.152: stopping secondarynamenode 9[hadoop@node1 name]$ hadoop namenode-format1013/11/1406:21:37 INFO namenode.NameNode: STARTUP_MSG: 11/************************************************************12STARTUP_MSG: Starting NameNode13STARTUP_MSG: host = node1/192.168.1.15114STARTUP_MSG: args = [-format]15STARTUP_MSG: version = 0.20.216STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 201017************************************************************/18Re-format filesystemin/app/user/hdfs/name ? (Yor N) Y1913/11/1406:21:39INFO namenode.FSNamesystem: fsOwner=hadoop,hadoop2013/11/1406:21:39INFO namenode.FSNamesystem: supergroup=supergroup2113/11/1406:21:39INFO namenode.FSNamesystem: isPermissionEnabled=true2213/11/1406:21:39INFO common.Storage:Imagefileofsize96savedin0 seconds.2313/11/1406:21:39INFO common.Storage: Storage directory/app/user/hdfs/name has been successfully formatted.2413/11/1406:21:39 INFO namenode.NameNode: SHUTDOWN_MSG: 25/************************************************************26SHUTDOWN_MSG: Shutting down NameNode at node1/192.168.1.15127************************************************************/

    5、从任意datanode中获取namenode格式化之前namespaceID并修改namenode的namespaceID跟datanode一致：

1[hadoop@node2 current]$ cat VERSION 2#Thu Nov1402:27:10CST2013 3namespaceID=2062292356 4storageID=DS-107813142-192.168.1.152-50010-1379339943465 5cTime=0 6storageType=DATA_NODE 7layoutVersion=-18 8[hadoop@node2 current]$ pwd 9/app/user/hdfs/data/current10----修改namenode的namespaceID----

[hadoop@node1 current]$ cat VERSION 11#Thu Nov1406:29:31CST201312namespaceID=206229235613cTime=014storageType=NAME_NODE15layoutVersion=-18

    6、删除新的namenode的fsimage文件：

1[hadoop@node1 current]$ ll2total163-rw-rw-r-- 1 hadoop hadoop 4 Nov 14 06:21 edits4-rw-rw-r-- 1 hadoop hadoop 96 Nov 14 06:21 fsimage5-rw-rw-r-- 1 hadoop hadoop 8 Nov 14 06:21 fstime6-rw-rw-r-- 1 hadoop hadoop 101 Nov 14 06:22 VERSION7[hadoop@node1 current]$ rm fsimage

    7、从Secondarynamenode拷贝fsimage到Namenode的current目录下：

[hadoop@node2 current]$ ll

total 16-rw-rw-r-- 1 hadoop hadoop 4 Nov 14 05:38 edits-rw-rw-r-- 1 hadoop hadoop 2410 Nov 14 05:38 fsimage-rw-rw-r-- 1 hadoop hadoop 8 Nov 14 05:38 fstime-rw-rw-r-- 1 hadoop hadoop 101 Nov 14 05:38 VERSION[hadoop@node2 current]$ scp fsimage node1:/app/user/hdfs/name/currentThe authenticity ofhost'node1 (192.168.1.151)'can't be established.

RSA key fingerprint is ca:9a:7e:19:ee:a1:35:44:7e:9d:d4:09:5c:fc:c5:0a.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'node1,192.168.1.151' (RSA) to the list of known hosts.

fsimage 100% 2410 2.4KB/s 00:00

    8、重启集群：

[hadoop@node1 current]$ start-all.sh

starting namenode, logging to/app/hadoop/bin/../logs/hadoop-hadoop-namenode-node1.out192.168.1.152: starting datanode, loggingto/app/hadoop/bin/../logs/hadoop-hadoop-datanode-node2.out192.168.1.153: starting datanode, loggingto/app/hadoop/bin/../logs/hadoop-hadoop-datanode-node3.out192.168.1.152: starting secondarynamenode, loggingto/app/hadoop/bin/../logs/hadoop-hadoop-secondarynamenode-node2.out

starting jobtracker, logging to/app/hadoop/bin/../logs/hadoop-hadoop-jobtracker-node1.out192.168.1.152: starting tasktracker, loggingto/app/hadoop/bin/../logs/hadoop-hadoop-tasktracker-node2.out192.168.1.153: starting tasktracker, loggingto/app/hadoop/bin/../logs/hadoop-hadoop-tasktracker-node3.out[hadoop@node1 current]$ jps32486 Jps32419 JobTracker32271NameNode

    9、验证数据的完整性：

1[hadoop@node1 current]$ hadoop dfs-ls/user/hive/warehouse 2Found8 items 3drwxr-xr-x-hadoop supergroup02013-10-1716:18/user/hive/warehouse/echo 4drwxr-xr-x-hadoop supergroup02013-10-2813:48/user/hive/warehouse/jack 5drwxr-xr-x-hadoop supergroup02013-09-1815:54/user/hive/warehouse/table4 6drwxr-xr-x-hadoop supergroup02013-09-1815:53/user/hive/warehouse/table5 7drwxr-xr-x-hadoop supergroup02013-09-1815:48/user/hive/warehouse/test 8drwxr-xr-x-hadoop supergroup02013-10-2514:50/user/hive/warehouse/test1 9drwxr-xr-x-hadoop supergroup02013-10-2514:52/user/hive/warehouse/test210drwxr-xr-x-hadoop supergroup02013-10-2514:30/user/hive/warehouse/test31112[hadoop@node3 conf]$ hive1314Logging initialized using configurationinjar:file:/app/hive/lib/hive-common-0.11.0.jar!/hive-log4j.properties15Hive historyfile=/tmp/hadoop/hive_job_log_hadoop_7451@node3_201311111325_424288589.txt16hive> show tables;17OK18echo19jack20table421table522test23test124test225test326Time taken:27.589seconds, Fetched:8 row(s)27hive>select*from table4;28OK29NULLNULLNULL301153124532345334563456735615367563783638NULLNULLNULL39Time taken:2.124seconds, Fetched:10row(s)

    之前里面的数据没有丢失。

方法二：使用hadoop namenode -importCheckpoint

    1、删除name目录：

1[hadoop@node1 hdfs]$ rm-rf name