2014-02-21 13:38:25,996 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: NameNode calls recoverBlock(block=blk_-6695300470410774365_837638, targets=[10.0.0.151:50010, 10.0.0.108:50010, 10.0.0.96:50010])
2014-02-21 13:38:26,117 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_3396426774893476207_837791 src: /10.0.0.91:48688 dest: /10.0.0.151:50010
2014-02-21 13:38:26,168 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: oldblock=blk_-6695300470410774365_837638(length=50386058), newblock=blk_-6695300470410774365_837793(length=50386058), datanode=10.0.0.151:50010


.
.
.


2014-02-21 13:39:27,453 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: oldblock=blk_-6695300470410774365_837638(length=50386058), newblock=blk_-6695300470410774365_837801(length=50386058), datanode=10.0.0.151:50010
2014-02-21 13:40:27,348 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: oldblock=blk_-6695300470410774365_837638(length=50386058), newblock=blk_-6695300470410774365_837802(length=50386058), datanode=10.0.0.151:50010
2014-02-21 13:40:39,997 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.0.0.91:48823, dest: /10.0.0.151:50010, bytes: 67108864, op: HDFS_WRITE, cliID: DFSClient_205598139, srvID: DS-1932679773-10.0.0.151-50010-1342621185881, blockid: blk_-4032627633308718992_837800
2014-02-21 13:40:39,997 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 1 for block blk_-4032627633308718992_837800 terminating
2014-02-21 13:41:00,273 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in receiveBlock for block blk_-6695300470410774365_837638 java.nio.channels.ClosedByInterruptException
2014-02-21 13:41:00,273 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock blk_-6695300470410774365_837638 received exception java.io.IOException: Interrupted receiveBlock
2014-02-21 13:41:00,273 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(10.0.0.151:50010, storageID=DS-1932679773-10.0.0.151-50010-1342621185881, infoPort=50075, ipcPort=50020):DataXceiver
java.io.IOException: Interrupted receiveBlock
    at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:578)
    at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:358)
    at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)
    at java.lang.Thread.run(Thread.java:662)
2014-02-21 13:41:00,274 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder blk_-6695300470410774365_837638 2 : Thread is interrupted.
2014-02-21 13:41:00,274 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 2 for block blk_-6695300470410774365_837638 terminating
2014-02-21 13:41:00,274 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Received block blk_-6695300470410774365_837802 of size 50386058 as part of lease recovery.
2014-02-21 13:41:00,275 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 50020, call updateBlock(blk_-6695300470410774365_837638, blk_-6695300470410774365_837801, true) from 10.0.0.108:40317: error: java.io.IOException: Cannot update block (id=-6695300470410774365) generation stamp from 837802 to 837801
java.io.IOException: Cannot update block (id=-6695300470410774365) generation stamp from 837802 to 837801
    at org.apache.hadoop.hdfs.server.datanode.FSDataset.tryUpdateBlock(FSDataset.java:1098)
    at org.apache.hadoop.hdfs.server.datanode.FSDataset.updateBlock(FSDataset.java:1034)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.updateBlock(DataNode.java:1453)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:961)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:957)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:955)
2014-02-21 13:41:00,275 WARN org.apache.hadoop.hdfs.server.protocol.InterDatanodeProtocol: Failed to updateBlock (newblock=blk_-6695300470410774365_837793, datanode=10.0.0.151:50010)
java.io.IOException: Meta file not found, blockFile=/mnt/hadoop/hadoopdata/hdfs/data/current/subdir3/subdir29/blk_-6695300470410774365
    at org.apache.hadoop.hdfs.server.datanode.FSDataset.findMetaFile(FSDataset.java:771)
    at org.apache.hadoop.hdfs.server.datanode.FSDataset.tryUpdateBlock(FSDataset.java:1086)
    at org.apache.hadoop.hdfs.server.datanode.FSDataset.updateBlock(FSDataset.java:1034)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.updateBlock(DataNode.java:1453)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.syncBlock(DataNode.java:1582)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.recoverBlock(DataNode.java:1551)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.access$100(DataNode.java:127)
    at org.apache.hadoop.hdfs.server.datanode.DataNode$1.run(DataNode.java:1437)
    at java.lang.Thread.run(Thread.java:662)