API tools faq
paste
Advertisement
Guest User

va-p-hbase-*-* crash log

a guest
May 23rd, 2013
74
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 207.79 KB | None | 0 0
raw download clone embed print report
  1. ++++++++++++++++++++++++++++++++++++++++++++
  2. va-p-hbase-01-c (no warning prior to fatal)
  3. ++++++++++++++++++++++++++++++++++++++++++++
  4. 2013-05-22 14:58:58,923 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server va-p-hbase-01-c,60020,1369233253404: Writing replication status
  5. org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/replication/rs/va-p-hbase-01-c,60020,1369233253404/1-va-p-hbase-01-c,60020,1369042378287-va-p-hbase-02-c,60020,1369042377731/va-p-hbase-01-c%2C60020%2C1369042378287.1369220050719
  6. at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
  7. at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
  8. at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1266)
  9. at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.setData(RecoverableZooKeeper.java:354)
  10. at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:846)
  11. at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:898)
  12. at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:892)
  13. at org.apache.hadoop.hbase.replication.ReplicationZookeeper.writeReplicationStatus(ReplicationZookeeper.java:558)
  14. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.logPositionAndCleanOldLogs(ReplicationSourceManager.java:154)
  15. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:638)
  16. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:387)
  17.  
  18. ++++++++++++++++++++++++++++++++++++++++++++
  19. va-p-hbase-01-d
  20. ++++++++++++++++++++++++++++++++++++++++++++
  21. 2013-05-22 14:59:01,080 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager: Atomically moving va-p-hbase-01-c,60020,1369233253404's hlogs to my queue
  22. 2013-05-22 14:59:01,152 WARN org.apache.hadoop.hbase.replication.ReplicationZookeeper: Got exception in copyQueuesFromRSUsingMulti:
  23. org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
  24. ...
  25. 2013-05-22 14:59:02,500 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1-va-p-hbase-01-c,60020,1369233253404 Got:
  26. java.io.IOException: File from recovered queue is nowhere to be found
  27. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:535)
  28. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:312)
  29. Caused by: java.io.FileNotFoundException: File does not exist: hdfs://HBaseNameService/hbase/.oldlogs/va-p-hbase-01-c%2C60020%2C1369233253404.1369233257396
  30. at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:807)
  31. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1704)
  32. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
  33. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
  34. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
  35. at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:720)
  36. at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:66)
  37. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:501)
  38. ... 1 more
  39.  
  40. .....
  41. 2013-05-22 14:59:03,555 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: NB dead servers : 5
  42. 2013-05-22 14:59:03,555 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/c,60020,1369233253404/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  43. 2013-05-22 14:59:03,558 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/c,60020,1369233253404-splitting/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  44. 2013-05-22 14:59:03,561 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  45. 2013-05-22 14:59:03,564 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01-splitting/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  46. 2013-05-22 14:59:03,567 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  47. 2013-05-22 14:59:03,570 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase-splitting/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  48. 2013-05-22 14:59:03,572 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  49. 2013-05-22 14:59:03,575 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p-splitting/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  50. 2013-05-22 14:59:03,578 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  51. 2013-05-22 14:59:03,580 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va-splitting/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  52. 2013-05-22 14:59:03,583 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1-va-p-hbase-01-c,60020,1369233253404 Got:
  53. java.io.IOException: File from recovered queue is nowhere to be found
  54. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:535)
  55. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:312)
  56. Caused by: java.io.FileNotFoundException: File does not exist: hdfs://HBaseNameService/hbase/.oldlogs/va-p-hbase-01-c%2C60020%2C1369233253404.1369233257396
  57. at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:807)
  58. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1704)
  59. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
  60. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
  61. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
  62. at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:720)
  63. at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:66)
  64. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:501)
  65. ... 1 more
  66. 2013-05-22 14:59:05,144 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Processed 195997 edits across 14 regions threw away edits for 0 regions; log file=hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-c,60020,1369233253404-splitting/va-p-hbase-01-c%2C60020%2C1369233253404.1369236857704 is corrupted = false progress failed = false
  67. 2013-05-22 14:59:05,158 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: successfully transitioned task /hbase/splitlog/hdfs%3A%2F%2FHBaseNameService%2Fhbase%2F.logs%2Fva-p-hbase-01-c%2C60020%2C1369233253404-splitting%2Fva-p-hbase-01-c%252C60020%252C1369233253404.1369236857704 to final state done
  68. 2013-05-22 14:59:05,159 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker va-p-hbase-01-d,60020,1369233257617 done with task /hbase/splitlog/hdfs%3A%2F%2FHBaseNameService%2Fhbase%2F.logs%2Fva-p-hbase-01-c%2C60020%2C1369233253404-splitting%2Fva-p-hbase-01-c%252C60020%252C1369233253404.1369236857704 in 5824ms
  69. 2013-05-22 14:59:07,201 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: t,CJ6K\xFE\xFA;`Xn\x1E\x0F\"\xD9\x1B\x7F\xFF\xFCD\xD3r,\x1F,1365680880843.6c86914244889b033d248e0628f28dcd.
  70. 2013-05-22 14:59:07,219 INFO org.apache.hadoop.hbase.regionserver.HRegion: Setting up tabledescriptor config now ...
  71. 2013-05-22 14:59:07,229 INFO org.apache.hadoop.hbase.regionserver.Store: time to purge deletes set to 0ms in store u
  72. 2013-05-22 14:59:07,273 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW (CompoundBloomFilter) metadata for 0ecb2704f2ad4506a75c1ddda49879ab
  73. 2013-05-22 14:59:07,347 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW (CompoundBloomFilter) metadata for 3a8d6872c7824e4c9b6675616c785f8c
  74. 2013-05-22 14:59:07,591 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW (CompoundBloomFilter) metadata for ca43b1db2aa7468990cb2974eda28760
  75. 2013-05-22 14:59:07,614 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://HBaseNameService/hbase/t/6c86914244889b033d248e0628f28dcd/recovered.edits/0000000000579753730; minSequenceid=579366218; path=hdfs://HBaseNameService/hbase/t/6c86914244889b033d248e0628f28dcd/recovered.edits/0000000000579753730
  76. 2013-05-22 14:59:08,228 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://HBaseNameService/hbase/t/6c86914244889b033d248e0628f28dcd/recovered.edits/0000000000579949695; minSequenceid=579753730; path=hdfs://HBaseNameService/hbase/t/6c86914244889b033d248e0628f28dcd/recovered.edits/0000000000579949695
  77. 2013-05-22 14:59:08,665 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://HBaseNameService/hbase/t/6c86914244889b033d248e0628f28dcd/recovered.edits/0000000000580169826; minSequenceid=579949695; path=hdfs://HBaseNameService/hbase/t/6c86914244889b033d248e0628f28dcd/recovered.edits/0000000000580169826
  78. 2013-05-22 14:59:09,096 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://HBaseNameService/hbase/t/6c86914244889b033d248e0628f28dcd/recovered.edits/0000000000580388868; minSequenceid=580169826; path=hdfs://HBaseNameService/hbase/t/6c86914244889b033d248e0628f28dcd/recovered.edits/0000000000580388868
  79. 2013-05-22 14:59:09,581 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://HBaseNameService/hbase/t/6c86914244889b033d248e0628f28dcd/recovered.edits/0000000000580478758; minSequenceid=580388868; path=hdfs://HBaseNameService/hbase/t/6c86914244889b033d248e0628f28dcd/recovered.edits/0000000000580478758
  80. 2013-05-22 14:59:09,759 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom filter type for hdfs://HBaseNameService/hbase/t/6c86914244889b033d248e0628f28dcd/.tmp/8095060ea2704f378e4c20093f12b6e9: ROW, CompoundBloomFilterWriter
  81. 2013-05-22 14:59:09,759 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter type for hdfs://HBaseNameService/hbase/t/6c86914244889b033d248e0628f28dcd/.tmp/8095060ea2704f378e4c20093f12b6e9: CompoundBloomFilterWriter
  82. 2013-05-22 14:59:09,976 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server va-p-hbase-01-d,60020,1369233257617: Writing replication status
  83. org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/replication/rs/va-p-hbase-01-d,60020,1369233257617/1-va-p-hbase-01-c,60020,1369233253404/va-p-hbase-01-c%2C60020%2C1369233253404.1369233257396
  84. at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
  85. at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
  86. at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1266)
  87. at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.setData(RecoverableZooKeeper.java:354)
  88. at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:846)
  89. at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:898)
  90. at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:892)
  91. at org.apache.hadoop.hbase.replication.ReplicationZookeeper.writeReplicationStatus(ReplicationZookeeper.java:558)
  92. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.logPositionAndCleanOldLogs(ReplicationSourceManager.java:154)
  93. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:638)
  94. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:387)
  95.  
  96. ++++++++++++++++++++++++++++++++++++++++++++
  97. va-p-hbase-02-c
  98. ++++++++++++++++++++++++++++++++++++++++++++
  99. 2013-05-22 14:59:14,338 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager: Atomically moving va-p-hbase-01-e,60020,1369233254969's hlogs to my queue
  100. 2013-05-22 14:59:14,458 WARN org.apache.hadoop.hbase.replication.ReplicationZookeeper: Got exception in copyQueuesFromRSUsingMulti:
  101. org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
  102. at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
  103. at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:945)
  104. at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:911)
  105. at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.multi(RecoverableZooKeeper.java:538)
  106. at org.apache.hadoop.hbase.zookeeper.ZKUtil.multiOrSequential(ZKUtil.java:1457)
  107. at org.apache.hadoop.hbase.replication.ReplicationZookeeper.copyQueuesFromRSUsingMulti(ReplicationZookeeper.java:705)
  108. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$NodeFailoverWorker.run(ReplicationSourceManager.java:585)
  109. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  110. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  111. at java.lang.Thread.run(Thread.java:722)
  112. 2013-05-22 14:59:14,707 INFO org.apache.hadoop.ipc.HBaseRPC: Server at va-p-hbase-01-e/172.16.28.25:60020 could not be reached after 1 tries, giving up.
  113. 2013-05-22 14:59:15,719 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Processed 226957 edits across 15 regions threw away edits for 0 regions; log file=hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-d,60020,1369233257617-splitting/va-p-hbase-01-d%2C60020%2C1369233257617.1369236860339 is corrupted = false progress failed = false
  114. 2013-05-22 14:59:15,725 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: successfully transitioned task /hbase/splitlog/hdfs%3A%2F%2FHBaseNameService%2Fhbase%2F.logs%2Fva-p-hbase-01-d%2C60020%2C1369233257617-splitting%2Fva-p-hbase-01-d%252C60020%252C1369233257617.1369236860339 to final state done
  115. 2013-05-22 14:59:15,726 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker va-p-hbase-02-c,60020,1369233268385 done with task /hbase/splitlog/hdfs%3A%2F%2FHBaseNameService%2Fhbase%2F.logs%2Fva-p-hbase-01-d%2C60020%2C1369233257617-splitting%2Fva-p-hbase-01-d%252C60020%252C1369233257617.1369236860339 in 5409ms
  116. 2013-05-22 14:59:15,735 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker va-p-hbase-02-c,60020,1369233268385 acquired task /hbase/splitlog/hdfs%3A%2F%2FHBaseNameService%2Fhbase%2F.logs%2Fva-p-hbase-01-d%2C60020%2C1369233257617-splitting%2Fva-p-hbase-01-d%252C60020%252C1369233257617.1369244060972
  117. 2013-05-22 14:59:15,737 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog: hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-d,60020,1369233257617-splitting/va-p-hbase-01-d%2C60020%2C1369233257617.1369244060972, length=104757621
  118. 2013-05-22 14:59:15,737 INFO org.apache.hadoop.hbase.util.FSHDFSUtils: Recovering file hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-d,60020,1369233257617-splitting/va-p-hbase-01-d%2C60020%2C1369233257617.1369244060972
  119. 2013-05-22 14:59:16,755 INFO org.apache.hadoop.hbase.util.FSHDFSUtils: Finished lease recover attempt for hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-d,60020,1369233257617-splitting/va-p-hbase-01-d%2C60020%2C1369233257617.1369244060972
  120. 2013-05-22 14:59:16,757 INFO org.apache.hadoop.ipc.HBaseRPC: Server at va-p-hbase-01-e/172.16.28.25:60020 could not be reached after 1 tries, giving up.
  121. 2013-05-22 14:59:18,760 INFO org.apache.hadoop.ipc.HBaseRPC: Server at va-p-hbase-01-e/172.16.28.25:60020 could not be reached after 1 tries, giving up.
  122. 2013-05-22 14:59:20,769 INFO org.apache.hadoop.ipc.HBaseRPC: Server at va-p-hbase-01-e/172.16.28.25:60020 could not be reached after 1 tries, giving up.
  123. 2013-05-22 14:59:22,390 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Processed 249707 edits across 15 regions threw away edits for 0 regions; log file=hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-d,60020,1369233257617-splitting/va-p-hbase-01-d%2C60020%2C1369233257617.1369244060972 is corrupted = false progress failed = false
  124. 2013-05-22 14:59:22,397 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: successfully transitioned task /hbase/splitlog/hdfs%3A%2F%2FHBaseNameService%2Fhbase%2F.logs%2Fva-p-hbase-01-d%2C60020%2C1369233257617-splitting%2Fva-p-hbase-01-d%252C60020%252C1369233257617.1369244060972 to final state done
  125. 2013-05-22 14:59:22,397 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker va-p-hbase-02-c,60020,1369233268385 done with task /hbase/splitlog/hdfs%3A%2F%2FHBaseNameService%2Fhbase%2F.logs%2Fva-p-hbase-01-d%2C60020%2C1369233257617-splitting%2Fva-p-hbase-01-d%252C60020%252C1369233257617.1369244060972 in 6662ms
  126. 2013-05-22 14:59:22,419 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker va-p-hbase-02-c,60020,1369233268385 acquired task /hbase/splitlog/hdfs%3A%2F%2FHBaseNameService%2Fhbase%2F.logs%2Fva-p-hbase-01-e%2C60020%2C1369233254969-splitting%2Fva-p-hbase-01-e%252C60020%252C1369233254969.1369244058266
  127. 2013-05-22 14:59:22,421 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog: hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-e,60020,1369233254969-splitting/va-p-hbase-01-e%2C60020%2C1369233254969.1369244058266, length=119573166
  128. 2013-05-22 14:59:22,421 INFO org.apache.hadoop.hbase.util.FSHDFSUtils: Recovering file hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-e,60020,1369233254969-splitting/va-p-hbase-01-e%2C60020%2C1369233254969.1369244058266
  129. 2013-05-22 14:59:22,547 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: t,\x18\xCC\x93{[\x8E\xB4ZO\x16\xD4K\xC0Q\x03P\x7F\xFF\xFE\xC2/mw\xB4,1365678032208.cdbc00d110f5b05c25262b0461729be6.
  130. 2013-05-22 14:59:22,564 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: t,&\xFA&\xCFC\xE2\xD8.\x8Ea\xDF\xA5\xE1\x1A\x1B\x8A\x7F\xFF\xFE\xC2]<\xD8\xC7,1365666895911.bff3e0821851865d1d691cbbefebf8ea.
  131. 2013-05-22 14:59:22,726 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: t,\x98E\x0A\xCB\xBC\x88d%\xEA\xCCO\x85\xD1\xBF\xD7\x97\x7F\xFF\xFE\xC2G\x93\xA3=,1365635480906.0390a40c1bb87421164d3b736b40f941.
  132. 2013-05-22 14:59:22,745 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: t,\xA6t\xDF\x0E\xEC\xDA5\xEF\xA3\x0F!\xD4\x08v\x10x\x7F\xFF\xFE\xC24LA\,1365669900093.5e55b320c9d5fa247a14146e73decbf2.
  133. 2013-05-22 14:59:22,773 INFO org.apache.hadoop.ipc.HBaseRPC: Server at va-p-hbase-01-e/172.16.28.25:60020 could not be reached after 1 tries, giving up.
  134. 2013-05-22 14:59:22,894 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: t,\xC2\xC4\xD9;\xF6\x0C\x8A\x93m\xE3\xDCo>-\xE1\x16\x7F\xFF\xFCD\xD3r,\x1F,1365324796091.e5da4d15df46398aaefb4dfb2755f5a7.
  135. 2013-05-22 14:59:23,077 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: t,\xE9\xB8\xC78#\x0F\xEC\x90\xB8\x03\xE8\xDC\xCD\x02\x1C\xC9\x7F\xFF\xFE\xC2g\x8B;\x1D,1365636281939.87546db48b07085b6f7d8899a51e9cfb.
  136. 2013-05-22 14:59:23,092 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: t,\xF0\xD5UUUUUUUUUUUUUJ,1365533087697.121aebabdc8875ecd8c184ebe69c838c.
  137. 2013-05-22 14:59:23,423 INFO org.apache.hadoop.hbase.util.FSHDFSUtils: Finished lease recover attempt for hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-e,60020,1369233254969-splitting/va-p-hbase-01-e%2C60020%2C1369233254969.1369244058266
  138. 2013-05-22 14:59:24,351 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager: /hbase/rs/va-p-hbase-02-e,60020,1369233253407 znode expired, trying to lock it
  139. 2013-05-22 14:59:24,777 INFO org.apache.hadoop.ipc.HBaseRPC: Server at va-p-hbase-01-e/172.16.28.25:60020 could not be reached after 1 tries, giving up.
  140. 2013-05-22 14:59:26,351 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager: Atomically moving va-p-hbase-02-e,60020,1369233253407's hlogs to my queue
  141. 2013-05-22 14:59:26,379 WARN org.apache.hadoop.hbase.replication.ReplicationZookeeper: Got exception in copyQueuesFromRSUsingMulti:
  142. org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
  143. at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
  144. at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:945)
  145. at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:911)
  146. at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.multi(RecoverableZooKeeper.java:538)
  147. at org.apache.hadoop.hbase.zookeeper.ZKUtil.multiOrSequential(ZKUtil.java:1457)
  148. at org.apache.hadoop.hbase.replication.ReplicationZookeeper.copyQueuesFromRSUsingMulti(ReplicationZookeeper.java:705)
  149. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$NodeFailoverWorker.run(ReplicationSourceManager.java:585)
  150. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  151. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  152. at java.lang.Thread.run(Thread.java:722)
  153. 2013-05-22 14:59:26,383 INFO org.apache.hadoop.hbase.metrics: new MBeanInfo
  154. 2013-05-22 14:59:26,511 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Getting 1 rs from peer cluster # 1
  155. 2013-05-22 14:59:26,513 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Choosing peer CA-P-HBase-01-c.private.mtlink.biz,60020,1369249119538
  156. 2013-05-22 14:59:26,782 INFO org.apache.hadoop.ipc.HBaseRPC: Server at va-p-hbase-01-e/172.16.28.25:60020 could not be reached after 1 tries, giving up.
  157. 2013-05-22 14:59:27,709 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Replicating 8b11c86d-1ae8-419b-a492-ced43b65290d -> 2b509cc1-433d-41f6-98a2-6b8e7a9b2d32
  158. 2013-05-22 14:59:28,694 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Processed 275495 edits across 15 regions threw away edits for 0 regions; log file=hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-e,60020,1369233254969-splitting/va-p-hbase-01-e%2C60020%2C1369233254969.1369244058266 is corrupted = false progress failed = false
  159. 2013-05-22 14:59:28,700 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: successfully transitioned task /hbase/splitlog/hdfs%3A%2F%2FHBaseNameService%2Fhbase%2F.logs%2Fva-p-hbase-01-e%2C60020%2C1369233254969-splitting%2Fva-p-hbase-01-e%252C60020%252C1369233254969.1369244058266 to final state done
  160. 2013-05-22 14:59:28,700 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker va-p-hbase-02-c,60020,1369233268385 done with task /hbase/splitlog/hdfs%3A%2F%2FHBaseNameService%2Fhbase%2F.logs%2Fva-p-hbase-01-e%2C60020%2C1369233254969-splitting%2Fva-p-hbase-01-e%252C60020%252C1369233254969.1369244058266 in 6281ms
  161. 2013-05-22 14:59:28,707 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker va-p-hbase-02-c,60020,1369233268385 acquired task /hbase/splitlog/hdfs%3A%2F%2FHBaseNameService%2Fhbase%2F.logs%2Fva-p-hbase-01-e%2C60020%2C1369233254969-splitting%2Fva-p-hbase-01-e%252C60020%252C1369233254969.1369247658450
  162. 2013-05-22 14:59:28,709 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog: hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-e,60020,1369233254969-splitting/va-p-hbase-01-e%2C60020%2C1369233254969.1369247658450, length=0
  163. 2013-05-22 14:59:28,710 WARN org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: File hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-e,60020,1369233254969-splitting/va-p-hbase-01-e%2C60020%2C1369233254969.1369247658450 might be still open, length is 0
  164. 2013-05-22 14:59:28,710 INFO org.apache.hadoop.hbase.util.FSHDFSUtils: Recovering file hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-e,60020,1369233254969-splitting/va-p-hbase-01-e%2C60020%2C1369233254969.1369247658450
  165. 2013-05-22 14:59:28,786 INFO org.apache.hadoop.ipc.HBaseRPC: Server at va-p-hbase-01-e/172.16.28.25:60020 could not be reached after 1 tries, giving up.
  166. 2013-05-22 14:59:29,714 INFO org.apache.hadoop.hbase.util.FSHDFSUtils: Finished lease recover attempt for hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-e,60020,1369233254969-splitting/va-p-hbase-01-e%2C60020%2C1369233254969.1369247658450
  167. 2013-05-22 14:59:30,789 INFO org.apache.hadoop.ipc.HBaseRPC: Server at va-p-hbase-01-e/172.16.28.25:60020 could not be reached after 1 tries, giving up.
  168. 2013-05-22 14:59:31,707 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server va-p-hbase-02-c,60020,1369233268385: Writing replication status
  169. org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/replication/rs/va-p-hbase-02-c,60020,1369233268385/1-va-p-hbase-01-c,60020,1369233253404-va-p-hbase-02-e,60020,1369233253407/va-p-hbase-01-c%2C60020%2C1369233253404.1369233257396
  170. at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
  171. at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
  172. at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1266)
  173. at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.setData(RecoverableZooKeeper.java:354)
  174. at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:846)
  175. at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:898)
  176. at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:892)
  177. at org.apache.hadoop.hbase.replication.ReplicationZookeeper.writeReplicationStatus(ReplicationZookeeper.java:558)
  178. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.logPositionAndCleanOldLogs(ReplicationSourceManager.java:154)
  179. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:638)
  180. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:387)
  181.  
  182.  
  183. ++++++++++++++++++++++++++++++++++++++++++++
  184. va-p-hbase-02-d
  185. ++++++++++++++++++++++++++++++++++++++++++++
  186. 2013-05-22 14:58:43,083 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Getting 0 rs from peer cluster # 1
  187. 2013-05-22 14:58:43,154 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Getting 1 rs from peer cluster # 1
  188. 2013-05-22 14:58:43,154 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Choosing peer CA-P-HBase-03-b.private.mtlink.biz,60020,1369249119927
  189. 2013-05-22 14:58:44,166 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Getting 1 rs from peer cluster # 1
  190. 2013-05-22 14:58:44,166 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Choosing peer CA-P-HBase-01-b.private.mtlink.biz,60020,1369249119729
  191. 2013-05-22 14:58:44,166 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Getting 1 rs from peer cluster # 1
  192. 2013-05-22 14:58:44,166 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Choosing peer CA-P-HBase-02-c.private.mtlink.biz,60020,1369249119155
  193. 2013-05-22 14:58:44,166 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Getting 1 rs from peer cluster # 1
  194. 2013-05-22 14:58:44,167 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Choosing peer CA-P-HBase-02-c.private.mtlink.biz,60020,1369249119155
  195. 2013-05-22 14:58:44,167 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Getting 1 rs from peer cluster # 1
  196. 2013-05-22 14:58:44,167 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Getting 1 rs from peer cluster # 1
  197. 2013-05-22 14:58:44,167 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Choosing peer CA-P-HBase-01-c.private.mtlink.biz,60020,1369249119538
  198. 2013-05-22 14:58:44,167 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Choosing peer CA-P-HBase-02-c.private.mtlink.biz,60020,1369249119155
  199. 2013-05-22 14:58:44,167 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Getting 1 rs from peer cluster # 1
  200. 2013-05-22 14:58:44,167 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Choosing peer CA-P-HBase-03-b.private.mtlink.biz,60020,1369249119927
  201. 2013-05-22 14:58:44,346 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Replicating 8b11c86d-1ae8-419b-a492-ced43b65290d -> 2b509cc1-433d-41f6-98a2-6b8e7a9b2d32
  202. 2013-05-22 14:58:45,368 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Replicating 8b11c86d-1ae8-419b-a492-ced43b65290d -> 2b509cc1-433d-41f6-98a2-6b8e7a9b2d32
  203. 2013-05-22 14:58:45,369 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Replicating 8b11c86d-1ae8-419b-a492-ced43b65290d -> 2b509cc1-433d-41f6-98a2-6b8e7a9b2d32
  204. 2013-05-22 14:58:45,369 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Replicating 8b11c86d-1ae8-419b-a492-ced43b65290d -> 2b509cc1-433d-41f6-98a2-6b8e7a9b2d32
  205. 2013-05-22 14:58:45,369 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Replicating 8b11c86d-1ae8-419b-a492-ced43b65290d -> 2b509cc1-433d-41f6-98a2-6b8e7a9b2d32
  206. 2013-05-22 14:58:45,369 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Replicating 8b11c86d-1ae8-419b-a492-ced43b65290d -> 2b509cc1-433d-41f6-98a2-6b8e7a9b2d32
  207. 2013-05-22 14:58:45,369 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Replicating 8b11c86d-1ae8-419b-a492-ced43b65290d -> 2b509cc1-433d-41f6-98a2-6b8e7a9b2d32
  208. 2013-05-22 14:58:59,065 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager: /hbase/rs/va-p-hbase-01-c,60020,1369233253404 znode expired, trying to lock it
  209. 2013-05-22 14:58:59,264 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker va-p-hbase-02-d,60020,1369233252475 acquired task /hbase/splitlog/hdfs%3A%2F%2FHBaseNameService%2Fhbase%2F.logs%2Fva-p-hbase-01-c%2C60020%2C1369233253404-splitting%2Fva-p-hbase-01-c%252C60020%252C1369233253404.1369244058363
  210. 2013-05-22 14:58:59,314 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog: hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-c,60020,1369233253404-splitting/va-p-hbase-01-c%2C60020%2C1369233253404.1369244058363, length=89549472
  211. 2013-05-22 14:58:59,317 INFO org.apache.hadoop.hbase.util.FSHDFSUtils: Recovering file hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-c,60020,1369233253404-splitting/va-p-hbase-01-c%2C60020%2C1369233253404.1369244058363
  212. 2013-05-22 14:59:00,375 INFO org.apache.hadoop.hbase.util.FSHDFSUtils: Finished lease recover attempt for hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-c,60020,1369233253404-splitting/va-p-hbase-01-c%2C60020%2C1369233253404.1369244058363
  213. 2013-05-22 14:59:00,682 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  214. 2013-05-22 14:59:00,710 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  215. 2013-05-22 14:59:00,842 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  216. 2013-05-22 14:59:00,871 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  217. 2013-05-22 14:59:00,898 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  218. 2013-05-22 14:59:01,014 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  219. 2013-05-22 14:59:01,048 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  220. 2013-05-22 14:59:01,066 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager: Atomically moving va-p-hbase-01-c,60020,1369233253404's hlogs to my queue
  221. 2013-05-22 14:59:01,194 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  222. 2013-05-22 14:59:01,226 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  223. 2013-05-22 14:59:01,262 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  224. 2013-05-22 14:59:01,294 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  225. 2013-05-22 14:59:01,358 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  226. 2013-05-22 14:59:01,410 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  227. 2013-05-22 14:59:07,050 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Processed 219038 edits across 14 regions threw away edits for 0 regions; log file=hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-c,60020,1369233253404-splitting/va-p-hbase-01-c%2C60020%2C1369233253404.1369244058363 is corrupted = false progress failed = false
  228. 2013-05-22 14:59:07,078 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: successfully transitioned task /hbase/splitlog/hdfs%3A%2F%2FHBaseNameService%2Fhbase%2F.logs%2Fva-p-hbase-01-c%2C60020%2C1369233253404-splitting%2Fva-p-hbase-01-c%252C60020%252C1369233253404.1369244058363 to final state done
  229. 2013-05-22 14:59:07,078 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker va-p-hbase-02-d,60020,1369233252475 done with task /hbase/splitlog/hdfs%3A%2F%2FHBaseNameService%2Fhbase%2F.logs%2Fva-p-hbase-01-c%2C60020%2C1369233253404-splitting%2Fva-p-hbase-01-c%252C60020%252C1369233253404.1369244058363 in 7813ms
  230. 2013-05-22 14:59:07,221 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: t,T\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFC,1365676551323.877df3f346db7508323b5169f80c74f9.
  231. 2013-05-22 14:59:07,236 INFO org.apache.hadoop.hbase.regionserver.HRegion: Setting up tabledescriptor config now ...
  232. 2013-05-22 14:59:07,239 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: t,c*\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xA6,1365673449134.1e393bfc6f2ffc5854fd45dc7343491a.
  233. 2013-05-22 14:59:07,248 INFO org.apache.hadoop.hbase.regionserver.Store: time to purge deletes set to 0ms in store u
  234. 2013-05-22 14:59:07,257 INFO org.apache.hadoop.hbase.regionserver.HRegion: Setting up tabledescriptor config now ...
  235. 2013-05-22 14:59:07,258 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: t,\x91+\xBB\xB9?\x07\x8F5j%\xCE\xE4\xFB\xB8\x03\xF5\x7F\xFF\xFE\xC2<\xE9e\xA2,1365678702133.004cd73ae42ccb7939221962ad9f0683.
  236. 2013-05-22 14:59:07,272 INFO org.apache.hadoop.hbase.regionserver.HRegion: Setting up tabledescriptor config now ...
  237. 2013-05-22 14:59:07,279 INFO org.apache.hadoop.hbase.regionserver.Store: time to purge deletes set to 0ms in store u
  238. 2013-05-22 14:59:07,295 INFO org.apache.hadoop.hbase.regionserver.Store: time to purge deletes set to 0ms in store u
  239. 2013-05-22 14:59:07,307 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: t,\xB4\xA2\xC0n\x8A\x95\xF2\x83\xCC\xF7X\xCFv\x166\xBD\x7F\xFF\xFCD\xD3r,\x1F,1365679577226.307e5da004a9e527ec368acfd727741c.
  240. 2013-05-22 14:59:07,324 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW (CompoundBloomFilter) metadata for 48e78439dc7f460396a60d0c5b95732f
  241. 2013-05-22 14:59:07,325 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: t,\xC6UUUUUUUUUUUUUUL,1365674741137.b7dc6ebbebc1fca27d231266521a360f.
  242. 2013-05-22 14:59:07,342 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: t,\xD4\x7F\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xF6,1365671116522.7f464acf4e552ce69504e0ad6d6b7a53.
  243. 2013-05-22 14:59:07,359 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: x,\x15@\x00\x00,1363541667782.3ef27c7fad63ecb087890bf79b0d1553.
  244. 2013-05-22 14:59:07,374 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: x,?\xC0\x00\x00,1363541667783.4b0b58047668240ac9bb4d85fdf45c4a.
  245. 2013-05-22 14:59:07,388 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW (CompoundBloomFilter) metadata for 59d8a9e9059d48a8a1e4f9e08eb8c980
  246. 2013-05-22 14:59:07,432 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW (CompoundBloomFilter) metadata for f0fc71cf1d8c4594b66faa7decc344a9
  247. 2013-05-22 14:59:07,531 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://HBaseNameService/hbase/t/004cd73ae42ccb7939221962ad9f0683/recovered.edits/0000000000579753729; minSequenceid=579366208; path=hdfs://HBaseNameService/hbase/t/004cd73ae42ccb7939221962ad9f0683/recovered.edits/0000000000579753729
  248. 2013-05-22 14:59:07,582 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW (CompoundBloomFilter) metadata for 9b98d79e23684461b0268264882aad11
  249. 2013-05-22 14:59:07,606 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://HBaseNameService/hbase/t/877df3f346db7508323b5169f80c74f9/recovered.edits/0000000000579753717; minSequenceid=579366161; path=hdfs://HBaseNameService/hbase/t/877df3f346db7508323b5169f80c74f9/recovered.edits/0000000000579753717
  250. 2013-05-22 14:59:07,923 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW (CompoundBloomFilter) metadata for 1a68af2a3efd4889b80a7ed11eccf24b
  251. 2013-05-22 14:59:07,995 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://HBaseNameService/hbase/t/1e393bfc6f2ffc5854fd45dc7343491a/recovered.edits/0000000000579753718; minSequenceid=579366212; path=hdfs://HBaseNameService/hbase/t/1e393bfc6f2ffc5854fd45dc7343491a/recovered.edits/0000000000579753718
  252. 2013-05-22 14:59:08,084 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://HBaseNameService/hbase/t/004cd73ae42ccb7939221962ad9f0683/recovered.edits/0000000000579949727; minSequenceid=579753729; path=hdfs://HBaseNameService/hbase/t/004cd73ae42ccb7939221962ad9f0683/recovered.edits/0000000000579949727
  253. 2013-05-22 14:59:08,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://HBaseNameService/hbase/t/877df3f346db7508323b5169f80c74f9/recovered.edits/0000000000579949717; minSequenceid=579753717; path=hdfs://HBaseNameService/hbase/t/877df3f346db7508323b5169f80c74f9/recovered.edits/0000000000579949717
  254. 2013-05-22 14:59:08,569 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://HBaseNameService/hbase/t/1e393bfc6f2ffc5854fd45dc7343491a/recovered.edits/0000000000579949682; minSequenceid=579753718; path=hdfs://HBaseNameService/hbase/t/1e393bfc6f2ffc5854fd45dc7343491a/recovered.edits/0000000000579949682
  255. 2013-05-22 14:59:08,569 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://HBaseNameService/hbase/t/004cd73ae42ccb7939221962ad9f0683/recovered.edits/0000000000580169806; minSequenceid=579949727; path=hdfs://HBaseNameService/hbase/t/004cd73ae42ccb7939221962ad9f0683/recovered.edits/0000000000580169806
  256. 2013-05-22 14:59:08,746 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://HBaseNameService/hbase/t/877df3f346db7508323b5169f80c74f9/recovered.edits/0000000000580169820; minSequenceid=579949717; path=hdfs://HBaseNameService/hbase/t/877df3f346db7508323b5169f80c74f9/recovered.edits/0000000000580169820
  257. 2013-05-22 14:59:09,129 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://HBaseNameService/hbase/t/1e393bfc6f2ffc5854fd45dc7343491a/recovered.edits/0000000000580169811; minSequenceid=579949682; path=hdfs://HBaseNameService/hbase/t/1e393bfc6f2ffc5854fd45dc7343491a/recovered.edits/0000000000580169811
  258. 2013-05-22 14:59:09,442 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://HBaseNameService/hbase/t/877df3f346db7508323b5169f80c74f9/recovered.edits/0000000000580388880; minSequenceid=580169820; path=hdfs://HBaseNameService/hbase/t/877df3f346db7508323b5169f80c74f9/recovered.edits/0000000000580388880
  259. 2013-05-22 14:59:09,510 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://HBaseNameService/hbase/t/004cd73ae42ccb7939221962ad9f0683/recovered.edits/0000000000580388876; minSequenceid=580169806; path=hdfs://HBaseNameService/hbase/t/004cd73ae42ccb7939221962ad9f0683/recovered.edits/0000000000580388876
  260. 2013-05-22 14:59:10,232 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager: /hbase/rs/va-p-hbase-01-d,60020,1369233257617 znode expired, trying to lock it
  261. 2013-05-22 14:59:10,372 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker va-p-hbase-02-d,60020,1369233252475 acquired task /hbase/splitlog/hdfs%3A%2F%2FHBaseNameService%2Fhbase%2F.logs%2Fva-p-hbase-01-d%2C60020%2C1369233257617-splitting%2Fva-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  262. 2013-05-22 14:59:10,375 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog: hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-d,60020,1369233257617-splitting/va-p-hbase-01-d%2C60020%2C1369233257617.1369233259640, length=83964033
  263. 2013-05-22 14:59:10,375 INFO org.apache.hadoop.hbase.util.FSHDFSUtils: Recovering file hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-d,60020,1369233257617-splitting/va-p-hbase-01-d%2C60020%2C1369233257617.1369233259640
  264. 2013-05-22 14:59:10,399 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://HBaseNameService/hbase/t/1e393bfc6f2ffc5854fd45dc7343491a/recovered.edits/0000000000580388873; minSequenceid=580169811; path=hdfs://HBaseNameService/hbase/t/1e393bfc6f2ffc5854fd45dc7343491a/recovered.edits/0000000000580388873
  265. 2013-05-22 14:59:10,583 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager: /hbase/rs/va-p-hbase-01-e,60020,1369233254969 znode expired, trying to lock it
  266. 2013-05-22 14:59:10,627 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://HBaseNameService/hbase/t/877df3f346db7508323b5169f80c74f9/recovered.edits/0000000000580478761; minSequenceid=580388880; path=hdfs://HBaseNameService/hbase/t/877df3f346db7508323b5169f80c74f9/recovered.edits/0000000000580478761
  267. 2013-05-22 14:59:11,159 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://HBaseNameService/hbase/t/004cd73ae42ccb7939221962ad9f0683/recovered.edits/0000000000580478738; minSequenceid=580388876; path=hdfs://HBaseNameService/hbase/t/004cd73ae42ccb7939221962ad9f0683/recovered.edits/0000000000580478738
  268. 2013-05-22 14:59:11,378 INFO org.apache.hadoop.hbase.util.FSHDFSUtils: Finished lease recover attempt for hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-d,60020,1369233257617-splitting/va-p-hbase-01-d%2C60020%2C1369233257617.1369233259640
  269. 2013-05-22 14:59:11,599 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom filter type for hdfs://HBaseNameService/hbase/t/877df3f346db7508323b5169f80c74f9/.tmp/27c94da41a5645718befd10941e3ce3f: ROW, CompoundBloomFilterWriter
  270. 2013-05-22 14:59:11,600 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter type for hdfs://HBaseNameService/hbase/t/877df3f346db7508323b5169f80c74f9/.tmp/27c94da41a5645718befd10941e3ce3f: CompoundBloomFilterWriter
  271. 2013-05-22 14:59:11,801 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom filter type for hdfs://HBaseNameService/hbase/t/004cd73ae42ccb7939221962ad9f0683/.tmp/826f9294a2a546e8b3b30bf8d162c02f: ROW, CompoundBloomFilterWriter
  272. 2013-05-22 14:59:11,801 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter type for hdfs://HBaseNameService/hbase/t/004cd73ae42ccb7939221962ad9f0683/.tmp/826f9294a2a546e8b3b30bf8d162c02f: CompoundBloomFilterWriter
  273. 2013-05-22 14:59:12,069 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://HBaseNameService/hbase/t/1e393bfc6f2ffc5854fd45dc7343491a/recovered.edits/0000000000580478725; minSequenceid=580388873; path=hdfs://HBaseNameService/hbase/t/1e393bfc6f2ffc5854fd45dc7343491a/recovered.edits/0000000000580478725
  274. 2013-05-22 14:59:12,266 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager: Atomically moving va-p-hbase-01-d,60020,1369233257617's hlogs to my queue
  275. 2013-05-22 14:59:12,292 INFO org.apache.hadoop.hbase.replication.ReplicationZookeeper: Atomically moved the dead regionserver logs.
  276. 2013-05-22 14:59:12,295 INFO org.apache.hadoop.hbase.metrics: new MBeanInfo
  277. 2013-05-22 14:59:12,352 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  278. 2013-05-22 14:59:12,412 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Getting 1 rs from peer cluster # 1
  279. 2013-05-22 14:59:12,413 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Choosing peer CA-P-HBase-02-b.private.mtlink.biz,60020,1369249119310
  280. 2013-05-22 14:59:12,524 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  281. 2013-05-22 14:59:12,568 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  282. 2013-05-22 14:59:13,053 INFO org.apache.hadoop.hbase.regionserver.StoreFile: General Bloom and NO DeleteFamily was added to HFile (hdfs://HBaseNameService/hbase/t/004cd73ae42ccb7939221962ad9f0683/.tmp/826f9294a2a546e8b3b30bf8d162c02f)
  283. 2013-05-22 14:59:13,053 INFO org.apache.hadoop.hbase.regionserver.Store: Flushed , sequenceid=580478738, memsize=30.7m, into tmp file hdfs://HBaseNameService/hbase/t/004cd73ae42ccb7939221962ad9f0683/.tmp/826f9294a2a546e8b3b30bf8d162c02f
  284. 2013-05-22 14:59:13,060 INFO org.apache.hadoop.hbase.regionserver.StoreFile: General Bloom and NO DeleteFamily was added to HFile (hdfs://HBaseNameService/hbase/t/877df3f346db7508323b5169f80c74f9/.tmp/27c94da41a5645718befd10941e3ce3f)
  285. 2013-05-22 14:59:13,060 INFO org.apache.hadoop.hbase.regionserver.Store: Flushed , sequenceid=580478761, memsize=31.7m, into tmp file hdfs://HBaseNameService/hbase/t/877df3f346db7508323b5169f80c74f9/.tmp/27c94da41a5645718befd10941e3ce3f
  286. 2013-05-22 14:59:13,080 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW (CompoundBloomFilter) metadata for 27c94da41a5645718befd10941e3ce3f
  287. 2013-05-22 14:59:13,107 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW (CompoundBloomFilter) metadata for 826f9294a2a546e8b3b30bf8d162c02f
  288. 2013-05-22 14:59:13,114 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW (CompoundBloomFilter) metadata for 27c94da41a5645718befd10941e3ce3f
  289. 2013-05-22 14:59:13,114 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://HBaseNameService/hbase/t/877df3f346db7508323b5169f80c74f9/u/27c94da41a5645718befd10941e3ce3f, entries=105827, sequenceid=580478761, filesize=13.9m
  290. 2013-05-22 14:59:13,114 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~52.0m/54545608, currentsize=0/0 for region t,T\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFC,1365676551323.877df3f346db7508323b5169f80c74f9. in 1660ms, sequenceid=580478761, compaction requested=true; wal=null
  291. 2013-05-22 14:59:13,121 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom filter type for hdfs://HBaseNameService/hbase/t/1e393bfc6f2ffc5854fd45dc7343491a/.tmp/4788b4740f1e4c5ebe2b2b2d7a9e5333: ROW, CompoundBloomFilterWriter
  292. 2013-05-22 14:59:13,121 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter type for hdfs://HBaseNameService/hbase/t/1e393bfc6f2ffc5854fd45dc7343491a/.tmp/4788b4740f1e4c5ebe2b2b2d7a9e5333: CompoundBloomFilterWriter
  293. 2013-05-22 14:59:13,150 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW (CompoundBloomFilter) metadata for 826f9294a2a546e8b3b30bf8d162c02f
  294. 2013-05-22 14:59:13,150 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://HBaseNameService/hbase/t/004cd73ae42ccb7939221962ad9f0683/u/826f9294a2a546e8b3b30bf8d162c02f, entries=102492, sequenceid=580478738, filesize=13.5m
  295. 2013-05-22 14:59:13,151 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~49.7m/52133408, currentsize=0/0 for region t,\x91+\xBB\xB9?\x07\x8F5j%\xCE\xE4\xFB\xB8\x03\xF5\x7F\xFF\xFE\xC2<\xE9e\xA2,1365678702133.004cd73ae42ccb7939221962ad9f0683. in 1418ms, sequenceid=580478738, compaction requested=false; wal=null
  296. 2013-05-22 14:59:13,157 INFO org.apache.hadoop.hbase.regionserver.HRegion: Onlined t,T\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFC,1365676551323.877df3f346db7508323b5169f80c74f9.; next sequenceid=580478762
  297. 2013-05-22 14:59:13,267 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks for region=t,T\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFC,1365676551323.877df3f346db7508323b5169f80c74f9., daughter=false
  298. 2013-05-22 14:59:13,271 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on u in region t,T\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFC,1365676551323.877df3f346db7508323b5169f80c74f9.
  299. 2013-05-22 14:59:13,274 INFO org.apache.hadoop.hbase.regionserver.Store: Starting compaction of 3 file(s) in u of t,T\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFC,1365676551323.877df3f346db7508323b5169f80c74f9. into tmpdir=hdfs://HBaseNameService/hbase/t/877df3f346db7508323b5169f80c74f9/.tmp, seqid=580478761, totalSize=42.3m
  300. 2013-05-22 14:59:13,289 INFO org.apache.hadoop.hbase.regionserver.HRegion: Onlined t,\x91+\xBB\xB9?\x07\x8F5j%\xCE\xE4\xFB\xB8\x03\xF5\x7F\xFF\xFE\xC2<\xE9e\xA2,1365678702133.004cd73ae42ccb7939221962ad9f0683.; next sequenceid=580478739
  301. 2013-05-22 14:59:13,305 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks for region=t,\x91+\xBB\xB9?\x07\x8F5j%\xCE\xE4\xFB\xB8\x03\xF5\x7F\xFF\xFE\xC2<\xE9e\xA2,1365678702133.004cd73ae42ccb7939221962ad9f0683., daughter=false
  302. 2013-05-22 14:59:13,369 INFO org.apache.hadoop.ipc.HBaseRPC: Server at va-p-hbase-01-e/172.16.28.25:60020 could not be reached after 1 tries, giving up.
  303. 2013-05-22 14:59:13,394 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom filter type for hdfs://HBaseNameService/hbase/t/877df3f346db7508323b5169f80c74f9/.tmp/ed6bdfb4f7d046beb8581573c1cf78a8: ROW, CompoundBloomFilterWriter
  304. 2013-05-22 14:59:13,394 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter type for hdfs://HBaseNameService/hbase/t/877df3f346db7508323b5169f80c74f9/.tmp/ed6bdfb4f7d046beb8581573c1cf78a8: CompoundBloomFilterWriter
  305. 2013-05-22 14:59:13,597 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Replicating 8b11c86d-1ae8-419b-a492-ced43b65290d -> 2b509cc1-433d-41f6-98a2-6b8e7a9b2d32
  306. 2013-05-22 14:59:13,611 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: NB dead servers : 5
  307. 2013-05-22 14:59:13,611 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/d,60020,1369233257617/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  308. 2013-05-22 14:59:13,614 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/d,60020,1369233257617-splitting/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  309. 2013-05-22 14:59:13,617 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  310. 2013-05-22 14:59:13,619 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01-splitting/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  311. 2013-05-22 14:59:13,622 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  312. 2013-05-22 14:59:13,626 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase-splitting/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  313. 2013-05-22 14:59:13,628 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  314. 2013-05-22 14:59:13,630 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p-splitting/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  315. 2013-05-22 14:59:13,634 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  316. 2013-05-22 14:59:13,636 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va-splitting/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  317. 2013-05-22 14:59:13,663 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1-va-p-hbase-01-d,60020,1369233257617 Got:
  318. java.io.IOException: File from recovered queue is nowhere to be found
  319. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:535)
  320. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:312)
  321. Caused by: java.io.FileNotFoundException: File does not exist: hdfs://HBaseNameService/hbase/.oldlogs/va-p-hbase-01-d%2C60020%2C1369233257617.1369233259640
  322. at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:807)
  323. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1704)
  324. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
  325. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
  326. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
  327. at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:720)
  328. at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:66)
  329. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:501)
  330. ... 1 more
  331. 2013-05-22 14:59:14,188 INFO org.apache.hadoop.hbase.regionserver.StoreFile: General Bloom and NO DeleteFamily was added to HFile (hdfs://HBaseNameService/hbase/t/1e393bfc6f2ffc5854fd45dc7343491a/.tmp/4788b4740f1e4c5ebe2b2b2d7a9e5333)
  332. 2013-05-22 14:59:14,188 INFO org.apache.hadoop.hbase.regionserver.Store: Flushed , sequenceid=580478725, memsize=31.9m, into tmp file hdfs://HBaseNameService/hbase/t/1e393bfc6f2ffc5854fd45dc7343491a/.tmp/4788b4740f1e4c5ebe2b2b2d7a9e5333
  333. 2013-05-22 14:59:14,206 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW (CompoundBloomFilter) metadata for 4788b4740f1e4c5ebe2b2b2d7a9e5333
  334. 2013-05-22 14:59:14,329 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager: Atomically moving va-p-hbase-01-e,60020,1369233254969's hlogs to my queue
  335. 2013-05-22 14:59:14,352 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW (CompoundBloomFilter) metadata for 4788b4740f1e4c5ebe2b2b2d7a9e5333
  336. 2013-05-22 14:59:14,352 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://HBaseNameService/hbase/t/1e393bfc6f2ffc5854fd45dc7343491a/u/4788b4740f1e4c5ebe2b2b2d7a9e5333, entries=106368, sequenceid=580478725, filesize=14.0m
  337. 2013-05-22 14:59:14,352 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~51.9m/54379888, currentsize=0/0 for region t,c*\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xA6,1365673449134.1e393bfc6f2ffc5854fd45dc7343491a. in 1298ms, sequenceid=580478725, compaction requested=false; wal=null
  338. 2013-05-22 14:59:14,365 WARN org.apache.hadoop.hbase.replication.ReplicationZookeeper: Got exception in copyQueuesFromRSUsingMulti:
  339. org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
  340. at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
  341. at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:945)
  342. at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:911)
  343. at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.multi(RecoverableZooKeeper.java:538)
  344. at org.apache.hadoop.hbase.zookeeper.ZKUtil.multiOrSequential(ZKUtil.java:1457)
  345. at org.apache.hadoop.hbase.replication.ReplicationZookeeper.copyQueuesFromRSUsingMulti(ReplicationZookeeper.java:705)
  346. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$NodeFailoverWorker.run(ReplicationSourceManager.java:585)
  347. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  348. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  349. at java.lang.Thread.run(Thread.java:722)
  350. 2013-05-22 14:59:14,367 INFO org.apache.hadoop.hbase.metrics: new MBeanInfo
  351. 2013-05-22 14:59:14,396 INFO org.apache.hadoop.hbase.regionserver.HRegion: Onlined t,c*\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xA6,1365673449134.1e393bfc6f2ffc5854fd45dc7343491a.; next sequenceid=580478726
  352. 2013-05-22 14:59:14,413 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks for region=t,c*\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xAA\xA6,1365673449134.1e393bfc6f2ffc5854fd45dc7343491a., daughter=false
  353. 2013-05-22 14:59:14,553 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Getting 1 rs from peer cluster # 1
  354. 2013-05-22 14:59:14,553 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Choosing peer CA-P-HBase-01-b.private.mtlink.biz,60020,1369249119729
  355. 2013-05-22 14:59:14,755 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: NB dead servers : 5
  356. 2013-05-22 14:59:14,755 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/d,60020,1369233257617/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  357. 2013-05-22 14:59:14,759 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/d,60020,1369233257617-splitting/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  358. 2013-05-22 14:59:14,764 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  359. 2013-05-22 14:59:14,767 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01-splitting/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  360. 2013-05-22 14:59:14,770 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  361. 2013-05-22 14:59:14,774 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase-splitting/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  362. 2013-05-22 14:59:14,776 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  363. 2013-05-22 14:59:14,780 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p-splitting/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  364. 2013-05-22 14:59:14,783 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  365. 2013-05-22 14:59:14,786 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va-splitting/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  366. 2013-05-22 14:59:14,789 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1-va-p-hbase-01-d,60020,1369233257617 Got:
  367. java.io.IOException: File from recovered queue is nowhere to be found
  368. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:535)
  369. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:312)
  370. Caused by: java.io.FileNotFoundException: File does not exist: hdfs://HBaseNameService/hbase/.oldlogs/va-p-hbase-01-d%2C60020%2C1369233257617.1369233259640
  371. at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:807)
  372. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1704)
  373. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
  374. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
  375. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
  376. at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:720)
  377. at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:66)
  378. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:501)
  379. ... 1 more
  380. 2013-05-22 14:59:15,385 INFO org.apache.hadoop.ipc.HBaseRPC: Server at va-p-hbase-01-e/172.16.28.25:60020 could not be reached after 1 tries, giving up.
  381. 2013-05-22 14:59:15,802 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Replicating 8b11c86d-1ae8-419b-a492-ced43b65290d -> 2b509cc1-433d-41f6-98a2-6b8e7a9b2d32
  382. 2013-05-22 14:59:15,811 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: NB dead servers : 5
  383. 2013-05-22 14:59:15,811 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/e,60020,1369233254969/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  384. 2013-05-22 14:59:15,814 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/e,60020,1369233254969-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  385. 2013-05-22 14:59:15,816 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  386. 2013-05-22 14:59:15,819 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  387. 2013-05-22 14:59:15,898 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  388. 2013-05-22 14:59:15,905 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  389. 2013-05-22 14:59:15,910 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  390. 2013-05-22 14:59:15,914 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  391. 2013-05-22 14:59:15,917 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  392. 2013-05-22 14:59:15,921 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  393. 2013-05-22 14:59:15,925 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1-va-p-hbase-01-e,60020,1369233254969 Got:
  394. java.io.IOException: File from recovered queue is nowhere to be found
  395. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:535)
  396. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:312)
  397. Caused by: java.io.FileNotFoundException: File does not exist: hdfs://HBaseNameService/hbase/.oldlogs/va-p-hbase-01-e%2C60020%2C1369233254969.1369233257503
  398. at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:807)
  399. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1704)
  400. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
  401. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
  402. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
  403. at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:720)
  404. at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:66)
  405. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:501)
  406. ... 1 more
  407. 2013-05-22 14:59:16,840 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: NB dead servers : 5
  408. 2013-05-22 14:59:16,840 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/d,60020,1369233257617/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  409. 2013-05-22 14:59:16,842 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/d,60020,1369233257617-splitting/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  410. 2013-05-22 14:59:16,845 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  411. 2013-05-22 14:59:16,847 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01-splitting/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  412. 2013-05-22 14:59:16,850 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  413. 2013-05-22 14:59:16,853 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase-splitting/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  414. 2013-05-22 14:59:16,856 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  415. 2013-05-22 14:59:16,858 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p-splitting/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  416. 2013-05-22 14:59:16,861 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  417. 2013-05-22 14:59:16,864 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va-splitting/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  418. 2013-05-22 14:59:16,867 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1-va-p-hbase-01-d,60020,1369233257617 Got:
  419. java.io.IOException: File from recovered queue is nowhere to be found
  420. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:535)
  421. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:312)
  422. Caused by: java.io.FileNotFoundException: File does not exist: hdfs://HBaseNameService/hbase/.oldlogs/va-p-hbase-01-d%2C60020%2C1369233257617.1369233259640
  423. at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:807)
  424. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1704)
  425. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
  426. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
  427. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
  428. at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:720)
  429. at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:66)
  430. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:501)
  431. ... 1 more
  432. 2013-05-22 14:59:16,929 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: NB dead servers : 5
  433. 2013-05-22 14:59:16,929 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/e,60020,1369233254969/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  434. 2013-05-22 14:59:16,932 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/e,60020,1369233254969-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  435. 2013-05-22 14:59:16,935 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  436. 2013-05-22 14:59:16,937 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  437. 2013-05-22 14:59:16,941 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  438. 2013-05-22 14:59:16,943 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  439. 2013-05-22 14:59:16,946 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  440. 2013-05-22 14:59:16,950 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  441. 2013-05-22 14:59:16,953 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  442. 2013-05-22 14:59:16,956 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  443. 2013-05-22 14:59:16,959 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1-va-p-hbase-01-e,60020,1369233254969 Got:
  444. java.io.IOException: File from recovered queue is nowhere to be found
  445. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:535)
  446. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:312)
  447. Caused by: java.io.FileNotFoundException: File does not exist: hdfs://HBaseNameService/hbase/.oldlogs/va-p-hbase-01-e%2C60020%2C1369233254969.1369233257503
  448. at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:807)
  449. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1704)
  450. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
  451. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
  452. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
  453. at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:720)
  454. at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:66)
  455. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:501)
  456. ... 1 more
  457. 2013-05-22 14:59:17,451 INFO org.apache.hadoop.ipc.HBaseRPC: Server at va-p-hbase-01-e/172.16.28.25:60020 could not be reached after 1 tries, giving up.
  458. 2013-05-22 14:59:17,518 INFO org.apache.hadoop.hbase.regionserver.StoreFile: General Bloom and NO DeleteFamily was added to HFile (hdfs://HBaseNameService/hbase/t/877df3f346db7508323b5169f80c74f9/.tmp/ed6bdfb4f7d046beb8581573c1cf78a8)
  459. 2013-05-22 14:59:17,538 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW (CompoundBloomFilter) metadata for ed6bdfb4f7d046beb8581573c1cf78a8
  460. 2013-05-22 14:59:17,557 INFO org.apache.hadoop.hbase.regionserver.Store: Renaming compacted file at hdfs://HBaseNameService/hbase/t/877df3f346db7508323b5169f80c74f9/.tmp/ed6bdfb4f7d046beb8581573c1cf78a8 to hdfs://HBaseNameService/hbase/t/877df3f346db7508323b5169f80c74f9/u/ed6bdfb4f7d046beb8581573c1cf78a8
  461. 2013-05-22 14:59:17,585 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW (CompoundBloomFilter) metadata for ed6bdfb4f7d046beb8581573c1cf78a8
  462. 2013-05-22 14:59:17,801 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 3 file(s) in u of t,T\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFC,1365676551323.877df3f346db7508323b5169f80c74f9. into ed6bdfb4f7d046beb8581573c1cf78a8, size=41.9m; total size for store is 2.9g
  463. 2013-05-22 14:59:17,801 INFO org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: completed compaction: regionName=t,T\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFC,1365676551323.877df3f346db7508323b5169f80c74f9., storeName=u, fileCount=3, fileSize=42.3m, priority=3, time=6851758289452066; duration=4sec
  464. 2013-05-22 14:59:18,404 INFO org.apache.hadoop.hbase.replication.regionserver.Replication: Normal source for cluster 1: Total replicated edits: 100000, currently replicating from: hdfs://HBaseNameService/hbase/.logs/va-p-hbase-02-d,60020,1369233252475/va-p-hbase-02-d%2C60020%2C1369233252475.1369233257398 at position: 43575152
  465. Recovered source for cluster/machine(s) 1: Total replicated edits: 125000, currently replicating from: hdfs://HBaseNameService/hbase/.oldlogs/va-p-hbase-01-c%2C60020%2C1369042378287.1369223650837 at position: 53567414
  466. Recovered source for cluster/machine(s) 1: Total replicated edits: 100000, currently replicating from: hdfs://HBaseNameService/hbase/.oldlogs/va-p-hbase-01-d%2C60020%2C1369042382584.1369222441570 at position: 43319077
  467. Recovered source for cluster/machine(s) 1: Total replicated edits: 125000, currently replicating from: hdfs://HBaseNameService/hbase/.oldlogs/va-p-hbase-01-e%2C60020%2C1369042368595.1369225661802 at position: 52807702
  468. Recovered source for cluster/machine(s) 1: Total replicated edits: 100000, currently replicating from: hdfs://HBaseNameService/hbase/.oldlogs/va-p-hbase-02-c%2C60020%2C1369042377731.1369223039701 at position: 43083315
  469. Recovered source for cluster/machine(s) 1: Total replicated edits: 100000, currently replicating from: hdfs://HBaseNameService/hbase/.oldlogs/va-p-hbase-02-d%2C60020%2C1369042368330.1369225444475 at position: 42337982
  470. Recovered source for cluster/machine(s) 1: Total replicated edits: 125000, currently replicating from: hdfs://HBaseNameService/hbase/.oldlogs/va-p-hbase-02-e%2C60020%2C1369042377129.1369223874741 at position: 53700152
  471. Recovered source for cluster/machine(s) 1: Total replicated edits: 0, currently replicating from: hdfs://HBaseNameService/hbase/.oldlogs/va-p-hbase-01-d%2C60020%2C1369233257617.1369233259640 at position: -1
  472. Recovered source for cluster/machine(s) 1: Total replicated edits: 0, currently replicating from: hdfs://HBaseNameService/hbase/.oldlogs/va-p-hbase-01-e%2C60020%2C1369233254969.1369233257503 at position: -1
  473.  
  474. 2013-05-22 14:59:18,962 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: NB dead servers : 5
  475. 2013-05-22 14:59:18,962 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/e,60020,1369233254969/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  476. 2013-05-22 14:59:18,965 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/e,60020,1369233254969-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  477. 2013-05-22 14:59:18,972 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  478. 2013-05-22 14:59:18,975 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  479. 2013-05-22 14:59:18,979 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  480. 2013-05-22 14:59:18,987 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  481. 2013-05-22 14:59:18,989 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  482. 2013-05-22 14:59:18,996 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  483. 2013-05-22 14:59:18,999 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  484. 2013-05-22 14:59:19,001 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  485. 2013-05-22 14:59:19,004 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1-va-p-hbase-01-e,60020,1369233254969 Got:
  486. java.io.IOException: File from recovered queue is nowhere to be found
  487. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:535)
  488. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:312)
  489. Caused by: java.io.FileNotFoundException: File does not exist: hdfs://HBaseNameService/hbase/.oldlogs/va-p-hbase-01-e%2C60020%2C1369233254969.1369233257503
  490. at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:807)
  491. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1704)
  492. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
  493. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
  494. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
  495. at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:720)
  496. at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:66)
  497. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:501)
  498. ... 1 more
  499. 2013-05-22 14:59:19,435 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Processed 202038 edits across 17 regions threw away edits for 0 regions; log file=hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-d,60020,1369233257617-splitting/va-p-hbase-01-d%2C60020%2C1369233257617.1369233259640 is corrupted = false progress failed = false
  500. 2013-05-22 14:59:19,440 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: successfully transitioned task /hbase/splitlog/hdfs%3A%2F%2FHBaseNameService%2Fhbase%2F.logs%2Fva-p-hbase-01-d%2C60020%2C1369233257617-splitting%2Fva-p-hbase-01-d%252C60020%252C1369233257617.1369233259640 to final state done
  501. 2013-05-22 14:59:19,440 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker va-p-hbase-02-d,60020,1369233252475 done with task /hbase/splitlog/hdfs%3A%2F%2FHBaseNameService%2Fhbase%2F.logs%2Fva-p-hbase-01-d%2C60020%2C1369233257617-splitting%2Fva-p-hbase-01-d%252C60020%252C1369233257617.1369233259640 in 9068ms
  502. 2013-05-22 14:59:19,456 INFO org.apache.hadoop.ipc.HBaseRPC: Server at va-p-hbase-01-e/172.16.28.25:60020 could not be reached after 1 tries, giving up.
  503. 2013-05-22 14:59:19,459 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker va-p-hbase-02-d,60020,1369233252475 acquired task /hbase/splitlog/hdfs%3A%2F%2FHBaseNameService%2Fhbase%2F.logs%2Fva-p-hbase-01-e%2C60020%2C1369233254969-splitting%2Fva-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  504. 2013-05-22 14:59:19,463 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog: hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-e,60020,1369233254969-splitting/va-p-hbase-01-e%2C60020%2C1369233254969.1369233257503, length=95430265
  505. 2013-05-22 14:59:19,463 INFO org.apache.hadoop.hbase.util.FSHDFSUtils: Recovering file hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-e,60020,1369233254969-splitting/va-p-hbase-01-e%2C60020%2C1369233254969.1369233257503
  506. 2013-05-22 14:59:20,467 INFO org.apache.hadoop.hbase.util.FSHDFSUtils: Finished lease recover attempt for hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-e,60020,1369233254969-splitting/va-p-hbase-01-e%2C60020%2C1369233254969.1369233257503
  507. 2013-05-22 14:59:21,306 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  508. 2013-05-22 14:59:21,462 INFO org.apache.hadoop.ipc.HBaseRPC: Server at va-p-hbase-01-e/172.16.28.25:60020 could not be reached after 1 tries, giving up.
  509. 2013-05-22 14:59:22,009 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: NB dead servers : 5
  510. 2013-05-22 14:59:22,009 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/e,60020,1369233254969/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  511. 2013-05-22 14:59:22,012 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/e,60020,1369233254969-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  512. 2013-05-22 14:59:22,018 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  513. 2013-05-22 14:59:22,022 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  514. 2013-05-22 14:59:22,025 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  515. 2013-05-22 14:59:22,028 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  516. 2013-05-22 14:59:22,031 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  517. 2013-05-22 14:59:22,033 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  518. 2013-05-22 14:59:22,036 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  519. 2013-05-22 14:59:22,038 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  520. 2013-05-22 14:59:22,041 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1-va-p-hbase-01-e,60020,1369233254969 Got:
  521. java.io.IOException: File from recovered queue is nowhere to be found
  522. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:535)
  523. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:312)
  524. Caused by: java.io.FileNotFoundException: File does not exist: hdfs://HBaseNameService/hbase/.oldlogs/va-p-hbase-01-e%2C60020%2C1369233254969.1369233257503
  525. at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:807)
  526. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1704)
  527. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
  528. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
  529. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
  530. at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:720)
  531. at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:66)
  532. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:501)
  533. ... 1 more
  534. 2013-05-22 14:59:22,661 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: t,_\x9C\xBCg\x8F\xE0\x1E!\x03\x1Fr\xAF0\xE3&\xB5\x7F\xFF\xFE\xC2/\xA6"\xE5,1365675574573.1edf77f78147247e6ee7a8e336ecc387.
  535. 2013-05-22 14:59:22,709 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: t,f\xB6\xF2"\xC82\xF5\x072R\x04kV\x9BP\xD9\x7F\xFF\xFE\xC2H_\xBB+,1365673449134.d7ebb92fdac5572fdeb8aede13e9fd69.
  536. 2013-05-22 14:59:22,853 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: t,\xBB\xB0\x84KH\x96\x093\xE7t\xFEe\x13\xC4;\x1D\x7F\xFF\xFE\xC2zJ\x8F\x19,1365680900479.51390267a8e89867e9f62a2708a9a097.
  537. 2013-05-22 14:59:23,036 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: t,\xD0\xF6W'\x85\xAB\x13d\xC5\xEBSJ\x02b~\xB6\x7F\xFF\xFE\xC2K\xE0\x9D\x0B,1365667186136.0250b09d6925deb9ddb6d2b6b50061c4.
  538. 2013-05-22 14:59:23,173 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: t,\xF4\xA6>\xC9\x818\xDB\x0B\xFD\xDD,\x98\x9E\xAB\xEFM\x7F\xFF\xFE\xC2\x1CaM\x91,1365533087697.fe66529186e9aefbda60371901994ce7.
  539. 2013-05-22 14:59:23,208 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: x,\xD4\x80\x00\x00,1366787449767.0fbb85e8b00bf8728a706f277e0669ec.
  540. 2013-05-22 14:59:23,561 INFO org.apache.hadoop.ipc.HBaseRPC: Server at va-p-hbase-01-e/172.16.28.25:60020 could not be reached after 1 tries, giving up.
  541. 2013-05-22 14:59:24,313 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager: /hbase/rs/va-p-hbase-02-e,60020,1369233253407 znode expired, trying to lock it
  542. 2013-05-22 14:59:25,582 INFO org.apache.hadoop.ipc.HBaseRPC: Server at va-p-hbase-01-e/172.16.28.25:60020 could not be reached after 1 tries, giving up.
  543. 2013-05-22 14:59:26,079 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: NB dead servers : 5
  544. 2013-05-22 14:59:26,079 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/e,60020,1369233254969/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  545. 2013-05-22 14:59:26,083 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/e,60020,1369233254969-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  546. 2013-05-22 14:59:26,088 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  547. 2013-05-22 14:59:26,090 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  548. 2013-05-22 14:59:26,092 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  549. 2013-05-22 14:59:26,095 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  550. 2013-05-22 14:59:26,098 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  551. 2013-05-22 14:59:26,101 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  552. 2013-05-22 14:59:26,104 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  553. 2013-05-22 14:59:26,106 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  554. 2013-05-22 14:59:26,109 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1-va-p-hbase-01-e,60020,1369233254969 Got:
  555. java.io.IOException: File from recovered queue is nowhere to be found
  556. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:535)
  557. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:312)
  558. Caused by: java.io.FileNotFoundException: File does not exist: hdfs://HBaseNameService/hbase/.oldlogs/va-p-hbase-01-e%2C60020%2C1369233254969.1369233257503
  559. at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:807)
  560. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1704)
  561. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
  562. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
  563. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
  564. at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:720)
  565. at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:66)
  566. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:501)
  567. ... 1 more
  568. 2013-05-22 14:59:26,313 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager: Atomically moving va-p-hbase-02-e,60020,1369233253407's hlogs to my queue
  569. 2013-05-22 14:59:26,431 INFO org.apache.hadoop.hbase.replication.ReplicationZookeeper: Atomically moved the dead regionserver logs.
  570. 2013-05-22 14:59:26,434 INFO org.apache.hadoop.hbase.metrics: new MBeanInfo
  571. 2013-05-22 14:59:26,462 INFO org.apache.hadoop.hbase.metrics: new MBeanInfo
  572. 2013-05-22 14:59:26,471 INFO org.apache.hadoop.hbase.metrics: new MBeanInfo
  573. 2013-05-22 14:59:26,607 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Getting 1 rs from peer cluster # 1
  574. 2013-05-22 14:59:26,607 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Choosing peer CA-P-HBase-03-b.private.mtlink.biz,60020,1369249119927
  575. 2013-05-22 14:59:26,607 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Getting 1 rs from peer cluster # 1
  576. 2013-05-22 14:59:26,608 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Choosing peer CA-P-HBase-03-b.private.mtlink.biz,60020,1369249119927
  577. 2013-05-22 14:59:26,609 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Getting 1 rs from peer cluster # 1
  578. 2013-05-22 14:59:26,609 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Choosing peer CA-P-HBase-02-b.private.mtlink.biz,60020,1369249119310
  579. 2013-05-22 14:59:27,610 INFO org.apache.hadoop.ipc.HBaseRPC: Server at va-p-hbase-01-e/172.16.28.25:60020 could not be reached after 1 tries, giving up.
  580. 2013-05-22 14:59:27,773 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Replicating 8b11c86d-1ae8-419b-a492-ced43b65290d -> 2b509cc1-433d-41f6-98a2-6b8e7a9b2d32
  581. 2013-05-22 14:59:27,774 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Replicating 8b11c86d-1ae8-419b-a492-ced43b65290d -> 2b509cc1-433d-41f6-98a2-6b8e7a9b2d32
  582. 2013-05-22 14:59:27,776 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Replicating 8b11c86d-1ae8-419b-a492-ced43b65290d -> 2b509cc1-433d-41f6-98a2-6b8e7a9b2d32
  583. 2013-05-22 14:59:27,783 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: NB dead servers : 5
  584. 2013-05-22 14:59:27,783 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: NB dead servers : 10
  585. 2013-05-22 14:59:27,783 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/e,60020,1369233253407/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  586. 2013-05-22 14:59:27,784 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/e,60020,1369233253407/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  587. 2013-05-22 14:59:27,787 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/e,60020,1369233253407-splitting/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  588. 2013-05-22 14:59:27,787 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/e,60020,1369233253407-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  589. 2013-05-22 14:59:27,790 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/02/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  590. 2013-05-22 14:59:27,792 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/02/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  591. 2013-05-22 14:59:27,793 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/02-splitting/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  592. 2013-05-22 14:59:27,795 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/02-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  593. 2013-05-22 14:59:27,797 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  594. 2013-05-22 14:59:27,800 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  595. 2013-05-22 14:59:27,801 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase-splitting/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  596. 2013-05-22 14:59:27,803 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  597. 2013-05-22 14:59:27,805 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  598. 2013-05-22 14:59:27,807 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  599. 2013-05-22 14:59:27,809 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p-splitting/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  600. 2013-05-22 14:59:27,812 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  601. 2013-05-22 14:59:27,813 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  602. 2013-05-22 14:59:27,816 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  603. 2013-05-22 14:59:27,817 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va-splitting/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  604. 2013-05-22 14:59:27,819 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  605. 2013-05-22 14:59:27,820 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1-va-p-hbase-02-e,60020,1369233253407 Got:
  606. java.io.IOException: File from recovered queue is nowhere to be found
  607. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:535)
  608. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:312)
  609. Caused by: java.io.FileNotFoundException: File does not exist: hdfs://HBaseNameService/hbase/.oldlogs/va-p-hbase-02-e%2C60020%2C1369233253407.1369233257432
  610. at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:807)
  611. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1704)
  612. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
  613. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
  614. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
  615. at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:720)
  616. at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:66)
  617. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:501)
  618. ... 1 more
  619. 2013-05-22 14:59:27,822 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/e,60020,1369233254969/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  620. 2013-05-22 14:59:27,825 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/e,60020,1369233254969-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  621. 2013-05-22 14:59:27,828 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  622. 2013-05-22 14:59:27,830 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  623. 2013-05-22 14:59:27,834 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  624. 2013-05-22 14:59:27,836 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  625. 2013-05-22 14:59:27,839 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  626. 2013-05-22 14:59:27,841 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  627. 2013-05-22 14:59:27,844 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  628. 2013-05-22 14:59:27,846 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  629. 2013-05-22 14:59:27,849 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1-va-p-hbase-01-e,60020,1369233254969-va-p-hbase-02-e,60020,1369233253407 Got:
  630. java.io.IOException: File from recovered queue is nowhere to be found
  631. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:535)
  632. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:312)
  633. Caused by: java.io.FileNotFoundException: File does not exist: hdfs://HBaseNameService/hbase/.oldlogs/va-p-hbase-01-e%2C60020%2C1369233254969.1369233257503
  634. at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:807)
  635. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1704)
  636. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
  637. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
  638. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
  639. at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:720)
  640. at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:66)
  641. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:501)
  642. ... 1 more
  643. 2013-05-22 14:59:28,902 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: NB dead servers : 5
  644. 2013-05-22 14:59:28,902 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/e,60020,1369233253407/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  645. 2013-05-22 14:59:28,903 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: NB dead servers : 10
  646. 2013-05-22 14:59:28,904 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/e,60020,1369233253407/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  647. 2013-05-22 14:59:28,906 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/e,60020,1369233253407-splitting/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  648. 2013-05-22 14:59:28,907 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/e,60020,1369233253407-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  649. 2013-05-22 14:59:28,909 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/02/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  650. 2013-05-22 14:59:28,916 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/02/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  651. 2013-05-22 14:59:28,916 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/02-splitting/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  652. 2013-05-22 14:59:28,920 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/02-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  653. 2013-05-22 14:59:28,921 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  654. 2013-05-22 14:59:28,924 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  655. 2013-05-22 14:59:28,925 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase-splitting/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  656. 2013-05-22 14:59:28,931 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  657. 2013-05-22 14:59:28,932 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  658. 2013-05-22 14:59:28,934 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  659. 2013-05-22 14:59:28,936 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p-splitting/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  660. 2013-05-22 14:59:28,937 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  661. 2013-05-22 14:59:28,940 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  662. 2013-05-22 14:59:28,942 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  663. 2013-05-22 14:59:28,944 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va-splitting/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  664. 2013-05-22 14:59:28,946 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  665. 2013-05-22 14:59:28,947 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1-va-p-hbase-02-e,60020,1369233253407 Got:
  666. java.io.IOException: File from recovered queue is nowhere to be found
  667. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:535)
  668. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:312)
  669. Caused by: java.io.FileNotFoundException: File does not exist: hdfs://HBaseNameService/hbase/.oldlogs/va-p-hbase-02-e%2C60020%2C1369233253407.1369233257432
  670. at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:807)
  671. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1704)
  672. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
  673. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
  674. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
  675. at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:720)
  676. at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:66)
  677. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:501)
  678. ... 1 more
  679. 2013-05-22 14:59:28,949 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/e,60020,1369233254969/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  680. 2013-05-22 14:59:28,954 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/e,60020,1369233254969-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  681. 2013-05-22 14:59:28,959 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  682. 2013-05-22 14:59:28,964 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  683. 2013-05-22 14:59:28,972 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  684. 2013-05-22 14:59:28,975 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  685. 2013-05-22 14:59:28,978 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  686. 2013-05-22 14:59:28,980 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  687. 2013-05-22 14:59:28,987 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  688. 2013-05-22 14:59:28,991 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  689. 2013-05-22 14:59:28,995 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1-va-p-hbase-01-e,60020,1369233254969-va-p-hbase-02-e,60020,1369233253407 Got:
  690. java.io.IOException: File from recovered queue is nowhere to be found
  691. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:535)
  692. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:312)
  693. Caused by: java.io.FileNotFoundException: File does not exist: hdfs://HBaseNameService/hbase/.oldlogs/va-p-hbase-01-e%2C60020%2C1369233254969.1369233257503
  694. at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:807)
  695. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1704)
  696. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
  697. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
  698. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
  699. at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:720)
  700. at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:66)
  701. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:501)
  702. ... 1 more
  703. 2013-05-22 14:59:29,000 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Processed 220558 edits across 18 regions threw away edits for 0 regions; log file=hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-e,60020,1369233254969-splitting/va-p-hbase-01-e%2C60020%2C1369233254969.1369233257503 is corrupted = false progress failed = false
  704. 2013-05-22 14:59:29,004 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: successfully transitioned task /hbase/splitlog/hdfs%3A%2F%2FHBaseNameService%2Fhbase%2F.logs%2Fva-p-hbase-01-e%2C60020%2C1369233254969-splitting%2Fva-p-hbase-01-e%252C60020%252C1369233254969.1369233257503 to final state done
  705. 2013-05-22 14:59:29,004 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker va-p-hbase-02-d,60020,1369233252475 done with task /hbase/splitlog/hdfs%3A%2F%2FHBaseNameService%2Fhbase%2F.logs%2Fva-p-hbase-01-e%2C60020%2C1369233254969-splitting%2Fva-p-hbase-01-e%252C60020%252C1369233254969.1369233257503 in 9545ms
  706. 2013-05-22 14:59:29,011 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker va-p-hbase-02-d,60020,1369233252475 acquired task /hbase/splitlog/hdfs%3A%2F%2FHBaseNameService%2Fhbase%2F.logs%2Fva-p-hbase-01-e%2C60020%2C1369233254969-splitting%2Fva-p-hbase-01-e%252C60020%252C1369233254969.1369240458040
  707. 2013-05-22 14:59:29,013 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog: hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-e,60020,1369233254969-splitting/va-p-hbase-01-e%2C60020%2C1369233254969.1369240458040, length=117830309
  708. 2013-05-22 14:59:29,014 INFO org.apache.hadoop.hbase.util.FSHDFSUtils: Recovering file hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-e,60020,1369233254969-splitting/va-p-hbase-01-e%2C60020%2C1369233254969.1369240458040
  709. 2013-05-22 14:59:29,613 INFO org.apache.hadoop.ipc.HBaseRPC: Server at va-p-hbase-01-e/172.16.28.25:60020 could not be reached after 1 tries, giving up.
  710. 2013-05-22 14:59:30,016 INFO org.apache.hadoop.hbase.util.FSHDFSUtils: Finished lease recover attempt for hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-e,60020,1369233254969-splitting/va-p-hbase-01-e%2C60020%2C1369233254969.1369240458040
  711. 2013-05-22 14:59:30,951 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: NB dead servers : 5
  712. 2013-05-22 14:59:30,951 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/e,60020,1369233253407/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  713. 2013-05-22 14:59:30,954 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/e,60020,1369233253407-splitting/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  714. 2013-05-22 14:59:30,956 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/02/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  715. 2013-05-22 14:59:30,959 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/02-splitting/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  716. 2013-05-22 14:59:31,037 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  717. 2013-05-22 14:59:31,041 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase-splitting/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  718. 2013-05-22 14:59:31,046 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  719. 2013-05-22 14:59:31,049 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p-splitting/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  720. 2013-05-22 14:59:31,055 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  721. 2013-05-22 14:59:31,059 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va-splitting/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  722. 2013-05-22 14:59:31,063 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1-va-p-hbase-02-e,60020,1369233253407 Got:
  723. java.io.IOException: File from recovered queue is nowhere to be found
  724. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:535)
  725. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:312)
  726. Caused by: java.io.FileNotFoundException: File does not exist: hdfs://HBaseNameService/hbase/.oldlogs/va-p-hbase-02-e%2C60020%2C1369233253407.1369233257432
  727. at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:807)
  728. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1704)
  729. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
  730. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
  731. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
  732. at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:720)
  733. at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:66)
  734. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:501)
  735. ... 1 more
  736. 2013-05-22 14:59:31,643 INFO org.apache.hadoop.ipc.HBaseRPC: Server at va-p-hbase-01-e/172.16.28.25:60020 could not be reached after 1 tries, giving up.
  737. 2013-05-22 14:59:31,839 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager: /hbase/rs/va-p-hbase-02-c,60020,1369233268385 znode expired, trying to lock it
  738. 2013-05-22 14:59:33,758 INFO org.apache.hadoop.ipc.HBaseRPC: Server at va-p-hbase-01-e/172.16.28.25:60020 could not be reached after 1 tries, giving up.
  739. 2013-05-22 14:59:33,844 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager: Atomically moving va-p-hbase-02-c,60020,1369233268385's hlogs to my queue
  740. 2013-05-22 14:59:33,868 INFO org.apache.hadoop.hbase.replication.ReplicationZookeeper: Atomically moved the dead regionserver logs.
  741. 2013-05-22 14:59:33,869 INFO org.apache.hadoop.hbase.metrics: new MBeanInfo
  742. 2013-05-22 14:59:33,969 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Getting 1 rs from peer cluster # 1
  743. 2013-05-22 14:59:33,969 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Choosing peer CA-P-HBase-03-b.private.mtlink.biz,60020,1369249119927
  744. 2013-05-22 14:59:34,072 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: NB dead servers : 5
  745. 2013-05-22 14:59:34,073 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/e,60020,1369233253407/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  746. 2013-05-22 14:59:34,075 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/e,60020,1369233253407-splitting/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  747. 2013-05-22 14:59:34,078 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/02/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  748. 2013-05-22 14:59:34,081 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/02-splitting/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  749. 2013-05-22 14:59:34,084 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  750. 2013-05-22 14:59:34,086 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase-splitting/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  751. 2013-05-22 14:59:34,089 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  752. 2013-05-22 14:59:34,092 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p-splitting/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  753. 2013-05-22 14:59:34,096 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  754. 2013-05-22 14:59:34,099 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va-splitting/va-p-hbase-02-e%252C60020%252C1369233253407.1369233257432
  755. 2013-05-22 14:59:34,101 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1-va-p-hbase-02-e,60020,1369233253407 Got:
  756. java.io.IOException: File from recovered queue is nowhere to be found
  757. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:535)
  758. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:312)
  759. Caused by: java.io.FileNotFoundException: File does not exist: hdfs://HBaseNameService/hbase/.oldlogs/va-p-hbase-02-e%2C60020%2C1369233253407.1369233257432
  760. at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:807)
  761. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1704)
  762. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
  763. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
  764. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
  765. at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:720)
  766. at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:66)
  767. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:501)
  768. ... 1 more
  769. 2013-05-22 14:59:34,245 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server va-p-hbase-02-d,60020,1369233252475: Writing replication status
  770. org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/replication/rs/va-p-hbase-02-d,60020,1369233252475/1-va-p-hbase-01-e,60020,1369233254969/va-p-hbase-01-e%2C60020%2C1369233254969.1369233257503
  771. at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
  772. at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
  773. at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1266)
  774. at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.setData(RecoverableZooKeeper.java:354)
  775. at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:846)
  776. at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:898)
  777. at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:892)
  778. at org.apache.hadoop.hbase.replication.ReplicationZookeeper.writeReplicationStatus(ReplicationZookeeper.java:558)
  779. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.logPositionAndCleanOldLogs(ReplicationSourceManager.java:154)
  780. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:638)
  781. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:387)
  782.  
  783. ++++++++++++++++++++++++++++++++++++++++++
  784. va-p-hbase-02-e
  785. ++++++++++++++++++++++++++++++++++++++++++
  786. 2013-05-22 14:58:42,606 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Getting 0 rs from peer cluster # 1
  787. 2013-05-22 14:58:43,708 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Getting 1 rs from peer cluster # 1
  788. 2013-05-22 14:58:43,708 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Choosing peer CA-P-HBase-01-c.private.mtlink.biz,60020,1369249119538
  789. 2013-05-22 14:58:44,874 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Replicating 8b11c86d-1ae8-419b-a492-ced43b65290d -> 2b509cc1-433d-41f6-98a2-6b8e7a9b2d32
  790. 2013-05-22 14:58:59,061 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager: /hbase/rs/va-p-hbase-01-c,60020,1369233253404 znode expired, trying to lock it
  791. 2013-05-22 14:58:59,283 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker va-p-hbase-02-e,60020,1369233253407 acquired task /hbase/splitlog/hdfs%3A%2F%2FHBaseNameService%2Fhbase%2F.logs%2Fva-p-hbase-01-c%2C60020%2C1369233253404-splitting%2Fva-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  792. 2013-05-22 14:58:59,396 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog: hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-c,60020,1369233253404-splitting/va-p-hbase-01-c%2C60020%2C1369233253404.1369233257396, length=72058933
  793. 2013-05-22 14:58:59,402 INFO org.apache.hadoop.hbase.util.FSHDFSUtils: Recovering file hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-c,60020,1369233253404-splitting/va-p-hbase-01-c%2C60020%2C1369233253404.1369233257396
  794. 2013-05-22 14:59:00,461 INFO org.apache.hadoop.hbase.util.FSHDFSUtils: Finished lease recover attempt for hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-c,60020,1369233253404-splitting/va-p-hbase-01-c%2C60020%2C1369233253404.1369233257396
  795. 2013-05-22 14:59:00,654 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  796. 2013-05-22 14:59:00,678 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  797. 2013-05-22 14:59:00,702 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  798. 2013-05-22 14:59:00,731 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  799. 2013-05-22 14:59:00,849 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  800. 2013-05-22 14:59:00,872 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  801. 2013-05-22 14:59:00,895 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  802. 2013-05-22 14:59:00,918 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  803. 2013-05-22 14:59:00,943 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  804. 2013-05-22 14:59:00,973 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  805. 2013-05-22 14:59:00,993 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  806. 2013-05-22 14:59:01,023 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  807. 2013-05-22 14:59:01,047 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  808. 2013-05-22 14:59:01,061 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager: Atomically moving va-p-hbase-01-c,60020,1369233253404's hlogs to my queue
  809. 2013-05-22 14:59:01,076 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  810. 2013-05-22 14:59:01,105 INFO org.apache.hadoop.hbase.replication.ReplicationZookeeper: Atomically moved the dead regionserver logs.
  811. 2013-05-22 14:59:01,108 INFO org.apache.hadoop.hbase.metrics: new MBeanInfo
  812. 2013-05-22 14:59:01,110 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  813. 2013-05-22 14:59:01,136 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  814. 2013-05-22 14:59:01,158 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  815. 2013-05-22 14:59:01,181 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  816. 2013-05-22 14:59:01,196 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Getting 1 rs from peer cluster # 1
  817. 2013-05-22 14:59:01,198 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Choosing peer CA-P-HBase-02-b.private.mtlink.biz,60020,1369249119310
  818. 2013-05-22 14:59:02,392 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Replicating 8b11c86d-1ae8-419b-a492-ced43b65290d -> 2b509cc1-433d-41f6-98a2-6b8e7a9b2d32
  819. 2013-05-22 14:59:02,400 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: NB dead servers : 5
  820. 2013-05-22 14:59:02,401 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/c,60020,1369233253404/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  821. 2013-05-22 14:59:02,402 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/c,60020,1369233253404-splitting/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  822. 2013-05-22 14:59:02,404 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  823. 2013-05-22 14:59:02,405 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01-splitting/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  824. 2013-05-22 14:59:02,407 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  825. 2013-05-22 14:59:02,409 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase-splitting/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  826. 2013-05-22 14:59:02,411 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  827. 2013-05-22 14:59:02,413 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p-splitting/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  828. 2013-05-22 14:59:02,415 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  829. 2013-05-22 14:59:02,416 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va-splitting/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  830. 2013-05-22 14:59:02,436 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1-va-p-hbase-01-c,60020,1369233253404 Got:
  831. java.io.IOException: File from recovered queue is nowhere to be found
  832. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:535)
  833. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:312)
  834. Caused by: java.io.FileNotFoundException: File does not exist: hdfs://HBaseNameService/hbase/.oldlogs/va-p-hbase-01-c%2C60020%2C1369233253404.1369233257396
  835. at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:807)
  836. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1704)
  837. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
  838. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
  839. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
  840. at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:720)
  841. at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:66)
  842. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:501)
  843. ... 1 more
  844. 2013-05-22 14:59:03,449 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: NB dead servers : 5
  845. 2013-05-22 14:59:03,450 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/c,60020,1369233253404/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  846. 2013-05-22 14:59:03,451 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/c,60020,1369233253404-splitting/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  847. 2013-05-22 14:59:03,453 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  848. 2013-05-22 14:59:03,454 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01-splitting/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  849. 2013-05-22 14:59:03,456 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  850. 2013-05-22 14:59:03,458 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase-splitting/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  851. 2013-05-22 14:59:03,460 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  852. 2013-05-22 14:59:03,461 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p-splitting/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  853. 2013-05-22 14:59:03,463 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  854. 2013-05-22 14:59:03,464 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va-splitting/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  855. 2013-05-22 14:59:03,466 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1-va-p-hbase-01-c,60020,1369233253404 Got:
  856. java.io.IOException: File from recovered queue is nowhere to be found
  857. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:535)
  858. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:312)
  859. Caused by: java.io.FileNotFoundException: File does not exist: hdfs://HBaseNameService/hbase/.oldlogs/va-p-hbase-01-c%2C60020%2C1369233253404.1369233257396
  860. at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:807)
  861. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1704)
  862. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
  863. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
  864. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
  865. at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:720)
  866. at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:66)
  867. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:501)
  868. ... 1 more
  869. 2013-05-22 14:59:05,472 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: NB dead servers : 5
  870. 2013-05-22 14:59:05,473 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/c,60020,1369233253404/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  871. 2013-05-22 14:59:05,474 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/c,60020,1369233253404-splitting/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  872. 2013-05-22 14:59:05,476 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  873. 2013-05-22 14:59:05,480 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Processed 177425 edits across 19 regions threw away edits for 0 regions; log file=hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-c,60020,1369233253404-splitting/va-p-hbase-01-c%2C60020%2C1369233253404.1369233257396 is corrupted = false progress failed = false
  874. 2013-05-22 14:59:05,481 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01-splitting/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  875. 2013-05-22 14:59:05,482 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  876. 2013-05-22 14:59:05,484 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase-splitting/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  877. 2013-05-22 14:59:05,486 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  878. 2013-05-22 14:59:05,487 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p-splitting/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  879. 2013-05-22 14:59:05,489 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  880. 2013-05-22 14:59:05,490 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va-splitting/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  881. 2013-05-22 14:59:05,491 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1-va-p-hbase-01-c,60020,1369233253404 Got:
  882. java.io.IOException: File from recovered queue is nowhere to be found
  883. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:535)
  884. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:312)
  885. Caused by: java.io.FileNotFoundException: File does not exist: hdfs://HBaseNameService/hbase/.oldlogs/va-p-hbase-01-c%2C60020%2C1369233253404.1369233257396
  886. at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:807)
  887. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1704)
  888. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
  889. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
  890. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
  891. at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:720)
  892. at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:66)
  893. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:501)
  894. ... 1 more
  895. 2013-05-22 14:59:05,519 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: successfully transitioned task /hbase/splitlog/hdfs%3A%2F%2FHBaseNameService%2Fhbase%2F.logs%2Fva-p-hbase-01-c%2C60020%2C1369233253404-splitting%2Fva-p-hbase-01-c%252C60020%252C1369233253404.1369233257396 to final state done
  896. 2013-05-22 14:59:05,519 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker va-p-hbase-02-e,60020,1369233253407 done with task /hbase/splitlog/hdfs%3A%2F%2FHBaseNameService%2Fhbase%2F.logs%2Fva-p-hbase-01-c%2C60020%2C1369233253404-splitting%2Fva-p-hbase-01-c%252C60020%252C1369233253404.1369233257396 in 6210ms
  897. 2013-05-22 14:59:07,389 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: x,\x94\xC0\x00\x00,1366787449573.9bd74e2e8d225374249fe03d9761a4bb.
  898. 2013-05-22 14:59:07,398 INFO org.apache.hadoop.hbase.regionserver.HRegion: Setting up tabledescriptor config now ...
  899. 2013-05-22 14:59:07,405 INFO org.apache.hadoop.hbase.regionserver.Store: time to purge deletes set to 0ms in store m
  900. 2013-05-22 14:59:07,454 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW (CompoundBloomFilter) metadata for 2b1290a263404351908f29c80d397049
  901. 2013-05-22 14:59:07,499 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW (CompoundBloomFilter) metadata for 4de1cd323b28404e88cf8e6eb5ab7dea
  902. 2013-05-22 14:59:07,565 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW (CompoundBloomFilter) metadata for 5ec0b64978274cfda27441e0935ae886
  903. 2013-05-22 14:59:07,583 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://HBaseNameService/hbase/x/9bd74e2e8d225374249fe03d9761a4bb/recovered.edits/0000000000579753650; minSequenceid=579366135; path=hdfs://HBaseNameService/hbase/x/9bd74e2e8d225374249fe03d9761a4bb/recovered.edits/0000000000579753650
  904. 2013-05-22 14:59:07,706 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://HBaseNameService/hbase/x/9bd74e2e8d225374249fe03d9761a4bb/recovered.edits/0000000000579949700; minSequenceid=579753650; path=hdfs://HBaseNameService/hbase/x/9bd74e2e8d225374249fe03d9761a4bb/recovered.edits/0000000000579949700
  905. 2013-05-22 14:59:07,787 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://HBaseNameService/hbase/x/9bd74e2e8d225374249fe03d9761a4bb/recovered.edits/0000000000580169839; minSequenceid=579949700; path=hdfs://HBaseNameService/hbase/x/9bd74e2e8d225374249fe03d9761a4bb/recovered.edits/0000000000580169839
  906. 2013-05-22 14:59:07,886 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://HBaseNameService/hbase/x/9bd74e2e8d225374249fe03d9761a4bb/recovered.edits/0000000000580388821; minSequenceid=580169839; path=hdfs://HBaseNameService/hbase/x/9bd74e2e8d225374249fe03d9761a4bb/recovered.edits/0000000000580388821
  907. 2013-05-22 14:59:07,981 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://HBaseNameService/hbase/x/9bd74e2e8d225374249fe03d9761a4bb/recovered.edits/0000000000580478727; minSequenceid=580388821; path=hdfs://HBaseNameService/hbase/x/9bd74e2e8d225374249fe03d9761a4bb/recovered.edits/0000000000580478727
  908. 2013-05-22 14:59:08,037 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom filter type for hdfs://HBaseNameService/hbase/x/9bd74e2e8d225374249fe03d9761a4bb/.tmp/63835ca9946643d8a21504e07652a562: ROW, CompoundBloomFilterWriter
  909. 2013-05-22 14:59:08,037 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter type for hdfs://HBaseNameService/hbase/x/9bd74e2e8d225374249fe03d9761a4bb/.tmp/63835ca9946643d8a21504e07652a562: CompoundBloomFilterWriter
  910. 2013-05-22 14:59:08,116 INFO org.apache.hadoop.hbase.regionserver.StoreFile: General Bloom and NO DeleteFamily was added to HFile (hdfs://HBaseNameService/hbase/x/9bd74e2e8d225374249fe03d9761a4bb/.tmp/63835ca9946643d8a21504e07652a562)
  911. 2013-05-22 14:59:08,116 INFO org.apache.hadoop.hbase.regionserver.Store: Flushed , sequenceid=580478727, memsize=4.0m, into tmp file hdfs://HBaseNameService/hbase/x/9bd74e2e8d225374249fe03d9761a4bb/.tmp/63835ca9946643d8a21504e07652a562
  912. 2013-05-22 14:59:08,127 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW (CompoundBloomFilter) metadata for 63835ca9946643d8a21504e07652a562
  913. 2013-05-22 14:59:08,142 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW (CompoundBloomFilter) metadata for 63835ca9946643d8a21504e07652a562
  914. 2013-05-22 14:59:08,143 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://HBaseNameService/hbase/x/9bd74e2e8d225374249fe03d9761a4bb/m/63835ca9946643d8a21504e07652a562, entries=15899, sequenceid=580478727, filesize=1.3m
  915. 2013-05-22 14:59:08,143 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~4.1m/4282536, currentsize=0/0 for region x,\x94\xC0\x00\x00,1366787449573.9bd74e2e8d225374249fe03d9761a4bb. in 126ms, sequenceid=580478727, compaction requested=true; wal=null
  916. 2013-05-22 14:59:08,169 INFO org.apache.hadoop.hbase.regionserver.HRegion: Onlined x,\x94\xC0\x00\x00,1366787449573.9bd74e2e8d225374249fe03d9761a4bb.; next sequenceid=580478728
  917. 2013-05-22 14:59:08,174 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks for region=x,\x94\xC0\x00\x00,1366787449573.9bd74e2e8d225374249fe03d9761a4bb., daughter=false
  918. 2013-05-22 14:59:08,176 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on m in region x,\x94\xC0\x00\x00,1366787449573.9bd74e2e8d225374249fe03d9761a4bb.
  919. 2013-05-22 14:59:08,176 INFO org.apache.hadoop.hbase.regionserver.Store: Starting compaction of 3 file(s) in m of x,\x94\xC0\x00\x00,1366787449573.9bd74e2e8d225374249fe03d9761a4bb. into tmpdir=hdfs://HBaseNameService/hbase/x/9bd74e2e8d225374249fe03d9761a4bb/.tmp, seqid=580478727, totalSize=4.8m
  920. 2013-05-22 14:59:08,244 INFO org.apache.hadoop.hbase.catalog.MetaEditor: Updated row x,\x94\xC0\x00\x00,1366787449573.9bd74e2e8d225374249fe03d9761a4bb. with server=va-p-hbase-02-e,60020,1369233253407
  921. 2013-05-22 14:59:08,244 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Done with post open deploy task for region=x,\x94\xC0\x00\x00,1366787449573.9bd74e2e8d225374249fe03d9761a4bb., daughter=false
  922. 2013-05-22 14:59:08,305 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom filter type for hdfs://HBaseNameService/hbase/x/9bd74e2e8d225374249fe03d9761a4bb/.tmp/db4e1fcf3d9b43ec9cbffc2f4ad346d5: ROW, CompoundBloomFilterWriter
  923. 2013-05-22 14:59:08,306 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter type for hdfs://HBaseNameService/hbase/x/9bd74e2e8d225374249fe03d9761a4bb/.tmp/db4e1fcf3d9b43ec9cbffc2f4ad346d5: CompoundBloomFilterWriter
  924. 2013-05-22 14:59:08,747 INFO org.apache.hadoop.hbase.regionserver.StoreFile: General Bloom and NO DeleteFamily was added to HFile (hdfs://HBaseNameService/hbase/x/9bd74e2e8d225374249fe03d9761a4bb/.tmp/db4e1fcf3d9b43ec9cbffc2f4ad346d5)
  925. 2013-05-22 14:59:08,759 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW (CompoundBloomFilter) metadata for db4e1fcf3d9b43ec9cbffc2f4ad346d5
  926. 2013-05-22 14:59:08,759 INFO org.apache.hadoop.hbase.regionserver.Store: Renaming compacted file at hdfs://HBaseNameService/hbase/x/9bd74e2e8d225374249fe03d9761a4bb/.tmp/db4e1fcf3d9b43ec9cbffc2f4ad346d5 to hdfs://HBaseNameService/hbase/x/9bd74e2e8d225374249fe03d9761a4bb/m/db4e1fcf3d9b43ec9cbffc2f4ad346d5
  927. 2013-05-22 14:59:08,775 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW (CompoundBloomFilter) metadata for db4e1fcf3d9b43ec9cbffc2f4ad346d5
  928. 2013-05-22 14:59:08,947 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 3 file(s) in m of x,\x94\xC0\x00\x00,1366787449573.9bd74e2e8d225374249fe03d9761a4bb. into db4e1fcf3d9b43ec9cbffc2f4ad346d5, size=4.8m; total size for store is 179.7m
  929. 2013-05-22 14:59:08,947 INFO org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: completed compaction: regionName=x,\x94\xC0\x00\x00,1366787449573.9bd74e2e8d225374249fe03d9761a4bb., storeName=m, fileCount=3, fileSize=4.8m, priority=3, time=6851640489576901; duration=0sec
  930. 2013-05-22 14:59:10,227 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager: /hbase/rs/va-p-hbase-01-d,60020,1369233257617 znode expired, trying to lock it
  931. 2013-05-22 14:59:10,325 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker va-p-hbase-02-e,60020,1369233253407 acquired task /hbase/splitlog/hdfs%3A%2F%2FHBaseNameService%2Fhbase%2F.logs%2Fva-p-hbase-01-d%2C60020%2C1369233257617-splitting%2Fva-p-hbase-01-d%252C60020%252C1369233257617.1369240460695
  932. 2013-05-22 14:59:10,329 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog: hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-d,60020,1369233257617-splitting/va-p-hbase-01-d%2C60020%2C1369233257617.1369240460695, length=103316548
  933. 2013-05-22 14:59:10,329 INFO org.apache.hadoop.hbase.util.FSHDFSUtils: Recovering file hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-d,60020,1369233257617-splitting/va-p-hbase-01-d%2C60020%2C1369233257617.1369240460695
  934. 2013-05-22 14:59:10,573 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager: /hbase/rs/va-p-hbase-01-e,60020,1369233254969 znode expired, trying to lock it
  935. 2013-05-22 14:59:11,331 INFO org.apache.hadoop.hbase.util.FSHDFSUtils: Finished lease recover attempt for hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-d,60020,1369233257617-splitting/va-p-hbase-01-d%2C60020%2C1369233257617.1369240460695
  936. 2013-05-22 14:59:12,274 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager: Atomically moving va-p-hbase-01-d,60020,1369233257617's hlogs to my queue
  937. 2013-05-22 14:59:12,304 WARN org.apache.hadoop.hbase.replication.ReplicationZookeeper: Got exception in copyQueuesFromRSUsingMulti:
  938. org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
  939. at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
  940. at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:945)
  941. at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:911)
  942. at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.multi(RecoverableZooKeeper.java:538)
  943. at org.apache.hadoop.hbase.zookeeper.ZKUtil.multiOrSequential(ZKUtil.java:1457)
  944. at org.apache.hadoop.hbase.replication.ReplicationZookeeper.copyQueuesFromRSUsingMulti(ReplicationZookeeper.java:705)
  945. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$NodeFailoverWorker.run(ReplicationSourceManager.java:585)
  946. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  947. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  948. at java.lang.Thread.run(Thread.java:722)
  949. 2013-05-22 14:59:12,309 INFO org.apache.hadoop.hbase.metrics: new MBeanInfo
  950. 2013-05-22 14:59:12,523 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Getting 1 rs from peer cluster # 1
  951. 2013-05-22 14:59:12,523 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Choosing peer CA-P-HBase-03-b.private.mtlink.biz,60020,1369249119927
  952. 2013-05-22 14:59:13,690 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Replicating 8b11c86d-1ae8-419b-a492-ced43b65290d -> 2b509cc1-433d-41f6-98a2-6b8e7a9b2d32
  953. 2013-05-22 14:59:13,694 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: NB dead servers : 5
  954. 2013-05-22 14:59:13,695 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/d,60020,1369233257617/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  955. 2013-05-22 14:59:13,697 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/d,60020,1369233257617-splitting/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  956. 2013-05-22 14:59:13,699 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  957. 2013-05-22 14:59:13,701 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01-splitting/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  958. 2013-05-22 14:59:13,703 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  959. 2013-05-22 14:59:13,705 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase-splitting/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  960. 2013-05-22 14:59:13,708 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  961. 2013-05-22 14:59:13,709 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p-splitting/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  962. 2013-05-22 14:59:13,712 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  963. 2013-05-22 14:59:13,714 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va-splitting/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  964. 2013-05-22 14:59:13,717 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1-va-p-hbase-01-d,60020,1369233257617 Got:
  965. java.io.IOException: File from recovered queue is nowhere to be found
  966. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:535)
  967. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:312)
  968. Caused by: java.io.FileNotFoundException: File does not exist: hdfs://HBaseNameService/hbase/.oldlogs/va-p-hbase-01-d%2C60020%2C1369233257617.1369233259640
  969. at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:807)
  970. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1704)
  971. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
  972. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
  973. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
  974. at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:720)
  975. at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:66)
  976. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:501)
  977. ... 1 more
  978. 2013-05-22 14:59:14,323 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager: Atomically moving va-p-hbase-01-e,60020,1369233254969's hlogs to my queue
  979. 2013-05-22 14:59:14,344 INFO org.apache.hadoop.hbase.replication.ReplicationZookeeper: Atomically moved the dead regionserver logs.
  980. 2013-05-22 14:59:14,347 INFO org.apache.hadoop.hbase.metrics: new MBeanInfo
  981. 2013-05-22 14:59:14,436 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Getting 1 rs from peer cluster # 1
  982. 2013-05-22 14:59:14,436 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Choosing peer CA-P-HBase-03-b.private.mtlink.biz,60020,1369249119927
  983. 2013-05-22 14:59:14,720 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: NB dead servers : 5
  984. 2013-05-22 14:59:14,721 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/d,60020,1369233257617/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  985. 2013-05-22 14:59:14,722 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/d,60020,1369233257617-splitting/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  986. 2013-05-22 14:59:14,724 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  987. 2013-05-22 14:59:14,725 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01-splitting/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  988. 2013-05-22 14:59:14,727 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  989. 2013-05-22 14:59:14,729 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase-splitting/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  990. 2013-05-22 14:59:14,731 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  991. 2013-05-22 14:59:14,734 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p-splitting/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  992. 2013-05-22 14:59:14,737 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  993. 2013-05-22 14:59:14,739 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va-splitting/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  994. 2013-05-22 14:59:14,740 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1-va-p-hbase-01-d,60020,1369233257617 Got:
  995. java.io.IOException: File from recovered queue is nowhere to be found
  996. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:535)
  997. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:312)
  998. Caused by: java.io.FileNotFoundException: File does not exist: hdfs://HBaseNameService/hbase/.oldlogs/va-p-hbase-01-d%2C60020%2C1369233257617.1369233259640
  999. at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:807)
  1000. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1704)
  1001. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
  1002. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
  1003. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
  1004. at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:720)
  1005. at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:66)
  1006. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:501)
  1007. ... 1 more
  1008. 2013-05-22 14:59:15,618 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Replicating 8b11c86d-1ae8-419b-a492-ced43b65290d -> 2b509cc1-433d-41f6-98a2-6b8e7a9b2d32
  1009. 2013-05-22 14:59:15,622 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: NB dead servers : 5
  1010. 2013-05-22 14:59:15,623 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/e,60020,1369233254969/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1011. 2013-05-22 14:59:15,625 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/e,60020,1369233254969-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1012. 2013-05-22 14:59:15,627 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1013. 2013-05-22 14:59:15,628 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1014. 2013-05-22 14:59:15,630 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1015. 2013-05-22 14:59:15,632 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1016. 2013-05-22 14:59:15,645 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1017. 2013-05-22 14:59:15,647 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1018. 2013-05-22 14:59:15,649 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1019. 2013-05-22 14:59:15,651 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1020. 2013-05-22 14:59:15,653 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1-va-p-hbase-01-e,60020,1369233254969 Got:
  1021. java.io.IOException: File from recovered queue is nowhere to be found
  1022. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:535)
  1023. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:312)
  1024. Caused by: java.io.FileNotFoundException: File does not exist: hdfs://HBaseNameService/hbase/.oldlogs/va-p-hbase-01-e%2C60020%2C1369233254969.1369233257503
  1025. at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:807)
  1026. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1704)
  1027. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
  1028. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
  1029. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
  1030. at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:720)
  1031. at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:66)
  1032. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:501)
  1033. ... 1 more
  1034. 2013-05-22 14:59:16,657 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: NB dead servers : 5
  1035. 2013-05-22 14:59:16,658 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/e,60020,1369233254969/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1036. 2013-05-22 14:59:16,660 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/e,60020,1369233254969-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1037. 2013-05-22 14:59:16,662 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1038. 2013-05-22 14:59:16,664 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1039. 2013-05-22 14:59:16,666 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1040. 2013-05-22 14:59:16,667 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1041. 2013-05-22 14:59:16,669 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1042. 2013-05-22 14:59:16,675 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1043. 2013-05-22 14:59:16,677 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1044. 2013-05-22 14:59:16,679 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1045. 2013-05-22 14:59:16,684 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Processed 247189 edits across 15 regions threw away edits for 0 regions; log file=hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-d,60020,1369233257617-splitting/va-p-hbase-01-d%2C60020%2C1369233257617.1369240460695 is corrupted = false progress failed = false
  1046. 2013-05-22 14:59:16,684 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1-va-p-hbase-01-e,60020,1369233254969 Got:
  1047. java.io.IOException: File from recovered queue is nowhere to be found
  1048. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:535)
  1049. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:312)
  1050. Caused by: java.io.FileNotFoundException: File does not exist: hdfs://HBaseNameService/hbase/.oldlogs/va-p-hbase-01-e%2C60020%2C1369233254969.1369233257503
  1051. at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:807)
  1052. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1704)
  1053. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
  1054. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
  1055. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
  1056. at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:720)
  1057. at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:66)
  1058. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:501)
  1059. ... 1 more
  1060. 2013-05-22 14:59:16,691 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: successfully transitioned task /hbase/splitlog/hdfs%3A%2F%2FHBaseNameService%2Fhbase%2F.logs%2Fva-p-hbase-01-d%2C60020%2C1369233257617-splitting%2Fva-p-hbase-01-d%252C60020%252C1369233257617.1369240460695 to final state done
  1061. 2013-05-22 14:59:16,691 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker va-p-hbase-02-e,60020,1369233253407 done with task /hbase/splitlog/hdfs%3A%2F%2FHBaseNameService%2Fhbase%2F.logs%2Fva-p-hbase-01-d%2C60020%2C1369233257617-splitting%2Fva-p-hbase-01-d%252C60020%252C1369233257617.1369240460695 in 6366ms
  1062. 2013-05-22 14:59:16,699 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker va-p-hbase-02-e,60020,1369233253407 acquired task /hbase/splitlog/hdfs%3A%2F%2FHBaseNameService%2Fhbase%2F.logs%2Fva-p-hbase-01-d%2C60020%2C1369233257617-splitting%2Fva-p-hbase-01-d%252C60020%252C1369233257617.1369247661135
  1063. 2013-05-22 14:59:16,701 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog: hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-d,60020,1369233257617-splitting/va-p-hbase-01-d%2C60020%2C1369233257617.1369247661135, length=0
  1064. 2013-05-22 14:59:16,701 WARN org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: File hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-d,60020,1369233257617-splitting/va-p-hbase-01-d%2C60020%2C1369233257617.1369247661135 might be still open, length is 0
  1065. 2013-05-22 14:59:16,701 INFO org.apache.hadoop.hbase.util.FSHDFSUtils: Recovering file hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-d,60020,1369233257617-splitting/va-p-hbase-01-d%2C60020%2C1369233257617.1369247661135
  1066. 2013-05-22 14:59:16,743 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: NB dead servers : 5
  1067. 2013-05-22 14:59:16,744 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/d,60020,1369233257617/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  1068. 2013-05-22 14:59:16,746 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/d,60020,1369233257617-splitting/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  1069. 2013-05-22 14:59:16,748 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  1070. 2013-05-22 14:59:16,750 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01-splitting/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  1071. 2013-05-22 14:59:16,753 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  1072. 2013-05-22 14:59:16,754 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase-splitting/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  1073. 2013-05-22 14:59:16,756 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  1074. 2013-05-22 14:59:16,757 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p-splitting/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  1075. 2013-05-22 14:59:16,759 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  1076. 2013-05-22 14:59:16,761 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va-splitting/va-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  1077. 2013-05-22 14:59:16,762 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1-va-p-hbase-01-d,60020,1369233257617 Got:
  1078. java.io.IOException: File from recovered queue is nowhere to be found
  1079. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:535)
  1080. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:312)
  1081. Caused by: java.io.FileNotFoundException: File does not exist: hdfs://HBaseNameService/hbase/.oldlogs/va-p-hbase-01-d%2C60020%2C1369233257617.1369233259640
  1082. at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:807)
  1083. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1704)
  1084. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
  1085. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
  1086. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
  1087. at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:720)
  1088. at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:66)
  1089. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:501)
  1090. ... 1 more
  1091. 2013-05-22 14:59:17,707 INFO org.apache.hadoop.hbase.util.FSHDFSUtils: Finished lease recover attempt for hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-d,60020,1369233257617-splitting/va-p-hbase-01-d%2C60020%2C1369233257617.1369247661135
  1092. 2013-05-22 14:59:18,578 INFO org.apache.hadoop.hbase.replication.regionserver.Replication: Normal source for cluster 1: Total replicated edits: 125000, currently replicating from: hdfs://HBaseNameService/hbase/.logs/va-p-hbase-02-e,60020,1369233253407/va-p-hbase-02-e%2C60020%2C1369233253407.1369233257432 at position: 56427397
  1093. Recovered source for cluster/machine(s) 1: Total replicated edits: 75000, currently replicating from: hdfs://HBaseNameService/hbase/.oldlogs/va-p-hbase-01-c%2C60020%2C1369233253404.1369233257396 at position: 45137674
  1094. Recovered source for cluster/machine(s) 1: Total replicated edits: 0, currently replicating from: hdfs://HBaseNameService/hbase/.oldlogs/va-p-hbase-01-d%2C60020%2C1369233257617.1369233259640 at position: -1
  1095. Recovered source for cluster/machine(s) 1: Total replicated edits: 0, currently replicating from: hdfs://HBaseNameService/hbase/.oldlogs/va-p-hbase-01-e%2C60020%2C1369233254969.1369233257503 at position: -1
  1096.  
  1097. 2013-05-22 14:59:18,688 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: NB dead servers : 5
  1098. 2013-05-22 14:59:18,688 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/e,60020,1369233254969/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1099. 2013-05-22 14:59:18,690 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/e,60020,1369233254969-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1100. 2013-05-22 14:59:18,692 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1101. 2013-05-22 14:59:18,710 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1102. 2013-05-22 14:59:18,712 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1103. 2013-05-22 14:59:18,714 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1104. 2013-05-22 14:59:18,716 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1105. 2013-05-22 14:59:18,717 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1106. 2013-05-22 14:59:18,720 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1107. 2013-05-22 14:59:18,722 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1108. 2013-05-22 14:59:18,724 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1-va-p-hbase-01-e,60020,1369233254969 Got:
  1109. java.io.IOException: File from recovered queue is nowhere to be found
  1110. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:535)
  1111. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:312)
  1112. Caused by: java.io.FileNotFoundException: File does not exist: hdfs://HBaseNameService/hbase/.oldlogs/va-p-hbase-01-e%2C60020%2C1369233254969.1369233257503
  1113. at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:807)
  1114. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1704)
  1115. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
  1116. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
  1117. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
  1118. at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:720)
  1119. at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:66)
  1120. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:501)
  1121. ... 1 more
  1122. 2013-05-22 14:59:20,518 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Processed 102447 edits across 15 regions threw away edits for 0 regions; log file=hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-d,60020,1369233257617-splitting/va-p-hbase-01-d%2C60020%2C1369233257617.1369247661135 is corrupted = false progress failed = false
  1123. 2013-05-22 14:59:20,530 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: successfully transitioned task /hbase/splitlog/hdfs%3A%2F%2FHBaseNameService%2Fhbase%2F.logs%2Fva-p-hbase-01-d%2C60020%2C1369233257617-splitting%2Fva-p-hbase-01-d%252C60020%252C1369233257617.1369247661135 to final state done
  1124. 2013-05-22 14:59:20,530 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker va-p-hbase-02-e,60020,1369233253407 done with task /hbase/splitlog/hdfs%3A%2F%2FHBaseNameService%2Fhbase%2F.logs%2Fva-p-hbase-01-d%2C60020%2C1369233257617-splitting%2Fva-p-hbase-01-d%252C60020%252C1369233257617.1369247661135 in 3831ms
  1125. 2013-05-22 14:59:20,540 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker va-p-hbase-02-e,60020,1369233253407 acquired task /hbase/splitlog/hdfs%3A%2F%2FHBaseNameService%2Fhbase%2F.logs%2Fva-p-hbase-01-e%2C60020%2C1369233254969-splitting%2Fva-p-hbase-01-e%252C60020%252C1369233254969.1369240458040
  1126. 2013-05-22 14:59:20,544 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog: hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-e,60020,1369233254969-splitting/va-p-hbase-01-e%2C60020%2C1369233254969.1369240458040, length=117830309
  1127. 2013-05-22 14:59:20,544 INFO org.apache.hadoop.hbase.util.FSHDFSUtils: Recovering file hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-e,60020,1369233254969-splitting/va-p-hbase-01-e%2C60020%2C1369233254969.1369240458040
  1128. 2013-05-22 14:59:21,545 INFO org.apache.hadoop.hbase.util.FSHDFSUtils: Finished lease recover attempt for hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-e,60020,1369233254969-splitting/va-p-hbase-01-e%2C60020%2C1369233254969.1369240458040
  1129. 2013-05-22 14:59:21,726 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: NB dead servers : 5
  1130. 2013-05-22 14:59:21,727 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/e,60020,1369233254969/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1131. 2013-05-22 14:59:21,728 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/e,60020,1369233254969-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1132. 2013-05-22 14:59:21,730 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1133. 2013-05-22 14:59:21,733 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1134. 2013-05-22 14:59:21,739 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1135. 2013-05-22 14:59:21,742 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1136. 2013-05-22 14:59:21,744 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1137. 2013-05-22 14:59:21,746 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1138. 2013-05-22 14:59:21,749 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1139. 2013-05-22 14:59:21,752 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va-splitting/va-p-hbase-01-e%252C60020%252C1369233254969.1369233257503
  1140. 2013-05-22 14:59:21,754 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1-va-p-hbase-01-e,60020,1369233254969 Got:
  1141. java.io.IOException: File from recovered queue is nowhere to be found
  1142. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:535)
  1143. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:312)
  1144. Caused by: java.io.FileNotFoundException: File does not exist: hdfs://HBaseNameService/hbase/.oldlogs/va-p-hbase-01-e%2C60020%2C1369233254969.1369233257503
  1145. at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:807)
  1146. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1704)
  1147. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
  1148. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
  1149. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
  1150. at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:720)
  1151. at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:66)
  1152. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:501)
  1153. ... 1 more
  1154. 2013-05-22 14:59:23,057 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: t,\xE61\x90|\xD2\xD4\xDD\xCE\x1D\x7F\x0A\xAE\xD4\x91? \x7F\xFF\xFE\xC2z\xBC\xB0\x84,1365680820583.b59ac1a329308e861d6fb3cc9ad539b5.
  1155. 2013-05-22 14:59:23,075 INFO org.apache.hadoop.hbase.regionserver.HRegion: Setting up tabledescriptor config now ...
  1156. 2013-05-22 14:59:23,090 INFO org.apache.hadoop.hbase.regionserver.Store: time to purge deletes set to 0ms in store u
  1157. 2013-05-22 14:59:23,195 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: x,\x7F\x80\x00\x00,1363541667783.d81cc5e3d5b949ec8235449812dbbaca.
  1158. 2013-05-22 14:59:23,202 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW (CompoundBloomFilter) metadata for 5109974d33514ab8bed440a17a0450f3
  1159. 2013-05-22 14:59:23,204 INFO org.apache.hadoop.hbase.regionserver.HRegion: Setting up tabledescriptor config now ...
  1160. 2013-05-22 14:59:23,213 INFO org.apache.hadoop.hbase.regionserver.Store: time to purge deletes set to 0ms in store m
  1161. 2013-05-22 14:59:23,285 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW (CompoundBloomFilter) metadata for a8c29fe044374516aad149f16a1af677
  1162. 2013-05-22 14:59:23,301 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://HBaseNameService/hbase/t/b59ac1a329308e861d6fb3cc9ad539b5/recovered.edits/0000000000579817511; minSequenceid=579615470; path=hdfs://HBaseNameService/hbase/t/b59ac1a329308e861d6fb3cc9ad539b5/recovered.edits/0000000000579817511
  1163. 2013-05-22 14:59:23,329 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW (CompoundBloomFilter) metadata for 174cce203d184ba3861da1050c328482
  1164. 2013-05-22 14:59:23,461 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW (CompoundBloomFilter) metadata for 96714765a8d04e5b9fc8131332282044
  1165. 2013-05-22 14:59:23,527 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW (CompoundBloomFilter) metadata for c53a2e23d0a6479a895bc202e6507f99
  1166. 2013-05-22 14:59:23,541 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://HBaseNameService/hbase/x/d81cc5e3d5b949ec8235449812dbbaca/recovered.edits/0000000000579817478; minSequenceid=579615398; path=hdfs://HBaseNameService/hbase/x/d81cc5e3d5b949ec8235449812dbbaca/recovered.edits/0000000000579817478
  1167. 2013-05-22 14:59:23,757 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://HBaseNameService/hbase/x/d81cc5e3d5b949ec8235449812dbbaca/recovered.edits/0000000000580044395; minSequenceid=579817478; path=hdfs://HBaseNameService/hbase/x/d81cc5e3d5b949ec8235449812dbbaca/recovered.edits/0000000000580044395
  1168. 2013-05-22 14:59:23,817 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://HBaseNameService/hbase/t/b59ac1a329308e861d6fb3cc9ad539b5/recovered.edits/0000000000580044468; minSequenceid=579817511; path=hdfs://HBaseNameService/hbase/t/b59ac1a329308e861d6fb3cc9ad539b5/recovered.edits/0000000000580044468
  1169. 2013-05-22 14:59:23,955 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://HBaseNameService/hbase/x/d81cc5e3d5b949ec8235449812dbbaca/recovered.edits/0000000000580291638; minSequenceid=580044395; path=hdfs://HBaseNameService/hbase/x/d81cc5e3d5b949ec8235449812dbbaca/recovered.edits/0000000000580291638
  1170. 2013-05-22 14:59:24,093 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server va-p-hbase-02-e,60020,1369233253407: Writing replication status
  1171. org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/replication/rs/va-p-hbase-02-e,60020,1369233253407/1-va-p-hbase-01-d,60020,1369233257617/va-p-hbase-01-d%2C60020%2C1369233257617.1369233259640
  1172. at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
  1173. at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
  1174. at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1266)
  1175. at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.setData(RecoverableZooKeeper.java:354)
  1176. at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:846)
  1177. at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:898)
  1178. at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:892)
  1179. at org.apache.hadoop.hbase.replication.ReplicationZookeeper.writeReplicationStatus(ReplicationZookeeper.java:558)
  1180. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.logPositionAndCleanOldLogs(ReplicationSourceManager.java:154)
  1181. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:638)
  1182. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:387)
  1183.  
  1184. +++++++++++++++++++++++++++++++++++++++++++
  1185. va-p-hbase-01-e
  1186. +++++++++++++++++++++++++++++++++++++++++++
  1187. 2013-05-22 14:58:42,580 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Getting 0 rs from peer cluster # 1
  1188. 2013-05-22 14:58:43,698 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Getting 1 rs from peer cluster # 1
  1189. 2013-05-22 14:58:43,698 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Choosing peer CA-P-HBase-01-c.private.mtlink.biz,60020,1369249119538
  1190. 2013-05-22 14:58:44,866 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Replicating 8b11c86d-1ae8-419b-a492-ced43b65290d -> 2b509cc1-433d-41f6-98a2-6b8e7a9b2d32
  1191. 2013-05-22 14:58:59,063 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager: /hbase/rs/va-p-hbase-01-c,60020,1369233253404 znode expired, trying to lock it
  1192. 2013-05-22 14:58:59,288 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker va-p-hbase-01-e,60020,1369233254969 acquired task /hbase/splitlog/hdfs%3A%2F%2FHBaseNameService%2Fhbase%2F.logs%2Fva-p-hbase-01-c%2C60020%2C1369233253404-splitting%2Fva-p-hbase-01-c%252C60020%252C1369233253404.1369240458110
  1193. 2013-05-22 14:58:59,387 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog: hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-c,60020,1369233253404-splitting/va-p-hbase-01-c%2C60020%2C1369233253404.1369240458110, length=89932863
  1194. 2013-05-22 14:58:59,397 INFO org.apache.hadoop.hbase.util.FSHDFSUtils: Recovering file hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-c,60020,1369233253404-splitting/va-p-hbase-01-c%2C60020%2C1369233253404.1369240458110
  1195. 2013-05-22 14:59:00,457 INFO org.apache.hadoop.hbase.util.FSHDFSUtils: Finished lease recover attempt for hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-c,60020,1369233253404-splitting/va-p-hbase-01-c%2C60020%2C1369233253404.1369240458110
  1196. 2013-05-22 14:59:00,630 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  1197. 2013-05-22 14:59:00,652 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  1198. 2013-05-22 14:59:00,669 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  1199. 2013-05-22 14:59:00,692 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  1200. 2013-05-22 14:59:00,712 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  1201. 2013-05-22 14:59:00,730 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  1202. 2013-05-22 14:59:00,844 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  1203. 2013-05-22 14:59:00,864 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  1204. 2013-05-22 14:59:00,892 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  1205. 2013-05-22 14:59:00,912 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  1206. 2013-05-22 14:59:00,929 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  1207. 2013-05-22 14:59:00,947 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  1208. 2013-05-22 14:59:00,970 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
  1209. 2013-05-22 14:59:01,063 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager: Atomically moving va-p-hbase-01-c,60020,1369233253404's hlogs to my queue
  1210. 2013-05-22 14:59:01,238 WARN org.apache.hadoop.hbase.replication.ReplicationZookeeper: Got exception in copyQueuesFromRSUsingMulti:
  1211. org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
  1212. at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
  1213. at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:945)
  1214. at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:911)
  1215. at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.multi(RecoverableZooKeeper.java:538)
  1216. at org.apache.hadoop.hbase.zookeeper.ZKUtil.multiOrSequential(ZKUtil.java:1457)
  1217. at org.apache.hadoop.hbase.replication.ReplicationZookeeper.copyQueuesFromRSUsingMulti(ReplicationZookeeper.java:705)
  1218. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$NodeFailoverWorker.run(ReplicationSourceManager.java:585)
  1219. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  1220. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  1221. at java.lang.Thread.run(Thread.java:722)
  1222. 2013-05-22 14:59:01,250 INFO org.apache.hadoop.hbase.metrics: new MBeanInfo
  1223. 2013-05-22 14:59:01,344 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Getting 1 rs from peer cluster # 1
  1224. 2013-05-22 14:59:01,345 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Choosing peer CA-P-HBase-03-b.private.mtlink.biz,60020,1369249119927
  1225. 2013-05-22 14:59:02,513 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Replicating 8b11c86d-1ae8-419b-a492-ced43b65290d -> 2b509cc1-433d-41f6-98a2-6b8e7a9b2d32
  1226. 2013-05-22 14:59:02,519 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: NB dead servers : 5
  1227. 2013-05-22 14:59:02,520 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/c,60020,1369233253404/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  1228. 2013-05-22 14:59:02,522 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/c,60020,1369233253404-splitting/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  1229. 2013-05-22 14:59:02,524 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  1230. 2013-05-22 14:59:02,526 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01-splitting/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  1231. 2013-05-22 14:59:02,529 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  1232. 2013-05-22 14:59:02,531 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase-splitting/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  1233. 2013-05-22 14:59:02,533 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  1234. 2013-05-22 14:59:02,535 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p-splitting/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  1235. 2013-05-22 14:59:02,537 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  1236. 2013-05-22 14:59:02,538 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va-splitting/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  1237. 2013-05-22 14:59:02,540 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1-va-p-hbase-01-c,60020,1369233253404 Got:
  1238. java.io.IOException: File from recovered queue is nowhere to be found
  1239. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:535)
  1240. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:312)
  1241. Caused by: java.io.FileNotFoundException: File does not exist: hdfs://HBaseNameService/hbase/.oldlogs/va-p-hbase-01-c%2C60020%2C1369233253404.1369233257396
  1242. at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:807)
  1243. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1704)
  1244. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
  1245. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
  1246. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
  1247. at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:720)
  1248. at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:66)
  1249. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:501)
  1250. ... 1 more
  1251. 2013-05-22 14:59:03,547 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: NB dead servers : 5
  1252. 2013-05-22 14:59:03,548 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/c,60020,1369233253404/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  1253. 2013-05-22 14:59:03,551 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/c,60020,1369233253404-splitting/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  1254. 2013-05-22 14:59:03,553 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  1255. 2013-05-22 14:59:03,555 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/01-splitting/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  1256. 2013-05-22 14:59:03,557 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  1257. 2013-05-22 14:59:03,559 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/hbase-splitting/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  1258. 2013-05-22 14:59:03,561 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  1259. 2013-05-22 14:59:03,563 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/p-splitting/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  1260. 2013-05-22 14:59:03,565 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  1261. 2013-05-22 14:59:03,567 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Possible location hdfs://HBaseNameService/hbase/.logs/va-splitting/va-p-hbase-01-c%252C60020%252C1369233253404.1369233257396
  1262. 2013-05-22 14:59:03,569 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1-va-p-hbase-01-c,60020,1369233253404 Got:
  1263. java.io.IOException: File from recovered queue is nowhere to be found
  1264. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:535)
  1265. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:312)
  1266. Caused by: java.io.FileNotFoundException: File does not exist: hdfs://HBaseNameService/hbase/.oldlogs/va-p-hbase-01-c%2C60020%2C1369233253404.1369233257396
  1267. at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:807)
  1268. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1704)
  1269. at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
  1270. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
  1271. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
  1272. at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:720)
  1273. at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:66)
  1274. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:501)
  1275. ... 1 more
  1276. 2013-05-22 14:59:06,050 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Processed 220114 edits across 14 regions threw away edits for 0 regions; log file=hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-c,60020,1369233253404-splitting/va-p-hbase-01-c%2C60020%2C1369233253404.1369240458110 is corrupted = false progress failed = false
  1277. 2013-05-22 14:59:06,081 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: successfully transitioned task /hbase/splitlog/hdfs%3A%2F%2FHBaseNameService%2Fhbase%2F.logs%2Fva-p-hbase-01-c%2C60020%2C1369233253404-splitting%2Fva-p-hbase-01-c%252C60020%252C1369233253404.1369240458110 to final state done
  1278. 2013-05-22 14:59:06,081 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker va-p-hbase-01-e,60020,1369233254969 done with task /hbase/splitlog/hdfs%3A%2F%2FHBaseNameService%2Fhbase%2F.logs%2Fva-p-hbase-01-c%2C60020%2C1369233253404-splitting%2Fva-p-hbase-01-c%252C60020%252C1369233253404.1369240458110 in 6768ms
  1279. 2013-05-22 14:59:07,127 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: -ROOT-,,0.70236052
  1280. 2013-05-22 14:59:07,139 INFO org.apache.hadoop.hbase.regionserver.HRegion: Setting up tabledescriptor config now ...
  1281. 2013-05-22 14:59:07,146 INFO org.apache.hadoop.hbase.regionserver.Store: time to purge deletes set to 0ms in store info
  1282. 2013-05-22 14:59:07,160 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: t,,1365678423110.b92c72c086066be1de695d4df5876102.
  1283. 2013-05-22 14:59:07,180 INFO org.apache.hadoop.hbase.regionserver.HRegion: Setting up tabledescriptor config now ...
  1284. 2013-05-22 14:59:07,186 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://HBaseNameService/hbase/-ROOT-/70236052/recovered.edits/0000000000571019703; minSequenceid=571019701; path=hdfs://HBaseNameService/hbase/-ROOT-/70236052/recovered.edits/0000000000571019703
  1285. 2013-05-22 14:59:07,187 INFO org.apache.hadoop.hbase.regionserver.Store: time to purge deletes set to 0ms in store u
  1286. 2013-05-22 14:59:07,218 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter type for hdfs://HBaseNameService/hbase/-ROOT-/70236052/.tmp/c646f80479b64016a5d573dd69d4f307: CompoundBloomFilterWriter
  1287. 2013-05-22 14:59:07,240 INFO org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom and NO DeleteFamily was added to HFile (hdfs://HBaseNameService/hbase/-ROOT-/70236052/.tmp/c646f80479b64016a5d573dd69d4f307)
  1288. 2013-05-22 14:59:07,240 INFO org.apache.hadoop.hbase.regionserver.Store: Flushed , sequenceid=571019703, memsize=472, into tmp file hdfs://HBaseNameService/hbase/-ROOT-/70236052/.tmp/c646f80479b64016a5d573dd69d4f307
  1289. 2013-05-22 14:59:07,265 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://HBaseNameService/hbase/-ROOT-/70236052/info/c646f80479b64016a5d573dd69d4f307, entries=2, sequenceid=571019703, filesize=809
  1290. 2013-05-22 14:59:07,266 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~472/472, currentsize=0/0 for region -ROOT-,,0.70236052 in 62ms, sequenceid=571019703, compaction requested=false; wal=null
  1291. 2013-05-22 14:59:07,277 INFO org.apache.hadoop.hbase.regionserver.HRegion: Onlined -ROOT-,,0.70236052; next sequenceid=571019704
  1292. 2013-05-22 14:59:07,280 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW (CompoundBloomFilter) metadata for 1b9797729b624a24b6304a15f866315a
  1293. 2013-05-22 14:59:07,288 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks for region=-ROOT-,,0.70236052, daughter=false
  1294. 2013-05-22 14:59:07,322 INFO org.apache.hadoop.hbase.catalog.RootLocationEditor: Setting ROOT region location in ZooKeeper as va-p-hbase-01-e,60020,1369233254969
  1295. 2013-05-22 14:59:07,331 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Done with post open deploy task for region=-ROOT-,,0.70236052, daughter=false
  1296. 2013-05-22 14:59:07,899 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW (CompoundBloomFilter) metadata for 41e34f73ebd44d00aa3f7fa259247360
  1297. 2013-05-22 14:59:07,985 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW (CompoundBloomFilter) metadata for cf9fa6f1d03b427babd48bbd1dc76764
  1298. 2013-05-22 14:59:07,999 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://HBaseNameService/hbase/t/b92c72c086066be1de695d4df5876102/recovered.edits/0000000000579753725; minSequenceid=579576306; path=hdfs://HBaseNameService/hbase/t/b92c72c086066be1de695d4df5876102/recovered.edits/0000000000579753725
  1299. 2013-05-22 14:59:08,554 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://HBaseNameService/hbase/t/b92c72c086066be1de695d4df5876102/recovered.edits/0000000000579949722; minSequenceid=579753725; path=hdfs://HBaseNameService/hbase/t/b92c72c086066be1de695d4df5876102/recovered.edits/0000000000579949722
  1300. 2013-05-22 14:59:08,992 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://HBaseNameService/hbase/t/b92c72c086066be1de695d4df5876102/recovered.edits/0000000000580169832; minSequenceid=579949722; path=hdfs://HBaseNameService/hbase/t/b92c72c086066be1de695d4df5876102/recovered.edits/0000000000580169832
  1301. 2013-05-22 14:59:09,368 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://HBaseNameService/hbase/t/b92c72c086066be1de695d4df5876102/recovered.edits/0000000000580388875; minSequenceid=580169832; path=hdfs://HBaseNameService/hbase/t/b92c72c086066be1de695d4df5876102/recovered.edits/0000000000580388875
  1302. 2013-05-22 14:59:09,682 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://HBaseNameService/hbase/t/b92c72c086066be1de695d4df5876102/recovered.edits/0000000000580478759; minSequenceid=580388875; path=hdfs://HBaseNameService/hbase/t/b92c72c086066be1de695d4df5876102/recovered.edits/0000000000580478759
  1303. 2013-05-22 14:59:09,908 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom filter type for hdfs://HBaseNameService/hbase/t/b92c72c086066be1de695d4df5876102/.tmp/b989bd12b507457b9d84035fbdbdf3cb: ROW, CompoundBloomFilterWriter
  1304. 2013-05-22 14:59:09,908 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter type for hdfs://HBaseNameService/hbase/t/b92c72c086066be1de695d4df5876102/.tmp/b989bd12b507457b9d84035fbdbdf3cb: CompoundBloomFilterWriter
  1305. 2013-05-22 14:59:10,229 INFO org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager: /hbase/rs/va-p-hbase-01-d,60020,1369233257617 znode expired, trying to lock it
  1306. 2013-05-22 14:59:10,250 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker va-p-hbase-01-e,60020,1369233254969 acquired task /hbase/splitlog/hdfs%3A%2F%2FHBaseNameService%2Fhbase%2F.logs%2Fva-p-hbase-01-d%2C60020%2C1369233257617-splitting%2Fva-p-hbase-01-d%252C60020%252C1369233257617.1369233259640
  1307. 2013-05-22 14:59:10,252 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog: hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-d,60020,1369233257617-splitting/va-p-hbase-01-d%2C60020%2C1369233257617.1369233259640, length=83964033
  1308. 2013-05-22 14:59:10,252 INFO org.apache.hadoop.hbase.util.FSHDFSUtils: Recovering file hdfs://HBaseNameService/hbase/.logs/va-p-hbase-01-d,60020,1369233257617-splitting/va-p-hbase-01-d%2C60020%2C1369233257617.1369233259640
  1309. 2013-05-22 14:59:10,278 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server va-p-hbase-01-e,60020,1369233254969: Writing replication status
  1310. org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/replication/rs/va-p-hbase-01-e,60020,1369233254969/1-va-p-hbase-01-c,60020,1369233253404/va-p-hbase-01-c%2C60020%2C1369233253404.1369233257396
  1311. at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
  1312. at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
  1313. at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1266)
  1314. at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.setData(RecoverableZooKeeper.java:354)
  1315. at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:846)
  1316. at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:898)
  1317. at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUtil.java:892)
  1318. at org.apache.hadoop.hbase.replication.ReplicationZookeeper.writeReplicationStatus(ReplicationZookeeper.java:558)
  1319. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.logPositionAndCleanOldLogs(ReplicationSourceManager.java:154)
  1320. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:638)
  1321. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:387)
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!