mivkovic

Hmasterlog

Sep 9th, 2017
479
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 8.21 KB | None | 0 0
  1.  
  2. ..............................
  3. 2017-09-08 14:18:16,880 INFO [wdc01is-ja-prod-hbase5:16000.activeMasterManager] util.FSHDFSUtils: Recovered lease, attempt=0 on file=hdfs://hbase5:9000/hbase/MasterProcWALs/state-00000000000000066826.log after 0ms
  4. 2017-09-08 14:18:16,890 INFO [wdc01is-ja-prod-hbase5:16000.activeMasterManager] util.FSHDFSUtils: Recover lease on dfs file hdfs://hbase5:9000/hbase/MasterProcWALs/state-00000000000000066827.log
  5. 2017-09-08 14:18:16,890 INFO [wdc01is-ja-prod-hbase5:16000.activeMasterManager] util.FSHDFSUtils: Recovered lease, attempt=0 on file=hdfs://hbase5:9000/hbase/MasterProcWALs/state-00000000000000066827.log after 0ms
  6. 2017-09-08 14:18:16,900 INFO [wdc01is-ja-prod-hbase5:16000.activeMasterManager] util.FSHDFSUtils: Recover lease on dfs file hdfs://hbase5:9000/hbase/MasterProcWALs/state-00000000000000066828.log
  7. 2017-09-08 14:18:16,900 INFO [wdc01is-ja-prod-hbase5:16000.activeMasterManager] util.FSHDFSUtils: Recovered lease, attempt=0 on file=hdfs://hbase5:9000/hbase/MasterProcWALs/state-00000000000000066828.log after 0ms
  8. 2017-09-08 14:18:18,461 INFO [wdc01is-ja-prod-hbase5.adm01.com,16000,1504878827926_splitLogManager__ChoreService_1] hbase.ScheduledChore: Chore: SplitLogManager Timeout Monitor missed its start time
  9. 2017-09-08 14:18:18,464 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1343ms
  10. GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1556ms
  11. 2017-09-08 14:18:18,561 INFO [wdc01is-ja-prod-hbase5:16000.activeMasterManager] util.FSHDFSUtils: Recover lease on dfs file hdfs://hbase5:9000/hbase/MasterProcWALs/state-00000000000000066829.log
  12. 2017-09-08 14:18:18,561 INFO [wdc01is-ja-prod-hbase5:16000.activeMasterManager] util.FSHDFSUtils: Recovered lease, attempt=0 on file=hdfs://hbase5:9000/hbase/MasterProcWALs/state-00000000000000066829.log after 0ms
  13. 2017-09-08 14:18:18,566 INFO [wdc01is-ja-prod-hbase5:16000.activeMasterManager] util.FSHDFSUtils: Recover lease on dfs file hdfs://hbase5:9000/hbase/MasterProcWALs/state-00000000000000066830.log
  14. 2017-09-08 14:18:18,566 INFO [wdc01is-ja-prod-hbase5:16000.activeMasterManager] util.FSHDFSUtils: Recovered lease, attempt=0 on file=hdfs://hbase5:9000/hbase/MasterProcWALs/state-00000000000000066830.log after 0ms
  15. 2017-09-08 14:18:20,036 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1072ms
  16. GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1463ms
  17. 2017-09-08 14:18:20,037 INFO [wdc01is-ja-prod-hbase5.adm01.com,16000,1504878827926_splitLogManager__ChoreService_1] hbase.ScheduledChore: Chore: SplitLogManager Timeout Monitor missed its start time
  18. 2017-09-08 14:18:20,037 INFO [wdc01is-ja-prod-hbase5.adm01.com,16000,1504878827926_splitLogManager__ChoreService_1] master.SplitLogManager: total tasks = 1 unassigned = 1 tasks={/hbase/splitWAL/WALs%2Fwdc01is-ja-prod-hbase9.adm01.com%2C16020%2C1504096956944-splitting%2Fwdc01is-ja-prod-hbase9.adm01.com%252C16020%252C1504096956944..meta.1504165675027.meta=last_update = 1504878829356 last_version = -1 cur_worker_name = null status = in_progress incarnation = 1 resubmits = 0 batch = null}
  19. 2017-09-08 14:18:20,136 FATAL [wdc01is-ja-prod-hbase5:16000.activeMasterManager] master.HMaster: Failed to become active master
  20. java.lang.OutOfMemoryError: Direct buffer memory
  21. at java.nio.Bits.reserveMemory(Bits.java:658)
  22. at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
  23. at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
  24. at org.apache.hadoop.hdfs.util.DirectBufferPool.getBuffer(DirectBufferPool.java:59)
  25. at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.reallocPacketBuf(PacketReceiver.java:272)
  26. at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:165)
  27. at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:102)
  28. at org.apache.hadoop.hdfs.RemoteBlockReader2.readNextPacket(RemoteBlockReader2.java:186)
  29. at org.apache.hadoop.hdfs.RemoteBlockReader2.read(RemoteBlockReader2.java:146)
  30. at org.apache.hadoop.hdfs.DFSInputStream$ByteArrayStrategy.doRead(DFSInputStream.java:686)
  31. at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:742)
  32. at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:799)
  33. at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:840)
  34. at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:648)
  35. at java.io.FilterInputStream.read(FilterInputStream.java:83)
  36. at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:232)
  37. at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
  38. at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
  39. at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
  40. at org.apache.hadoop.hbase.protobuf.generated.ProcedureProtos$ProcedureWALHeader.parseDelimitedFrom(ProcedureProtos.java:3870)
  41. at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat.readHeader(ProcedureWALFormat.java:137)
  42. at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFile.open(ProcedureWALFile.java:71)
  43. at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.initOldLog(WALProcedureStore.java:940)
  44. at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.initOldLogs(WALProcedureStore.java:903)
  45. at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:260)
  46. at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:508)
  47. at org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1157)
  48. at org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1079)
  49. at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:670)
  50. at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:184)
  51. at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1729)
  52. at java.lang.Thread.run(Thread.java:745)
  53.  
  54.  
  55. ........
  56. 2017-09-08 14:18:20,238 INFO [master/wdc01is-ja-prod-hbase5.adm01.com/10.125.19.34:16000] wal.WALProcedureStore: Stopping the WAL Procedure Store
  57.  
  58. ......
  59. 2017-09-08 14:28:51,312 INFO [wdc01is-ja-prod-hbase5:16000.activeMasterManager] wal.WALProcedureStore: Starting WAL Procedure Store lease recovery
  60. 2017-09-08 14:28:51,329 INFO [wdc01is-ja-prod-hbase5:16000.activeMasterManager] wal.WALProcedureStore: Lease acquired for flushLogId: 1
  61.  
  62. ........................
  63. 2017-09-08 14:29:08,442 INFO [ProcedureExecutor-18] procedure.ServerCrashProcedure: Finished processing of crashed wdc01is-ja-prod-hbase10.adm01.com,16020,1504096963019
  64. 2017-09-08 14:44:22,158 INFO [WALProcedureStoreSyncThread] wal.WALProcedureStore: Remove log: hdfs://hbase5:9000/hbase/MasterProcWALs/state-00000000000000000001.log
  65. 2017-09-08 14:44:22,158 INFO [WALProcedureStoreSyncThread] wal.WALProcedureStore: Removed logs: [hdfs://hbase5:9000/hbase/MasterProcWALs/state-00000000000000000002.log]
  66. 2017-09-08 15:44:22,164 INFO [WALProcedureStoreSyncThread] wal.WALProcedureStore: Remove log: hdfs://hbase5:9000/hbase/MasterProcWALs/state-00000000000000000002.log
  67. 2017-09-08 15:44:22,164 INFO [WALProcedureStoreSyncThread] wal.WALProcedureStore: Removed logs: [hdfs://hbase5:9000/hbase/MasterProcWALs/state-00000000000000000003.log]
  68. 2017-09-08 16:44:22,171 INFO [WALProcedureStoreSyncThread] wal.WALProcedureStore: Remove log: hdfs://hbase5:9000/hbase/MasterProcWALs/state-00000000000000000003.log
  69. 2017-09-08 16:44:22,171 INFO [WALProcedureStoreSyncThread] wal.WALProcedureStore: Removed logs: [hdfs://hbase5:9000/hbase/MasterProcWALs/state-00000000000000000004.log]
  70. ......
  71. 2017-09-08 20:44:22,202 INFO [WALProcedureStoreSyncThread] wal.WALProcedureStore: Remove log: hdfs://hbase5:9000/hbase/MasterProcWALs/state-00000000000000000007.log
  72. 2017-09-08 20:44:22,202 INFO [WALProcedureStoreSyncThread] wal.WALProcedureStore: Removed logs: [hdfs://hbase5:9000/hbase/MasterProcWALs/state-00000000000000000008.log]
Advertisement
Add Comment
Please, Sign In to add comment