Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- ..............................
- 2017-09-08 14:18:16,880 INFO [wdc01is-ja-prod-hbase5:16000.activeMasterManager] util.FSHDFSUtils: Recovered lease, attempt=0 on file=hdfs://hbase5:9000/hbase/MasterProcWALs/state-00000000000000066826.log after 0ms
- 2017-09-08 14:18:16,890 INFO [wdc01is-ja-prod-hbase5:16000.activeMasterManager] util.FSHDFSUtils: Recover lease on dfs file hdfs://hbase5:9000/hbase/MasterProcWALs/state-00000000000000066827.log
- 2017-09-08 14:18:16,890 INFO [wdc01is-ja-prod-hbase5:16000.activeMasterManager] util.FSHDFSUtils: Recovered lease, attempt=0 on file=hdfs://hbase5:9000/hbase/MasterProcWALs/state-00000000000000066827.log after 0ms
- 2017-09-08 14:18:16,900 INFO [wdc01is-ja-prod-hbase5:16000.activeMasterManager] util.FSHDFSUtils: Recover lease on dfs file hdfs://hbase5:9000/hbase/MasterProcWALs/state-00000000000000066828.log
- 2017-09-08 14:18:16,900 INFO [wdc01is-ja-prod-hbase5:16000.activeMasterManager] util.FSHDFSUtils: Recovered lease, attempt=0 on file=hdfs://hbase5:9000/hbase/MasterProcWALs/state-00000000000000066828.log after 0ms
- 2017-09-08 14:18:18,461 INFO [wdc01is-ja-prod-hbase5.adm01.com,16000,1504878827926_splitLogManager__ChoreService_1] hbase.ScheduledChore: Chore: SplitLogManager Timeout Monitor missed its start time
- 2017-09-08 14:18:18,464 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1343ms
- GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1556ms
- 2017-09-08 14:18:18,561 INFO [wdc01is-ja-prod-hbase5:16000.activeMasterManager] util.FSHDFSUtils: Recover lease on dfs file hdfs://hbase5:9000/hbase/MasterProcWALs/state-00000000000000066829.log
- 2017-09-08 14:18:18,561 INFO [wdc01is-ja-prod-hbase5:16000.activeMasterManager] util.FSHDFSUtils: Recovered lease, attempt=0 on file=hdfs://hbase5:9000/hbase/MasterProcWALs/state-00000000000000066829.log after 0ms
- 2017-09-08 14:18:18,566 INFO [wdc01is-ja-prod-hbase5:16000.activeMasterManager] util.FSHDFSUtils: Recover lease on dfs file hdfs://hbase5:9000/hbase/MasterProcWALs/state-00000000000000066830.log
- 2017-09-08 14:18:18,566 INFO [wdc01is-ja-prod-hbase5:16000.activeMasterManager] util.FSHDFSUtils: Recovered lease, attempt=0 on file=hdfs://hbase5:9000/hbase/MasterProcWALs/state-00000000000000066830.log after 0ms
- 2017-09-08 14:18:20,036 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1072ms
- GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1463ms
- 2017-09-08 14:18:20,037 INFO [wdc01is-ja-prod-hbase5.adm01.com,16000,1504878827926_splitLogManager__ChoreService_1] hbase.ScheduledChore: Chore: SplitLogManager Timeout Monitor missed its start time
- 2017-09-08 14:18:20,037 INFO [wdc01is-ja-prod-hbase5.adm01.com,16000,1504878827926_splitLogManager__ChoreService_1] master.SplitLogManager: total tasks = 1 unassigned = 1 tasks={/hbase/splitWAL/WALs%2Fwdc01is-ja-prod-hbase9.adm01.com%2C16020%2C1504096956944-splitting%2Fwdc01is-ja-prod-hbase9.adm01.com%252C16020%252C1504096956944..meta.1504165675027.meta=last_update = 1504878829356 last_version = -1 cur_worker_name = null status = in_progress incarnation = 1 resubmits = 0 batch = null}
- 2017-09-08 14:18:20,136 FATAL [wdc01is-ja-prod-hbase5:16000.activeMasterManager] master.HMaster: Failed to become active master
- java.lang.OutOfMemoryError: Direct buffer memory
- at java.nio.Bits.reserveMemory(Bits.java:658)
- at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
- at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
- at org.apache.hadoop.hdfs.util.DirectBufferPool.getBuffer(DirectBufferPool.java:59)
- at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.reallocPacketBuf(PacketReceiver.java:272)
- at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:165)
- at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:102)
- at org.apache.hadoop.hdfs.RemoteBlockReader2.readNextPacket(RemoteBlockReader2.java:186)
- at org.apache.hadoop.hdfs.RemoteBlockReader2.read(RemoteBlockReader2.java:146)
- at org.apache.hadoop.hdfs.DFSInputStream$ByteArrayStrategy.doRead(DFSInputStream.java:686)
- at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:742)
- at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:799)
- at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:840)
- at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:648)
- at java.io.FilterInputStream.read(FilterInputStream.java:83)
- at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:232)
- at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
- at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
- at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
- at org.apache.hadoop.hbase.protobuf.generated.ProcedureProtos$ProcedureWALHeader.parseDelimitedFrom(ProcedureProtos.java:3870)
- at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat.readHeader(ProcedureWALFormat.java:137)
- at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFile.open(ProcedureWALFile.java:71)
- at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.initOldLog(WALProcedureStore.java:940)
- at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.initOldLogs(WALProcedureStore.java:903)
- at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:260)
- at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:508)
- at org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1157)
- at org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1079)
- at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:670)
- at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:184)
- at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1729)
- at java.lang.Thread.run(Thread.java:745)
- ........
- 2017-09-08 14:18:20,238 INFO [master/wdc01is-ja-prod-hbase5.adm01.com/10.125.19.34:16000] wal.WALProcedureStore: Stopping the WAL Procedure Store
- ......
- 2017-09-08 14:28:51,312 INFO [wdc01is-ja-prod-hbase5:16000.activeMasterManager] wal.WALProcedureStore: Starting WAL Procedure Store lease recovery
- 2017-09-08 14:28:51,329 INFO [wdc01is-ja-prod-hbase5:16000.activeMasterManager] wal.WALProcedureStore: Lease acquired for flushLogId: 1
- ........................
- 2017-09-08 14:29:08,442 INFO [ProcedureExecutor-18] procedure.ServerCrashProcedure: Finished processing of crashed wdc01is-ja-prod-hbase10.adm01.com,16020,1504096963019
- 2017-09-08 14:44:22,158 INFO [WALProcedureStoreSyncThread] wal.WALProcedureStore: Remove log: hdfs://hbase5:9000/hbase/MasterProcWALs/state-00000000000000000001.log
- 2017-09-08 14:44:22,158 INFO [WALProcedureStoreSyncThread] wal.WALProcedureStore: Removed logs: [hdfs://hbase5:9000/hbase/MasterProcWALs/state-00000000000000000002.log]
- 2017-09-08 15:44:22,164 INFO [WALProcedureStoreSyncThread] wal.WALProcedureStore: Remove log: hdfs://hbase5:9000/hbase/MasterProcWALs/state-00000000000000000002.log
- 2017-09-08 15:44:22,164 INFO [WALProcedureStoreSyncThread] wal.WALProcedureStore: Removed logs: [hdfs://hbase5:9000/hbase/MasterProcWALs/state-00000000000000000003.log]
- 2017-09-08 16:44:22,171 INFO [WALProcedureStoreSyncThread] wal.WALProcedureStore: Remove log: hdfs://hbase5:9000/hbase/MasterProcWALs/state-00000000000000000003.log
- 2017-09-08 16:44:22,171 INFO [WALProcedureStoreSyncThread] wal.WALProcedureStore: Removed logs: [hdfs://hbase5:9000/hbase/MasterProcWALs/state-00000000000000000004.log]
- ......
- 2017-09-08 20:44:22,202 INFO [WALProcedureStoreSyncThread] wal.WALProcedureStore: Remove log: hdfs://hbase5:9000/hbase/MasterProcWALs/state-00000000000000000007.log
- 2017-09-08 20:44:22,202 INFO [WALProcedureStoreSyncThread] wal.WALProcedureStore: Removed logs: [hdfs://hbase5:9000/hbase/MasterProcWALs/state-00000000000000000008.log]
Advertisement
Add Comment
Please, Sign In to add comment