API tools faq
paste
Guest User

Untitled

a guest
Jul 23rd, 2013
220
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 91.23 KB | None | 0 0
raw download clone embed print report
  1. Error Message
  2.  
  3. Unexpected exception, expected<org.apache.hadoop.hbase.exceptions.RegionServerStoppedException> but was<junit.framework.AssertionFailedError>
  4. Stacktrace
  5.  
  6. java.lang.Exception: Unexpected exception, expected<org.apache.hadoop.hbase.exceptions.RegionServerStoppedException> but was<junit.framework.AssertionFailedError>
  7. at org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:28)
  8. at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
  9. at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
  10. at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
  11. at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
  12. at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
  13. at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
  14. at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
  15. at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
  16. at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
  17. at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
  18. at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
  19. at org.junit.runners.Suite.runChild(Suite.java:127)
  20. at org.junit.runners.Suite.runChild(Suite.java:26)
  21. at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
  22. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
  23. at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
  24. at java.util.concurrent.FutureTask.run(FutureTask.java:138)
  25. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  26. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  27. at java.lang.Thread.run(Thread.java:662)
  28. Caused by: junit.framework.AssertionFailedError: Waiting timed out after [40,000] msec
  29. at junit.framework.Assert.fail(Assert.java:57)
  30. at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:193)
  31. at org.apache.hadoop.hbase.HBaseTestingUtility.waitFor(HBaseTestingUtility.java:2774)
  32. at org.apache.hadoop.hbase.client.TestHCM.testClusterStatus(TestHCM.java:158)
  33. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  34. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
  35. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  36. at java.lang.reflect.Method.invoke(Method.java:597)
  37. at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
  38. at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
  39. at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
  40. at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
  41. at org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:19)
  42. ... 20 more
  43. Standard Output
  44.  
  45. 2013-07-21 22:12:11,453 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(802): Starting up minicluster with 1 master(s) and 2 regionserver(s) and 2 datanode(s)
  46. 2013-07-21 22:12:11,483 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(359): Created new mini-cluster data directory: /var/lib/jenkins/workspace/hbase/hbase-server/target/test-data/50101747-f2a6-45c2-92d6-effdbda427e1/dfscluster_ba212393-fac8-4cf7-b6bc-9b754afcab15
  47. 2013-07-21 22:12:11,483 INFO [pool-1-thread-1] hbase.HBaseTestingUtil
  48. ...[truncated 1144000 bytes]...
  49. e has already shut down.
  50. 2013-07-21 22:14:02,655 INFO [pool-1-thread-1] log.Slf4jLog(67): Stopped SelectChannelConnector@localhost:0
  51. 2013-07-21 22:14:02,757 WARN [org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@256eb6bc] namenode.FSNamesystem$ReplicationMonitor(2868): ReplicationMonitor thread received InterruptedException.java.lang.InterruptedException: sleep interrupted
  52. 2013-07-21 22:14:02,791 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(923): Minicluster is down
  53. Standard Error
  54.  
  55. 2013-07-21 22:12:27,551 INFO [pool-1-thread-1] hbase.ResourceChecker(147): before: client.TestHCM#testClusterStatus Thread=275, OpenFileDescriptor=440, MaxFileDescriptor=4096, SystemLoadAverage=265, ProcessCount=240, AvailableMemoryMB=1991, ConnectionCount=6
  56. 2013-07-21 22:12:27,561 DEBUG [pool-1-thread-1] client.HConnectionManager(2466): regionserver/lars-francke.com/176.9.40.230:0 HConnection server-to-server retries=350
  57. 2013-07-21 22:12:27,567 INFO [RS:2;lars-francke:49775] zookeeper.RecoverableZooKeeper(120): Process identifier=regionserver:49775 connecting to ZooKeeper ensemble=localhost:59753
  58. 2013-07-21 22:12:27,574 DEBUG [RS:2;lars-francke:49775-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:49775 Received ZooKeeper Event, type=None, state=SyncConnected, path=null
  59. 2013-07-21 22:12:27,574 DEBUG [RS:2;lars-francke:49775] zookeeper.ZKUtil(431): regionserver:49775 Set watcher on existing znode=/hbase/master
  60. 2013-07-21 22:12:27,575 DEBUG [RS:2;lars-francke:49775-EventThread] zookeeper.ZooKeeperWatcher(384): regionserver:49775-0x14002dc7119000e connected
  61. 2013-07-21 22:12:27,576 DEBUG [RS:2;lars-francke:49775] zookeeper.ZKUtil(431): regionserver:49775-0x14002dc7119000e Set watcher on existing znode=/hbase/running
  62. 2013-07-21 22:12:27,578 INFO [RS:2;lars-francke:49775] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x47f9d0e connecting to ZooKeeper ensemble=localhost:59753
  63. 2013-07-21 22:12:27,582 DEBUG [RS:2;lars-francke:49775-EventThread] zookeeper.ZooKeeperWatcher(307): hconnection-0x47f9d0e Received ZooKeeper Event, type=None, state=SyncConnected, path=null
  64. 2013-07-21 22:12:27,584 DEBUG [RS:2;lars-francke:49775-EventThread] zookeeper.ZooKeeperWatcher(384): hconnection-0x47f9d0e-0x14002dc7119000f connected
  65. 2013-07-21 22:12:27,584 DEBUG [RS:2;lars-francke:49775] catalog.CatalogTracker(192): Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@4d2ad974
  66. 2013-07-21 22:12:27,585 DEBUG [RS:2;lars-francke:49775] zookeeper.ZKUtil(431): regionserver:49775-0x14002dc7119000e Set watcher on existing znode=/hbase/meta-region-server
  67. 2013-07-21 22:12:27,587 INFO [RS:2;lars-francke:49775] regionserver.HRegionServer(708): ClusterId : 38f444ec-34cf-4be3-8191-972a4f6cfaab
  68. 2013-07-21 22:12:27,591 INFO [RS:2;lars-francke:49775] zookeeper.RecoverableZooKeeper(462): Node /hbase/online-snapshot/acquired already exists and this is not a retry
  69. 2013-07-21 22:12:27,592 INFO [RS:2;lars-francke:49775] regionserver.MemStoreFlusher(117): globalMemStoreLimit=675.6m, globalMemStoreLimitLowMark=641.8m, maxHeap=1.6g
  70. 2013-07-21 22:12:27,592 INFO [RS:2;lars-francke:49775] regionserver.HRegionServer$CompactionChecker(1327): CompactionChecker runs every 1sec
  71. 2013-07-21 22:12:27,593 INFO [RS:2;lars-francke:49775] regionserver.HRegionServer(1933): reportForDuty to master=lars-francke.com,59060,1374437536563 with port=49775, startcode=1374437547564
  72. 2013-07-21 22:12:27,594 INFO [RpcServer.handler=3,port=59060] master.ServerManager(367): Registering server=lars-francke.com,49775,1374437547564
  73. 2013-07-21 22:12:27,594 DEBUG [RS:2;lars-francke:49775] regionserver.HRegionServer(1172): Config from master: hbase.rootdir=hdfs://localhost:42066/user/jenkins/hbase
  74. 2013-07-21 22:12:27,595 DEBUG [RS:2;lars-francke:49775] regionserver.HRegionServer(1172): Config from master: fs.default.name=hdfs://localhost:42066
  75. 2013-07-21 22:12:27,599 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:59060-0x14002dc71190000 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs
  76. 2013-07-21 22:12:27,599 WARN [RS:2;lars-francke:49775] hbase.ZNodeClearer(57): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!)
  77. 2013-07-21 22:12:27,601 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(431): master:59060-0x14002dc71190000 Set watcher on existing znode=/hbase/rs/lars-francke.com,49775,1374437547564
  78. 2013-07-21 22:12:27,601 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(431): master:59060-0x14002dc71190000 Set watcher on existing znode=/hbase/rs/lars-francke.com,56454,1374437536906
  79. 2013-07-21 22:12:27,602 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(431): master:59060-0x14002dc71190000 Set watcher on existing znode=/hbase/rs/lars-francke.com,38941,1374437536925
  80. 2013-07-21 22:12:27,602 INFO [RS:2;lars-francke:49775] fs.HFileSystem(244): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks
  81. 2013-07-21 22:12:27,602 DEBUG [RS:2;lars-francke:49775] regionserver.HRegionServer(1424): logdir=hdfs://localhost:42066/user/jenkins/hbase/.logs/lars-francke.com,49775,1374437547564
  82. 2013-07-21 22:12:27,603 INFO [RS:2;lars-francke:49775] wal.FSHLog(350): WAL/HLog configuration: blocksize=64 MB, rollsize=60.8 MB, enabled=true, optionallogflushinternal=1000ms
  83. 2013-07-21 22:12:27,632 INFO [RS:2;lars-francke:49775] wal.FSHLog(522): New WAL /user/jenkins/hbase/.logs/lars-francke.com,49775,1374437547564/lars-francke.com%2C49775%2C1374437547564.1374437547615
  84. 2013-07-21 22:12:27,633 DEBUG [RS:2;lars-francke:49775] executor.ExecutorService(99): Starting executor service name=RS_OPEN_REGION-lars-francke:49775, corePoolSize=3, maxPoolSize=3
  85. 2013-07-21 22:12:27,633 DEBUG [RS:2;lars-francke:49775] executor.ExecutorService(99): Starting executor service name=RS_OPEN_META-lars-francke:49775, corePoolSize=1, maxPoolSize=1
  86. 2013-07-21 22:12:27,633 DEBUG [RS:2;lars-francke:49775] executor.ExecutorService(99): Starting executor service name=RS_CLOSE_REGION-lars-francke:49775, corePoolSize=3, maxPoolSize=3
  87. 2013-07-21 22:12:27,633 DEBUG [RS:2;lars-francke:49775] executor.ExecutorService(99): Starting executor service name=RS_CLOSE_META-lars-francke:49775, corePoolSize=1, maxPoolSize=1
  88. 2013-07-21 22:12:27,647 INFO [RS:2;lars-francke:49775] regionserver.HRegionServer(1203): Serving as lars-francke.com,49775,1374437547564, RpcServer on lars-francke.com/176.9.40.230:49775, sessionid=0x14002dc7119000e
  89. 2013-07-21 22:12:27,647 INFO [SplitLogWorker-lars-francke.com,49775,1374437547564] regionserver.SplitLogWorker(170): SplitLogWorker lars-francke.com,49775,1374437547564 starting
  90. 2013-07-21 22:12:27,647 DEBUG [RS:2;lars-francke:49775] snapshot.RegionServerSnapshotManager(140): Start Snapshot Manager lars-francke.com,49775,1374437547564
  91. 2013-07-21 22:12:27,648 DEBUG [RS:2;lars-francke:49775] procedure.ZKProcedureMemberRpcs(339): Starting procedure member 'lars-francke.com,49775,1374437547564'
  92. 2013-07-21 22:12:27,648 DEBUG [RS:2;lars-francke:49775] procedure.ZKProcedureMemberRpcs(138): Checking for aborted procedures on node: '/hbase/online-snapshot/abort'
  93. 2013-07-21 22:12:27,649 DEBUG [RS:2;lars-francke:49775] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired'
  94. 2013-07-21 22:12:27,649 INFO [SplitLogWorker-lars-francke.com,49775,1374437547564] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x249fe45c connecting to ZooKeeper ensemble=localhost:59753
  95. 2013-07-21 22:12:27,666 DEBUG [SplitLogWorker-lars-francke.com,49775,1374437547564-EventThread] zookeeper.ZooKeeperWatcher(307): hconnection-0x249fe45c Received ZooKeeper Event, type=None, state=SyncConnected, path=null
  96. 2013-07-21 22:12:27,667 DEBUG [SplitLogWorker-lars-francke.com,49775,1374437547564-EventThread] zookeeper.ZooKeeperWatcher(384): hconnection-0x249fe45c-0x14002dc71190010 connected
  97. 2013-07-21 22:12:27,711 DEBUG [RpcServer.handler=0,port=59060] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testClusterStatus/write-master:590600000000000
  98. 2013-07-21 22:12:27,714 DEBUG [RpcServer.handler=0,port=59060] client.ClientScanner(198): Finished region={ENCODED => 1028785192, NAME => '.META.,,1', STARTKEY => '', ENDKEY => ''}
  99. 2013-07-21 22:12:27,732 INFO [MASTER_TABLE_OPERATIONS-lars-francke:59060-0] handler.CreateTableHandler(146): Create table testClusterStatus
  100. 2013-07-21 22:12:27,736 DEBUG [pool-1-thread-1] client.ClientScanner(198): Finished region={ENCODED => 1028785192, NAME => '.META.,,1', STARTKEY => '', ENDKEY => ''}
  101. 2013-07-21 22:12:27,839 DEBUG [pool-1-thread-1] client.ClientScanner(198): Finished region={ENCODED => 1028785192, NAME => '.META.,,1', STARTKEY => '', ENDKEY => ''}
  102. 2013-07-21 22:12:28,043 DEBUG [pool-1-thread-1] client.ClientScanner(198): Finished region={ENCODED => 1028785192, NAME => '.META.,,1', STARTKEY => '', ENDKEY => ''}
  103. 2013-07-21 22:12:28,198 INFO [RegionOpenAndInitThread-testClusterStatus-1] regionserver.HRegion(3921): creating HRegion testClusterStatus HTD == 'testClusterStatus', {NAME => 'cf', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => '2147483647', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', ENCODE_ON_DISK => 'true', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:42066/user/jenkins/hbase/.tmp Table name == testClusterStatus
  104. 2013-07-21 22:12:28,346 DEBUG [pool-1-thread-1] client.ClientScanner(198): Finished region={ENCODED => 1028785192, NAME => '.META.,,1', STARTKEY => '', ENDKEY => ''}
  105. 2013-07-21 22:12:28,681 DEBUG [RegionOpenAndInitThread-testClusterStatus-1] regionserver.HRegion(535): Instantiated testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09.
  106. 2013-07-21 22:12:28,682 DEBUG [RegionOpenAndInitThread-testClusterStatus-1] regionserver.HRegion(950): Closing testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09.: disabling compactions & flushes
  107. 2013-07-21 22:12:28,682 DEBUG [RegionOpenAndInitThread-testClusterStatus-1] regionserver.HRegion(972): Updates disabled for region testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09.
  108. 2013-07-21 22:12:28,682 INFO [RegionOpenAndInitThread-testClusterStatus-1] regionserver.HRegion(1030): Closed testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09.
  109. 2013-07-21 22:12:28,701 INFO [MASTER_TABLE_OPERATIONS-lars-francke:59060-0] catalog.MetaEditor(254): Added 1 regions in META
  110. 2013-07-21 22:12:28,701 INFO [MASTER_TABLE_OPERATIONS-lars-francke:59060-0] master.AssignmentManager(2458): Bulk assigning 1 region(s) across 3 server(s), round-robin=true
  111. 2013-07-21 22:12:28,702 DEBUG [lars-francke.com,59060,1374437536563-GeneralBulkAssigner-2] master.AssignmentManager(1503): Assigning 1 region(s) to lars-francke.com,56454,1374437536906
  112. 2013-07-21 22:12:28,702 DEBUG [MASTER_TABLE_OPERATIONS-lars-francke:59060-0] master.GeneralBulkAssigner(177): Timeout-on-RIT=151000
  113. 2013-07-21 22:12:28,702 DEBUG [lars-francke.com,59060,1374437536563-GeneralBulkAssigner-2] zookeeper.ZKAssign(177): master:59060-0x14002dc71190000 Async create of unassigned node for f02a31af489a311d2a63454a13db9b09 with OFFLINE state
  114. 2013-07-21 22:12:28,706 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:59060-0x14002dc71190000 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition
  115. 2013-07-21 22:12:28,707 DEBUG [pool-1-thread-1-EventThread] master.OfflineCallback(69): rs={f02a31af489a311d2a63454a13db9b09 state=OFFLINE, ts=1374437548701, server=null}, server=lars-francke.com,56454,1374437536906
  116. 2013-07-21 22:12:28,707 DEBUG [pool-1-thread-1-EventThread] master.OfflineCallback$ExistCallback(106): rs={f02a31af489a311d2a63454a13db9b09 state=OFFLINE, ts=1374437548701, server=null}, server=lars-francke.com,56454,1374437536906
  117. 2013-07-21 22:12:28,707 INFO [lars-francke.com,59060,1374437536563-GeneralBulkAssigner-2] master.AssignmentManager(1539): lars-francke.com,56454,1374437536906 unassigned znodes=1 of total=1
  118. 2013-07-21 22:12:28,707 INFO [lars-francke.com,59060,1374437536563-GeneralBulkAssigner-2] master.RegionStates(265): Transitioned from {f02a31af489a311d2a63454a13db9b09 state=OFFLINE, ts=1374437548702, server=null} to {f02a31af489a311d2a63454a13db9b09 state=PENDING_OPEN, ts=1374437548707, server=lars-francke.com,56454,1374437536906}
  119. 2013-07-21 22:12:28,708 INFO [RpcServer.handler=3,port=56454] regionserver.HRegionServer(3452): Open testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09.
  120. 2013-07-21 22:12:28,711 DEBUG [RS_OPEN_REGION-lars-francke:56454-2] zookeeper.ZKAssign(786): regionserver:56454-0x14002dc71190001 Transitioning f02a31af489a311d2a63454a13db9b09 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
  121. 2013-07-21 22:12:28,711 DEBUG [lars-francke.com,59060,1374437536563-GeneralBulkAssigner-2] master.AssignmentManager(1661): Bulk assigning done for lars-francke.com,56454,1374437536906
  122. 2013-07-21 22:12:28,711 DEBUG [MASTER_TABLE_OPERATIONS-lars-francke:59060-0] master.GeneralBulkAssigner(153): bulk assigning total 1 regions to 3 servers, took 9ms, with 1 regions still in transition
  123. 2013-07-21 22:12:28,711 INFO [MASTER_TABLE_OPERATIONS-lars-francke:59060-0] master.AssignmentManager(2465): Bulk assigning done
  124. 2013-07-21 22:12:28,799 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:59060-0x14002dc71190000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/f02a31af489a311d2a63454a13db9b09
  125. 2013-07-21 22:12:28,799 DEBUG [RS_OPEN_REGION-lars-francke:56454-2] zookeeper.ZKAssign(861): regionserver:56454-0x14002dc71190001 Successfully transitioned node f02a31af489a311d2a63454a13db9b09 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
  126. 2013-07-21 22:12:28,799 INFO [RS_OPEN_REGION-lars-francke:56454-2] regionserver.HRegion(4082): Open {ENCODED => f02a31af489a311d2a63454a13db9b09, NAME => 'testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09.', STARTKEY => '', ENDKEY => ''}
  127. 2013-07-21 22:12:28,799 DEBUG [RS_OPEN_REGION-lars-francke:56454-2] regionserver.MetricsRegionSourceImpl(58): Creating new MetricsRegionSourceImpl for table testClusterStatus f02a31af489a311d2a63454a13db9b09
  128. 2013-07-21 22:12:28,800 DEBUG [RS_OPEN_REGION-lars-francke:56454-2] regionserver.HRegion(535): Instantiated testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09.
  129. 2013-07-21 22:12:28,800 DEBUG [AM.ZK.Worker-pool-4-thread-4] master.AssignmentManager(767): Handling transition=RS_ZK_REGION_OPENING, server=lars-francke.com,56454,1374437536906, region=f02a31af489a311d2a63454a13db9b09, current state from region state map ={f02a31af489a311d2a63454a13db9b09 state=PENDING_OPEN, ts=1374437548707, server=lars-francke.com,56454,1374437536906}
  130. 2013-07-21 22:12:28,800 INFO [AM.ZK.Worker-pool-4-thread-4] master.RegionStates(265): Transitioned from {f02a31af489a311d2a63454a13db9b09 state=PENDING_OPEN, ts=1374437548707, server=lars-francke.com,56454,1374437536906} to {f02a31af489a311d2a63454a13db9b09 state=OPENING, ts=1374437548800, server=lars-francke.com,56454,1374437536906}
  131. 2013-07-21 22:12:28,824 DEBUG [MASTER_TABLE_OPERATIONS-lars-francke:59060-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testClusterStatus/write-master:590600000000000
  132. 2013-07-21 22:12:28,840 INFO [StoreOpener-f02a31af489a311d2a63454a13db9b09-1] compactions.CompactionConfiguration(85): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000
  133. 2013-07-21 22:12:28,843 INFO [RS_OPEN_REGION-lars-francke:56454-2] regionserver.HRegion(630): Onlined f02a31af489a311d2a63454a13db9b09; next sequenceid=1
  134. 2013-07-21 22:12:28,843 DEBUG [RS_OPEN_REGION-lars-francke:56454-2] zookeeper.ZKAssign(598): regionserver:56454-0x14002dc71190001 Attempting to retransition the opening state of node f02a31af489a311d2a63454a13db9b09
  135. 2013-07-21 22:12:28,844 INFO [PostOpenDeployTasks:f02a31af489a311d2a63454a13db9b09] regionserver.HRegionServer(1701): Post open deploy tasks for region=testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09.
  136. 2013-07-21 22:12:28,847 INFO [PostOpenDeployTasks:f02a31af489a311d2a63454a13db9b09] catalog.MetaEditor(432): Updated row testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09. with server=lars-francke.com,56454,1374437536906
  137. 2013-07-21 22:12:28,847 INFO [PostOpenDeployTasks:f02a31af489a311d2a63454a13db9b09] regionserver.HRegionServer(1726): Done with post open deploy task for region=testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09.
  138. 2013-07-21 22:12:28,847 DEBUG [RS_OPEN_REGION-lars-francke:56454-2] zookeeper.ZKAssign(786): regionserver:56454-0x14002dc71190001 Transitioning f02a31af489a311d2a63454a13db9b09 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED
  139. 2013-07-21 22:12:28,856 DEBUG [pool-1-thread-1] client.ClientScanner(198): Finished region={ENCODED => 1028785192, NAME => '.META.,,1', STARTKEY => '', ENDKEY => ''}
  140. 2013-07-21 22:12:28,859 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:59060-0x14002dc71190000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/f02a31af489a311d2a63454a13db9b09
  141. 2013-07-21 22:12:28,859 DEBUG [RS_OPEN_REGION-lars-francke:56454-2] zookeeper.ZKAssign(861): regionserver:56454-0x14002dc71190001 Successfully transitioned node f02a31af489a311d2a63454a13db9b09 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED
  142. 2013-07-21 22:12:28,859 DEBUG [RS_OPEN_REGION-lars-francke:56454-2] handler.OpenRegionHandler(373): region transitioned to opened in zookeeper: {ENCODED => f02a31af489a311d2a63454a13db9b09, NAME => 'testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09.', STARTKEY => '', ENDKEY => ''}, server: lars-francke.com,56454,1374437536906
  143. 2013-07-21 22:12:28,859 DEBUG [RS_OPEN_REGION-lars-francke:56454-2] handler.OpenRegionHandler(186): Opened testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09. on server:lars-francke.com,56454,1374437536906
  144. 2013-07-21 22:12:28,860 DEBUG [AM.ZK.Worker-pool-4-thread-2] master.AssignmentManager(767): Handling transition=RS_ZK_REGION_OPENED, server=lars-francke.com,56454,1374437536906, region=f02a31af489a311d2a63454a13db9b09, current state from region state map ={f02a31af489a311d2a63454a13db9b09 state=OPENING, ts=1374437548800, server=lars-francke.com,56454,1374437536906}
  145. 2013-07-21 22:12:28,860 INFO [AM.ZK.Worker-pool-4-thread-2] master.RegionStates(265): Transitioned from {f02a31af489a311d2a63454a13db9b09 state=OPENING, ts=1374437548800, server=lars-francke.com,56454,1374437536906} to {f02a31af489a311d2a63454a13db9b09 state=OPEN, ts=1374437548860, server=lars-francke.com,56454,1374437536906}
  146. 2013-07-21 22:12:28,860 INFO [pool-1-thread-1] hbase.Waiter(174): Waiting up to [60,000] milli-secs(wait.for.ratio=[1])
  147. 2013-07-21 22:12:28,860 DEBUG [MASTER_OPEN_REGION-lars-francke:59060-4] handler.OpenedRegionHandler(145): Handling OPENED event for f02a31af489a311d2a63454a13db9b09 from lars-francke.com,56454,1374437536906; deleting unassigned node
  148. 2013-07-21 22:12:28,860 DEBUG [MASTER_OPEN_REGION-lars-francke:59060-4] zookeeper.ZKAssign(405): master:59060-0x14002dc71190000 Deleting existing unassigned node for f02a31af489a311d2a63454a13db9b09 that is in expected state RS_ZK_REGION_OPENED
  149. 2013-07-21 22:12:28,864 DEBUG [pool-1-thread-1] client.ClientScanner(198): Finished region={ENCODED => 1028785192, NAME => '.META.,,1', STARTKEY => '', ENDKEY => ''}
  150. 2013-07-21 22:12:28,924 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:59060-0x14002dc71190000 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/f02a31af489a311d2a63454a13db9b09
  151. 2013-07-21 22:12:28,924 DEBUG [AM.ZK.Worker-pool-4-thread-3] master.AssignmentManager$4(1218): The znode of testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09. has been deleted, region state: {f02a31af489a311d2a63454a13db9b09 state=OPEN, ts=1374437548860, server=lars-francke.com,56454,1374437536906}
  152. 2013-07-21 22:12:28,924 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:59060-0x14002dc71190000 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition
  153. 2013-07-21 22:12:28,924 INFO [AM.ZK.Worker-pool-4-thread-3] master.RegionStates(301): Onlined f02a31af489a311d2a63454a13db9b09 on lars-francke.com,56454,1374437536906
  154. 2013-07-21 22:12:28,925 INFO [AM.ZK.Worker-pool-4-thread-3] master.AssignmentManager$4(1223): The master has opened testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09. that was online on lars-francke.com,56454,1374437536906
  155. 2013-07-21 22:12:28,925 DEBUG [MASTER_OPEN_REGION-lars-francke:59060-4] zookeeper.ZKAssign(434): master:59060-0x14002dc71190000 Successfully deleted unassigned node for region f02a31af489a311d2a63454a13db9b09 in expected state RS_ZK_REGION_OPENED
  156. 2013-07-21 22:12:28,930 DEBUG [pool-1-thread-1] client.ClientScanner(198): Finished region={ENCODED => 1028785192, NAME => '.META.,,1', STARTKEY => '', ENDKEY => ''}
  157. 2013-07-21 22:12:28,934 DEBUG [pool-1-thread-1] client.ClientScanner(198): Finished region={ENCODED => 1028785192, NAME => '.META.,,1', STARTKEY => '', ENDKEY => ''}
  158. 2013-07-21 22:12:28,935 INFO [RpcServer.handler=4,port=59060] master.HMaster(1610): Added move plan hri=testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09., src=lars-francke.com,56454,1374437536906, dest=lars-francke.com,49775,1374437547564, running balancer
  159. 2013-07-21 22:12:28,935 DEBUG [RpcServer.handler=4,port=59060] master.AssignmentManager(2205): Starting unassign of testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09. (offlining)
  160. 2013-07-21 22:12:28,935 DEBUG [RpcServer.handler=4,port=59060] zookeeper.ZKAssign(481): master:59060-0x14002dc71190000 Creating unassigned node for f02a31af489a311d2a63454a13db9b09 in a CLOSING state
  161. 2013-07-21 22:12:28,965 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:59060-0x14002dc71190000 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition
  162. 2013-07-21 22:12:28,965 INFO [RpcServer.handler=4,port=59060] master.RegionStates(265): Transitioned from {f02a31af489a311d2a63454a13db9b09 state=OPEN, ts=1374437548924, server=lars-francke.com,56454,1374437536906} to {f02a31af489a311d2a63454a13db9b09 state=PENDING_CLOSE, ts=1374437548965, server=lars-francke.com,56454,1374437536906}
  163. 2013-07-21 22:12:28,966 INFO [RpcServer.handler=2,port=56454] regionserver.HRegionServer(3572): Received close region: f02a31af489a311d2a63454a13db9b09Transitioning in ZK: yes. Version of ZK closing node:0. Destination server:lars-francke.com,49775,1374437547564
  164. 2013-07-21 22:12:28,966 DEBUG [RS_CLOSE_REGION-lars-francke:56454-1] handler.CloseRegionHandler(125): Processing close of testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09.
  165. 2013-07-21 22:12:28,966 DEBUG [RpcServer.handler=4,port=59060] master.AssignmentManager(1699): Sent CLOSE to lars-francke.com,56454,1374437536906 for region testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09.
  166. 2013-07-21 22:12:28,967 DEBUG [RS_CLOSE_REGION-lars-francke:56454-1] regionserver.HRegion(950): Closing testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09.: disabling compactions & flushes
  167. 2013-07-21 22:12:28,967 DEBUG [RS_CLOSE_REGION-lars-francke:56454-1] regionserver.HRegion(972): Updates disabled for region testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09.
  168. 2013-07-21 22:12:28,968 INFO [StoreCloserThread-testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09.-1] regionserver.HStore(662): Closed cf
  169. 2013-07-21 22:12:28,968 INFO [RS_CLOSE_REGION-lars-francke:56454-1] regionserver.HRegion(1030): Closed testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09.
  170. 2013-07-21 22:12:28,968 INFO [RS_CLOSE_REGION-lars-francke:56454-1] regionserver.HRegionServer(4105): Adding moved region record: f02a31af489a311d2a63454a13db9b09 to lars-francke.com,49775,1374437547564:49775 as of 1
  171. 2013-07-21 22:12:28,968 DEBUG [RS_CLOSE_REGION-lars-francke:56454-1] zookeeper.ZKAssign(786): regionserver:56454-0x14002dc71190001 Transitioning f02a31af489a311d2a63454a13db9b09 from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED
  172. 2013-07-21 22:12:29,016 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:59060-0x14002dc71190000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/f02a31af489a311d2a63454a13db9b09
  173. 2013-07-21 22:12:29,016 DEBUG [RS_CLOSE_REGION-lars-francke:56454-1] zookeeper.ZKAssign(861): regionserver:56454-0x14002dc71190001 Successfully transitioned node f02a31af489a311d2a63454a13db9b09 from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED
  174. 2013-07-21 22:12:29,016 DEBUG [RS_CLOSE_REGION-lars-francke:56454-1] handler.CloseRegionHandler(168): set region closed state in zk successfully for region testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09. sn name: lars-francke.com,56454,1374437536906
  175. 2013-07-21 22:12:29,016 DEBUG [RS_CLOSE_REGION-lars-francke:56454-1] handler.CloseRegionHandler(177): Closed region testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09.
  176. 2013-07-21 22:12:29,016 DEBUG [AM.ZK.Worker-pool-4-thread-8] master.AssignmentManager(767): Handling transition=RS_ZK_REGION_CLOSED, server=lars-francke.com,56454,1374437536906, region=f02a31af489a311d2a63454a13db9b09, current state from region state map ={f02a31af489a311d2a63454a13db9b09 state=PENDING_CLOSE, ts=1374437548965, server=lars-francke.com,56454,1374437536906}
  177. 2013-07-21 22:12:29,017 WARN [AM.ZK.Worker-pool-4-thread-8] master.RegionStates(250): Closed region f02a31af489a311d2a63454a13db9b09 still on lars-francke.com,56454,1374437536906? Ignored, reset it to null
  178. 2013-07-21 22:12:29,017 INFO [AM.ZK.Worker-pool-4-thread-8] master.RegionStates(265): Transitioned from {f02a31af489a311d2a63454a13db9b09 state=PENDING_CLOSE, ts=1374437548965, server=lars-francke.com,56454,1374437536906} to {f02a31af489a311d2a63454a13db9b09 state=CLOSED, ts=1374437549017, server=null}
  179. 2013-07-21 22:12:29,017 DEBUG [MASTER_CLOSE_REGION-lars-francke:59060-1] handler.ClosedRegionHandler(92): Handling CLOSED event for f02a31af489a311d2a63454a13db9b09
  180. 2013-07-21 22:12:29,017 DEBUG [MASTER_CLOSE_REGION-lars-francke:59060-1] master.AssignmentManager(1765): Forcing OFFLINE; was={f02a31af489a311d2a63454a13db9b09 state=CLOSED, ts=1374437549017, server=null}
  181. 2013-07-21 22:12:29,017 INFO [MASTER_CLOSE_REGION-lars-francke:59060-1] master.RegionStates(265): Transitioned from {f02a31af489a311d2a63454a13db9b09 state=CLOSED, ts=1374437549017, server=null} to {f02a31af489a311d2a63454a13db9b09 state=OFFLINE, ts=1374437549017, server=null}
  182. 2013-07-21 22:12:29,017 DEBUG [MASTER_CLOSE_REGION-lars-francke:59060-1] master.AssignmentManager(2096): Found an existing plan for testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09. destination server is lars-francke.com,49775,1374437547564 accepted as a dest server = true
  183. 2013-07-21 22:12:29,017 DEBUG [MASTER_CLOSE_REGION-lars-francke:59060-1] master.AssignmentManager(2135): Using pre-existing plan for testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09.; plan=hri=testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09., src=lars-francke.com,56454,1374437536906, dest=lars-francke.com,49775,1374437547564
  184. 2013-07-21 22:12:29,017 DEBUG [MASTER_CLOSE_REGION-lars-francke:59060-1] zookeeper.ZKAssign(208): master:59060-0x14002dc71190000 Creating (or updating) unassigned node for f02a31af489a311d2a63454a13db9b09 with OFFLINE state
  185. 2013-07-21 22:12:29,031 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:59060-0x14002dc71190000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/f02a31af489a311d2a63454a13db9b09
  186. 2013-07-21 22:12:29,032 INFO [MASTER_CLOSE_REGION-lars-francke:59060-1] master.AssignmentManager(1854): Assigning testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09. to lars-francke.com,49775,1374437547564
  187. 2013-07-21 22:12:29,032 INFO [MASTER_CLOSE_REGION-lars-francke:59060-1] master.RegionStates(265): Transitioned from {f02a31af489a311d2a63454a13db9b09 state=OFFLINE, ts=1374437549017, server=null} to {f02a31af489a311d2a63454a13db9b09 state=PENDING_OPEN, ts=1374437549032, server=lars-francke.com,49775,1374437547564}
  188. 2013-07-21 22:12:29,032 DEBUG [MASTER_CLOSE_REGION-lars-francke:59060-1] master.ServerManager(735): New admin connection to lars-francke.com,49775,1374437547564
  189. 2013-07-21 22:12:29,032 DEBUG [AM.ZK.Worker-pool-4-thread-9] master.AssignmentManager(767): Handling transition=M_ZK_REGION_OFFLINE, server=lars-francke.com,49775,1374437547564, region=f02a31af489a311d2a63454a13db9b09, current state from region state map ={f02a31af489a311d2a63454a13db9b09 state=PENDING_OPEN, ts=1374437549032, server=lars-francke.com,49775,1374437547564}
  190. 2013-07-21 22:12:29,033 INFO [RpcServer.handler=0,port=49775] regionserver.HRegionServer(3452): Open testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09.
  191. 2013-07-21 22:12:29,036 DEBUG [RS_OPEN_REGION-lars-francke:49775-0] zookeeper.ZKAssign(786): regionserver:49775-0x14002dc7119000e Transitioning f02a31af489a311d2a63454a13db9b09 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
  192. 2013-07-21 22:12:29,049 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:59060-0x14002dc71190000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/f02a31af489a311d2a63454a13db9b09
  193. 2013-07-21 22:12:29,049 DEBUG [RS_OPEN_REGION-lars-francke:49775-0] zookeeper.ZKAssign(861): regionserver:49775-0x14002dc7119000e Successfully transitioned node f02a31af489a311d2a63454a13db9b09 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
  194. 2013-07-21 22:12:29,050 INFO [RS_OPEN_REGION-lars-francke:49775-0] regionserver.HRegion(4082): Open {ENCODED => f02a31af489a311d2a63454a13db9b09, NAME => 'testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09.', STARTKEY => '', ENDKEY => ''}
  195. 2013-07-21 22:12:29,050 DEBUG [AM.ZK.Worker-pool-4-thread-12] master.AssignmentManager(767): Handling transition=RS_ZK_REGION_OPENING, server=lars-francke.com,49775,1374437547564, region=f02a31af489a311d2a63454a13db9b09, current state from region state map ={f02a31af489a311d2a63454a13db9b09 state=PENDING_OPEN, ts=1374437549032, server=lars-francke.com,49775,1374437547564}
  196. 2013-07-21 22:12:29,050 INFO [AM.ZK.Worker-pool-4-thread-12] master.RegionStates(265): Transitioned from {f02a31af489a311d2a63454a13db9b09 state=PENDING_OPEN, ts=1374437549032, server=lars-francke.com,49775,1374437547564} to {f02a31af489a311d2a63454a13db9b09 state=OPENING, ts=1374437549050, server=lars-francke.com,49775,1374437547564}
  197. 2013-07-21 22:12:29,051 DEBUG [RS_OPEN_REGION-lars-francke:49775-0] regionserver.MetricsRegionSourceImpl(58): Creating new MetricsRegionSourceImpl for table testClusterStatus f02a31af489a311d2a63454a13db9b09
  198. 2013-07-21 22:12:29,051 DEBUG [RS_OPEN_REGION-lars-francke:49775-0] regionserver.HRegion(535): Instantiated testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09.
  199. 2013-07-21 22:12:29,053 INFO [StoreOpener-f02a31af489a311d2a63454a13db9b09-1] compactions.CompactionConfiguration(85): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000
  200. 2013-07-21 22:12:29,056 INFO [RS_OPEN_REGION-lars-francke:49775-0] regionserver.HRegion(630): Onlined f02a31af489a311d2a63454a13db9b09; next sequenceid=1
  201. 2013-07-21 22:12:29,056 DEBUG [RS_OPEN_REGION-lars-francke:49775-0] zookeeper.ZKAssign(598): regionserver:49775-0x14002dc7119000e Attempting to retransition the opening state of node f02a31af489a311d2a63454a13db9b09
  202. 2013-07-21 22:12:29,058 INFO [PostOpenDeployTasks:f02a31af489a311d2a63454a13db9b09] regionserver.HRegionServer(1701): Post open deploy tasks for region=testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09.
  203. 2013-07-21 22:12:29,063 INFO [PostOpenDeployTasks:f02a31af489a311d2a63454a13db9b09] catalog.MetaEditor(432): Updated row testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09. with server=lars-francke.com,49775,1374437547564
  204. 2013-07-21 22:12:29,063 INFO [PostOpenDeployTasks:f02a31af489a311d2a63454a13db9b09] regionserver.HRegionServer(1726): Done with post open deploy task for region=testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09.
  205. 2013-07-21 22:12:29,063 DEBUG [RS_OPEN_REGION-lars-francke:49775-0] zookeeper.ZKAssign(786): regionserver:49775-0x14002dc7119000e Transitioning f02a31af489a311d2a63454a13db9b09 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED
  206. 2013-07-21 22:12:29,073 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:59060-0x14002dc71190000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/f02a31af489a311d2a63454a13db9b09
  207. 2013-07-21 22:12:29,073 DEBUG [RS_OPEN_REGION-lars-francke:49775-0] zookeeper.ZKAssign(861): regionserver:49775-0x14002dc7119000e Successfully transitioned node f02a31af489a311d2a63454a13db9b09 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED
  208. 2013-07-21 22:12:29,073 DEBUG [RS_OPEN_REGION-lars-francke:49775-0] handler.OpenRegionHandler(373): region transitioned to opened in zookeeper: {ENCODED => f02a31af489a311d2a63454a13db9b09, NAME => 'testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09.', STARTKEY => '', ENDKEY => ''}, server: lars-francke.com,49775,1374437547564
  209. 2013-07-21 22:12:29,073 DEBUG [RS_OPEN_REGION-lars-francke:49775-0] handler.OpenRegionHandler(186): Opened testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09. on server:lars-francke.com,49775,1374437547564
  210. 2013-07-21 22:12:29,073 DEBUG [AM.ZK.Worker-pool-4-thread-10] master.AssignmentManager(767): Handling transition=RS_ZK_REGION_OPENED, server=lars-francke.com,49775,1374437547564, region=f02a31af489a311d2a63454a13db9b09, current state from region state map ={f02a31af489a311d2a63454a13db9b09 state=OPENING, ts=1374437549050, server=lars-francke.com,49775,1374437547564}
  211. 2013-07-21 22:12:29,074 INFO [AM.ZK.Worker-pool-4-thread-10] master.RegionStates(265): Transitioned from {f02a31af489a311d2a63454a13db9b09 state=OPENING, ts=1374437549050, server=lars-francke.com,49775,1374437547564} to {f02a31af489a311d2a63454a13db9b09 state=OPEN, ts=1374437549074, server=lars-francke.com,49775,1374437547564}
  212. 2013-07-21 22:12:29,074 DEBUG [MASTER_OPEN_REGION-lars-francke:59060-0] handler.OpenedRegionHandler(145): Handling OPENED event for f02a31af489a311d2a63454a13db9b09 from lars-francke.com,49775,1374437547564; deleting unassigned node
  213. 2013-07-21 22:12:29,074 DEBUG [MASTER_OPEN_REGION-lars-francke:59060-0] zookeeper.ZKAssign(405): master:59060-0x14002dc71190000 Deleting existing unassigned node for f02a31af489a311d2a63454a13db9b09 that is in expected state RS_ZK_REGION_OPENED
  214. 2013-07-21 22:12:29,081 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:59060-0x14002dc71190000 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/f02a31af489a311d2a63454a13db9b09
  215. 2013-07-21 22:12:29,081 DEBUG [AM.ZK.Worker-pool-4-thread-11] master.AssignmentManager$4(1218): The znode of testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09. has been deleted, region state: {f02a31af489a311d2a63454a13db9b09 state=OPEN, ts=1374437549074, server=lars-francke.com,49775,1374437547564}
  216. 2013-07-21 22:12:29,081 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:59060-0x14002dc71190000 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition
  217. 2013-07-21 22:12:29,081 INFO [AM.ZK.Worker-pool-4-thread-11] master.RegionStates(301): Onlined f02a31af489a311d2a63454a13db9b09 on lars-francke.com,49775,1374437547564
  218. 2013-07-21 22:12:29,081 INFO [AM.ZK.Worker-pool-4-thread-11] master.RegionStates(309): Offlined f02a31af489a311d2a63454a13db9b09 from lars-francke.com,56454,1374437536906
  219. 2013-07-21 22:12:29,081 DEBUG [MASTER_OPEN_REGION-lars-francke:59060-0] zookeeper.ZKAssign(434): master:59060-0x14002dc71190000 Successfully deleted unassigned node for region f02a31af489a311d2a63454a13db9b09 in expected state RS_ZK_REGION_OPENED
  220. 2013-07-21 22:12:29,081 INFO [AM.ZK.Worker-pool-4-thread-11] master.AssignmentManager$4(1223): The master has opened testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09. that was online on lars-francke.com,49775,1374437547564
  221. 2013-07-21 22:12:29,088 DEBUG [pool-1-thread-1] client.ClientScanner(198): Finished region={ENCODED => 1028785192, NAME => '.META.,,1', STARTKEY => '', ENDKEY => ''}
  222. 2013-07-21 22:12:29,114 FATAL [pool-1-thread-1] regionserver.HRegionServer(1752): ABORTING region server lars-francke.com,49775,1374437547564: I'm dead
  223. 2013-07-21 22:12:29,115 FATAL [pool-1-thread-1] regionserver.HRegionServer(1758): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint]
  224. 2013-07-21 22:12:29,116 ERROR [RpcServer.handler=2,port=59060] master.HMaster(1289): Region server lars-francke.com,49775,1374437547564 reported a fatal error:
  225. ABORTING region server lars-francke.com,49775,1374437547564: I'm dead
  226. 2013-07-21 22:12:29,117 INFO [pool-1-thread-1] regionserver.HRegionServer(1683): STOPPED: I'm dead
  227. 2013-07-21 22:12:29,117 INFO [RS:2;lars-francke:49775] regionserver.SplitLogWorker(596): Sending interrupt to stop the worker thread
  228. 2013-07-21 22:12:29,117 INFO [pool-1-thread-1] hbase.Waiter(174): Waiting up to [40,000] milli-secs(wait.for.ratio=[1])
  229. 2013-07-21 22:12:29,118 INFO [SplitLogWorker-lars-francke.com,49775,1374437547564] regionserver.SplitLogWorker(281): SplitLogWorker interrupted while waiting for task, exiting: java.lang.InterruptedException
  230. 2013-07-21 22:12:29,118 INFO [SplitLogWorker-lars-francke.com,49775,1374437547564] regionserver.SplitLogWorker(205): SplitLogWorker lars-francke.com,49775,1374437547564 exiting
  231. 2013-07-21 22:12:29,118 INFO [Thread-385] regionserver.MemStoreFlusher$FlushHandler(267): Thread-385 exiting
  232. 2013-07-21 22:12:29,118 INFO [RS:2;lars-francke:49775.logRoller] regionserver.LogRoller(119): LogRoller exiting.
  233. 2013-07-21 22:12:29,118 INFO [RS:2;lars-francke:49775.compactionChecker] hbase.Chore(93): RS:2;lars-francke:49775.compactionChecker exiting
  234. 2013-07-21 22:12:29,118 INFO [RS:2;lars-francke:49775] snapshot.RegionServerSnapshotManager(151): Stopping RegionServerSnapshotManager abruptly.
  235. 2013-07-21 22:12:29,119 INFO [RS:2;lars-francke:49775] regionserver.HRegionServer(916): aborting server lars-francke.com,49775,1374437547564
  236. 2013-07-21 22:12:29,119 DEBUG [RS_CLOSE_REGION-lars-francke:49775-0] handler.CloseRegionHandler(125): Processing close of testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09.
  237. 2013-07-21 22:12:29,119 DEBUG [RS:2;lars-francke:49775] catalog.CatalogTracker(208): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@4d2ad974
  238. 2013-07-21 22:12:29,119 DEBUG [RS_CLOSE_REGION-lars-francke:49775-0] regionserver.HRegion(950): Closing testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09.: disabling compactions & flushes
  239. 2013-07-21 22:12:29,119 INFO [RS:2;lars-francke:49775] client.HConnectionManager$HConnectionImplementation(1628): Closing zookeeper sessionid=0x14002dc7119000f
  240. 2013-07-21 22:12:29,119 DEBUG [RS_CLOSE_REGION-lars-francke:49775-0] regionserver.HRegion(972): Updates disabled for region testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09.
  241. 2013-07-21 22:12:29,120 INFO [StoreCloserThread-testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09.-1] regionserver.HStore(662): Closed cf
  242. 2013-07-21 22:12:29,120 INFO [RS_CLOSE_REGION-lars-francke:49775-0] regionserver.HRegion(1030): Closed testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09.
  243. 2013-07-21 22:12:29,120 DEBUG [RS_CLOSE_REGION-lars-francke:49775-0] handler.CloseRegionHandler(177): Closed region testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09.
  244. 2013-07-21 22:12:29,123 INFO [RS:2;lars-francke:49775] regionserver.HRegionServer(939): stopping server lars-francke.com,49775,1374437547564; all regions closed.
  245. 2013-07-21 22:12:29,123 INFO [RS:2;lars-francke:49775.logSyncer] wal.FSHLog$LogSyncer(967): RS:2;lars-francke:49775.logSyncer exiting
  246. 2013-07-21 22:12:29,123 DEBUG [RS:2;lars-francke:49775] wal.FSHLog(808): Closing WAL writer in hdfs://localhost:42066/user/jenkins/hbase/.logs/lars-francke.com,49775,1374437547564
  247. 2013-07-21 22:12:29,239 INFO [RS:2;lars-francke:49775] regionserver.Leases(124): RS:2;lars-francke:49775 closing leases
  248. 2013-07-21 22:12:29,239 INFO [RS:2;lars-francke:49775] regionserver.Leases(131): RS:2;lars-francke:49775 closed leases
  249. 2013-07-21 22:12:29,634 INFO [RS:2;lars-francke:49775.periodicFlusher] hbase.Chore(93): RS:2;lars-francke:49775.periodicFlusher exiting
  250. 2013-07-21 22:12:29,634 INFO [RS:2;lars-francke:49775] regionserver.CompactSplitThread(357): Waiting for Split Thread to finish...
  251. 2013-07-21 22:12:29,634 INFO [RS:2;lars-francke:49775.leaseChecker] regionserver.Leases(124): RS:2;lars-francke:49775.leaseChecker closing leases
  252. 2013-07-21 22:12:29,634 INFO [RS:2;lars-francke:49775.leaseChecker] regionserver.Leases(131): RS:2;lars-francke:49775.leaseChecker closed leases
  253. 2013-07-21 22:12:29,634 INFO [RS:2;lars-francke:49775] regionserver.CompactSplitThread(357): Waiting for Merge Thread to finish...
  254. 2013-07-21 22:12:29,634 INFO [RS:2;lars-francke:49775] regionserver.CompactSplitThread(357): Waiting for Large Compaction Thread to finish...
  255. 2013-07-21 22:12:29,634 INFO [RS:2;lars-francke:49775] regionserver.CompactSplitThread(357): Waiting for Small Compaction Thread to finish...
  256. 2013-07-21 22:12:29,664 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:59060-0x14002dc71190000 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/lars-francke.com,49775,1374437547564
  257. 2013-07-21 22:12:29,664 INFO [pool-1-thread-1-EventThread] zookeeper.RegionServerTracker(94): RegionServer ephemeral node deleted, processing expiration [lars-francke.com,49775,1374437547564]
  258. 2013-07-21 22:12:29,672 DEBUG [pool-1-thread-1-EventThread] master.AssignmentManager(3032): based on AM, current region=.META.,,1.1028785192 is on server=lars-francke.com,56454,1374437536906 server being checked: lars-francke.com,49775,1374437547564
  259. 2013-07-21 22:12:29,672 INFO [RS:2;lars-francke:49775] regionserver.HRegionServer(968): stopping server lars-francke.com,49775,1374437547564; zookeeper connection closed.
  260. 2013-07-21 22:12:29,673 INFO [RS:2;lars-francke:49775] regionserver.HRegionServer(971): RS:2;lars-francke:49775 exiting
  261. 2013-07-21 22:12:29,673 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@1d7b51e8] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(193): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@1d7b51e8
  262. 2013-07-21 22:12:29,675 DEBUG [pool-1-thread-1-EventThread] master.ServerManager(510): Added=lars-francke.com,49775,1374437547564 to dead servers, submitted shutdown handler to be executed meta=false
  263. 2013-07-21 22:12:29,675 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:59060-0x14002dc71190000 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs
  264. 2013-07-21 22:12:29,676 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(431): master:59060-0x14002dc71190000 Set watcher on existing znode=/hbase/rs/lars-francke.com,56454,1374437536906
  265. 2013-07-21 22:12:29,677 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(431): master:59060-0x14002dc71190000 Set watcher on existing znode=/hbase/rs/lars-francke.com,38941,1374437536925
  266. 2013-07-21 22:12:29,682 DEBUG [MASTER_SERVER_OPERATIONS-lars-francke:59060-0] client.ClientScanner(198): Finished region={ENCODED => 1028785192, NAME => '.META.,,1', STARTKEY => '', ENDKEY => ''}
  267. 2013-07-21 22:12:29,683 INFO [MASTER_SERVER_OPERATIONS-lars-francke:59060-0] handler.ServerShutdownHandler(185): Splitting logs for lars-francke.com,49775,1374437547564 before assignment.
  268. 2013-07-21 22:12:29,696 DEBUG [MASTER_SERVER_OPERATIONS-lars-francke:59060-0] master.MasterFileSystem(322): Renamed region directory: hdfs://localhost:42066/user/jenkins/hbase/.logs/lars-francke.com,49775,1374437547564-splitting
  269. 2013-07-21 22:12:29,697 INFO [MASTER_SERVER_OPERATIONS-lars-francke:59060-0] master.SplitLogManager(1324): dead splitlog workers [lars-francke.com,49775,1374437547564]
  270. 2013-07-21 22:12:29,698 DEBUG [MASTER_SERVER_OPERATIONS-lars-francke:59060-0] master.SplitLogManager(326): Scheduling batch of logs to split
  271. 2013-07-21 22:12:29,698 INFO [MASTER_SERVER_OPERATIONS-lars-francke:59060-0] master.SplitLogManager(328): started splitting 1 logs in [hdfs://localhost:42066/user/jenkins/hbase/.logs/lars-francke.com,49775,1374437547564-splitting]
  272. 2013-07-21 22:12:29,705 DEBUG [RS:1;lars-francke:38941-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:38941-0x14002dc71190002 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitlog
  273. 2013-07-21 22:12:29,706 DEBUG [RS:0;lars-francke:56454-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:56454-0x14002dc71190001 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitlog
  274. 2013-07-21 22:12:29,706 DEBUG [RS:1;lars-francke:38941-EventThread] regionserver.SplitLogWorker(583): tasks arrived or departed
  275. 2013-07-21 22:12:29,706 DEBUG [RS:0;lars-francke:56454-EventThread] regionserver.SplitLogWorker(583): tasks arrived or departed
  276. 2013-07-21 22:12:29,706 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(707): put up splitlog task at znode /hbase/splitlog/.logs%2Flars-francke.com%2C49775%2C1374437547564-splitting%2Flars-francke.com%252C49775%252C1374437547564.1374437547615
  277. 2013-07-21 22:12:29,709 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(749): task not yet acquired /hbase/splitlog/.logs%2Flars-francke.com%2C49775%2C1374437547564-splitting%2Flars-francke.com%252C49775%252C1374437547564.1374437547615 ver = 0
  278. 2013-07-21 22:12:29,714 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:59060-0x14002dc71190000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitlog/.logs%2Flars-francke.com%2C49775%2C1374437547564-splitting%2Flars-francke.com%252C49775%252C1374437547564.1374437547615
  279. 2013-07-21 22:12:29,714 INFO [SplitLogWorker-lars-francke.com,56454,1374437536906] regionserver.SplitLogWorker(344): worker lars-francke.com,56454,1374437536906 acquired task /hbase/splitlog/.logs%2Flars-francke.com%2C49775%2C1374437547564-splitting%2Flars-francke.com%252C49775%252C1374437547564.1374437547615
  280. 2013-07-21 22:12:29,719 INFO [SplitLogWorker-lars-francke.com,56454,1374437536906] wal.HLogSplitter(247): Splitting hlog: hdfs://localhost:42066/user/jenkins/hbase/.logs/lars-francke.com,49775,1374437547564-splitting/lars-francke.com%2C49775%2C1374437547564.1374437547615, length=116
  281. 2013-07-21 22:12:29,719 INFO [SplitLogWorker-lars-francke.com,56454,1374437536906] wal.HLogSplitter(248): DistributedLogReplay = false
  282. 2013-07-21 22:12:29,723 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(822): task /hbase/splitlog/.logs%2Flars-francke.com%2C49775%2C1374437547564-splitting%2Flars-francke.com%252C49775%252C1374437547564.1374437547615 acquired by lars-francke.com,56454,1374437536906
  283. 2013-07-21 22:12:29,731 DEBUG [RS:0;lars-francke:56454-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:56454-0x14002dc71190001 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitlog/.logs%2Flars-francke.com%2C49775%2C1374437547564-splitting%2Flars-francke.com%252C49775%252C1374437547564.1374437547615
  284. 2013-07-21 22:12:29,731 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:59060-0x14002dc71190000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitlog/.logs%2Flars-francke.com%2C49775%2C1374437547564-splitting%2Flars-francke.com%252C49775%252C1374437547564.1374437547615
  285. 2013-07-21 22:12:29,732 INFO [SplitLogWorker-lars-francke.com,56454,1374437536906] util.FSHDFSUtils(86): Recovering lease on dfs file hdfs://localhost:42066/user/jenkins/hbase/.logs/lars-francke.com,49775,1374437547564-splitting/lars-francke.com%2C49775%2C1374437547564.1374437547615
  286. 2013-07-21 22:12:29,732 INFO [SplitLogWorker-lars-francke.com,56454,1374437536906] util.FSHDFSUtils(173): recoverLease=true, attempt=0 on file=hdfs://localhost:42066/user/jenkins/hbase/.logs/lars-francke.com,49775,1374437547564-splitting/lars-francke.com%2C49775%2C1374437547564.1374437547615 after 0ms
  287. 2013-07-21 22:12:29,733 DEBUG [SplitLogWorker-lars-francke.com,56454,1374437536906] fs.HFileSystem$ReorderWALBlocks(327): /user/jenkins/hbase/.logs/lars-francke.com,49775,1374437547564-splitting/lars-francke.com%2C49775%2C1374437547564.1374437547615 is an HLog file, so reordering blocks, last hostname will be:lars-francke.com
  288. 2013-07-21 22:12:29,735 DEBUG [SplitLogWorker-lars-francke.com,56454,1374437536906] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 108, fileLength: 116, trailerPresent: true
  289. 2013-07-21 22:12:29,738 DEBUG [WriterThread-0] wal.HLogSplitter$WriterThread(799): Writer thread Thread[WriterThread-0,5,main]: starting
  290. 2013-07-21 22:12:29,738 DEBUG [WriterThread-1] wal.HLogSplitter$WriterThread(799): Writer thread Thread[WriterThread-1,5,main]: starting
  291. 2013-07-21 22:12:29,738 DEBUG [WriterThread-2] wal.HLogSplitter$WriterThread(799): Writer thread Thread[WriterThread-2,5,main]: starting
  292. 2013-07-21 22:12:29,741 INFO [SplitLogWorker-lars-francke.com,56454,1374437536906] wal.HLogSplitter(334): Finishing writing output logs and closing down.
  293. 2013-07-21 22:12:29,741 INFO [SplitLogWorker-lars-francke.com,56454,1374437536906] wal.HLogSplitter$OutputSink(925): Waiting for split writer threads to finish
  294. 2013-07-21 22:12:29,772 DEBUG [WriterThread-2] wal.HLogSplitter$LogRecoveredEditsOutputSink(1200): Creating writer path=hdfs://localhost:42066/user/jenkins/hbase/testClusterStatus/f02a31af489a311d2a63454a13db9b09/recovered.edits/0000000000000000002.temp region=f02a31af489a311d2a63454a13db9b09
  295. 2013-07-21 22:12:29,772 INFO [SplitLogWorker-lars-francke.com,56454,1374437536906] wal.HLogSplitter$OutputSink(943): Split writers finished
  296. 2013-07-21 22:12:29,986 INFO [lars-francke.com,59060,1374437536563.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1383): total tasks = 1 unassigned = 0 tasks={/hbase/splitlog/.logs%2Flars-francke.com%2C49775%2C1374437547564-splitting%2Flars-francke.com%252C49775%252C1374437547564.1374437547615=last_update = 1374437549731 last_version = 2 cur_worker_name = lars-francke.com,56454,1374437536906 status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0}
  297. 2013-07-21 22:12:30,119 INFO [pool-1-thread-1] hbase.Waiter(174): Waiting up to [40,000] milli-secs(wait.for.ratio=[1])
  298. 2013-07-21 22:12:30,188 INFO [split-log-closeStream-1] wal.HLogSplitter$LogRecoveredEditsOutputSink$2(1042): Closed path hdfs://localhost:42066/user/jenkins/hbase/testClusterStatus/f02a31af489a311d2a63454a13db9b09/recovered.edits/0000000000000000002.temp (wrote 1 edits in 31ms)
  299. 2013-07-21 22:12:30,204 DEBUG [split-log-closeStream-1] wal.HLogSplitter$LogRecoveredEditsOutputSink$2(1073): Rename hdfs://localhost:42066/user/jenkins/hbase/testClusterStatus/f02a31af489a311d2a63454a13db9b09/recovered.edits/0000000000000000002.temp to hdfs://localhost:42066/user/jenkins/hbase/testClusterStatus/f02a31af489a311d2a63454a13db9b09/recovered.edits/0000000000000000002
  300. 2013-07-21 22:12:30,205 INFO [SplitLogWorker-lars-francke.com,56454,1374437536906] wal.HLogSplitter(341): Processed 1 edits across 1 regions; log file=hdfs://localhost:42066/user/jenkins/hbase/.logs/lars-francke.com,49775,1374437547564-splitting/lars-francke.com%2C49775%2C1374437547564.1374437547615 is corrupted = false progress failed = false
  301. 2013-07-21 22:12:30,213 DEBUG [RS:0;lars-francke:56454-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:56454-0x14002dc71190001 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitlog/.logs%2Flars-francke.com%2C49775%2C1374437547564-splitting%2Flars-francke.com%252C49775%252C1374437547564.1374437547615
  302. 2013-07-21 22:12:30,214 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:59060-0x14002dc71190000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitlog/.logs%2Flars-francke.com%2C49775%2C1374437547564-splitting%2Flars-francke.com%252C49775%252C1374437547564.1374437547615
  303. 2013-07-21 22:12:30,214 INFO [SplitLogWorker-lars-francke.com,56454,1374437536906] regionserver.SplitLogWorker(462): successfully transitioned task /hbase/splitlog/.logs%2Flars-francke.com%2C49775%2C1374437547564-splitting%2Flars-francke.com%252C49775%252C1374437547564.1374437547615 to final state DONE lars-francke.com,56454,1374437536906
  304. 2013-07-21 22:12:30,214 INFO [SplitLogWorker-lars-francke.com,56454,1374437536906] regionserver.SplitLogWorker(396): worker lars-francke.com,56454,1374437536906 done with task /hbase/splitlog/.logs%2Flars-francke.com%2C49775%2C1374437547564-splitting%2Flars-francke.com%252C49775%252C1374437547564.1374437547615 in 500ms
  305. 2013-07-21 22:12:30,214 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(757): task /hbase/splitlog/.logs%2Flars-francke.com%2C49775%2C1374437547564-splitting%2Flars-francke.com%252C49775%252C1374437547564.1374437547615 entered state: DONE lars-francke.com,56454,1374437536906
  306. 2013-07-21 22:12:30,246 DEBUG [pool-1-thread-1-EventThread] wal.HLogSplitter(430): Archived processed log hdfs://localhost:42066/user/jenkins/hbase/.logs/lars-francke.com,49775,1374437547564-splitting/lars-francke.com%2C49775%2C1374437547564.1374437547615 to hdfs://localhost:42066/user/jenkins/hbase/.oldlogs/lars-francke.com%2C49775%2C1374437547564.1374437547615
  307. 2013-07-21 22:12:30,247 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(670): Done splitting /hbase/splitlog/.logs%2Flars-francke.com%2C49775%2C1374437547564-splitting%2Flars-francke.com%252C49775%252C1374437547564.1374437547615
  308. 2013-07-21 22:12:30,266 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:59060-0x14002dc71190000 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitlog/.logs%2Flars-francke.com%2C49775%2C1374437547564-splitting%2Flars-francke.com%252C49775%252C1374437547564.1374437547615
  309. 2013-07-21 22:12:30,266 DEBUG [RS:1;lars-francke:38941-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:38941-0x14002dc71190002 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitlog
  310. 2013-07-21 22:12:30,266 DEBUG [RS:1;lars-francke:38941-EventThread] regionserver.SplitLogWorker(583): tasks arrived or departed
  311. 2013-07-21 22:12:30,266 DEBUG [RS:0;lars-francke:56454-EventThread] zookeeper.ZooKeeperWatcher(307): regionserver:56454-0x14002dc71190001 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitlog
  312. 2013-07-21 22:12:30,266 DEBUG [RS:0;lars-francke:56454-EventThread] regionserver.SplitLogWorker(583): tasks arrived or departed
  313. 2013-07-21 22:12:30,266 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager$DeleteAsyncCallback(1580): deleted /hbase/splitlog/.logs%2Flars-francke.com%2C49775%2C1374437547564-splitting%2Flars-francke.com%252C49775%252C1374437547564.1374437547615
  314. 2013-07-21 22:12:30,271 INFO [MASTER_SERVER_OPERATIONS-lars-francke:59060-0] master.SplitLogManager(385): finished splitting (more than or equal to) 116 bytes in 1 log files in [hdfs://localhost:42066/user/jenkins/hbase/.logs/lars-francke.com,49775,1374437547564-splitting] in 573ms
  315. 2013-07-21 22:12:30,271 INFO [MASTER_SERVER_OPERATIONS-lars-francke:59060-0] handler.ServerShutdownHandler(206): Reassigning 1 region(s) that lars-francke.com,49775,1374437547564 was carrying (and 0 regions(s) that were opening on this server)
  316. 2013-07-21 22:12:30,272 DEBUG [MASTER_SERVER_OPERATIONS-lars-francke:59060-0] master.AssignmentManager(1503): Assigning 1 region(s) to lars-francke.com,38941,1374437536925
  317. 2013-07-21 22:12:30,272 DEBUG [MASTER_SERVER_OPERATIONS-lars-francke:59060-0] master.RegionStates(327): Offline a region f02a31af489a311d2a63454a13db9b09 with current state=OPEN, expected state=OFFLINE, assigned to server: lars-francke.com,49775,1374437547564, expected null
  318. 2013-07-21 22:12:30,272 INFO [MASTER_SERVER_OPERATIONS-lars-francke:59060-0] master.RegionStates(265): Transitioned from {f02a31af489a311d2a63454a13db9b09 state=OPEN, ts=1374437549081, server=lars-francke.com,49775,1374437547564} to {f02a31af489a311d2a63454a13db9b09 state=OFFLINE, ts=1374437550272, server=null}
  319. 2013-07-21 22:12:30,272 DEBUG [MASTER_SERVER_OPERATIONS-lars-francke:59060-0] zookeeper.ZKAssign(177): master:59060-0x14002dc71190000 Async create of unassigned node for f02a31af489a311d2a63454a13db9b09 with OFFLINE state
  320. 2013-07-21 22:12:30,280 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:59060-0x14002dc71190000 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition
  321. 2013-07-21 22:12:30,280 DEBUG [pool-1-thread-1-EventThread] master.OfflineCallback(69): rs={f02a31af489a311d2a63454a13db9b09 state=OFFLINE, ts=1374437550272, server=null}, server=lars-francke.com,38941,1374437536925
  322. 2013-07-21 22:12:30,281 DEBUG [pool-1-thread-1-EventThread] master.OfflineCallback$ExistCallback(106): rs={f02a31af489a311d2a63454a13db9b09 state=OFFLINE, ts=1374437550272, server=null}, server=lars-francke.com,38941,1374437536925
  323. 2013-07-21 22:12:30,282 INFO [MASTER_SERVER_OPERATIONS-lars-francke:59060-0] master.AssignmentManager(1539): lars-francke.com,38941,1374437536925 unassigned znodes=1 of total=1
  324. 2013-07-21 22:12:30,283 INFO [MASTER_SERVER_OPERATIONS-lars-francke:59060-0] master.RegionStates(265): Transitioned from {f02a31af489a311d2a63454a13db9b09 state=OFFLINE, ts=1374437550272, server=null} to {f02a31af489a311d2a63454a13db9b09 state=PENDING_OPEN, ts=1374437550283, server=lars-francke.com,38941,1374437536925}
  325. 2013-07-21 22:12:30,283 INFO [RpcServer.handler=4,port=38941] regionserver.HRegionServer(3452): Open testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09.
  326. 2013-07-21 22:12:30,286 DEBUG [RS_OPEN_REGION-lars-francke:38941-0] zookeeper.ZKAssign(786): regionserver:38941-0x14002dc71190002 Transitioning f02a31af489a311d2a63454a13db9b09 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
  327. 2013-07-21 22:12:30,286 DEBUG [MASTER_SERVER_OPERATIONS-lars-francke:59060-0] master.AssignmentManager(1661): Bulk assigning done for lars-francke.com,38941,1374437536925
  328. 2013-07-21 22:12:30,286 INFO [MASTER_SERVER_OPERATIONS-lars-francke:59060-0] handler.ServerShutdownHandler(301): Finished processing of shutdown of lars-francke.com,49775,1374437547564
  329. 2013-07-21 22:12:30,313 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:59060-0x14002dc71190000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/f02a31af489a311d2a63454a13db9b09
  330. 2013-07-21 22:12:30,314 DEBUG [RS_OPEN_REGION-lars-francke:38941-0] zookeeper.ZKAssign(861): regionserver:38941-0x14002dc71190002 Successfully transitioned node f02a31af489a311d2a63454a13db9b09 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
  331. 2013-07-21 22:12:30,314 INFO [RS_OPEN_REGION-lars-francke:38941-0] regionserver.HRegion(4082): Open {ENCODED => f02a31af489a311d2a63454a13db9b09, NAME => 'testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09.', STARTKEY => '', ENDKEY => ''}
  332. 2013-07-21 22:12:30,314 DEBUG [RS_OPEN_REGION-lars-francke:38941-0] regionserver.MetricsRegionSourceImpl(58): Creating new MetricsRegionSourceImpl for table testClusterStatus f02a31af489a311d2a63454a13db9b09
  333. 2013-07-21 22:12:30,314 DEBUG [AM.ZK.Worker-pool-4-thread-15] master.AssignmentManager(767): Handling transition=RS_ZK_REGION_OPENING, server=lars-francke.com,38941,1374437536925, region=f02a31af489a311d2a63454a13db9b09, current state from region state map ={f02a31af489a311d2a63454a13db9b09 state=PENDING_OPEN, ts=1374437550283, server=lars-francke.com,38941,1374437536925}
  334. 2013-07-21 22:12:30,314 DEBUG [RS_OPEN_REGION-lars-francke:38941-0] regionserver.HRegion(535): Instantiated testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09.
  335. 2013-07-21 22:12:30,314 INFO [AM.ZK.Worker-pool-4-thread-15] master.RegionStates(265): Transitioned from {f02a31af489a311d2a63454a13db9b09 state=PENDING_OPEN, ts=1374437550283, server=lars-francke.com,38941,1374437536925} to {f02a31af489a311d2a63454a13db9b09 state=OPENING, ts=1374437550314, server=lars-francke.com,38941,1374437536925}
  336. 2013-07-21 22:12:30,316 INFO [StoreOpener-f02a31af489a311d2a63454a13db9b09-1] compactions.CompactionConfiguration(85): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000
  337. 2013-07-21 22:12:30,320 INFO [RS_OPEN_REGION-lars-francke:38941-0] regionserver.HRegion(2893): Replaying edits from hdfs://localhost:42066/user/jenkins/hbase/testClusterStatus/f02a31af489a311d2a63454a13db9b09/recovered.edits/0000000000000000002
  338. 2013-07-21 22:12:30,322 DEBUG [RS_OPEN_REGION-lars-francke:38941-0] wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset: 108, fileLength: 116, trailerPresent: true
  339. 2013-07-21 22:12:30,323 DEBUG [RS_OPEN_REGION-lars-francke:38941-0] zookeeper.ZKAssign(598): regionserver:38941-0x14002dc71190002 Attempting to retransition the opening state of node f02a31af489a311d2a63454a13db9b09
  340. 2013-07-21 22:12:30,324 DEBUG [RS_OPEN_REGION-lars-francke:38941-0] regionserver.HRegion(3037): Applied 1, skipped 0, firstSequenceidInLog=2, maxSequenceidInLog=2, path=hdfs://localhost:42066/user/jenkins/hbase/testClusterStatus/f02a31af489a311d2a63454a13db9b09/recovered.edits/0000000000000000002
  341. 2013-07-21 22:12:30,324 DEBUG [RS_OPEN_REGION-lars-francke:38941-0] regionserver.HRegion(1477): Started memstore flush for testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09., current region memstore size 168.0; wal is null, using passed sequenceid=2
  342. 2013-07-21 22:12:30,825 DEBUG [RS_OPEN_REGION-lars-francke:38941-0] util.FSUtils(298): DFS Client does not support most favored nodes create; using default create
  343. 2013-07-21 22:12:30,905 INFO [RS_OPEN_REGION-lars-francke:38941-0] regionserver.DefaultStoreFlusher(88): Flushed, sequenceid=2, memsize=168.0, hasBloomFilter=false, into tmp file hdfs://localhost:42066/user/jenkins/hbase/testClusterStatus/f02a31af489a311d2a63454a13db9b09/.tmp/502fe66506fb4572b5acbcf2e2b07191
  344. 2013-07-21 22:12:30,911 DEBUG [RS_OPEN_REGION-lars-francke:38941-0] regionserver.HRegionFileSystem(338): Committing store file hdfs://localhost:42066/user/jenkins/hbase/testClusterStatus/f02a31af489a311d2a63454a13db9b09/.tmp/502fe66506fb4572b5acbcf2e2b07191 as hdfs://localhost:42066/user/jenkins/hbase/testClusterStatus/f02a31af489a311d2a63454a13db9b09/cf/502fe66506fb4572b5acbcf2e2b07191
  345. 2013-07-21 22:12:30,932 INFO [RS_OPEN_REGION-lars-francke:38941-0] regionserver.HStore(760): Added hdfs://localhost:42066/user/jenkins/hbase/testClusterStatus/f02a31af489a311d2a63454a13db9b09/cf/502fe66506fb4572b5acbcf2e2b07191, entries=1, sequenceid=2, filesize=782.0
  346. 2013-07-21 22:12:30,932 INFO [RS_OPEN_REGION-lars-francke:38941-0] regionserver.HRegion(1621): Finished memstore flush of ~168.0/168, currentsize=0.0/0 for region testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09. in 608ms, sequenceid=2, compaction requested=false; wal=null
  347. 2013-07-21 22:12:30,954 DEBUG [RS_OPEN_REGION-lars-francke:38941-0] regionserver.HRegion(2874): Deleted recovered.edits file=hdfs://localhost:42066/user/jenkins/hbase/testClusterStatus/f02a31af489a311d2a63454a13db9b09/recovered.edits/0000000000000000002
  348. 2013-07-21 22:12:30,955 INFO [RS_OPEN_REGION-lars-francke:38941-0] regionserver.HRegion(630): Onlined f02a31af489a311d2a63454a13db9b09; next sequenceid=3
  349. 2013-07-21 22:12:30,955 DEBUG [RS_OPEN_REGION-lars-francke:38941-0] zookeeper.ZKAssign(598): regionserver:38941-0x14002dc71190002 Attempting to retransition the opening state of node f02a31af489a311d2a63454a13db9b09
  350. 2013-07-21 22:12:30,956 INFO [PostOpenDeployTasks:f02a31af489a311d2a63454a13db9b09] regionserver.HRegionServer(1701): Post open deploy tasks for region=testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09.
  351. 2013-07-21 22:12:30,959 INFO [PostOpenDeployTasks:f02a31af489a311d2a63454a13db9b09] catalog.MetaEditor(432): Updated row testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09. with server=lars-francke.com,38941,1374437536925
  352. 2013-07-21 22:12:30,959 INFO [PostOpenDeployTasks:f02a31af489a311d2a63454a13db9b09] regionserver.HRegionServer(1726): Done with post open deploy task for region=testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09.
  353. 2013-07-21 22:12:30,959 DEBUG [RS_OPEN_REGION-lars-francke:38941-0] zookeeper.ZKAssign(786): regionserver:38941-0x14002dc71190002 Transitioning f02a31af489a311d2a63454a13db9b09 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED
  354. 2013-07-21 22:12:31,024 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:59060-0x14002dc71190000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/f02a31af489a311d2a63454a13db9b09
  355. 2013-07-21 22:12:31,024 DEBUG [RS_OPEN_REGION-lars-francke:38941-0] zookeeper.ZKAssign(861): regionserver:38941-0x14002dc71190002 Successfully transitioned node f02a31af489a311d2a63454a13db9b09 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED
  356. 2013-07-21 22:12:31,025 DEBUG [RS_OPEN_REGION-lars-francke:38941-0] handler.OpenRegionHandler(373): region transitioned to opened in zookeeper: {ENCODED => f02a31af489a311d2a63454a13db9b09, NAME => 'testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09.', STARTKEY => '', ENDKEY => ''}, server: lars-francke.com,38941,1374437536925
  357. 2013-07-21 22:12:31,025 DEBUG [RS_OPEN_REGION-lars-francke:38941-0] handler.OpenRegionHandler(186): Opened testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09. on server:lars-francke.com,38941,1374437536925
  358. 2013-07-21 22:12:31,025 DEBUG [AM.ZK.Worker-pool-4-thread-20] master.AssignmentManager(767): Handling transition=RS_ZK_REGION_OPENED, server=lars-francke.com,38941,1374437536925, region=f02a31af489a311d2a63454a13db9b09, current state from region state map ={f02a31af489a311d2a63454a13db9b09 state=OPENING, ts=1374437550314, server=lars-francke.com,38941,1374437536925}
  359. 2013-07-21 22:12:31,025 INFO [AM.ZK.Worker-pool-4-thread-20] master.RegionStates(265): Transitioned from {f02a31af489a311d2a63454a13db9b09 state=OPENING, ts=1374437550314, server=lars-francke.com,38941,1374437536925} to {f02a31af489a311d2a63454a13db9b09 state=OPEN, ts=1374437551025, server=lars-francke.com,38941,1374437536925}
  360. 2013-07-21 22:12:31,025 DEBUG [MASTER_OPEN_REGION-lars-francke:59060-1] handler.OpenedRegionHandler(145): Handling OPENED event for f02a31af489a311d2a63454a13db9b09 from lars-francke.com,38941,1374437536925; deleting unassigned node
  361. 2013-07-21 22:12:31,025 DEBUG [MASTER_OPEN_REGION-lars-francke:59060-1] zookeeper.ZKAssign(405): master:59060-0x14002dc71190000 Deleting existing unassigned node for f02a31af489a311d2a63454a13db9b09 that is in expected state RS_ZK_REGION_OPENED
  362. 2013-07-21 22:12:31,038 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:59060-0x14002dc71190000 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/f02a31af489a311d2a63454a13db9b09
  363. 2013-07-21 22:12:31,038 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(307): master:59060-0x14002dc71190000 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition
  364. 2013-07-21 22:12:31,038 DEBUG [AM.ZK.Worker-pool-4-thread-16] master.AssignmentManager$4(1218): The znode of testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09. has been deleted, region state: {f02a31af489a311d2a63454a13db9b09 state=OPEN, ts=1374437551025, server=lars-francke.com,38941,1374437536925}
  365. 2013-07-21 22:12:31,038 INFO [AM.ZK.Worker-pool-4-thread-16] master.RegionStates(301): Onlined f02a31af489a311d2a63454a13db9b09 on lars-francke.com,38941,1374437536925
  366. 2013-07-21 22:12:31,038 DEBUG [MASTER_OPEN_REGION-lars-francke:59060-1] zookeeper.ZKAssign(434): master:59060-0x14002dc71190000 Successfully deleted unassigned node for region f02a31af489a311d2a63454a13db9b09 in expected state RS_ZK_REGION_OPENED
  367. 2013-07-21 22:12:31,038 INFO [AM.ZK.Worker-pool-4-thread-16] master.AssignmentManager$4(1223): The master has opened testClusterStatus,,1374437547649.f02a31af489a311d2a63454a13db9b09. that was online on lars-francke.com,38941,1374437536925
  368. 2013-07-21 22:13:10,133 INFO [pool-1-thread-1] hbase.ResourceChecker(171): after: client.TestHCM#testClusterStatus Thread=288 (was 275)
  369. Potentially hanging thread: SplitLogWorker-lars-francke.com,49775,1374437547564-EventThread
  370. sun.misc.Unsafe.park(Native Method)
  371. java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
  372. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  373. java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  374. org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
  375.  
  376. Potentially hanging thread: Priority.RpcServer.handler=9,port=49775
  377. sun.misc.Unsafe.park(Native Method)
  378. java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
  379. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  380. java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  381. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
  382. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
  383. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
  384. java.lang.Thread.run(Thread.java:662)
  385.  
  386. Potentially hanging thread: Thread-410
  387. sun.misc.Unsafe.park(Native Method)
  388. java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
  389. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
  390. java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424)
  391. java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
  392. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  393. java.lang.Thread.run(Thread.java:662)
  394.  
  395. Potentially hanging thread: Priority.RpcServer.handler=3,port=49775
  396. sun.misc.Unsafe.park(Native Method)
  397. java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
  398. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  399. java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  400. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
  401. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
  402. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
  403. java.lang.Thread.run(Thread.java:662)
  404.  
  405. Potentially hanging thread: Priority.RpcServer.handler=7,port=49775
  406. sun.misc.Unsafe.park(Native Method)
  407. java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
  408. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  409. java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  410. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
  411. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
  412. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
  413. java.lang.Thread.run(Thread.java:662)
  414.  
  415. Potentially hanging thread: SplitLogWorker-lars-francke.com,49775,1374437547564-SendThread(localhost:59753)
  416. sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  417. sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  418. sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  419. sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  420. sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  421. org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
  422. org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
  423.  
  424. Potentially hanging thread: Priority.RpcServer.handler=0,port=49775
  425. sun.misc.Unsafe.park(Native Method)
  426. java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
  427. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  428. java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  429. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
  430. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
  431. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
  432. java.lang.Thread.run(Thread.java:662)
  433.  
  434. Potentially hanging thread: RpcServer.handler=3,port=49775
  435. sun.misc.Unsafe.park(Native Method)
  436. java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
  437. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  438. java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  439. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
  440. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
  441. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
  442. java.lang.Thread.run(Thread.java:662)
  443.  
  444. Potentially hanging thread: MASTER_CLOSE_REGION-lars-francke:59060-1
  445. sun.misc.Unsafe.park(Native Method)
  446. java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
  447. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  448. java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  449. java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
  450. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  451. java.lang.Thread.run(Thread.java:662)
  452.  
  453. Potentially hanging thread: Priority.RpcServer.handler=4,port=49775
  454. sun.misc.Unsafe.park(Native Method)
  455. java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
  456. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  457. java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  458. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
  459. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
  460. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
  461. java.lang.Thread.run(Thread.java:662)
  462.  
  463. Potentially hanging thread: Priority.RpcServer.handler=6,port=49775
  464. sun.misc.Unsafe.park(Native Method)
  465. java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
  466. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  467. java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  468. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
  469. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
  470. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
  471. java.lang.Thread.run(Thread.java:662)
  472.  
  473. Potentially hanging thread: Thread-449
  474. sun.misc.Unsafe.park(Native Method)
  475. java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
  476. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
  477. java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424)
  478. java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
  479. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  480. java.lang.Thread.run(Thread.java:662)
  481.  
  482. Potentially hanging thread: Replication.RpcServer.handler=1,port=49775
  483. sun.misc.Unsafe.park(Native Method)
  484. java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
  485. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  486. java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  487. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
  488. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
  489. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
  490. java.lang.Thread.run(Thread.java:662)
  491.  
  492. Potentially hanging thread: Priority.RpcServer.handler=8,port=49775
  493. sun.misc.Unsafe.park(Native Method)
  494. java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
  495. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  496. java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  497. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
  498. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
  499. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
  500. java.lang.Thread.run(Thread.java:662)
  501.  
  502. Potentially hanging thread: RpcServer.handler=1,port=49775
  503. sun.misc.Unsafe.park(Native Method)
  504. java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
  505. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  506. java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  507. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
  508. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
  509. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
  510. java.lang.Thread.run(Thread.java:662)
  511.  
  512. Potentially hanging thread: RpcServer.handler=0,port=49775
  513. sun.misc.Unsafe.park(Native Method)
  514. java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
  515. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  516. java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  517. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
  518. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
  519. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
  520. java.lang.Thread.run(Thread.java:662)
  521.  
  522. Potentially hanging thread: MASTER_SERVER_OPERATIONS-lars-francke:59060-0
  523. sun.misc.Unsafe.park(Native Method)
  524. java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
  525. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  526. java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  527. java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
  528. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  529. java.lang.Thread.run(Thread.java:662)
  530.  
  531. Potentially hanging thread: Priority.RpcServer.handler=2,port=49775
  532. sun.misc.Unsafe.park(Native Method)
  533. java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
  534. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  535. java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  536. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
  537. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
  538. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
  539. java.lang.Thread.run(Thread.java:662)
  540.  
  541. Potentially hanging thread: Replication.RpcServer.handler=2,port=49775
  542. sun.misc.Unsafe.park(Native Method)
  543. java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
  544. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  545. java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  546. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
  547. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
  548. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
  549. java.lang.Thread.run(Thread.java:662)
  550.  
  551. Potentially hanging thread: RpcServer.handler=2,port=49775
  552. sun.misc.Unsafe.park(Native Method)
  553. java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
  554. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  555. java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  556. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
  557. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
  558. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
  559. java.lang.Thread.run(Thread.java:662)
  560.  
  561. Potentially hanging thread: Thread-450
  562. sun.misc.Unsafe.park(Native Method)
  563. java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
  564. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
  565. java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424)
  566. java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
  567. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  568. java.lang.Thread.run(Thread.java:662)
  569.  
  570. Potentially hanging thread: Old I/O datagram worker ([id: 0xf7f8bd64, /226.1.1.3:60100])
  571. java.net.PlainDatagramSocketImpl.receive0(Native Method)
  572. java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:145)
  573. java.net.DatagramSocket.receive(DatagramSocket.java:725)
  574. org.jboss.netty.channel.socket.oio.OioDatagramWorker.process(OioDatagramWorker.java:47)
  575. org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:75)
  576. org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
  577. org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
  578. java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  579. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  580. java.lang.Thread.run(Thread.java:662)
  581.  
  582. Potentially hanging thread: RpcServer.handler=4,port=49775
  583. sun.misc.Unsafe.park(Native Method)
  584. java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
  585. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  586. java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  587. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
  588. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
  589. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
  590. java.lang.Thread.run(Thread.java:662)
  591.  
  592. Potentially hanging thread: Priority.RpcServer.handler=1,port=49775
  593. sun.misc.Unsafe.park(Native Method)
  594. java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
  595. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  596. java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  597. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
  598. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
  599. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
  600. java.lang.Thread.run(Thread.java:662)
  601.  
  602. Potentially hanging thread: hbase-table-pool-108-thread-1
  603. sun.misc.Unsafe.park(Native Method)
  604. java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
  605. java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
  606. java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
  607. java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
  608. java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
  609. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  610. java.lang.Thread.run(Thread.java:662)
  611.  
  612. Potentially hanging thread: RS_CLOSE_REGION-lars-francke:56454-1
  613. sun.misc.Unsafe.park(Native Method)
  614. java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
  615. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  616. java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  617. java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
  618. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  619. java.lang.Thread.run(Thread.java:662)
  620.  
  621. Potentially hanging thread: Replication.RpcServer.handler=0,port=49775
  622. sun.misc.Unsafe.park(Native Method)
  623. java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
  624. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  625. java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  626. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
  627. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
  628. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
  629. java.lang.Thread.run(Thread.java:662)
  630.  
  631. Potentially hanging thread: org.apache.hadoop.hdfs.server.datanode.DataXceiver@353c017
  632. sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  633. sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  634. sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  635. sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  636. sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  637. org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:332)
  638. org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
  639. org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
  640. org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
  641. java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
  642. java.io.BufferedInputStream.read(BufferedInputStream.java:237)
  643. java.io.DataInputStream.readShort(DataInputStream.java:295)
  644. org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:208)
  645. org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:104)
  646. java.lang.Thread.run(Thread.java:662)
  647.  
  648. Potentially hanging thread: org.apache.hadoop.hdfs.LeaseRenewer$1@23bffd0d
  649. java.lang.Thread.sleep(Native Method)
  650. org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:397)
  651. org.apache.hadoop.hdfs.LeaseRenewer.access$600(LeaseRenewer.java:69)
  652. org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:273)
  653. java.lang.Thread.run(Thread.java:662)
  654.  
  655. Potentially hanging thread: Priority.RpcServer.handler=5,port=49775
  656. sun.misc.Unsafe.park(Native Method)
  657. java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
  658. java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  659. java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  660. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:164)
  661. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
  662. org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
  663. java.lang.Thread.run(Thread.java:662)
  664. - Thread LEAK? -, OpenFileDescriptor=417 (was 440), MaxFileDescriptor=4096 (was 4096), SystemLoadAverage=227 (was 265), ProcessCount=243 (was 240) - ProcessCount LEAK? -, AvailableMemoryMB=2196 (was 1991) - AvailableMemoryMB LEAK? -, ConnectionCount=7 (was 6) - ConnectionCount LEAK? -
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!