Want more features on Pastebin? Sign Up, it's FREE!
Guest

HBase Region Server Stack Trace

By: ronbuckley on Aug 6th, 2012  |  syntax: None  |  size: 110.42 KB  |  views: 65  |  expires: Never
download  |  raw  |  embed  |  report abuse  |  print
Text below is selected. Please press Ctrl+C to copy to your clipboard. (⌘+C on Mac)
  1.  
  2. -PROD- prodcon@wchddb37pxdu:/prod/hadoop/prod/hbase-0.92.1/logs$cat  hbase-prodcon-regionserver-wchddb37pxdu.prod.oclc.org.out
  3. SLF4J: Class path contains multiple SLF4J bindings.
  4. SLF4J: Found binding in [jar:file:/drive1/hadoop/20120522/hbase-0.92.1/lib/slf4j-log4j12-1.5.8.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  5. SLF4J: Found binding in [jar:file:/drive1/hadoop/20120522/hadoop-0.20.2-cdh3u3/lib/slf4j-log4j12-1.4.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  6. SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
  7. 2012-08-06 11:46:46
  8. Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.0-b11 mixed mode):
  9.  
  10. "IPC Client (47) connection to /10.192.98.52:37073 from blk_81630121065287353_2238161" daemon prio=10 tid=0x00002aaacc7b7000 nid=0x1c8f in Object.wait() [0x000000004e0ef000]
  11.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  12.         at java.lang.Object.wait(Native Method)
  13.         at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:680)
  14.         - locked <0x00000005840a63a0> (a org.apache.hadoop.ipc.Client$Connection)
  15.         at org.apache.hadoop.ipc.Client$Connection.run(Client.java:723)
  16.  
  17. "IPC Client (47) connection to wchddb01pxdu.prod.oclc.org/10.192.98.16:9008 from prodcon" daemon prio=10 tid=0x00002aaacc3a5000 nid=0x1c8e in Object.wait() [0x000000004e3f2000]
  18.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  19.         at java.lang.Object.wait(Native Method)
  20.         at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:680)
  21.         - locked <0x0000000583b67d70> (a org.apache.hadoop.ipc.Client$Connection)
  22.         at org.apache.hadoop.ipc.Client$Connection.run(Client.java:723)
  23.  
  24. "1772284907@qtp-1068922666-14" prio=10 tid=0x00002aaad8488000 nid=0x1b27 in Object.wait() [0x000000004e5f4000]
  25.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  26.         at java.lang.Object.wait(Native Method)
  27.         at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
  28.         - locked <0x00000005835d94a0> (a org.mortbay.thread.QueuedThreadPool$PoolThread)
  29.  
  30. "RMI TCP Connection(idle)" daemon prio=10 tid=0x00002aaad8490000 nid=0x1985 waiting on condition [0x000000004f503000]
  31.    java.lang.Thread.State: TIMED_WAITING (parking)
  32.         at sun.misc.Unsafe.park(Native Method)
  33.         - parking to wait for  <0x000000058b4d4548> (a java.util.concurrent.SynchronousQueue$TransferStack)
  34.         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
  35.         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
  36.         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
  37.         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
  38.         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
  39.         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  40.         at java.lang.Thread.run(Thread.java:662)
  41.  
  42. "915001464@qtp-1068922666-13" prio=10 tid=0x00002aaad8287800 nid=0x1949 in Object.wait() [0x000000004e2f1000]
  43.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  44.         at java.lang.Object.wait(Native Method)
  45.         at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
  46.         - locked <0x000000058057a0f0> (a org.mortbay.thread.QueuedThreadPool$PoolThread)
  47.  
  48. "ResponseProcessor for block blk_81630121065287353_2238161" daemon prio=10 tid=0x00002aaabfe1c800 nid=0x13bb runnable [0x000000004e1f0000]
  49.    java.lang.Thread.State: RUNNABLE
  50.         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  51.         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  52.         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  53.         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  54.         - locked <0x0000000587f20758> (a sun.nio.ch.Util$2)
  55.         - locked <0x0000000587f20748> (a java.util.Collections$UnmodifiableSet)
  56.         - locked <0x0000000587f206b0> (a sun.nio.ch.EPollSelectorImpl)
  57.         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  58.         at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:332)
  59.         at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
  60.         at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
  61.         at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
  62.         at java.io.DataInputStream.readFully(DataInputStream.java:178)
  63.         at java.io.DataInputStream.readLong(DataInputStream.java:399)
  64.         at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:120)
  65.         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2956)
  66.  
  67. "DataStreamer for file /hbase/.logs/wchddb37pxdu.prod.oclc.org,9009,1337721421002/wchddb37pxdu.prod.oclc.org%2C9009%2C1337721421002.1344267769705 block blk_81630121065287353_2238161" daemon prio=10 tid=0x00002aaad81ab000 nid=0x13b4 in Object.wait() [0x000000004f200000]
  68.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  69.         at java.lang.Object.wait(Native Method)
  70.         at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2805)
  71.         - locked <0x00000006e8c6b168> (a java.util.LinkedList)
  72.  
  73. "sendParams-166" daemon prio=10 tid=0x00002aaacca57000 nid=0x2b0d waiting on condition [0x0000000066358000]
  74.    java.lang.Thread.State: TIMED_WAITING (parking)
  75.         at sun.misc.Unsafe.park(Native Method)
  76.         - parking to wait for  <0x000000058b300eb0> (a java.util.concurrent.SynchronousQueue$TransferStack)
  77.         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
  78.         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
  79.         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
  80.         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
  81.         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
  82.         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  83.         at java.lang.Thread.run(Thread.java:662)
  84.  
  85. "ReplicationExecutor-0" prio=10 tid=0x00002aaaccbd6000 nid=0x5e94 waiting on condition [0x0000000058a80000]
  86.    java.lang.Thread.State: WAITING (parking)
  87.         at sun.misc.Unsafe.park(Native Method)
  88.         - parking to wait for  <0x000000058b89f790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  89.         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  90.         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  91.         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  92.         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
  93.         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  94.         at java.lang.Thread.run(Thread.java:662)
  95.  
  96. "IPC Client (47) connection to wchddb02pxdu.prod.oclc.org/10.192.98.17:9010 from prodcon" daemon prio=10 tid=0x00002aaaccc76800 nid=0x59df in Object.wait() [0x000000007ff3f000]
  97.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  98.         at java.lang.Object.wait(Native Method)
  99.         at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.waitForWork(HBaseClient.java:459)
  100.         - locked <0x000000058fbd02e0> (a org.apache.hadoop.hbase.ipc.HBaseClient$Connection)
  101.         at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:504)
  102.  
  103. "RS_CLOSE_REGION-wchddb37pxdu.prod.oclc.org,9009,1337721421002-2" prio=10 tid=0x00002aaabf26d800 nid=0x3dd2 waiting on condition [0x00000000783c4000]
  104.    java.lang.Thread.State: WAITING (parking)
  105.         at sun.misc.Unsafe.park(Native Method)
  106.         - parking to wait for  <0x000000058b393590> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  107.         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  108.         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  109.         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  110.         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
  111.         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  112.         at java.lang.Thread.run(Thread.java:662)
  113.  
  114. "RS_CLOSE_REGION-wchddb37pxdu.prod.oclc.org,9009,1337721421002-1" prio=10 tid=0x0000000052de3800 nid=0x3dd1 waiting on condition [0x00000000782c3000]
  115.    java.lang.Thread.State: WAITING (parking)
  116.         at sun.misc.Unsafe.park(Native Method)
  117.         - parking to wait for  <0x000000058b393590> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  118.         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  119.         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  120.         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  121.         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
  122.         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  123.         at java.lang.Thread.run(Thread.java:662)
  124.  
  125. "RS_CLOSE_REGION-wchddb37pxdu.prod.oclc.org,9009,1337721421002-0" prio=10 tid=0x0000000052e1e000 nid=0x3dcc waiting on condition [0x0000000077abb000]
  126.    java.lang.Thread.State: WAITING (parking)
  127.         at sun.misc.Unsafe.park(Native Method)
  128.         - parking to wait for  <0x000000058b393590> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  129.         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  130.         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  131.         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  132.         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
  133.         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  134.         at java.lang.Thread.run(Thread.java:662)
  135.  
  136. "regionserver9009-splits-1337730044584" daemon prio=10 tid=0x0000000052868000 nid=0x1e64 waiting on condition [0x0000000056f65000]
  137.    java.lang.Thread.State: WAITING (parking)
  138.         at sun.misc.Unsafe.park(Native Method)
  139.         - parking to wait for  <0x000000058b8a3088> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  140.         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  141.         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  142.         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  143.         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
  144.         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  145.         at java.lang.Thread.run(Thread.java:662)
  146.  
  147. "regionserver9009-largeCompactions-1337726358700" daemon prio=10 tid=0x00002aaac0b55800 nid=0x973 waiting on condition [0x00000000777b8000]
  148.    java.lang.Thread.State: WAITING (parking)
  149.         at sun.misc.Unsafe.park(Native Method)
  150.         - parking to wait for  <0x000000058b8a16d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  151.         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  152.         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  153.         at java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:220)
  154.         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
  155.         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  156.         at java.lang.Thread.run(Thread.java:662)
  157.  
  158. "regionserver9009-EventThread.replicationSource,1" daemon prio=10 tid=0x00002aaacc980000 nid=0xcca waiting on condition [0x0000000043544000]
  159.    java.lang.Thread.State: TIMED_WAITING (sleeping)
  160.         at java.lang.Thread.sleep(Native Method)
  161.         at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.sleepForRetries(ReplicationSource.java:550)
  162.         at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:362)
  163.  
  164. "regionserver9009-EventThread" daemon prio=10 tid=0x0000000052a50800 nid=0xcc9 waiting on condition [0x000000004909f000]
  165.    java.lang.Thread.State: WAITING (parking)
  166.         at sun.misc.Unsafe.park(Native Method)
  167.         - parking to wait for  <0x000000058caaa060> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  168.         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  169.         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  170.         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  171.         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:493)
  172.  
  173. "regionserver9009-SendThread(wchddb02pxwe.prod.oclc.org:9011)" daemon prio=10 tid=0x0000000052946800 nid=0xcc8 runnable [0x0000000040cf6000]
  174.    java.lang.Thread.State: RUNNABLE
  175.         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  176.         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  177.         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  178.         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  179.         - locked <0x000000058caaa000> (a sun.nio.ch.Util$2)
  180.         - locked <0x000000058caaa018> (a java.util.Collections$UnmodifiableSet)
  181.         - locked <0x000000058cab5cf8> (a sun.nio.ch.EPollSelectorImpl)
  182.         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  183.         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:274)
  184.         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
  185.  
  186. "regionserver9009-smallCompactions-1337721650410" daemon prio=10 tid=0x0000000052866000 nid=0x130 waiting on condition [0x0000000049baa000]
  187.    java.lang.Thread.State: WAITING (parking)
  188.         at sun.misc.Unsafe.park(Native Method)
  189.         - parking to wait for  <0x000000058b8a1618> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  190.         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  191.         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  192.         at java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:220)
  193.         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
  194.         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  195.         at java.lang.Thread.run(Thread.java:662)
  196.  
  197. "RS_OPEN_REGION-wchddb37pxdu.prod.oclc.org,9009,1337721421002-2" prio=10 tid=0x00000000528f8000 nid=0x7fdd waiting on condition [0x00000000499a8000]
  198.    java.lang.Thread.State: WAITING (parking)
  199.         at sun.misc.Unsafe.park(Native Method)
  200.         - parking to wait for  <0x000000058b3938d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  201.         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  202.         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  203.         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  204.         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
  205.         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  206.         at java.lang.Thread.run(Thread.java:662)
  207.  
  208. "RS_OPEN_REGION-wchddb37pxdu.prod.oclc.org,9009,1337721421002-1" prio=10 tid=0x0000000053148000 nid=0x7fdc waiting on condition [0x000000004a5b4000]
  209.    java.lang.Thread.State: WAITING (parking)
  210.         at sun.misc.Unsafe.park(Native Method)
  211.         - parking to wait for  <0x000000058b3938d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  212.         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  213.         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  214.         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  215.         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
  216.         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  217.         at java.lang.Thread.run(Thread.java:662)
  218.  
  219. "RS_OPEN_REGION-wchddb37pxdu.prod.oclc.org,9009,1337721421002-0" prio=10 tid=0x0000000052e10000 nid=0x7fdb waiting on condition [0x00000000496a5000]
  220.    java.lang.Thread.State: WAITING (parking)
  221.         at sun.misc.Unsafe.park(Native Method)
  222.         - parking to wait for  <0x000000058b3938d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  223.         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  224.         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  225.         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  226.         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
  227.         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  228.         at java.lang.Thread.run(Thread.java:662)
  229.  
  230. "RMI Scheduler(0)" daemon prio=10 tid=0x0000000053126800 nid=0x7e4a waiting on condition [0x0000000049ead000]
  231.    java.lang.Thread.State: TIMED_WAITING (parking)
  232.         at sun.misc.Unsafe.park(Native Method)
  233.         - parking to wait for  <0x000000058b7f1150> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  234.         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
  235.         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
  236.         at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
  237.         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
  238.         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
  239.         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
  240.         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  241.         at java.lang.Thread.run(Thread.java:662)
  242.  
  243. "SplitLogWorker-wchddb37pxdu.prod.oclc.org,9009,1337721421002" prio=10 tid=0x00002aaac0a59800 nid=0x6d83 in Object.wait() [0x000000004808f000]
  244.    java.lang.Thread.State: WAITING (on object monitor)
  245.         at java.lang.Object.wait(Native Method)
  246.         at java.lang.Object.wait(Object.java:485)
  247.         at org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:205)
  248.         - locked <0x000000058c87ada0> (a java.lang.Object)
  249.         at org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:165)
  250.         at java.lang.Thread.run(Thread.java:662)
  251.  
  252. "PRI IPC Server handler 9 on 9009" daemon prio=10 tid=0x00002aaac0a57800 nid=0x6d82 waiting on condition [0x0000000047f8e000]
  253.    java.lang.Thread.State: WAITING (parking)
  254.         at sun.misc.Unsafe.park(Native Method)
  255.         - parking to wait for  <0x000000058b300df0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  256.         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  257.         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  258.         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  259.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1299)
  260.  
  261. "PRI IPC Server handler 8 on 9009" daemon prio=10 tid=0x00002aaac0a55800 nid=0x6d81 waiting on condition [0x0000000047e8d000]
  262.    java.lang.Thread.State: WAITING (parking)
  263.         at sun.misc.Unsafe.park(Native Method)
  264.         - parking to wait for  <0x000000058b300df0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  265.         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  266.         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  267.         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  268.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1299)
  269.  
  270. "PRI IPC Server handler 7 on 9009" daemon prio=10 tid=0x00002aaac0a53800 nid=0x6d80 waiting on condition [0x0000000047d8c000]
  271.    java.lang.Thread.State: WAITING (parking)
  272.         at sun.misc.Unsafe.park(Native Method)
  273.         - parking to wait for  <0x000000058b300df0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  274.         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  275.         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  276.         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  277.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1299)
  278.  
  279. "PRI IPC Server handler 6 on 9009" daemon prio=10 tid=0x00002aaac0730000 nid=0x6d7f waiting on condition [0x0000000047c8b000]
  280.    java.lang.Thread.State: WAITING (parking)
  281.         at sun.misc.Unsafe.park(Native Method)
  282.         - parking to wait for  <0x000000058b300df0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  283.         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  284.         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  285.         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  286.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1299)
  287.  
  288. "PRI IPC Server handler 5 on 9009" daemon prio=10 tid=0x00002aaac072e000 nid=0x6d7e waiting on condition [0x0000000047b8a000]
  289.    java.lang.Thread.State: WAITING (parking)
  290.         at sun.misc.Unsafe.park(Native Method)
  291.         - parking to wait for  <0x000000058b300df0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  292.         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  293.         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  294.         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  295.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1299)
  296.  
  297. "PRI IPC Server handler 4 on 9009" daemon prio=10 tid=0x00002aaac072c000 nid=0x6d7d waiting on condition [0x0000000047a89000]
  298.    java.lang.Thread.State: WAITING (parking)
  299.         at sun.misc.Unsafe.park(Native Method)
  300.         - parking to wait for  <0x000000058b300df0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  301.         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  302.         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  303.         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  304.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1299)
  305.  
  306. "PRI IPC Server handler 3 on 9009" daemon prio=10 tid=0x00002aaac072a000 nid=0x6d7c waiting on condition [0x0000000047988000]
  307.    java.lang.Thread.State: WAITING (parking)
  308.         at sun.misc.Unsafe.park(Native Method)
  309.         - parking to wait for  <0x000000058b300df0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  310.         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  311.         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  312.         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  313.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1299)
  314.  
  315. "PRI IPC Server handler 2 on 9009" daemon prio=10 tid=0x00002aaac0728000 nid=0x6d7b waiting on condition [0x0000000047887000]
  316.    java.lang.Thread.State: WAITING (parking)
  317.         at sun.misc.Unsafe.park(Native Method)
  318.         - parking to wait for  <0x000000058b300df0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  319.         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  320.         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  321.         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  322.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1299)
  323.  
  324. "PRI IPC Server handler 1 on 9009" daemon prio=10 tid=0x00002aaac0726000 nid=0x6d7a waiting on condition [0x0000000047786000]
  325.    java.lang.Thread.State: WAITING (parking)
  326.         at sun.misc.Unsafe.park(Native Method)
  327.         - parking to wait for  <0x000000058b300df0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  328.         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  329.         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  330.         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  331.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1299)
  332.  
  333. "PRI IPC Server handler 0 on 9009" daemon prio=10 tid=0x00002aaac0724000 nid=0x6d79 waiting on condition [0x0000000047685000]
  334.    java.lang.Thread.State: WAITING (parking)
  335.         at sun.misc.Unsafe.park(Native Method)
  336.         - parking to wait for  <0x000000058b300df0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  337.         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  338.         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  339.         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  340.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1299)
  341.  
  342. "IPC Server handler 49 on 9009" daemon prio=10 tid=0x00002aaac0722800 nid=0x6d78 in Object.wait() [0x0000000047584000]
  343.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  344.         at java.lang.Object.wait(Native Method)
  345.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  346.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  347.         at org.apache.hadoop.hbase.regionserver.HRegion.delete(HRegion.java:1469)
  348.         at org.apache.hadoop.hbase.regionserver.HRegionServer.delete(HRegionServer.java:2260)
  349.         at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
  350.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  351.         at java.lang.reflect.Method.invoke(Method.java:597)
  352.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  353.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  354.  
  355. "IPC Server handler 48 on 9009" daemon prio=10 tid=0x00002aaac0047000 nid=0x6d77 in Object.wait() [0x0000000047483000]
  356.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  357.         at java.lang.Object.wait(Native Method)
  358.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  359.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  360.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  361.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  362.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  363.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  364.         at java.lang.reflect.Method.invoke(Method.java:597)
  365.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  366.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  367.  
  368. "IPC Server handler 47 on 9009" daemon prio=10 tid=0x00002aaac0045000 nid=0x6d76 in Object.wait() [0x0000000047382000]
  369.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  370.         at java.lang.Object.wait(Native Method)
  371.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  372.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  373.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  374.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  375.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  376.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  377.         at java.lang.reflect.Method.invoke(Method.java:597)
  378.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  379.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  380.  
  381. "IPC Server handler 46 on 9009" daemon prio=10 tid=0x00002aaac0043800 nid=0x6d75 in Object.wait() [0x0000000047281000]
  382.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  383.         at java.lang.Object.wait(Native Method)
  384.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  385.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  386.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  387.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  388.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  389.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  390.         at java.lang.reflect.Method.invoke(Method.java:597)
  391.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  392.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  393.  
  394. "IPC Server handler 45 on 9009" daemon prio=10 tid=0x00002aaac0041800 nid=0x6d74 in Object.wait() [0x0000000047180000]
  395.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  396.         at java.lang.Object.wait(Native Method)
  397.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  398.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  399.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  400.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  401.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  402.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  403.         at java.lang.reflect.Method.invoke(Method.java:597)
  404.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  405.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  406.  
  407. "IPC Server handler 44 on 9009" daemon prio=10 tid=0x00002aaac003f800 nid=0x6d73 in Object.wait() [0x000000004707f000]
  408.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  409.         at java.lang.Object.wait(Native Method)
  410.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  411.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  412.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  413.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  414.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  415.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  416.         at java.lang.reflect.Method.invoke(Method.java:597)
  417.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  418.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  419.  
  420. "IPC Server handler 43 on 9009" daemon prio=10 tid=0x00002aaac003d800 nid=0x6d72 in Object.wait() [0x0000000046f7e000]
  421.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  422.         at java.lang.Object.wait(Native Method)
  423.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  424.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  425.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  426.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  427.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  428.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  429.         at java.lang.reflect.Method.invoke(Method.java:597)
  430.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  431.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  432.  
  433. "IPC Server handler 42 on 9009" daemon prio=10 tid=0x00002aaac003b800 nid=0x6d71 in Object.wait() [0x0000000046e7d000]
  434.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  435.         at java.lang.Object.wait(Native Method)
  436.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  437.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  438.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  439.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  440.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  441.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  442.         at java.lang.reflect.Method.invoke(Method.java:597)
  443.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  444.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  445.  
  446. "IPC Server handler 41 on 9009" daemon prio=10 tid=0x00002aaac0039800 nid=0x6d70 in Object.wait() [0x0000000046d7c000]
  447.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  448.         at java.lang.Object.wait(Native Method)
  449.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  450.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  451.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  452.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  453.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  454.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  455.         at java.lang.reflect.Method.invoke(Method.java:597)
  456.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  457.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  458.  
  459. "IPC Server handler 40 on 9009" daemon prio=10 tid=0x00002aaac0a00800 nid=0x6d6f in Object.wait() [0x0000000046c7b000]
  460.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  461.         at java.lang.Object.wait(Native Method)
  462.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  463.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  464.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  465.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  466.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  467.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  468.         at java.lang.reflect.Method.invoke(Method.java:597)
  469.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  470.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  471.  
  472. "IPC Server handler 39 on 9009" daemon prio=10 tid=0x00002aaac09fe800 nid=0x6d6e in Object.wait() [0x0000000046b7a000]
  473.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  474.         at java.lang.Object.wait(Native Method)
  475.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  476.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  477.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  478.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  479.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  480.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  481.         at java.lang.reflect.Method.invoke(Method.java:597)
  482.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  483.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  484.  
  485. "IPC Server handler 38 on 9009" daemon prio=10 tid=0x00002aaac09fc800 nid=0x6d6d in Object.wait() [0x0000000046a79000]
  486.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  487.         at java.lang.Object.wait(Native Method)
  488.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  489.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  490.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  491.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  492.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  493.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  494.         at java.lang.reflect.Method.invoke(Method.java:597)
  495.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  496.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  497.  
  498. "IPC Server handler 37 on 9009" daemon prio=10 tid=0x00002aaac09fa800 nid=0x6d6c in Object.wait() [0x0000000046978000]
  499.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  500.         at java.lang.Object.wait(Native Method)
  501.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  502.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  503.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  504.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  505.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  506.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  507.         at java.lang.reflect.Method.invoke(Method.java:597)
  508.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  509.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  510.  
  511. "IPC Server handler 36 on 9009" daemon prio=10 tid=0x00002aaac09f8800 nid=0x6d6b in Object.wait() [0x0000000046877000]
  512.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  513.         at java.lang.Object.wait(Native Method)
  514.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  515.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  516.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  517.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  518.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  519.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  520.         at java.lang.reflect.Method.invoke(Method.java:597)
  521.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  522.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  523.  
  524. "IPC Server handler 35 on 9009" daemon prio=10 tid=0x00002aaac09f6800 nid=0x6d6a in Object.wait() [0x0000000046776000]
  525.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  526.         at java.lang.Object.wait(Native Method)
  527.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  528.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  529.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  530.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  531.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  532.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  533.         at java.lang.reflect.Method.invoke(Method.java:597)
  534.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  535.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  536.  
  537. "IPC Server handler 34 on 9009" daemon prio=10 tid=0x00002aaac09f4800 nid=0x6d69 in Object.wait() [0x0000000046675000]
  538.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  539.         at java.lang.Object.wait(Native Method)
  540.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  541.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  542.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  543.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  544.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  545.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  546.         at java.lang.reflect.Method.invoke(Method.java:597)
  547.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  548.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  549.  
  550. "IPC Server handler 33 on 9009" daemon prio=10 tid=0x00002aaac09f2800 nid=0x6d68 in Object.wait() [0x0000000046574000]
  551.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  552.         at java.lang.Object.wait(Native Method)
  553.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  554.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  555.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  556.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  557.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  558.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  559.         at java.lang.reflect.Method.invoke(Method.java:597)
  560.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  561.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  562.  
  563. "IPC Server handler 32 on 9009" daemon prio=10 tid=0x00002aaac08a9800 nid=0x6d67 in Object.wait() [0x0000000046473000]
  564.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  565.         at java.lang.Object.wait(Native Method)
  566.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  567.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  568.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  569.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  570.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  571.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  572.         at java.lang.reflect.Method.invoke(Method.java:597)
  573.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  574.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  575.  
  576. "IPC Server handler 31 on 9009" daemon prio=10 tid=0x00002aaac08a7800 nid=0x6d66 in Object.wait() [0x0000000046372000]
  577.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  578.         at java.lang.Object.wait(Native Method)
  579.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  580.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  581.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  582.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  583.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  584.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  585.         at java.lang.reflect.Method.invoke(Method.java:597)
  586.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  587.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  588.  
  589. "IPC Server handler 30 on 9009" daemon prio=10 tid=0x00002aaac08a5800 nid=0x6d65 in Object.wait() [0x0000000046271000]
  590.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  591.         at java.lang.Object.wait(Native Method)
  592.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  593.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  594.         at org.apache.hadoop.hbase.regionserver.HRegion.delete(HRegion.java:1469)
  595.         at org.apache.hadoop.hbase.regionserver.HRegionServer.delete(HRegionServer.java:2260)
  596.         at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
  597.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  598.         at java.lang.reflect.Method.invoke(Method.java:597)
  599.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  600.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  601.  
  602. "IPC Server handler 29 on 9009" daemon prio=10 tid=0x00002aaac08a3800 nid=0x6d64 runnable [0x0000000046170000]
  603.    java.lang.Thread.State: RUNNABLE
  604.         at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:691)
  605.         at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.seekInSubLists(MemStore.java:726)
  606.         - locked <0x000000058d939c38> (a org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner)
  607.         at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:765)
  608.         - locked <0x000000058d939c38> (a org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner)
  609.         at org.apache.hadoop.hbase.regionserver.KeyValueHeap$SeekType$1.seek(KeyValueHeap.java:57)
  610.         at org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:277)
  611.         at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:248)
  612.         at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:436)
  613.         - locked <0x000000058fd24658> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  614.         at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:322)
  615.         - locked <0x000000058fd24658> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  616.         at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:138)
  617.         at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:2945)
  618.         at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:2901)
  619.         - locked <0x000000058d939c70> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl)
  620.         at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:2918)
  621.         - locked <0x000000058d939c70> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl)
  622.         at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:3693)
  623.         at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:3585)
  624.         at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1785)
  625.         at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source)
  626.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  627.         at java.lang.reflect.Method.invoke(Method.java:597)
  628.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  629.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  630.  
  631. "IPC Server handler 28 on 9009" daemon prio=10 tid=0x00002aaac08a1800 nid=0x6d63 in Object.wait() [0x000000004606f000]
  632.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  633.         at java.lang.Object.wait(Native Method)
  634.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  635.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  636.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  637.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  638.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  639.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  640.         at java.lang.reflect.Method.invoke(Method.java:597)
  641.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  642.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  643.  
  644. "IPC Server handler 27 on 9009" daemon prio=10 tid=0x00002aaac08a0000 nid=0x6d62 in Object.wait() [0x0000000045f6e000]
  645.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  646.         at java.lang.Object.wait(Native Method)
  647.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  648.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  649.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  650.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  651.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  652.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  653.         at java.lang.reflect.Method.invoke(Method.java:597)
  654.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  655.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  656.  
  657. "IPC Server handler 26 on 9009" daemon prio=10 tid=0x00002aaac089e000 nid=0x6d61 in Object.wait() [0x0000000045e6d000]
  658.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  659.         at java.lang.Object.wait(Native Method)
  660.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  661.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  662.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  663.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  664.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  665.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  666.         at java.lang.reflect.Method.invoke(Method.java:597)
  667.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  668.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  669.  
  670. "IPC Server handler 25 on 9009" daemon prio=10 tid=0x00002aaac0a7f800 nid=0x6d60 in Object.wait() [0x0000000045d6c000]
  671.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  672.         at java.lang.Object.wait(Native Method)
  673.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  674.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  675.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  676.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  677.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  678.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  679.         at java.lang.reflect.Method.invoke(Method.java:597)
  680.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  681.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  682.  
  683. "IPC Server handler 24 on 9009" daemon prio=10 tid=0x00002aaac0a7d800 nid=0x6d5f in Object.wait() [0x0000000045c6b000]
  684.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  685.         at java.lang.Object.wait(Native Method)
  686.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  687.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  688.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  689.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  690.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  691.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  692.         at java.lang.reflect.Method.invoke(Method.java:597)
  693.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  694.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  695.  
  696. "IPC Server handler 23 on 9009" daemon prio=10 tid=0x00002aaac0a7b800 nid=0x6d5e in Object.wait() [0x0000000045b6a000]
  697.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  698.         at java.lang.Object.wait(Native Method)
  699.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  700.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  701.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  702.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  703.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  704.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  705.         at java.lang.reflect.Method.invoke(Method.java:597)
  706.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  707.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  708.  
  709. "IPC Server handler 22 on 9009" daemon prio=10 tid=0x00002aaac055f000 nid=0x6d5d in Object.wait() [0x0000000045a69000]
  710.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  711.         at java.lang.Object.wait(Native Method)
  712.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  713.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  714.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  715.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  716.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  717.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  718.         at java.lang.reflect.Method.invoke(Method.java:597)
  719.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  720.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  721.  
  722. "IPC Server handler 21 on 9009" daemon prio=10 tid=0x00002aaac07f7800 nid=0x6d5c in Object.wait() [0x0000000045968000]
  723.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  724.         at java.lang.Object.wait(Native Method)
  725.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  726.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  727.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  728.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  729.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  730.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  731.         at java.lang.reflect.Method.invoke(Method.java:597)
  732.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  733.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  734.  
  735. "IPC Server handler 20 on 9009" daemon prio=10 tid=0x00002aaacc236800 nid=0x6d5b in Object.wait() [0x0000000045867000]
  736.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  737.         at java.lang.Object.wait(Native Method)
  738.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  739.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  740.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  741.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  742.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  743.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  744.         at java.lang.reflect.Method.invoke(Method.java:597)
  745.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  746.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  747.  
  748. "IPC Server handler 19 on 9009" daemon prio=10 tid=0x0000000052fc7800 nid=0x6d5a in Object.wait() [0x0000000045766000]
  749.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  750.         at java.lang.Object.wait(Native Method)
  751.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  752.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  753.         at org.apache.hadoop.hbase.regionserver.HRegion.delete(HRegion.java:1469)
  754.         at org.apache.hadoop.hbase.regionserver.HRegionServer.delete(HRegionServer.java:2260)
  755.         at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
  756.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  757.         at java.lang.reflect.Method.invoke(Method.java:597)
  758.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  759.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  760.  
  761. "IPC Server handler 18 on 9009" daemon prio=10 tid=0x0000000052c84000 nid=0x6d59 in Object.wait() [0x0000000045665000]
  762.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  763.         at java.lang.Object.wait(Native Method)
  764.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  765.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  766.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  767.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  768.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  769.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  770.         at java.lang.reflect.Method.invoke(Method.java:597)
  771.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  772.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  773.  
  774. "IPC Server handler 17 on 9009" daemon prio=10 tid=0x0000000052c82000 nid=0x6d58 in Object.wait() [0x0000000045564000]
  775.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  776.         at java.lang.Object.wait(Native Method)
  777.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  778.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  779.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  780.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  781.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  782.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  783.         at java.lang.reflect.Method.invoke(Method.java:597)
  784.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  785.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  786.  
  787. "IPC Server handler 16 on 9009" daemon prio=10 tid=0x0000000052c7f800 nid=0x6d57 in Object.wait() [0x0000000045463000]
  788.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  789.         at java.lang.Object.wait(Native Method)
  790.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  791.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  792.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  793.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  794.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  795.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  796.         at java.lang.reflect.Method.invoke(Method.java:597)
  797.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  798.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  799.  
  800. "IPC Server handler 15 on 9009" daemon prio=10 tid=0x0000000052a03800 nid=0x6d56 in Object.wait() [0x0000000045362000]
  801.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  802.         at java.lang.Object.wait(Native Method)
  803.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  804.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  805.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  806.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  807.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  808.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  809.         at java.lang.reflect.Method.invoke(Method.java:597)
  810.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  811.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  812.  
  813. "IPC Server handler 14 on 9009" daemon prio=10 tid=0x0000000052a01000 nid=0x6d55 in Object.wait() [0x0000000045261000]
  814.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  815.         at java.lang.Object.wait(Native Method)
  816.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  817.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  818.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  819.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  820.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  821.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  822.         at java.lang.reflect.Method.invoke(Method.java:597)
  823.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  824.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  825.  
  826. "IPC Server handler 13 on 9009" daemon prio=10 tid=0x00000000529ff000 nid=0x6d54 in Object.wait() [0x0000000045160000]
  827.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  828.         at java.lang.Object.wait(Native Method)
  829.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  830.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  831.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  832.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  833.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  834.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  835.         at java.lang.reflect.Method.invoke(Method.java:597)
  836.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  837.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  838.  
  839. "IPC Server handler 12 on 9009" daemon prio=10 tid=0x0000000052c65800 nid=0x6d53 in Object.wait() [0x000000004505f000]
  840.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  841.         at java.lang.Object.wait(Native Method)
  842.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  843.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  844.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  845.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  846.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  847.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  848.         at java.lang.reflect.Method.invoke(Method.java:597)
  849.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  850.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  851.  
  852. "IPC Server handler 11 on 9009" daemon prio=10 tid=0x0000000052c64000 nid=0x6d52 in Object.wait() [0x0000000044f5e000]
  853.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  854.         at java.lang.Object.wait(Native Method)
  855.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  856.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  857.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  858.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  859.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  860.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  861.         at java.lang.reflect.Method.invoke(Method.java:597)
  862.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  863.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  864.  
  865. "IPC Server handler 10 on 9009" daemon prio=10 tid=0x0000000053170800 nid=0x6d51 in Object.wait() [0x0000000044e5d000]
  866.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  867.         at java.lang.Object.wait(Native Method)
  868.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  869.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  870.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  871.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  872.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  873.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  874.         at java.lang.reflect.Method.invoke(Method.java:597)
  875.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  876.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  877.  
  878. "IPC Server handler 9 on 9009" daemon prio=10 tid=0x0000000052fd7000 nid=0x6d50 in Object.wait() [0x0000000044d5c000]
  879.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  880.         at java.lang.Object.wait(Native Method)
  881.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  882.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  883.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  884.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  885.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  886.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  887.         at java.lang.reflect.Method.invoke(Method.java:597)
  888.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  889.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  890.  
  891. "IPC Server handler 8 on 9009" daemon prio=10 tid=0x0000000052e1f000 nid=0x6d4f in Object.wait() [0x0000000044c5b000]
  892.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  893.         at java.lang.Object.wait(Native Method)
  894.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  895.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  896.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  897.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  898.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  899.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  900.         at java.lang.reflect.Method.invoke(Method.java:597)
  901.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  902.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  903.  
  904. "IPC Server handler 7 on 9009" daemon prio=10 tid=0x0000000052bd9000 nid=0x6d4e in Object.wait() [0x0000000044b5a000]
  905.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  906.         at java.lang.Object.wait(Native Method)
  907.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  908.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  909.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  910.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  911.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  912.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  913.         at java.lang.reflect.Method.invoke(Method.java:597)
  914.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  915.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  916.  
  917. "IPC Server handler 6 on 9009" daemon prio=10 tid=0x0000000052be9000 nid=0x6d4d in Object.wait() [0x0000000044a59000]
  918.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  919.         at java.lang.Object.wait(Native Method)
  920.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  921.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  922.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  923.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  924.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  925.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  926.         at java.lang.reflect.Method.invoke(Method.java:597)
  927.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  928.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  929.  
  930. "IPC Server handler 5 on 9009" daemon prio=10 tid=0x0000000053160800 nid=0x6d4c in Object.wait() [0x0000000044958000]
  931.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  932.         at java.lang.Object.wait(Native Method)
  933.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  934.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  935.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  936.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  937.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  938.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  939.         at java.lang.reflect.Method.invoke(Method.java:597)
  940.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  941.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  942.  
  943. "IPC Server handler 4 on 9009" daemon prio=10 tid=0x0000000052ee3800 nid=0x6d4b in Object.wait() [0x0000000044857000]
  944.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  945.         at java.lang.Object.wait(Native Method)
  946.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  947.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  948.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  949.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  950.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  951.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  952.         at java.lang.reflect.Method.invoke(Method.java:597)
  953.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  954.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  955.  
  956. "IPC Server handler 3 on 9009" daemon prio=10 tid=0x0000000052c02800 nid=0x6d4a in Object.wait() [0x0000000044756000]
  957.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  958.         at java.lang.Object.wait(Native Method)
  959.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  960.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  961.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  962.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  963.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  964.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  965.         at java.lang.reflect.Method.invoke(Method.java:597)
  966.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  967.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  968.  
  969. "IPC Server handler 2 on 9009" daemon prio=10 tid=0x0000000052c01000 nid=0x6d49 in Object.wait() [0x0000000044655000]
  970.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  971.         at java.lang.Object.wait(Native Method)
  972.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  973.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  974.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  975.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  976.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  977.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  978.         at java.lang.reflect.Method.invoke(Method.java:597)
  979.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  980.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  981.  
  982. "IPC Server handler 1 on 9009" daemon prio=10 tid=0x0000000052a22800 nid=0x6d48 in Object.wait() [0x0000000044554000]
  983.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  984.         at java.lang.Object.wait(Native Method)
  985.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  986.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  987.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  988.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  989.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  990.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  991.         at java.lang.reflect.Method.invoke(Method.java:597)
  992.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  993.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  994.  
  995. "IPC Server handler 0 on 9009" daemon prio=10 tid=0x0000000052c19800 nid=0x6d47 in Object.wait() [0x0000000044453000]
  996.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  997.         at java.lang.Object.wait(Native Method)
  998.         at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  999.         - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  1000.         at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  1001.         at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  1002.         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  1003.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  1004.         at java.lang.reflect.Method.invoke(Method.java:597)
  1005.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  1006.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  1007.  
  1008. "IPC Server listener on 9009" daemon prio=10 tid=0x0000000052c54800 nid=0x6d46 runnable [0x0000000044352000]
  1009.    java.lang.Thread.State: RUNNABLE
  1010.         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1011.         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  1012.         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  1013.         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  1014.         - locked <0x000000058b300e80> (a sun.nio.ch.Util$2)
  1015.         - locked <0x000000058b300e98> (a java.util.Collections$UnmodifiableSet)
  1016.         - locked <0x000000058b302f20> (a sun.nio.ch.EPollSelectorImpl)
  1017.         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  1018.         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
  1019.         at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:592)
  1020.  
  1021. "IPC Server Responder" daemon prio=10 tid=0x0000000052c54000 nid=0x6d45 runnable [0x0000000044251000]
  1022.    java.lang.Thread.State: RUNNABLE
  1023.         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1024.         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  1025.         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  1026.         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  1027.         - locked <0x000000058b33b220> (a sun.nio.ch.Util$2)
  1028.         - locked <0x000000058b33b238> (a java.util.Collections$UnmodifiableSet)
  1029.         - locked <0x000000058b33aba0> (a sun.nio.ch.EPollSelectorImpl)
  1030.         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  1031.         at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRunLoop(HBaseServer.java:782)
  1032.         at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:765)
  1033.  
  1034. "Timer-0" daemon prio=10 tid=0x0000000053121000 nid=0x6d44 in Object.wait() [0x0000000044150000]
  1035.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  1036.         at java.lang.Object.wait(Native Method)
  1037.         at java.util.TimerThread.mainLoop(Timer.java:509)
  1038.         - locked <0x000000058b7e83a8> (a java.util.TaskQueue)
  1039.         at java.util.TimerThread.run(Timer.java:462)
  1040.  
  1041. "1861006321@qtp-1068922666-1 - Acceptor0 SelectChannelConnector@0.0.0.0:9004" prio=10 tid=0x00002aaac0079800 nid=0x6d41 runnable [0x000000004404f000]
  1042.    java.lang.Thread.State: RUNNABLE
  1043.         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1044.         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  1045.         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  1046.         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  1047.         - locked <0x000000058b959868> (a sun.nio.ch.Util$2)
  1048.         - locked <0x000000058b959880> (a java.util.Collections$UnmodifiableSet)
  1049.         - locked <0x000000058b3b5400> (a sun.nio.ch.EPollSelectorImpl)
  1050.         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  1051.         at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
  1052.         at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
  1053.         at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
  1054.         at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
  1055.         at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
  1056.  
  1057. "regionserver9009.leaseChecker" daemon prio=10 tid=0x00002aaac002b800 nid=0x6d3f waiting on condition [0x0000000043e4d000]
  1058.    java.lang.Thread.State: TIMED_WAITING (parking)
  1059.         at sun.misc.Unsafe.park(Native Method)
  1060.         - parking to wait for  <0x000000058b30a830> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1061.         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
  1062.         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
  1063.         at java.util.concurrent.DelayQueue.poll(DelayQueue.java:201)
  1064.         at org.apache.hadoop.hbase.regionserver.Leases.run(Leases.java:83)
  1065.         at java.lang.Thread.run(Thread.java:662)
  1066.  
  1067. "regionserver9009.compactionChecker" daemon prio=10 tid=0x00002aaac005d800 nid=0x6d3e in Object.wait() [0x0000000043d4c000]
  1068.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  1069.         at java.lang.Object.wait(Native Method)
  1070.         at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
  1071.         - locked <0x000000058b9597c0> (a java.lang.Object)
  1072.         at org.apache.hadoop.hbase.Chore.run(Chore.java:75)
  1073.         at java.lang.Thread.run(Thread.java:662)
  1074.  
  1075. "regionserver9009.cacheFlusher" daemon prio=10 tid=0x00002aaac005d000 nid=0x6d3d waiting for monitor entry [0x0000000043c4b000]
  1076.    java.lang.Thread.State: BLOCKED (on object monitor)
  1077.         at org.apache.hadoop.hbase.regionserver.StoreScanner.updateReaders(StoreScanner.java:360)
  1078.         - waiting to lock <0x000000058fd24658> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  1079.         at org.apache.hadoop.hbase.regionserver.Store.notifyChangedReadersObservers(Store.java:667)
  1080.         at org.apache.hadoop.hbase.regionserver.Store.updateStorefiles(Store.java:656)
  1081.         at org.apache.hadoop.hbase.regionserver.Store.access$200(Store.java:89)
  1082.         at org.apache.hadoop.hbase.regionserver.Store$StoreFlusherImpl.commit(Store.java:1916)
  1083.         at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1259)
  1084.         at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1162)
  1085.         at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1104)
  1086.         at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:400)
  1087.         at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:374)
  1088.         at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:243)
  1089.         at java.lang.Thread.run(Thread.java:662)
  1090.  
  1091. "regionserver9009.logRoller" daemon prio=10 tid=0x00002aaac00ca000 nid=0x6d3c waiting on condition [0x0000000043b4a000]
  1092.    java.lang.Thread.State: WAITING (parking)
  1093.         at sun.misc.Unsafe.park(Native Method)
  1094.         - parking to wait for  <0x000000058b8f29a8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
  1095.         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  1096.         at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
  1097.         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
  1098.         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
  1099.         at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
  1100.         at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
  1101.         at org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:549)
  1102.         at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
  1103.         at java.lang.Thread.run(Thread.java:662)
  1104.  
  1105. "Timer thread for monitoring jvm" daemon prio=10 tid=0x00002aaac00c8000 nid=0x6d3b in Object.wait() [0x0000000043a49000]
  1106.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  1107.         at java.lang.Object.wait(Native Method)
  1108.         at java.util.TimerThread.mainLoop(Timer.java:509)
  1109.         - locked <0x000000058b9597d8> (a java.util.TaskQueue)
  1110.         at java.util.TimerThread.run(Timer.java:462)
  1111.  
  1112. "Timer thread for monitoring hbase" daemon prio=10 tid=0x00002aaac0720800 nid=0x6d3a in Object.wait() [0x0000000043948000]
  1113.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  1114.         at java.lang.Object.wait(Native Method)
  1115.         at java.util.TimerThread.mainLoop(Timer.java:509)
  1116.         - locked <0x000000058b959790> (a java.util.TaskQueue)
  1117.         at java.util.TimerThread.run(Timer.java:462)
  1118.  
  1119. "regionserver9009.logSyncer" daemon prio=10 tid=0x00002aaac07ac000 nid=0x6d39 waiting on condition [0x0000000043847000]
  1120.    java.lang.Thread.State: TIMED_WAITING (sleeping)
  1121.         at java.lang.Thread.sleep(Native Method)
  1122.         at org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:1059)
  1123.         at java.lang.Thread.run(Thread.java:662)
  1124.  
  1125. "LeaseChecker" daemon prio=10 tid=0x00002aaac0698000 nid=0x6d38 waiting on condition [0x00000000411c5000]
  1126.    java.lang.Thread.State: TIMED_WAITING (sleeping)
  1127.         at java.lang.Thread.sleep(Native Method)
  1128.         at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1302)
  1129.         at java.lang.Thread.run(Thread.java:662)
  1130.  
  1131. "IncidentDistributionPoint-0/ProcessSyncQueue" daemon prio=10 tid=0x0000000052e99800 nid=0x6d32 in Object.wait() [0x0000000043645000]
  1132.    java.lang.Thread.State: WAITING (on object monitor)
  1133.         at java.lang.Object.wait(Native Method)
  1134.         at java.lang.Object.wait(Object.java:485)
  1135.         at ORG.oclc.infra.util.SyncQueue.waitForEntries(SyncQueue.java:276)
  1136.         - locked <0x000000058b304f08> (a ORG.oclc.infra.util.SyncQueue)
  1137.         at ORG.oclc.infra.drax.DestinationList.execute(DestinationList.java:115)
  1138.         at ORG.oclc.infra.drax.DistributionPoint.run(DistributionPoint.java:134)
  1139.         at java.lang.Thread.run(Thread.java:662)
  1140.  
  1141. "regionserver9009-EventThread" daemon prio=10 tid=0x0000000052f61000 nid=0x6cfc waiting on condition [0x0000000040df7000]
  1142.    java.lang.Thread.State: WAITING (parking)
  1143.         at sun.misc.Unsafe.park(Native Method)
  1144.         - parking to wait for  <0x000000058b300b40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1145.         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  1146.         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  1147.         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  1148.         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:493)
  1149.  
  1150. "regionserver9009-SendThread(wchddb35pxdu.prod.oclc.org:9011)" daemon prio=10 tid=0x000000005259e000 nid=0x6cfa runnable [0x0000000040bf5000]
  1151.    java.lang.Thread.State: RUNNABLE
  1152.         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1153.         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  1154.         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  1155.         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  1156.         - locked <0x000000058b300a38> (a sun.nio.ch.Util$2)
  1157.         - locked <0x000000058b300a50> (a java.util.Collections$UnmodifiableSet)
  1158.         - locked <0x000000058b3068d8> (a sun.nio.ch.EPollSelectorImpl)
  1159.         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  1160.         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:274)
  1161.         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
  1162.  
  1163. "DestroyJavaVM" prio=10 tid=0x00002aaac0ab5000 nid=0x6b6a waiting on condition [0x0000000000000000]
  1164.    java.lang.Thread.State: RUNNABLE
  1165.  
  1166. "regionserver9009-EventThread" daemon prio=10 tid=0x00002aaacc165800 nid=0x6bbc waiting on condition [0x0000000040af4000]
  1167.    java.lang.Thread.State: WAITING (parking)
  1168.         at sun.misc.Unsafe.park(Native Method)
  1169.         - parking to wait for  <0x000000058b300b88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1170.         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  1171.         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  1172.         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  1173.         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:493)
  1174.  
  1175. "regionserver9009-SendThread(wchddb02pxdu.prod.oclc.org:9011)" daemon prio=10 tid=0x00002aaacc162800 nid=0x6bbb runnable [0x000000004032e000]
  1176.    java.lang.Thread.State: RUNNABLE
  1177.         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1178.         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  1179.         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  1180.         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  1181.         - locked <0x000000058b300bd0> (a sun.nio.ch.Util$2)
  1182.         - locked <0x000000058b300be8> (a java.util.Collections$UnmodifiableSet)
  1183.         - locked <0x000000058b300858> (a sun.nio.ch.EPollSelectorImpl)
  1184.         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  1185.         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:274)
  1186.         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
  1187.  
  1188. "regionserver9009" prio=10 tid=0x00000000530ad000 nid=0x6bb9 in Object.wait() [0x0000000043241000]
  1189.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  1190.         at java.lang.Object.wait(Native Method)
  1191.         at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
  1192.         - locked <0x000000058b300a98> (a java.lang.Object)
  1193.         at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:55)
  1194.         at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:691)
  1195.         at java.lang.Thread.run(Thread.java:662)
  1196.  
  1197. "LRU Statistics #0" daemon prio=10 tid=0x00000000530d8800 nid=0x6bb8 waiting on condition [0x0000000043140000]
  1198.    java.lang.Thread.State: TIMED_WAITING (parking)
  1199.         at sun.misc.Unsafe.park(Native Method)
  1200.         - parking to wait for  <0x000000058b300c30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1201.         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
  1202.         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
  1203.         at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
  1204.         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
  1205.         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
  1206.         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
  1207.         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  1208.         at java.lang.Thread.run(Thread.java:662)
  1209.  
  1210. "LruBlockCache.EvictionThread" daemon prio=10 tid=0x00000000528c3000 nid=0x6bb7 in Object.wait() [0x0000000041c3d000]
  1211.    java.lang.Thread.State: WAITING (on object monitor)
  1212.         at java.lang.Object.wait(Native Method)
  1213.         at java.lang.Object.wait(Object.java:485)
  1214.         at org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:566)
  1215.         - locked <0x000000058b300ab0> (a org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread)
  1216.         at java.lang.Thread.run(Thread.java:662)
  1217.  
  1218. "Timer thread for monitoring rpc" daemon prio=10 tid=0x0000000052f48000 nid=0x6bb3 in Object.wait() [0x000000004303f000]
  1219.    java.lang.Thread.State: TIMED_WAITING (on object monitor)
  1220.         at java.lang.Object.wait(Native Method)
  1221.         at java.util.TimerThread.mainLoop(Timer.java:509)
  1222.         - locked <0x000000058b300ee0> (a java.util.TaskQueue)
  1223.         at java.util.TimerThread.run(Timer.java:462)
  1224.  
  1225. "IPC Reader 9 on port 9009" daemon prio=10 tid=0x0000000052b5d000 nid=0x6bb2 runnable [0x0000000042f3e000]
  1226.    java.lang.Thread.State: RUNNABLE
  1227.         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1228.         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  1229.         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  1230.         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  1231.         - locked <0x000000058b300ac8> (a sun.nio.ch.Util$2)
  1232.         - locked <0x000000058b300ae0> (a java.util.Collections$UnmodifiableSet)
  1233.         - locked <0x000000058b306920> (a sun.nio.ch.EPollSelectorImpl)
  1234.         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  1235.         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
  1236.         at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:484)
  1237.         - locked <0x000000058b300af8> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  1238.         at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:470)
  1239.         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  1240.         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  1241.         at java.lang.Thread.run(Thread.java:662)
  1242.  
  1243. "IPC Reader 8 on port 9009" daemon prio=10 tid=0x0000000052b5b000 nid=0x6bb1 runnable [0x0000000042e3d000]
  1244.    java.lang.Thread.State: RUNNABLE
  1245.         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1246.         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  1247.         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  1248.         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  1249.         - locked <0x000000058b300028> (a sun.nio.ch.Util$2)
  1250.         - locked <0x000000058b300040> (a java.util.Collections$UnmodifiableSet)
  1251.         - locked <0x000000058b30a9b0> (a sun.nio.ch.EPollSelectorImpl)
  1252.         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  1253.         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
  1254.         at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:484)
  1255.         - locked <0x000000058b300058> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  1256.         at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:470)
  1257.         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  1258.         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  1259.         at java.lang.Thread.run(Thread.java:662)
  1260.  
  1261. "IPC Reader 7 on port 9009" daemon prio=10 tid=0x0000000052a30800 nid=0x6bb0 runnable [0x0000000042d3c000]
  1262.    java.lang.Thread.State: RUNNABLE
  1263.         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1264.         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  1265.         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  1266.         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  1267.         - locked <0x000000058b300f10> (a sun.nio.ch.Util$2)
  1268.         - locked <0x000000058b300f28> (a java.util.Collections$UnmodifiableSet)
  1269.         - locked <0x000000058b302f68> (a sun.nio.ch.EPollSelectorImpl)
  1270.         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  1271.         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
  1272.         at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:484)
  1273.         - locked <0x000000058b300f40> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  1274.         at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:470)
  1275.         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  1276.         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  1277.         at java.lang.Thread.run(Thread.java:662)
  1278.  
  1279. "IPC Reader 6 on port 9009" daemon prio=10 tid=0x0000000052a2e800 nid=0x6bae runnable [0x0000000042c3b000]
  1280.    java.lang.Thread.State: RUNNABLE
  1281.         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1282.         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  1283.         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  1284.         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  1285.         - locked <0x000000058b30ab18> (a sun.nio.ch.Util$2)
  1286.         - locked <0x000000058b30ab30> (a java.util.Collections$UnmodifiableSet)
  1287.         - locked <0x000000058b302fb0> (a sun.nio.ch.EPollSelectorImpl)
  1288.         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  1289.         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
  1290.         at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:484)
  1291.         - locked <0x000000058b30ab48> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  1292.         at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:470)
  1293.         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  1294.         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  1295.         at java.lang.Thread.run(Thread.java:662)
  1296.  
  1297. "IPC Reader 5 on port 9009" daemon prio=10 tid=0x0000000053128800 nid=0x6bad runnable [0x0000000042b3a000]
  1298.    java.lang.Thread.State: RUNNABLE
  1299.         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1300.         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  1301.         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  1302.         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  1303.         - locked <0x000000058b300088> (a sun.nio.ch.Util$2)
  1304.         - locked <0x000000058b3000a0> (a java.util.Collections$UnmodifiableSet)
  1305.         - locked <0x000000058b30a9f8> (a sun.nio.ch.EPollSelectorImpl)
  1306.         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  1307.         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
  1308.         at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:484)
  1309.         - locked <0x000000058b3000b8> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  1310.         at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:470)
  1311.         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  1312.         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  1313.         at java.lang.Thread.run(Thread.java:662)
  1314.  
  1315. "IPC Reader 4 on port 9009" daemon prio=10 tid=0x0000000052c86800 nid=0x6bac runnable [0x0000000042a39000]
  1316.    java.lang.Thread.State: RUNNABLE
  1317.         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1318.         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  1319.         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  1320.         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  1321.         - locked <0x000000058b30a878> (a sun.nio.ch.Util$2)
  1322.         - locked <0x000000058b30a890> (a java.util.Collections$UnmodifiableSet)
  1323.         - locked <0x000000058b30b738> (a sun.nio.ch.EPollSelectorImpl)
  1324.         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  1325.         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
  1326.         at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:484)
  1327.         - locked <0x000000058b30a8a8> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  1328.         at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:470)
  1329.         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  1330.         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  1331.         at java.lang.Thread.run(Thread.java:662)
  1332.  
  1333. "IPC Reader 3 on port 9009" daemon prio=10 tid=0x0000000052f51800 nid=0x6bab runnable [0x0000000042938000]
  1334.    java.lang.Thread.State: RUNNABLE
  1335.         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1336.         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  1337.         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  1338.         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  1339.         - locked <0x000000058b30ab78> (a sun.nio.ch.Util$2)
  1340.         - locked <0x000000058b30ab90> (a java.util.Collections$UnmodifiableSet)
  1341.         - locked <0x000000058b302ff8> (a sun.nio.ch.EPollSelectorImpl)
  1342.         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  1343.         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
  1344.         at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:484)
  1345.         - locked <0x000000058b30aba8> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  1346.         at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:470)
  1347.         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  1348.         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  1349.         at java.lang.Thread.run(Thread.java:662)
  1350.  
  1351. "IPC Reader 2 on port 9009" daemon prio=10 tid=0x00000000528fc800 nid=0x6baa runnable [0x0000000042837000]
  1352.    java.lang.Thread.State: RUNNABLE
  1353.         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1354.         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  1355.         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  1356.         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  1357.         - locked <0x000000058b3000e8> (a sun.nio.ch.Util$2)
  1358.         - locked <0x000000058b300100> (a java.util.Collections$UnmodifiableSet)
  1359.         - locked <0x000000058b30aa40> (a sun.nio.ch.EPollSelectorImpl)
  1360.         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  1361.         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
  1362.         at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:484)
  1363.         - locked <0x000000058b300118> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  1364.         at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:470)
  1365.         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  1366.         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  1367.         at java.lang.Thread.run(Thread.java:662)
  1368.  
  1369. "IPC Reader 1 on port 9009" daemon prio=10 tid=0x0000000052ecb000 nid=0x6ba9 runnable [0x0000000042736000]
  1370.    java.lang.Thread.State: RUNNABLE
  1371.         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1372.         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  1373.         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  1374.         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  1375.         - locked <0x000000058b300b28> (a sun.nio.ch.Util$2)
  1376.         - locked <0x000000058b30c0f8> (a java.util.Collections$UnmodifiableSet)
  1377.         - locked <0x000000058b306968> (a sun.nio.ch.EPollSelectorImpl)
  1378.         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  1379.         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
  1380.         at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:484)
  1381.         - locked <0x000000058b30c110> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  1382.         at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:470)
  1383.         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  1384.         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  1385.         at java.lang.Thread.run(Thread.java:662)
  1386.  
  1387. "IPC Reader 0 on port 9009" daemon prio=10 tid=0x0000000052c8a000 nid=0x6ba8 runnable [0x0000000042635000]
  1388.    java.lang.Thread.State: RUNNABLE
  1389.         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1390.         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  1391.         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  1392.         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  1393.         - locked <0x000000058b30c278> (a sun.nio.ch.Util$2)
  1394.         - locked <0x000000058b30c260> (a java.util.Collections$UnmodifiableSet)
  1395.         - locked <0x000000058b303088> (a sun.nio.ch.EPollSelectorImpl)
  1396.         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  1397.         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
  1398.         at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:484)
  1399.         - locked <0x000000058b74e918> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  1400.         at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:470)
  1401.         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  1402.         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  1403.         at java.lang.Thread.run(Thread.java:662)
  1404.  
  1405. "RMI TCP Accept-0" daemon prio=10 tid=0x00002aaac065e000 nid=0x6b83 runnable [0x0000000040fc3000]
  1406.    java.lang.Thread.State: RUNNABLE
  1407.         at java.net.PlainSocketImpl.socketAccept(Native Method)
  1408.         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
  1409.         - locked <0x000000058b30bae0> (a java.net.SocksSocketImpl)
  1410.         at java.net.ServerSocket.implAccept(ServerSocket.java:462)
  1411.         at java.net.ServerSocket.accept(ServerSocket.java:430)
  1412.         at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
  1413.         at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
  1414.         at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
  1415.         at java.lang.Thread.run(Thread.java:662)
  1416.  
  1417. "RMI TCP Accept-9021" daemon prio=10 tid=0x00002aaac065a800 nid=0x6b82 runnable [0x00000000409f3000]
  1418.    java.lang.Thread.State: RUNNABLE
  1419.         at java.net.PlainSocketImpl.socketAccept(Native Method)
  1420.         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
  1421.         - locked <0x000000058b30c3e0> (a java.net.SocksSocketImpl)
  1422.         at java.net.ServerSocket.implAccept(ServerSocket.java:462)
  1423.         at java.net.ServerSocket.accept(ServerSocket.java:430)
  1424.         at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
  1425.         at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
  1426.         at java.lang.Thread.run(Thread.java:662)
  1427.  
  1428. "RMI TCP Accept-0" daemon prio=10 tid=0x00002aaac0648000 nid=0x6b81 runnable [0x00000000408f2000]
  1429.    java.lang.Thread.State: RUNNABLE
  1430.         at java.net.PlainSocketImpl.socketAccept(Native Method)
  1431.         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
  1432.         - locked <0x000000058b30ca70> (a java.net.SocksSocketImpl)
  1433.         at java.net.ServerSocket.implAccept(ServerSocket.java:462)
  1434.         at java.net.ServerSocket.accept(ServerSocket.java:430)
  1435.         at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
  1436.         at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
  1437.         at java.lang.Thread.run(Thread.java:662)
  1438.  
  1439. "Low Memory Detector" daemon prio=10 tid=0x00002aaac0026800 nid=0x6b7d runnable [0x0000000000000000]
  1440.    java.lang.Thread.State: RUNNABLE
  1441.  
  1442. "C2 CompilerThread1" daemon prio=10 tid=0x00002aaac0024000 nid=0x6b7c waiting on condition [0x0000000000000000]
  1443.    java.lang.Thread.State: RUNNABLE
  1444.  
  1445. "C2 CompilerThread0" daemon prio=10 tid=0x00002aaac0022000 nid=0x6b7b waiting on condition [0x0000000000000000]
  1446.    java.lang.Thread.State: RUNNABLE
  1447.  
  1448. "Signal Dispatcher" daemon prio=10 tid=0x00002aaac001f800 nid=0x6b7a waiting on condition [0x0000000000000000]
  1449.    java.lang.Thread.State: RUNNABLE
  1450.  
  1451. "Surrogate Locker Thread (Concurrent GC)" daemon prio=10 tid=0x00002aaac001d800 nid=0x6b79 waiting on condition [0x0000000000000000]
  1452.    java.lang.Thread.State: RUNNABLE
  1453.  
  1454. "Finalizer" daemon prio=10 tid=0x00002aaac0001000 nid=0x6b78 in Object.wait() [0x000000004171c000]
  1455.    java.lang.Thread.State: WAITING (on object monitor)
  1456.         at java.lang.Object.wait(Native Method)
  1457.         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
  1458.         - locked <0x000000058b300160> (a java.lang.ref.ReferenceQueue$Lock)
  1459.         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
  1460.         at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
  1461.  
  1462. "Reference Handler" daemon prio=10 tid=0x00000000527e3000 nid=0x6b77 in Object.wait() [0x000000004161b000]
  1463.    java.lang.Thread.State: WAITING (on object monitor)
  1464.         at java.lang.Object.wait(Native Method)
  1465.         at java.lang.Object.wait(Object.java:485)
  1466.         at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
  1467.         - locked <0x000000058b303988> (a java.lang.ref.Reference$Lock)
  1468.  
  1469. "VM Thread" prio=10 tid=0x00000000527dc800 nid=0x6b76 runnable
  1470.  
  1471. "Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x00000000525ac800 nid=0x6b6b runnable
  1472.  
  1473. "Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x00000000525ae800 nid=0x6b6c runnable
  1474.  
  1475. "Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x00000000525b0800 nid=0x6b6d runnable
  1476.  
  1477. "Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x00000000525b2000 nid=0x6b6e runnable
  1478.  
  1479. "Gang worker#4 (Parallel GC Threads)" prio=10 tid=0x00000000525b4000 nid=0x6b6f runnable
  1480.  
  1481. "Gang worker#5 (Parallel GC Threads)" prio=10 tid=0x00000000525b6000 nid=0x6b70 runnable
  1482.  
  1483. "Gang worker#6 (Parallel GC Threads)" prio=10 tid=0x00000000525b7800 nid=0x6b71 runnable
  1484.  
  1485. "Gang worker#7 (Parallel GC Threads)" prio=10 tid=0x00000000525b9800 nid=0x6b72 runnable
  1486.  
  1487. "Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00000000526ad800 nid=0x6b75 runnable
  1488. "Gang worker#0 (Parallel CMS Threads)" prio=10 tid=0x00000000526a9800 nid=0x6b73 runnable
  1489.  
  1490. "Gang worker#1 (Parallel CMS Threads)" prio=10 tid=0x00000000526ab000 nid=0x6b74 runnable
  1491.  
  1492. "VM Periodic Task Thread" prio=10 tid=0x00002aaac0661800 nid=0x6b84 waiting on condition
  1493.  
  1494. JNI global references: 1248
  1495.  
  1496. Heap
  1497.  par new generation   total 235968K, used 176200K [0x000000057ae00000, 0x000000058ae00000, 0x000000058ae00000)
  1498.   eden space 209792K,  72% used [0x000000057ae00000, 0x0000000584214c88, 0x0000000587ae0000)
  1499.   from space 26176K,  93% used [0x0000000587ae0000, 0x00000005892dd6c8, 0x0000000589470000)
  1500.   to   space 26176K,   0% used [0x0000000589470000, 0x0000000589470000, 0x000000058ae00000)
  1501.  concurrent mark-sweep generation total 10223616K, used 4535508K [0x000000058ae00000, 0x00000007fae00000, 0x00000007fae00000)
  1502.  concurrent-mark-sweep perm gen total 49196K, used 29423K [0x00000007fae00000, 0x00000007fde0b000, 0x0000000800000000)
  1503.  
  1504. -PROD- prodcon@wchddb37pxdu:/prod/hadoop/prod/hbase-0.92.1/logs$
clone this paste RAW Paste Data