API tools faq
paste
Advertisement
ronbuckley

HBase Region Server Stack Trace

ronbuckley
Aug 6th, 2012
151
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 110.42 KB | None | 0 0
raw download clone embed print report
  1.  
  2. -PROD- prodcon@wchddb37pxdu:/prod/hadoop/prod/hbase-0.92.1/logs$cat hbase-prodcon-regionserver-wchddb37pxdu.prod.oclc.org.out
  3. SLF4J: Class path contains multiple SLF4J bindings.
  4. SLF4J: Found binding in [jar:file:/drive1/hadoop/20120522/hbase-0.92.1/lib/slf4j-log4j12-1.5.8.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  5. SLF4J: Found binding in [jar:file:/drive1/hadoop/20120522/hadoop-0.20.2-cdh3u3/lib/slf4j-log4j12-1.4.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  6. SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
  7. 2012-08-06 11:46:46
  8. Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.0-b11 mixed mode):
  9.  
  10. "IPC Client (47) connection to /10.192.98.52:37073 from blk_81630121065287353_2238161" daemon prio=10 tid=0x00002aaacc7b7000 nid=0x1c8f in Object.wait() [0x000000004e0ef000]
  11. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  12. at java.lang.Object.wait(Native Method)
  13. at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:680)
  14. - locked <0x00000005840a63a0> (a org.apache.hadoop.ipc.Client$Connection)
  15. at org.apache.hadoop.ipc.Client$Connection.run(Client.java:723)
  16.  
  17. "IPC Client (47) connection to wchddb01pxdu.prod.oclc.org/10.192.98.16:9008 from prodcon" daemon prio=10 tid=0x00002aaacc3a5000 nid=0x1c8e in Object.wait() [0x000000004e3f2000]
  18. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  19. at java.lang.Object.wait(Native Method)
  20. at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:680)
  21. - locked <0x0000000583b67d70> (a org.apache.hadoop.ipc.Client$Connection)
  22. at org.apache.hadoop.ipc.Client$Connection.run(Client.java:723)
  23.  
  24. "1772284907@qtp-1068922666-14" prio=10 tid=0x00002aaad8488000 nid=0x1b27 in Object.wait() [0x000000004e5f4000]
  25. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  26. at java.lang.Object.wait(Native Method)
  27. at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
  28. - locked <0x00000005835d94a0> (a org.mortbay.thread.QueuedThreadPool$PoolThread)
  29.  
  30. "RMI TCP Connection(idle)" daemon prio=10 tid=0x00002aaad8490000 nid=0x1985 waiting on condition [0x000000004f503000]
  31. java.lang.Thread.State: TIMED_WAITING (parking)
  32. at sun.misc.Unsafe.park(Native Method)
  33. - parking to wait for <0x000000058b4d4548> (a java.util.concurrent.SynchronousQueue$TransferStack)
  34. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
  35. at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
  36. at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
  37. at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
  38. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
  39. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  40. at java.lang.Thread.run(Thread.java:662)
  41.  
  42. "915001464@qtp-1068922666-13" prio=10 tid=0x00002aaad8287800 nid=0x1949 in Object.wait() [0x000000004e2f1000]
  43. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  44. at java.lang.Object.wait(Native Method)
  45. at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
  46. - locked <0x000000058057a0f0> (a org.mortbay.thread.QueuedThreadPool$PoolThread)
  47.  
  48. "ResponseProcessor for block blk_81630121065287353_2238161" daemon prio=10 tid=0x00002aaabfe1c800 nid=0x13bb runnable [0x000000004e1f0000]
  49. java.lang.Thread.State: RUNNABLE
  50. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  51. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  52. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  53. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  54. - locked <0x0000000587f20758> (a sun.nio.ch.Util$2)
  55. - locked <0x0000000587f20748> (a java.util.Collections$UnmodifiableSet)
  56. - locked <0x0000000587f206b0> (a sun.nio.ch.EPollSelectorImpl)
  57. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  58. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:332)
  59. at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
  60. at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
  61. at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
  62. at java.io.DataInputStream.readFully(DataInputStream.java:178)
  63. at java.io.DataInputStream.readLong(DataInputStream.java:399)
  64. at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:120)
  65. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2956)
  66.  
  67. "DataStreamer for file /hbase/.logs/wchddb37pxdu.prod.oclc.org,9009,1337721421002/wchddb37pxdu.prod.oclc.org%2C9009%2C1337721421002.1344267769705 block blk_81630121065287353_2238161" daemon prio=10 tid=0x00002aaad81ab000 nid=0x13b4 in Object.wait() [0x000000004f200000]
  68. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  69. at java.lang.Object.wait(Native Method)
  70. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2805)
  71. - locked <0x00000006e8c6b168> (a java.util.LinkedList)
  72.  
  73. "sendParams-166" daemon prio=10 tid=0x00002aaacca57000 nid=0x2b0d waiting on condition [0x0000000066358000]
  74. java.lang.Thread.State: TIMED_WAITING (parking)
  75. at sun.misc.Unsafe.park(Native Method)
  76. - parking to wait for <0x000000058b300eb0> (a java.util.concurrent.SynchronousQueue$TransferStack)
  77. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
  78. at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
  79. at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
  80. at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
  81. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
  82. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  83. at java.lang.Thread.run(Thread.java:662)
  84.  
  85. "ReplicationExecutor-0" prio=10 tid=0x00002aaaccbd6000 nid=0x5e94 waiting on condition [0x0000000058a80000]
  86. java.lang.Thread.State: WAITING (parking)
  87. at sun.misc.Unsafe.park(Native Method)
  88. - parking to wait for <0x000000058b89f790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  89. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  90. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  91. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  92. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
  93. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  94. at java.lang.Thread.run(Thread.java:662)
  95.  
  96. "IPC Client (47) connection to wchddb02pxdu.prod.oclc.org/10.192.98.17:9010 from prodcon" daemon prio=10 tid=0x00002aaaccc76800 nid=0x59df in Object.wait() [0x000000007ff3f000]
  97. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  98. at java.lang.Object.wait(Native Method)
  99. at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.waitForWork(HBaseClient.java:459)
  100. - locked <0x000000058fbd02e0> (a org.apache.hadoop.hbase.ipc.HBaseClient$Connection)
  101. at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:504)
  102.  
  103. "RS_CLOSE_REGION-wchddb37pxdu.prod.oclc.org,9009,1337721421002-2" prio=10 tid=0x00002aaabf26d800 nid=0x3dd2 waiting on condition [0x00000000783c4000]
  104. java.lang.Thread.State: WAITING (parking)
  105. at sun.misc.Unsafe.park(Native Method)
  106. - parking to wait for <0x000000058b393590> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  107. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  108. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  109. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  110. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
  111. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  112. at java.lang.Thread.run(Thread.java:662)
  113.  
  114. "RS_CLOSE_REGION-wchddb37pxdu.prod.oclc.org,9009,1337721421002-1" prio=10 tid=0x0000000052de3800 nid=0x3dd1 waiting on condition [0x00000000782c3000]
  115. java.lang.Thread.State: WAITING (parking)
  116. at sun.misc.Unsafe.park(Native Method)
  117. - parking to wait for <0x000000058b393590> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  118. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  119. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  120. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  121. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
  122. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  123. at java.lang.Thread.run(Thread.java:662)
  124.  
  125. "RS_CLOSE_REGION-wchddb37pxdu.prod.oclc.org,9009,1337721421002-0" prio=10 tid=0x0000000052e1e000 nid=0x3dcc waiting on condition [0x0000000077abb000]
  126. java.lang.Thread.State: WAITING (parking)
  127. at sun.misc.Unsafe.park(Native Method)
  128. - parking to wait for <0x000000058b393590> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  129. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  130. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  131. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  132. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
  133. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  134. at java.lang.Thread.run(Thread.java:662)
  135.  
  136. "regionserver9009-splits-1337730044584" daemon prio=10 tid=0x0000000052868000 nid=0x1e64 waiting on condition [0x0000000056f65000]
  137. java.lang.Thread.State: WAITING (parking)
  138. at sun.misc.Unsafe.park(Native Method)
  139. - parking to wait for <0x000000058b8a3088> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  140. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  141. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  142. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  143. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
  144. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  145. at java.lang.Thread.run(Thread.java:662)
  146.  
  147. "regionserver9009-largeCompactions-1337726358700" daemon prio=10 tid=0x00002aaac0b55800 nid=0x973 waiting on condition [0x00000000777b8000]
  148. java.lang.Thread.State: WAITING (parking)
  149. at sun.misc.Unsafe.park(Native Method)
  150. - parking to wait for <0x000000058b8a16d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  151. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  152. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  153. at java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:220)
  154. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
  155. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  156. at java.lang.Thread.run(Thread.java:662)
  157.  
  158. "regionserver9009-EventThread.replicationSource,1" daemon prio=10 tid=0x00002aaacc980000 nid=0xcca waiting on condition [0x0000000043544000]
  159. java.lang.Thread.State: TIMED_WAITING (sleeping)
  160. at java.lang.Thread.sleep(Native Method)
  161. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.sleepForRetries(ReplicationSource.java:550)
  162. at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:362)
  163.  
  164. "regionserver9009-EventThread" daemon prio=10 tid=0x0000000052a50800 nid=0xcc9 waiting on condition [0x000000004909f000]
  165. java.lang.Thread.State: WAITING (parking)
  166. at sun.misc.Unsafe.park(Native Method)
  167. - parking to wait for <0x000000058caaa060> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  168. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  169. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  170. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  171. at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:493)
  172.  
  173. "regionserver9009-SendThread(wchddb02pxwe.prod.oclc.org:9011)" daemon prio=10 tid=0x0000000052946800 nid=0xcc8 runnable [0x0000000040cf6000]
  174. java.lang.Thread.State: RUNNABLE
  175. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  176. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  177. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  178. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  179. - locked <0x000000058caaa000> (a sun.nio.ch.Util$2)
  180. - locked <0x000000058caaa018> (a java.util.Collections$UnmodifiableSet)
  181. - locked <0x000000058cab5cf8> (a sun.nio.ch.EPollSelectorImpl)
  182. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  183. at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:274)
  184. at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
  185.  
  186. "regionserver9009-smallCompactions-1337721650410" daemon prio=10 tid=0x0000000052866000 nid=0x130 waiting on condition [0x0000000049baa000]
  187. java.lang.Thread.State: WAITING (parking)
  188. at sun.misc.Unsafe.park(Native Method)
  189. - parking to wait for <0x000000058b8a1618> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  190. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  191. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  192. at java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:220)
  193. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
  194. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  195. at java.lang.Thread.run(Thread.java:662)
  196.  
  197. "RS_OPEN_REGION-wchddb37pxdu.prod.oclc.org,9009,1337721421002-2" prio=10 tid=0x00000000528f8000 nid=0x7fdd waiting on condition [0x00000000499a8000]
  198. java.lang.Thread.State: WAITING (parking)
  199. at sun.misc.Unsafe.park(Native Method)
  200. - parking to wait for <0x000000058b3938d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  201. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  202. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  203. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  204. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
  205. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  206. at java.lang.Thread.run(Thread.java:662)
  207.  
  208. "RS_OPEN_REGION-wchddb37pxdu.prod.oclc.org,9009,1337721421002-1" prio=10 tid=0x0000000053148000 nid=0x7fdc waiting on condition [0x000000004a5b4000]
  209. java.lang.Thread.State: WAITING (parking)
  210. at sun.misc.Unsafe.park(Native Method)
  211. - parking to wait for <0x000000058b3938d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  212. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  213. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  214. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  215. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
  216. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  217. at java.lang.Thread.run(Thread.java:662)
  218.  
  219. "RS_OPEN_REGION-wchddb37pxdu.prod.oclc.org,9009,1337721421002-0" prio=10 tid=0x0000000052e10000 nid=0x7fdb waiting on condition [0x00000000496a5000]
  220. java.lang.Thread.State: WAITING (parking)
  221. at sun.misc.Unsafe.park(Native Method)
  222. - parking to wait for <0x000000058b3938d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  223. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  224. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  225. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  226. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
  227. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  228. at java.lang.Thread.run(Thread.java:662)
  229.  
  230. "RMI Scheduler(0)" daemon prio=10 tid=0x0000000053126800 nid=0x7e4a waiting on condition [0x0000000049ead000]
  231. java.lang.Thread.State: TIMED_WAITING (parking)
  232. at sun.misc.Unsafe.park(Native Method)
  233. - parking to wait for <0x000000058b7f1150> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  234. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
  235. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
  236. at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
  237. at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
  238. at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
  239. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
  240. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  241. at java.lang.Thread.run(Thread.java:662)
  242.  
  243. "SplitLogWorker-wchddb37pxdu.prod.oclc.org,9009,1337721421002" prio=10 tid=0x00002aaac0a59800 nid=0x6d83 in Object.wait() [0x000000004808f000]
  244. java.lang.Thread.State: WAITING (on object monitor)
  245. at java.lang.Object.wait(Native Method)
  246. at java.lang.Object.wait(Object.java:485)
  247. at org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:205)
  248. - locked <0x000000058c87ada0> (a java.lang.Object)
  249. at org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:165)
  250. at java.lang.Thread.run(Thread.java:662)
  251.  
  252. "PRI IPC Server handler 9 on 9009" daemon prio=10 tid=0x00002aaac0a57800 nid=0x6d82 waiting on condition [0x0000000047f8e000]
  253. java.lang.Thread.State: WAITING (parking)
  254. at sun.misc.Unsafe.park(Native Method)
  255. - parking to wait for <0x000000058b300df0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  256. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  257. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  258. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  259. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1299)
  260.  
  261. "PRI IPC Server handler 8 on 9009" daemon prio=10 tid=0x00002aaac0a55800 nid=0x6d81 waiting on condition [0x0000000047e8d000]
  262. java.lang.Thread.State: WAITING (parking)
  263. at sun.misc.Unsafe.park(Native Method)
  264. - parking to wait for <0x000000058b300df0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  265. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  266. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  267. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  268. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1299)
  269.  
  270. "PRI IPC Server handler 7 on 9009" daemon prio=10 tid=0x00002aaac0a53800 nid=0x6d80 waiting on condition [0x0000000047d8c000]
  271. java.lang.Thread.State: WAITING (parking)
  272. at sun.misc.Unsafe.park(Native Method)
  273. - parking to wait for <0x000000058b300df0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  274. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  275. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  276. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  277. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1299)
  278.  
  279. "PRI IPC Server handler 6 on 9009" daemon prio=10 tid=0x00002aaac0730000 nid=0x6d7f waiting on condition [0x0000000047c8b000]
  280. java.lang.Thread.State: WAITING (parking)
  281. at sun.misc.Unsafe.park(Native Method)
  282. - parking to wait for <0x000000058b300df0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  283. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  284. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  285. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  286. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1299)
  287.  
  288. "PRI IPC Server handler 5 on 9009" daemon prio=10 tid=0x00002aaac072e000 nid=0x6d7e waiting on condition [0x0000000047b8a000]
  289. java.lang.Thread.State: WAITING (parking)
  290. at sun.misc.Unsafe.park(Native Method)
  291. - parking to wait for <0x000000058b300df0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  292. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  293. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  294. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  295. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1299)
  296.  
  297. "PRI IPC Server handler 4 on 9009" daemon prio=10 tid=0x00002aaac072c000 nid=0x6d7d waiting on condition [0x0000000047a89000]
  298. java.lang.Thread.State: WAITING (parking)
  299. at sun.misc.Unsafe.park(Native Method)
  300. - parking to wait for <0x000000058b300df0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  301. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  302. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  303. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  304. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1299)
  305.  
  306. "PRI IPC Server handler 3 on 9009" daemon prio=10 tid=0x00002aaac072a000 nid=0x6d7c waiting on condition [0x0000000047988000]
  307. java.lang.Thread.State: WAITING (parking)
  308. at sun.misc.Unsafe.park(Native Method)
  309. - parking to wait for <0x000000058b300df0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  310. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  311. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  312. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  313. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1299)
  314.  
  315. "PRI IPC Server handler 2 on 9009" daemon prio=10 tid=0x00002aaac0728000 nid=0x6d7b waiting on condition [0x0000000047887000]
  316. java.lang.Thread.State: WAITING (parking)
  317. at sun.misc.Unsafe.park(Native Method)
  318. - parking to wait for <0x000000058b300df0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  319. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  320. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  321. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  322. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1299)
  323.  
  324. "PRI IPC Server handler 1 on 9009" daemon prio=10 tid=0x00002aaac0726000 nid=0x6d7a waiting on condition [0x0000000047786000]
  325. java.lang.Thread.State: WAITING (parking)
  326. at sun.misc.Unsafe.park(Native Method)
  327. - parking to wait for <0x000000058b300df0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  328. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  329. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  330. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  331. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1299)
  332.  
  333. "PRI IPC Server handler 0 on 9009" daemon prio=10 tid=0x00002aaac0724000 nid=0x6d79 waiting on condition [0x0000000047685000]
  334. java.lang.Thread.State: WAITING (parking)
  335. at sun.misc.Unsafe.park(Native Method)
  336. - parking to wait for <0x000000058b300df0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  337. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  338. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  339. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  340. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1299)
  341.  
  342. "IPC Server handler 49 on 9009" daemon prio=10 tid=0x00002aaac0722800 nid=0x6d78 in Object.wait() [0x0000000047584000]
  343. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  344. at java.lang.Object.wait(Native Method)
  345. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  346. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  347. at org.apache.hadoop.hbase.regionserver.HRegion.delete(HRegion.java:1469)
  348. at org.apache.hadoop.hbase.regionserver.HRegionServer.delete(HRegionServer.java:2260)
  349. at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
  350. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  351. at java.lang.reflect.Method.invoke(Method.java:597)
  352. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  353. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  354.  
  355. "IPC Server handler 48 on 9009" daemon prio=10 tid=0x00002aaac0047000 nid=0x6d77 in Object.wait() [0x0000000047483000]
  356. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  357. at java.lang.Object.wait(Native Method)
  358. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  359. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  360. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  361. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  362. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  363. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  364. at java.lang.reflect.Method.invoke(Method.java:597)
  365. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  366. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  367.  
  368. "IPC Server handler 47 on 9009" daemon prio=10 tid=0x00002aaac0045000 nid=0x6d76 in Object.wait() [0x0000000047382000]
  369. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  370. at java.lang.Object.wait(Native Method)
  371. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  372. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  373. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  374. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  375. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  376. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  377. at java.lang.reflect.Method.invoke(Method.java:597)
  378. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  379. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  380.  
  381. "IPC Server handler 46 on 9009" daemon prio=10 tid=0x00002aaac0043800 nid=0x6d75 in Object.wait() [0x0000000047281000]
  382. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  383. at java.lang.Object.wait(Native Method)
  384. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  385. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  386. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  387. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  388. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  389. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  390. at java.lang.reflect.Method.invoke(Method.java:597)
  391. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  392. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  393.  
  394. "IPC Server handler 45 on 9009" daemon prio=10 tid=0x00002aaac0041800 nid=0x6d74 in Object.wait() [0x0000000047180000]
  395. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  396. at java.lang.Object.wait(Native Method)
  397. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  398. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  399. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  400. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  401. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  402. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  403. at java.lang.reflect.Method.invoke(Method.java:597)
  404. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  405. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  406.  
  407. "IPC Server handler 44 on 9009" daemon prio=10 tid=0x00002aaac003f800 nid=0x6d73 in Object.wait() [0x000000004707f000]
  408. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  409. at java.lang.Object.wait(Native Method)
  410. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  411. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  412. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  413. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  414. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  415. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  416. at java.lang.reflect.Method.invoke(Method.java:597)
  417. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  418. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  419.  
  420. "IPC Server handler 43 on 9009" daemon prio=10 tid=0x00002aaac003d800 nid=0x6d72 in Object.wait() [0x0000000046f7e000]
  421. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  422. at java.lang.Object.wait(Native Method)
  423. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  424. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  425. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  426. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  427. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  428. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  429. at java.lang.reflect.Method.invoke(Method.java:597)
  430. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  431. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  432.  
  433. "IPC Server handler 42 on 9009" daemon prio=10 tid=0x00002aaac003b800 nid=0x6d71 in Object.wait() [0x0000000046e7d000]
  434. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  435. at java.lang.Object.wait(Native Method)
  436. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  437. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  438. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  439. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  440. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  441. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  442. at java.lang.reflect.Method.invoke(Method.java:597)
  443. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  444. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  445.  
  446. "IPC Server handler 41 on 9009" daemon prio=10 tid=0x00002aaac0039800 nid=0x6d70 in Object.wait() [0x0000000046d7c000]
  447. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  448. at java.lang.Object.wait(Native Method)
  449. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  450. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  451. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  452. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  453. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  454. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  455. at java.lang.reflect.Method.invoke(Method.java:597)
  456. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  457. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  458.  
  459. "IPC Server handler 40 on 9009" daemon prio=10 tid=0x00002aaac0a00800 nid=0x6d6f in Object.wait() [0x0000000046c7b000]
  460. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  461. at java.lang.Object.wait(Native Method)
  462. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  463. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  464. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  465. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  466. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  467. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  468. at java.lang.reflect.Method.invoke(Method.java:597)
  469. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  470. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  471.  
  472. "IPC Server handler 39 on 9009" daemon prio=10 tid=0x00002aaac09fe800 nid=0x6d6e in Object.wait() [0x0000000046b7a000]
  473. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  474. at java.lang.Object.wait(Native Method)
  475. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  476. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  477. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  478. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  479. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  480. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  481. at java.lang.reflect.Method.invoke(Method.java:597)
  482. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  483. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  484.  
  485. "IPC Server handler 38 on 9009" daemon prio=10 tid=0x00002aaac09fc800 nid=0x6d6d in Object.wait() [0x0000000046a79000]
  486. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  487. at java.lang.Object.wait(Native Method)
  488. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  489. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  490. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  491. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  492. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  493. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  494. at java.lang.reflect.Method.invoke(Method.java:597)
  495. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  496. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  497.  
  498. "IPC Server handler 37 on 9009" daemon prio=10 tid=0x00002aaac09fa800 nid=0x6d6c in Object.wait() [0x0000000046978000]
  499. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  500. at java.lang.Object.wait(Native Method)
  501. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  502. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  503. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  504. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  505. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  506. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  507. at java.lang.reflect.Method.invoke(Method.java:597)
  508. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  509. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  510.  
  511. "IPC Server handler 36 on 9009" daemon prio=10 tid=0x00002aaac09f8800 nid=0x6d6b in Object.wait() [0x0000000046877000]
  512. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  513. at java.lang.Object.wait(Native Method)
  514. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  515. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  516. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  517. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  518. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  519. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  520. at java.lang.reflect.Method.invoke(Method.java:597)
  521. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  522. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  523.  
  524. "IPC Server handler 35 on 9009" daemon prio=10 tid=0x00002aaac09f6800 nid=0x6d6a in Object.wait() [0x0000000046776000]
  525. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  526. at java.lang.Object.wait(Native Method)
  527. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  528. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  529. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  530. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  531. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  532. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  533. at java.lang.reflect.Method.invoke(Method.java:597)
  534. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  535. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  536.  
  537. "IPC Server handler 34 on 9009" daemon prio=10 tid=0x00002aaac09f4800 nid=0x6d69 in Object.wait() [0x0000000046675000]
  538. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  539. at java.lang.Object.wait(Native Method)
  540. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  541. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  542. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  543. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  544. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  545. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  546. at java.lang.reflect.Method.invoke(Method.java:597)
  547. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  548. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  549.  
  550. "IPC Server handler 33 on 9009" daemon prio=10 tid=0x00002aaac09f2800 nid=0x6d68 in Object.wait() [0x0000000046574000]
  551. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  552. at java.lang.Object.wait(Native Method)
  553. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  554. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  555. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  556. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  557. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  558. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  559. at java.lang.reflect.Method.invoke(Method.java:597)
  560. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  561. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  562.  
  563. "IPC Server handler 32 on 9009" daemon prio=10 tid=0x00002aaac08a9800 nid=0x6d67 in Object.wait() [0x0000000046473000]
  564. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  565. at java.lang.Object.wait(Native Method)
  566. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  567. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  568. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  569. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  570. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  571. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  572. at java.lang.reflect.Method.invoke(Method.java:597)
  573. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  574. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  575.  
  576. "IPC Server handler 31 on 9009" daemon prio=10 tid=0x00002aaac08a7800 nid=0x6d66 in Object.wait() [0x0000000046372000]
  577. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  578. at java.lang.Object.wait(Native Method)
  579. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  580. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  581. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  582. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  583. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  584. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  585. at java.lang.reflect.Method.invoke(Method.java:597)
  586. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  587. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  588.  
  589. "IPC Server handler 30 on 9009" daemon prio=10 tid=0x00002aaac08a5800 nid=0x6d65 in Object.wait() [0x0000000046271000]
  590. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  591. at java.lang.Object.wait(Native Method)
  592. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  593. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  594. at org.apache.hadoop.hbase.regionserver.HRegion.delete(HRegion.java:1469)
  595. at org.apache.hadoop.hbase.regionserver.HRegionServer.delete(HRegionServer.java:2260)
  596. at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
  597. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  598. at java.lang.reflect.Method.invoke(Method.java:597)
  599. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  600. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  601.  
  602. "IPC Server handler 29 on 9009" daemon prio=10 tid=0x00002aaac08a3800 nid=0x6d64 runnable [0x0000000046170000]
  603. java.lang.Thread.State: RUNNABLE
  604. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:691)
  605. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.seekInSubLists(MemStore.java:726)
  606. - locked <0x000000058d939c38> (a org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner)
  607. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:765)
  608. - locked <0x000000058d939c38> (a org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner)
  609. at org.apache.hadoop.hbase.regionserver.KeyValueHeap$SeekType$1.seek(KeyValueHeap.java:57)
  610. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:277)
  611. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:248)
  612. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:436)
  613. - locked <0x000000058fd24658> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  614. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:322)
  615. - locked <0x000000058fd24658> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  616. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:138)
  617. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:2945)
  618. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:2901)
  619. - locked <0x000000058d939c70> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl)
  620. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:2918)
  621. - locked <0x000000058d939c70> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl)
  622. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:3693)
  623. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:3585)
  624. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1785)
  625. at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source)
  626. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  627. at java.lang.reflect.Method.invoke(Method.java:597)
  628. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  629. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  630.  
  631. "IPC Server handler 28 on 9009" daemon prio=10 tid=0x00002aaac08a1800 nid=0x6d63 in Object.wait() [0x000000004606f000]
  632. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  633. at java.lang.Object.wait(Native Method)
  634. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  635. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  636. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  637. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  638. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  639. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  640. at java.lang.reflect.Method.invoke(Method.java:597)
  641. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  642. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  643.  
  644. "IPC Server handler 27 on 9009" daemon prio=10 tid=0x00002aaac08a0000 nid=0x6d62 in Object.wait() [0x0000000045f6e000]
  645. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  646. at java.lang.Object.wait(Native Method)
  647. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  648. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  649. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  650. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  651. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  652. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  653. at java.lang.reflect.Method.invoke(Method.java:597)
  654. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  655. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  656.  
  657. "IPC Server handler 26 on 9009" daemon prio=10 tid=0x00002aaac089e000 nid=0x6d61 in Object.wait() [0x0000000045e6d000]
  658. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  659. at java.lang.Object.wait(Native Method)
  660. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  661. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  662. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  663. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  664. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  665. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  666. at java.lang.reflect.Method.invoke(Method.java:597)
  667. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  668. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  669.  
  670. "IPC Server handler 25 on 9009" daemon prio=10 tid=0x00002aaac0a7f800 nid=0x6d60 in Object.wait() [0x0000000045d6c000]
  671. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  672. at java.lang.Object.wait(Native Method)
  673. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  674. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  675. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  676. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  677. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  678. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  679. at java.lang.reflect.Method.invoke(Method.java:597)
  680. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  681. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  682.  
  683. "IPC Server handler 24 on 9009" daemon prio=10 tid=0x00002aaac0a7d800 nid=0x6d5f in Object.wait() [0x0000000045c6b000]
  684. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  685. at java.lang.Object.wait(Native Method)
  686. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  687. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  688. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  689. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  690. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  691. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  692. at java.lang.reflect.Method.invoke(Method.java:597)
  693. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  694. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  695.  
  696. "IPC Server handler 23 on 9009" daemon prio=10 tid=0x00002aaac0a7b800 nid=0x6d5e in Object.wait() [0x0000000045b6a000]
  697. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  698. at java.lang.Object.wait(Native Method)
  699. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  700. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  701. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  702. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  703. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  704. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  705. at java.lang.reflect.Method.invoke(Method.java:597)
  706. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  707. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  708.  
  709. "IPC Server handler 22 on 9009" daemon prio=10 tid=0x00002aaac055f000 nid=0x6d5d in Object.wait() [0x0000000045a69000]
  710. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  711. at java.lang.Object.wait(Native Method)
  712. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  713. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  714. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  715. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  716. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  717. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  718. at java.lang.reflect.Method.invoke(Method.java:597)
  719. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  720. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  721.  
  722. "IPC Server handler 21 on 9009" daemon prio=10 tid=0x00002aaac07f7800 nid=0x6d5c in Object.wait() [0x0000000045968000]
  723. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  724. at java.lang.Object.wait(Native Method)
  725. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  726. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  727. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  728. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  729. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  730. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  731. at java.lang.reflect.Method.invoke(Method.java:597)
  732. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  733. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  734.  
  735. "IPC Server handler 20 on 9009" daemon prio=10 tid=0x00002aaacc236800 nid=0x6d5b in Object.wait() [0x0000000045867000]
  736. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  737. at java.lang.Object.wait(Native Method)
  738. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  739. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  740. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  741. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  742. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  743. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  744. at java.lang.reflect.Method.invoke(Method.java:597)
  745. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  746. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  747.  
  748. "IPC Server handler 19 on 9009" daemon prio=10 tid=0x0000000052fc7800 nid=0x6d5a in Object.wait() [0x0000000045766000]
  749. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  750. at java.lang.Object.wait(Native Method)
  751. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  752. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  753. at org.apache.hadoop.hbase.regionserver.HRegion.delete(HRegion.java:1469)
  754. at org.apache.hadoop.hbase.regionserver.HRegionServer.delete(HRegionServer.java:2260)
  755. at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
  756. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  757. at java.lang.reflect.Method.invoke(Method.java:597)
  758. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  759. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  760.  
  761. "IPC Server handler 18 on 9009" daemon prio=10 tid=0x0000000052c84000 nid=0x6d59 in Object.wait() [0x0000000045665000]
  762. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  763. at java.lang.Object.wait(Native Method)
  764. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  765. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  766. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  767. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  768. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  769. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  770. at java.lang.reflect.Method.invoke(Method.java:597)
  771. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  772. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  773.  
  774. "IPC Server handler 17 on 9009" daemon prio=10 tid=0x0000000052c82000 nid=0x6d58 in Object.wait() [0x0000000045564000]
  775. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  776. at java.lang.Object.wait(Native Method)
  777. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  778. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  779. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  780. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  781. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  782. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  783. at java.lang.reflect.Method.invoke(Method.java:597)
  784. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  785. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  786.  
  787. "IPC Server handler 16 on 9009" daemon prio=10 tid=0x0000000052c7f800 nid=0x6d57 in Object.wait() [0x0000000045463000]
  788. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  789. at java.lang.Object.wait(Native Method)
  790. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  791. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  792. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  793. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  794. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  795. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  796. at java.lang.reflect.Method.invoke(Method.java:597)
  797. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  798. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  799.  
  800. "IPC Server handler 15 on 9009" daemon prio=10 tid=0x0000000052a03800 nid=0x6d56 in Object.wait() [0x0000000045362000]
  801. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  802. at java.lang.Object.wait(Native Method)
  803. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  804. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  805. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  806. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  807. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  808. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  809. at java.lang.reflect.Method.invoke(Method.java:597)
  810. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  811. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  812.  
  813. "IPC Server handler 14 on 9009" daemon prio=10 tid=0x0000000052a01000 nid=0x6d55 in Object.wait() [0x0000000045261000]
  814. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  815. at java.lang.Object.wait(Native Method)
  816. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  817. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  818. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  819. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  820. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  821. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  822. at java.lang.reflect.Method.invoke(Method.java:597)
  823. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  824. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  825.  
  826. "IPC Server handler 13 on 9009" daemon prio=10 tid=0x00000000529ff000 nid=0x6d54 in Object.wait() [0x0000000045160000]
  827. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  828. at java.lang.Object.wait(Native Method)
  829. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  830. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  831. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  832. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  833. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  834. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  835. at java.lang.reflect.Method.invoke(Method.java:597)
  836. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  837. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  838.  
  839. "IPC Server handler 12 on 9009" daemon prio=10 tid=0x0000000052c65800 nid=0x6d53 in Object.wait() [0x000000004505f000]
  840. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  841. at java.lang.Object.wait(Native Method)
  842. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  843. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  844. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  845. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  846. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  847. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  848. at java.lang.reflect.Method.invoke(Method.java:597)
  849. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  850. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  851.  
  852. "IPC Server handler 11 on 9009" daemon prio=10 tid=0x0000000052c64000 nid=0x6d52 in Object.wait() [0x0000000044f5e000]
  853. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  854. at java.lang.Object.wait(Native Method)
  855. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  856. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  857. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  858. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  859. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  860. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  861. at java.lang.reflect.Method.invoke(Method.java:597)
  862. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  863. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  864.  
  865. "IPC Server handler 10 on 9009" daemon prio=10 tid=0x0000000053170800 nid=0x6d51 in Object.wait() [0x0000000044e5d000]
  866. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  867. at java.lang.Object.wait(Native Method)
  868. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  869. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  870. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  871. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  872. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  873. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  874. at java.lang.reflect.Method.invoke(Method.java:597)
  875. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  876. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  877.  
  878. "IPC Server handler 9 on 9009" daemon prio=10 tid=0x0000000052fd7000 nid=0x6d50 in Object.wait() [0x0000000044d5c000]
  879. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  880. at java.lang.Object.wait(Native Method)
  881. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  882. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  883. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  884. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  885. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  886. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  887. at java.lang.reflect.Method.invoke(Method.java:597)
  888. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  889. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  890.  
  891. "IPC Server handler 8 on 9009" daemon prio=10 tid=0x0000000052e1f000 nid=0x6d4f in Object.wait() [0x0000000044c5b000]
  892. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  893. at java.lang.Object.wait(Native Method)
  894. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  895. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  896. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  897. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  898. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  899. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  900. at java.lang.reflect.Method.invoke(Method.java:597)
  901. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  902. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  903.  
  904. "IPC Server handler 7 on 9009" daemon prio=10 tid=0x0000000052bd9000 nid=0x6d4e in Object.wait() [0x0000000044b5a000]
  905. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  906. at java.lang.Object.wait(Native Method)
  907. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  908. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  909. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  910. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  911. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  912. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  913. at java.lang.reflect.Method.invoke(Method.java:597)
  914. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  915. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  916.  
  917. "IPC Server handler 6 on 9009" daemon prio=10 tid=0x0000000052be9000 nid=0x6d4d in Object.wait() [0x0000000044a59000]
  918. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  919. at java.lang.Object.wait(Native Method)
  920. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  921. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  922. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  923. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  924. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  925. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  926. at java.lang.reflect.Method.invoke(Method.java:597)
  927. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  928. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  929.  
  930. "IPC Server handler 5 on 9009" daemon prio=10 tid=0x0000000053160800 nid=0x6d4c in Object.wait() [0x0000000044958000]
  931. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  932. at java.lang.Object.wait(Native Method)
  933. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  934. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  935. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  936. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  937. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  938. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  939. at java.lang.reflect.Method.invoke(Method.java:597)
  940. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  941. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  942.  
  943. "IPC Server handler 4 on 9009" daemon prio=10 tid=0x0000000052ee3800 nid=0x6d4b in Object.wait() [0x0000000044857000]
  944. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  945. at java.lang.Object.wait(Native Method)
  946. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  947. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  948. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  949. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  950. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  951. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  952. at java.lang.reflect.Method.invoke(Method.java:597)
  953. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  954. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  955.  
  956. "IPC Server handler 3 on 9009" daemon prio=10 tid=0x0000000052c02800 nid=0x6d4a in Object.wait() [0x0000000044756000]
  957. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  958. at java.lang.Object.wait(Native Method)
  959. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  960. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  961. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  962. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  963. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  964. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  965. at java.lang.reflect.Method.invoke(Method.java:597)
  966. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  967. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  968.  
  969. "IPC Server handler 2 on 9009" daemon prio=10 tid=0x0000000052c01000 nid=0x6d49 in Object.wait() [0x0000000044655000]
  970. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  971. at java.lang.Object.wait(Native Method)
  972. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  973. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  974. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  975. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  976. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  977. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  978. at java.lang.reflect.Method.invoke(Method.java:597)
  979. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  980. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  981.  
  982. "IPC Server handler 1 on 9009" daemon prio=10 tid=0x0000000052a22800 nid=0x6d48 in Object.wait() [0x0000000044554000]
  983. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  984. at java.lang.Object.wait(Native Method)
  985. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  986. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  987. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  988. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  989. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  990. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  991. at java.lang.reflect.Method.invoke(Method.java:597)
  992. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  993. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  994.  
  995. "IPC Server handler 0 on 9009" daemon prio=10 tid=0x0000000052c19800 nid=0x6d47 in Object.wait() [0x0000000044453000]
  996. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  997. at java.lang.Object.wait(Native Method)
  998. at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2051)
  999. - locked <0x00000006a84b9218> (a org.apache.hadoop.hbase.regionserver.HRegion)
  1000. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1717)
  1001. at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3076)
  1002. at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
  1003. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  1004. at java.lang.reflect.Method.invoke(Method.java:597)
  1005. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  1006. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  1007.  
  1008. "IPC Server listener on 9009" daemon prio=10 tid=0x0000000052c54800 nid=0x6d46 runnable [0x0000000044352000]
  1009. java.lang.Thread.State: RUNNABLE
  1010. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1011. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  1012. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  1013. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  1014. - locked <0x000000058b300e80> (a sun.nio.ch.Util$2)
  1015. - locked <0x000000058b300e98> (a java.util.Collections$UnmodifiableSet)
  1016. - locked <0x000000058b302f20> (a sun.nio.ch.EPollSelectorImpl)
  1017. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  1018. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
  1019. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:592)
  1020.  
  1021. "IPC Server Responder" daemon prio=10 tid=0x0000000052c54000 nid=0x6d45 runnable [0x0000000044251000]
  1022. java.lang.Thread.State: RUNNABLE
  1023. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1024. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  1025. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  1026. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  1027. - locked <0x000000058b33b220> (a sun.nio.ch.Util$2)
  1028. - locked <0x000000058b33b238> (a java.util.Collections$UnmodifiableSet)
  1029. - locked <0x000000058b33aba0> (a sun.nio.ch.EPollSelectorImpl)
  1030. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  1031. at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRunLoop(HBaseServer.java:782)
  1032. at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:765)
  1033.  
  1034. "Timer-0" daemon prio=10 tid=0x0000000053121000 nid=0x6d44 in Object.wait() [0x0000000044150000]
  1035. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  1036. at java.lang.Object.wait(Native Method)
  1037. at java.util.TimerThread.mainLoop(Timer.java:509)
  1038. - locked <0x000000058b7e83a8> (a java.util.TaskQueue)
  1039. at java.util.TimerThread.run(Timer.java:462)
  1040.  
  1041. "1861006321@qtp-1068922666-1 - Acceptor0 SelectChannelConnector@0.0.0.0:9004" prio=10 tid=0x00002aaac0079800 nid=0x6d41 runnable [0x000000004404f000]
  1042. java.lang.Thread.State: RUNNABLE
  1043. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1044. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  1045. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  1046. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  1047. - locked <0x000000058b959868> (a sun.nio.ch.Util$2)
  1048. - locked <0x000000058b959880> (a java.util.Collections$UnmodifiableSet)
  1049. - locked <0x000000058b3b5400> (a sun.nio.ch.EPollSelectorImpl)
  1050. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  1051. at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
  1052. at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
  1053. at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
  1054. at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
  1055. at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
  1056.  
  1057. "regionserver9009.leaseChecker" daemon prio=10 tid=0x00002aaac002b800 nid=0x6d3f waiting on condition [0x0000000043e4d000]
  1058. java.lang.Thread.State: TIMED_WAITING (parking)
  1059. at sun.misc.Unsafe.park(Native Method)
  1060. - parking to wait for <0x000000058b30a830> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1061. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
  1062. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
  1063. at java.util.concurrent.DelayQueue.poll(DelayQueue.java:201)
  1064. at org.apache.hadoop.hbase.regionserver.Leases.run(Leases.java:83)
  1065. at java.lang.Thread.run(Thread.java:662)
  1066.  
  1067. "regionserver9009.compactionChecker" daemon prio=10 tid=0x00002aaac005d800 nid=0x6d3e in Object.wait() [0x0000000043d4c000]
  1068. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  1069. at java.lang.Object.wait(Native Method)
  1070. at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
  1071. - locked <0x000000058b9597c0> (a java.lang.Object)
  1072. at org.apache.hadoop.hbase.Chore.run(Chore.java:75)
  1073. at java.lang.Thread.run(Thread.java:662)
  1074.  
  1075. "regionserver9009.cacheFlusher" daemon prio=10 tid=0x00002aaac005d000 nid=0x6d3d waiting for monitor entry [0x0000000043c4b000]
  1076. java.lang.Thread.State: BLOCKED (on object monitor)
  1077. at org.apache.hadoop.hbase.regionserver.StoreScanner.updateReaders(StoreScanner.java:360)
  1078. - waiting to lock <0x000000058fd24658> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  1079. at org.apache.hadoop.hbase.regionserver.Store.notifyChangedReadersObservers(Store.java:667)
  1080. at org.apache.hadoop.hbase.regionserver.Store.updateStorefiles(Store.java:656)
  1081. at org.apache.hadoop.hbase.regionserver.Store.access$200(Store.java:89)
  1082. at org.apache.hadoop.hbase.regionserver.Store$StoreFlusherImpl.commit(Store.java:1916)
  1083. at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1259)
  1084. at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1162)
  1085. at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1104)
  1086. at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:400)
  1087. at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:374)
  1088. at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:243)
  1089. at java.lang.Thread.run(Thread.java:662)
  1090.  
  1091. "regionserver9009.logRoller" daemon prio=10 tid=0x00002aaac00ca000 nid=0x6d3c waiting on condition [0x0000000043b4a000]
  1092. java.lang.Thread.State: WAITING (parking)
  1093. at sun.misc.Unsafe.park(Native Method)
  1094. - parking to wait for <0x000000058b8f29a8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
  1095. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  1096. at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
  1097. at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
  1098. at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
  1099. at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
  1100. at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
  1101. at org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:549)
  1102. at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
  1103. at java.lang.Thread.run(Thread.java:662)
  1104.  
  1105. "Timer thread for monitoring jvm" daemon prio=10 tid=0x00002aaac00c8000 nid=0x6d3b in Object.wait() [0x0000000043a49000]
  1106. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  1107. at java.lang.Object.wait(Native Method)
  1108. at java.util.TimerThread.mainLoop(Timer.java:509)
  1109. - locked <0x000000058b9597d8> (a java.util.TaskQueue)
  1110. at java.util.TimerThread.run(Timer.java:462)
  1111.  
  1112. "Timer thread for monitoring hbase" daemon prio=10 tid=0x00002aaac0720800 nid=0x6d3a in Object.wait() [0x0000000043948000]
  1113. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  1114. at java.lang.Object.wait(Native Method)
  1115. at java.util.TimerThread.mainLoop(Timer.java:509)
  1116. - locked <0x000000058b959790> (a java.util.TaskQueue)
  1117. at java.util.TimerThread.run(Timer.java:462)
  1118.  
  1119. "regionserver9009.logSyncer" daemon prio=10 tid=0x00002aaac07ac000 nid=0x6d39 waiting on condition [0x0000000043847000]
  1120. java.lang.Thread.State: TIMED_WAITING (sleeping)
  1121. at java.lang.Thread.sleep(Native Method)
  1122. at org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:1059)
  1123. at java.lang.Thread.run(Thread.java:662)
  1124.  
  1125. "LeaseChecker" daemon prio=10 tid=0x00002aaac0698000 nid=0x6d38 waiting on condition [0x00000000411c5000]
  1126. java.lang.Thread.State: TIMED_WAITING (sleeping)
  1127. at java.lang.Thread.sleep(Native Method)
  1128. at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1302)
  1129. at java.lang.Thread.run(Thread.java:662)
  1130.  
  1131. "IncidentDistributionPoint-0/ProcessSyncQueue" daemon prio=10 tid=0x0000000052e99800 nid=0x6d32 in Object.wait() [0x0000000043645000]
  1132. java.lang.Thread.State: WAITING (on object monitor)
  1133. at java.lang.Object.wait(Native Method)
  1134. at java.lang.Object.wait(Object.java:485)
  1135. at ORG.oclc.infra.util.SyncQueue.waitForEntries(SyncQueue.java:276)
  1136. - locked <0x000000058b304f08> (a ORG.oclc.infra.util.SyncQueue)
  1137. at ORG.oclc.infra.drax.DestinationList.execute(DestinationList.java:115)
  1138. at ORG.oclc.infra.drax.DistributionPoint.run(DistributionPoint.java:134)
  1139. at java.lang.Thread.run(Thread.java:662)
  1140.  
  1141. "regionserver9009-EventThread" daemon prio=10 tid=0x0000000052f61000 nid=0x6cfc waiting on condition [0x0000000040df7000]
  1142. java.lang.Thread.State: WAITING (parking)
  1143. at sun.misc.Unsafe.park(Native Method)
  1144. - parking to wait for <0x000000058b300b40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1145. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  1146. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  1147. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  1148. at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:493)
  1149.  
  1150. "regionserver9009-SendThread(wchddb35pxdu.prod.oclc.org:9011)" daemon prio=10 tid=0x000000005259e000 nid=0x6cfa runnable [0x0000000040bf5000]
  1151. java.lang.Thread.State: RUNNABLE
  1152. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1153. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  1154. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  1155. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  1156. - locked <0x000000058b300a38> (a sun.nio.ch.Util$2)
  1157. - locked <0x000000058b300a50> (a java.util.Collections$UnmodifiableSet)
  1158. - locked <0x000000058b3068d8> (a sun.nio.ch.EPollSelectorImpl)
  1159. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  1160. at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:274)
  1161. at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
  1162.  
  1163. "DestroyJavaVM" prio=10 tid=0x00002aaac0ab5000 nid=0x6b6a waiting on condition [0x0000000000000000]
  1164. java.lang.Thread.State: RUNNABLE
  1165.  
  1166. "regionserver9009-EventThread" daemon prio=10 tid=0x00002aaacc165800 nid=0x6bbc waiting on condition [0x0000000040af4000]
  1167. java.lang.Thread.State: WAITING (parking)
  1168. at sun.misc.Unsafe.park(Native Method)
  1169. - parking to wait for <0x000000058b300b88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1170. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  1171. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  1172. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  1173. at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:493)
  1174.  
  1175. "regionserver9009-SendThread(wchddb02pxdu.prod.oclc.org:9011)" daemon prio=10 tid=0x00002aaacc162800 nid=0x6bbb runnable [0x000000004032e000]
  1176. java.lang.Thread.State: RUNNABLE
  1177. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1178. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  1179. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  1180. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  1181. - locked <0x000000058b300bd0> (a sun.nio.ch.Util$2)
  1182. - locked <0x000000058b300be8> (a java.util.Collections$UnmodifiableSet)
  1183. - locked <0x000000058b300858> (a sun.nio.ch.EPollSelectorImpl)
  1184. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  1185. at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:274)
  1186. at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
  1187.  
  1188. "regionserver9009" prio=10 tid=0x00000000530ad000 nid=0x6bb9 in Object.wait() [0x0000000043241000]
  1189. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  1190. at java.lang.Object.wait(Native Method)
  1191. at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
  1192. - locked <0x000000058b300a98> (a java.lang.Object)
  1193. at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:55)
  1194. at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:691)
  1195. at java.lang.Thread.run(Thread.java:662)
  1196.  
  1197. "LRU Statistics #0" daemon prio=10 tid=0x00000000530d8800 nid=0x6bb8 waiting on condition [0x0000000043140000]
  1198. java.lang.Thread.State: TIMED_WAITING (parking)
  1199. at sun.misc.Unsafe.park(Native Method)
  1200. - parking to wait for <0x000000058b300c30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1201. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
  1202. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
  1203. at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
  1204. at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
  1205. at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
  1206. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
  1207. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  1208. at java.lang.Thread.run(Thread.java:662)
  1209.  
  1210. "LruBlockCache.EvictionThread" daemon prio=10 tid=0x00000000528c3000 nid=0x6bb7 in Object.wait() [0x0000000041c3d000]
  1211. java.lang.Thread.State: WAITING (on object monitor)
  1212. at java.lang.Object.wait(Native Method)
  1213. at java.lang.Object.wait(Object.java:485)
  1214. at org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:566)
  1215. - locked <0x000000058b300ab0> (a org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread)
  1216. at java.lang.Thread.run(Thread.java:662)
  1217.  
  1218. "Timer thread for monitoring rpc" daemon prio=10 tid=0x0000000052f48000 nid=0x6bb3 in Object.wait() [0x000000004303f000]
  1219. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  1220. at java.lang.Object.wait(Native Method)
  1221. at java.util.TimerThread.mainLoop(Timer.java:509)
  1222. - locked <0x000000058b300ee0> (a java.util.TaskQueue)
  1223. at java.util.TimerThread.run(Timer.java:462)
  1224.  
  1225. "IPC Reader 9 on port 9009" daemon prio=10 tid=0x0000000052b5d000 nid=0x6bb2 runnable [0x0000000042f3e000]
  1226. java.lang.Thread.State: RUNNABLE
  1227. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1228. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  1229. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  1230. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  1231. - locked <0x000000058b300ac8> (a sun.nio.ch.Util$2)
  1232. - locked <0x000000058b300ae0> (a java.util.Collections$UnmodifiableSet)
  1233. - locked <0x000000058b306920> (a sun.nio.ch.EPollSelectorImpl)
  1234. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  1235. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
  1236. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:484)
  1237. - locked <0x000000058b300af8> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  1238. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:470)
  1239. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  1240. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  1241. at java.lang.Thread.run(Thread.java:662)
  1242.  
  1243. "IPC Reader 8 on port 9009" daemon prio=10 tid=0x0000000052b5b000 nid=0x6bb1 runnable [0x0000000042e3d000]
  1244. java.lang.Thread.State: RUNNABLE
  1245. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1246. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  1247. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  1248. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  1249. - locked <0x000000058b300028> (a sun.nio.ch.Util$2)
  1250. - locked <0x000000058b300040> (a java.util.Collections$UnmodifiableSet)
  1251. - locked <0x000000058b30a9b0> (a sun.nio.ch.EPollSelectorImpl)
  1252. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  1253. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
  1254. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:484)
  1255. - locked <0x000000058b300058> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  1256. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:470)
  1257. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  1258. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  1259. at java.lang.Thread.run(Thread.java:662)
  1260.  
  1261. "IPC Reader 7 on port 9009" daemon prio=10 tid=0x0000000052a30800 nid=0x6bb0 runnable [0x0000000042d3c000]
  1262. java.lang.Thread.State: RUNNABLE
  1263. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1264. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  1265. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  1266. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  1267. - locked <0x000000058b300f10> (a sun.nio.ch.Util$2)
  1268. - locked <0x000000058b300f28> (a java.util.Collections$UnmodifiableSet)
  1269. - locked <0x000000058b302f68> (a sun.nio.ch.EPollSelectorImpl)
  1270. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  1271. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
  1272. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:484)
  1273. - locked <0x000000058b300f40> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  1274. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:470)
  1275. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  1276. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  1277. at java.lang.Thread.run(Thread.java:662)
  1278.  
  1279. "IPC Reader 6 on port 9009" daemon prio=10 tid=0x0000000052a2e800 nid=0x6bae runnable [0x0000000042c3b000]
  1280. java.lang.Thread.State: RUNNABLE
  1281. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1282. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  1283. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  1284. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  1285. - locked <0x000000058b30ab18> (a sun.nio.ch.Util$2)
  1286. - locked <0x000000058b30ab30> (a java.util.Collections$UnmodifiableSet)
  1287. - locked <0x000000058b302fb0> (a sun.nio.ch.EPollSelectorImpl)
  1288. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  1289. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
  1290. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:484)
  1291. - locked <0x000000058b30ab48> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  1292. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:470)
  1293. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  1294. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  1295. at java.lang.Thread.run(Thread.java:662)
  1296.  
  1297. "IPC Reader 5 on port 9009" daemon prio=10 tid=0x0000000053128800 nid=0x6bad runnable [0x0000000042b3a000]
  1298. java.lang.Thread.State: RUNNABLE
  1299. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1300. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  1301. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  1302. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  1303. - locked <0x000000058b300088> (a sun.nio.ch.Util$2)
  1304. - locked <0x000000058b3000a0> (a java.util.Collections$UnmodifiableSet)
  1305. - locked <0x000000058b30a9f8> (a sun.nio.ch.EPollSelectorImpl)
  1306. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  1307. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
  1308. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:484)
  1309. - locked <0x000000058b3000b8> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  1310. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:470)
  1311. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  1312. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  1313. at java.lang.Thread.run(Thread.java:662)
  1314.  
  1315. "IPC Reader 4 on port 9009" daemon prio=10 tid=0x0000000052c86800 nid=0x6bac runnable [0x0000000042a39000]
  1316. java.lang.Thread.State: RUNNABLE
  1317. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1318. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  1319. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  1320. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  1321. - locked <0x000000058b30a878> (a sun.nio.ch.Util$2)
  1322. - locked <0x000000058b30a890> (a java.util.Collections$UnmodifiableSet)
  1323. - locked <0x000000058b30b738> (a sun.nio.ch.EPollSelectorImpl)
  1324. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  1325. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
  1326. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:484)
  1327. - locked <0x000000058b30a8a8> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  1328. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:470)
  1329. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  1330. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  1331. at java.lang.Thread.run(Thread.java:662)
  1332.  
  1333. "IPC Reader 3 on port 9009" daemon prio=10 tid=0x0000000052f51800 nid=0x6bab runnable [0x0000000042938000]
  1334. java.lang.Thread.State: RUNNABLE
  1335. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1336. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  1337. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  1338. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  1339. - locked <0x000000058b30ab78> (a sun.nio.ch.Util$2)
  1340. - locked <0x000000058b30ab90> (a java.util.Collections$UnmodifiableSet)
  1341. - locked <0x000000058b302ff8> (a sun.nio.ch.EPollSelectorImpl)
  1342. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  1343. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
  1344. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:484)
  1345. - locked <0x000000058b30aba8> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  1346. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:470)
  1347. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  1348. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  1349. at java.lang.Thread.run(Thread.java:662)
  1350.  
  1351. "IPC Reader 2 on port 9009" daemon prio=10 tid=0x00000000528fc800 nid=0x6baa runnable [0x0000000042837000]
  1352. java.lang.Thread.State: RUNNABLE
  1353. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1354. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  1355. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  1356. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  1357. - locked <0x000000058b3000e8> (a sun.nio.ch.Util$2)
  1358. - locked <0x000000058b300100> (a java.util.Collections$UnmodifiableSet)
  1359. - locked <0x000000058b30aa40> (a sun.nio.ch.EPollSelectorImpl)
  1360. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  1361. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
  1362. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:484)
  1363. - locked <0x000000058b300118> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  1364. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:470)
  1365. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  1366. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  1367. at java.lang.Thread.run(Thread.java:662)
  1368.  
  1369. "IPC Reader 1 on port 9009" daemon prio=10 tid=0x0000000052ecb000 nid=0x6ba9 runnable [0x0000000042736000]
  1370. java.lang.Thread.State: RUNNABLE
  1371. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1372. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  1373. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  1374. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  1375. - locked <0x000000058b300b28> (a sun.nio.ch.Util$2)
  1376. - locked <0x000000058b30c0f8> (a java.util.Collections$UnmodifiableSet)
  1377. - locked <0x000000058b306968> (a sun.nio.ch.EPollSelectorImpl)
  1378. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  1379. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
  1380. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:484)
  1381. - locked <0x000000058b30c110> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  1382. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:470)
  1383. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  1384. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  1385. at java.lang.Thread.run(Thread.java:662)
  1386.  
  1387. "IPC Reader 0 on port 9009" daemon prio=10 tid=0x0000000052c8a000 nid=0x6ba8 runnable [0x0000000042635000]
  1388. java.lang.Thread.State: RUNNABLE
  1389. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1390. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  1391. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  1392. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  1393. - locked <0x000000058b30c278> (a sun.nio.ch.Util$2)
  1394. - locked <0x000000058b30c260> (a java.util.Collections$UnmodifiableSet)
  1395. - locked <0x000000058b303088> (a sun.nio.ch.EPollSelectorImpl)
  1396. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  1397. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
  1398. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:484)
  1399. - locked <0x000000058b74e918> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  1400. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:470)
  1401. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  1402. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  1403. at java.lang.Thread.run(Thread.java:662)
  1404.  
  1405. "RMI TCP Accept-0" daemon prio=10 tid=0x00002aaac065e000 nid=0x6b83 runnable [0x0000000040fc3000]
  1406. java.lang.Thread.State: RUNNABLE
  1407. at java.net.PlainSocketImpl.socketAccept(Native Method)
  1408. at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
  1409. - locked <0x000000058b30bae0> (a java.net.SocksSocketImpl)
  1410. at java.net.ServerSocket.implAccept(ServerSocket.java:462)
  1411. at java.net.ServerSocket.accept(ServerSocket.java:430)
  1412. at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
  1413. at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
  1414. at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
  1415. at java.lang.Thread.run(Thread.java:662)
  1416.  
  1417. "RMI TCP Accept-9021" daemon prio=10 tid=0x00002aaac065a800 nid=0x6b82 runnable [0x00000000409f3000]
  1418. java.lang.Thread.State: RUNNABLE
  1419. at java.net.PlainSocketImpl.socketAccept(Native Method)
  1420. at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
  1421. - locked <0x000000058b30c3e0> (a java.net.SocksSocketImpl)
  1422. at java.net.ServerSocket.implAccept(ServerSocket.java:462)
  1423. at java.net.ServerSocket.accept(ServerSocket.java:430)
  1424. at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
  1425. at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
  1426. at java.lang.Thread.run(Thread.java:662)
  1427.  
  1428. "RMI TCP Accept-0" daemon prio=10 tid=0x00002aaac0648000 nid=0x6b81 runnable [0x00000000408f2000]
  1429. java.lang.Thread.State: RUNNABLE
  1430. at java.net.PlainSocketImpl.socketAccept(Native Method)
  1431. at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
  1432. - locked <0x000000058b30ca70> (a java.net.SocksSocketImpl)
  1433. at java.net.ServerSocket.implAccept(ServerSocket.java:462)
  1434. at java.net.ServerSocket.accept(ServerSocket.java:430)
  1435. at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
  1436. at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
  1437. at java.lang.Thread.run(Thread.java:662)
  1438.  
  1439. "Low Memory Detector" daemon prio=10 tid=0x00002aaac0026800 nid=0x6b7d runnable [0x0000000000000000]
  1440. java.lang.Thread.State: RUNNABLE
  1441.  
  1442. "C2 CompilerThread1" daemon prio=10 tid=0x00002aaac0024000 nid=0x6b7c waiting on condition [0x0000000000000000]
  1443. java.lang.Thread.State: RUNNABLE
  1444.  
  1445. "C2 CompilerThread0" daemon prio=10 tid=0x00002aaac0022000 nid=0x6b7b waiting on condition [0x0000000000000000]
  1446. java.lang.Thread.State: RUNNABLE
  1447.  
  1448. "Signal Dispatcher" daemon prio=10 tid=0x00002aaac001f800 nid=0x6b7a waiting on condition [0x0000000000000000]
  1449. java.lang.Thread.State: RUNNABLE
  1450.  
  1451. "Surrogate Locker Thread (Concurrent GC)" daemon prio=10 tid=0x00002aaac001d800 nid=0x6b79 waiting on condition [0x0000000000000000]
  1452. java.lang.Thread.State: RUNNABLE
  1453.  
  1454. "Finalizer" daemon prio=10 tid=0x00002aaac0001000 nid=0x6b78 in Object.wait() [0x000000004171c000]
  1455. java.lang.Thread.State: WAITING (on object monitor)
  1456. at java.lang.Object.wait(Native Method)
  1457. at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
  1458. - locked <0x000000058b300160> (a java.lang.ref.ReferenceQueue$Lock)
  1459. at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
  1460. at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
  1461.  
  1462. "Reference Handler" daemon prio=10 tid=0x00000000527e3000 nid=0x6b77 in Object.wait() [0x000000004161b000]
  1463. java.lang.Thread.State: WAITING (on object monitor)
  1464. at java.lang.Object.wait(Native Method)
  1465. at java.lang.Object.wait(Object.java:485)
  1466. at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
  1467. - locked <0x000000058b303988> (a java.lang.ref.Reference$Lock)
  1468.  
  1469. "VM Thread" prio=10 tid=0x00000000527dc800 nid=0x6b76 runnable
  1470.  
  1471. "Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x00000000525ac800 nid=0x6b6b runnable
  1472.  
  1473. "Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x00000000525ae800 nid=0x6b6c runnable
  1474.  
  1475. "Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x00000000525b0800 nid=0x6b6d runnable
  1476.  
  1477. "Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x00000000525b2000 nid=0x6b6e runnable
  1478.  
  1479. "Gang worker#4 (Parallel GC Threads)" prio=10 tid=0x00000000525b4000 nid=0x6b6f runnable
  1480.  
  1481. "Gang worker#5 (Parallel GC Threads)" prio=10 tid=0x00000000525b6000 nid=0x6b70 runnable
  1482.  
  1483. "Gang worker#6 (Parallel GC Threads)" prio=10 tid=0x00000000525b7800 nid=0x6b71 runnable
  1484.  
  1485. "Gang worker#7 (Parallel GC Threads)" prio=10 tid=0x00000000525b9800 nid=0x6b72 runnable
  1486.  
  1487. "Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00000000526ad800 nid=0x6b75 runnable
  1488. "Gang worker#0 (Parallel CMS Threads)" prio=10 tid=0x00000000526a9800 nid=0x6b73 runnable
  1489.  
  1490. "Gang worker#1 (Parallel CMS Threads)" prio=10 tid=0x00000000526ab000 nid=0x6b74 runnable
  1491.  
  1492. "VM Periodic Task Thread" prio=10 tid=0x00002aaac0661800 nid=0x6b84 waiting on condition
  1493.  
  1494. JNI global references: 1248
  1495.  
  1496. Heap
  1497. par new generation total 235968K, used 176200K [0x000000057ae00000, 0x000000058ae00000, 0x000000058ae00000)
  1498. eden space 209792K, 72% used [0x000000057ae00000, 0x0000000584214c88, 0x0000000587ae0000)
  1499. from space 26176K, 93% used [0x0000000587ae0000, 0x00000005892dd6c8, 0x0000000589470000)
  1500. to space 26176K, 0% used [0x0000000589470000, 0x0000000589470000, 0x000000058ae00000)
  1501. concurrent mark-sweep generation total 10223616K, used 4535508K [0x000000058ae00000, 0x00000007fae00000, 0x00000007fae00000)
  1502. concurrent-mark-sweep perm gen total 49196K, used 29423K [0x00000007fae00000, 0x00000007fde0b000, 0x0000000800000000)
  1503.  
  1504. -PROD- prodcon@wchddb37pxdu:/prod/hadoop/prod/hbase-0.92.1/logs$
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!