API tools faq
paste
Advertisement
Guest User

Untitled

a guest
May 14th, 2011
112
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 406.96 KB | None | 0 0
raw download clone embed print report
  1. "IPC Client (47) connection to mtag1/10.103.7.1:9000 from root" daemon prio=10 tid=0x00007f6f501ac000 nid=0xfa5 runnable [0x00007f6f5d041000]
  2. java.lang.Thread.State: RUNNABLE
  3. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  4. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  5. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  6. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  7. - locked <0x00000006173020d8> (a sun.nio.ch.Util$2)
  8. - locked <0x00000006173020c0> (a java.util.Collections$UnmodifiableSet)
  9. - locked <0x00000006173042d0> (a sun.nio.ch.EPollSelectorImpl)
  10. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  11. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:332)
  12. at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
  13. at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
  14. at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
  15. at java.io.FilterInputStream.read(FilterInputStream.java:116)
  16. at org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:304)
  17. at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
  18. at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
  19. - locked <0x00000006187af310> (a java.io.BufferedInputStream)
  20. at java.io.DataInputStream.readInt(DataInputStream.java:370)
  21. at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:569)
  22. at org.apache.hadoop.ipc.Client$Connection.run(Client.java:477)
  23.  
  24. "ResponseProcessor for block blk_1030279999998547108_1158319" daemon prio=10 tid=0x00007f6f5448f800 nid=0x24c0 runnable [0x00007f6f5cd3e000]
  25. java.lang.Thread.State: RUNNABLE
  26. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  27. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  28. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  29. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  30. - locked <0x0000000620a852f8> (a sun.nio.ch.Util$2)
  31. - locked <0x0000000620a852e0> (a java.util.Collections$UnmodifiableSet)
  32. - locked <0x0000000620a3f408> (a sun.nio.ch.EPollSelectorImpl)
  33. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  34. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:332)
  35. at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
  36. at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
  37. at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
  38. at java.io.DataInputStream.readFully(DataInputStream.java:178)
  39. at java.io.DataInputStream.readLong(DataInputStream.java:399)
  40. at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:119)
  41. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2468)
  42.  
  43. "DataStreamer for file /hbase/.logs/mtag27.prod.imageshack.com,60020,1305408211142/10.103.7.27%3A60020.1305411438693 block blk_1030279999998547108_1158319" daemon prio=10 tid=0x00007f6f2c009000 nid=0x24bc in Object.wait() [0x00007f6f03dbc000]
  44. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  45. at java.lang.Object.wait(Native Method)
  46. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2313)
  47. - locked <0x00000005dbac2f88> (a java.util.LinkedList)
  48.  
  49. "pool-3-thread-1" prio=10 tid=0x00007f6f141cb000 nid=0x6557 waiting on condition [0x00007f6f03fbe000]
  50. java.lang.Thread.State: TIMED_WAITING (parking)
  51. at sun.misc.Unsafe.park(Native Method)
  52. - parking to wait for <0x00000005138a8b30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  53. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
  54. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
  55. at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
  56. at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
  57. at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
  58. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
  59. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  60. at java.lang.Thread.run(Thread.java:662)
  61.  
  62. "LruBlockCache.EvictionThread" daemon prio=10 tid=0x00007f6f141c9800 nid=0x6556 in Object.wait() [0x00007f6f040bf000]
  63. java.lang.Thread.State: WAITING (on object monitor)
  64. at java.lang.Object.wait(Native Method)
  65. - waiting on <0x00000005138232e0> (a org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread)
  66. at java.lang.Object.wait(Object.java:485)
  67. at org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:514)
  68. - locked <0x00000005138232e0> (a org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread)
  69.  
  70. "PRI IPC Server handler 9 on 60020" daemon prio=10 tid=0x00007f6f141c7800 nid=0x6555 waiting on condition [0x00007f6f041c0000]
  71. java.lang.Thread.State: WAITING (parking)
  72. at sun.misc.Unsafe.park(Native Method)
  73. - parking to wait for <0x0000000513679408> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  74. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  75. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  76. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  77. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1013)
  78.  
  79. "PRI IPC Server handler 8 on 60020" daemon prio=10 tid=0x00007f6f141c5800 nid=0x6554 waiting on condition [0x00007f6f042c1000]
  80. java.lang.Thread.State: WAITING (parking)
  81. at sun.misc.Unsafe.park(Native Method)
  82. - parking to wait for <0x0000000513679408> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  83. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  84. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  85. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  86. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1013)
  87.  
  88. "PRI IPC Server handler 7 on 60020" daemon prio=10 tid=0x00007f6f141c3800 nid=0x6553 waiting on condition [0x00007f6f043c2000]
  89. java.lang.Thread.State: WAITING (parking)
  90. at sun.misc.Unsafe.park(Native Method)
  91. - parking to wait for <0x0000000513679408> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  92. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  93. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  94. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  95. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1013)
  96.  
  97. "PRI IPC Server handler 6 on 60020" daemon prio=10 tid=0x00007f6f141c1800 nid=0x6552 waiting on condition [0x00007f6f044c3000]
  98. java.lang.Thread.State: WAITING (parking)
  99. at sun.misc.Unsafe.park(Native Method)
  100. - parking to wait for <0x0000000513679408> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  101. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  102. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  103. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  104. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1013)
  105.  
  106. "PRI IPC Server handler 5 on 60020" daemon prio=10 tid=0x00007f6f141bf800 nid=0x6551 waiting on condition [0x00007f6f045c4000]
  107. java.lang.Thread.State: WAITING (parking)
  108. at sun.misc.Unsafe.park(Native Method)
  109. - parking to wait for <0x0000000513679408> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  110. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  111. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  112. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  113. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1013)
  114.  
  115. "PRI IPC Server handler 4 on 60020" daemon prio=10 tid=0x00007f6f141bd800 nid=0x6550 waiting on condition [0x00007f6f046c5000]
  116. java.lang.Thread.State: WAITING (parking)
  117. at sun.misc.Unsafe.park(Native Method)
  118. - parking to wait for <0x0000000513679408> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  119. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  120. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  121. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  122. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1013)
  123.  
  124. "PRI IPC Server handler 3 on 60020" daemon prio=10 tid=0x00007f6f141bb800 nid=0x654f waiting on condition [0x00007f6f047c6000]
  125. java.lang.Thread.State: WAITING (parking)
  126. at sun.misc.Unsafe.park(Native Method)
  127. - parking to wait for <0x0000000513679408> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  128. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  129. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  130. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  131. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1013)
  132.  
  133. "PRI IPC Server handler 2 on 60020" daemon prio=10 tid=0x00007f6f141b9800 nid=0x654e waiting on condition [0x00007f6f048c7000]
  134. java.lang.Thread.State: WAITING (parking)
  135. at sun.misc.Unsafe.park(Native Method)
  136. - parking to wait for <0x0000000513679408> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  137. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  138. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  139. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  140. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1013)
  141.  
  142. "PRI IPC Server handler 1 on 60020" daemon prio=10 tid=0x00007f6f141b7800 nid=0x654d waiting on condition [0x00007f6f049c8000]
  143. java.lang.Thread.State: WAITING (parking)
  144. at sun.misc.Unsafe.park(Native Method)
  145. - parking to wait for <0x0000000513679408> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  146. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  147. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  148. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  149. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1013)
  150.  
  151. "PRI IPC Server handler 0 on 60020" daemon prio=10 tid=0x00007f6f141b5800 nid=0x654c waiting on condition [0x00007f6f04ac9000]
  152. java.lang.Thread.State: WAITING (parking)
  153. at sun.misc.Unsafe.park(Native Method)
  154. - parking to wait for <0x0000000513679408> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  155. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  156. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  157. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  158. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1013)
  159.  
  160. "IPC Server handler 49 on 60020" daemon prio=10 tid=0x00007f6f141b3800 nid=0x654b runnable [0x00007f6f04bca000]
  161. java.lang.Thread.State: RUNNABLE
  162. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:566)
  163. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  164. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  165. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  166. - locked <0x000000050dea00d0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  167. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  168. - locked <0x000000050dea00d0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  169. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  170. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  171. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  172. - locked <0x000000050dea0138> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  173. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  174. - locked <0x000000050dea0138> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  175. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  176. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  177. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  178. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  179. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  180. at java.lang.reflect.Method.invoke(Method.java:597)
  181. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  182. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  183.  
  184. "IPC Server handler 48 on 60020" daemon prio=10 tid=0x00007f6f141b1800 nid=0x654a runnable [0x00007f6f04ccb000]
  185. java.lang.Thread.State: RUNNABLE
  186. at sun.misc.Unsafe.unpark(Native Method)
  187. at java.util.concurrent.locks.LockSupport.unpark(LockSupport.java:124)
  188. at java.util.concurrent.locks.AbstractQueuedSynchronizer.unparkSuccessor(AbstractQueuedSynchronizer.java:640)
  189. at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1242)
  190. at java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:431)
  191. at java.util.concurrent.LinkedBlockingQueue.signalNotFull(LinkedBlockingQueue.java:147)
  192. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:409)
  193. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1013)
  194.  
  195. "IPC Server handler 47 on 60020" daemon prio=10 tid=0x00007f6f141af000 nid=0x6549 runnable [0x00007f6f04dcc000]
  196. java.lang.Thread.State: RUNNABLE
  197. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:566)
  198. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  199. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  200. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  201. - locked <0x000000050cee2140> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  202. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  203. - locked <0x000000050cee2140> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  204. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  205. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  206. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  207. - locked <0x000000050cee2040> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  208. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  209. - locked <0x000000050cee2040> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  210. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  211. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  212. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  213. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  214. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  215. at java.lang.reflect.Method.invoke(Method.java:597)
  216. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  217. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  218.  
  219. "IPC Server handler 46 on 60020" daemon prio=10 tid=0x00007f6f141ad000 nid=0x6548 runnable [0x00007f6f04ecd000]
  220. java.lang.Thread.State: RUNNABLE
  221. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:74)
  222. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:62)
  223. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:561)
  224. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  225. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  226. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  227. - locked <0x000000050deecea0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  228. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  229. - locked <0x000000050deecea0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  230. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  231. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  232. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  233. - locked <0x000000050deecf08> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  234. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  235. - locked <0x000000050deecf08> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  236. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  237. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  238. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  239. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  240. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  241. at java.lang.reflect.Method.invoke(Method.java:597)
  242. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  243. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  244.  
  245. "IPC Server handler 45 on 60020" daemon prio=10 tid=0x00007f6f141ab000 nid=0x6547 in Object.wait() [0x00007f6f04fce000]
  246. java.lang.Thread.State: BLOCKED (on object monitor)
  247. at java.lang.Object.wait(Native Method)
  248. at java.lang.Object.wait(Object.java:485)
  249. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.waitForAckedSeqno(DFSClient.java:3284)
  250. - locked <0x00000005db9b7800> (a java.util.LinkedList)
  251. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.sync(DFSClient.java:3201)
  252. at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:97)
  253. at org.apache.hadoop.io.SequenceFile$Writer.syncFs(SequenceFile.java:944)
  254. at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
  255. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  256. at java.lang.reflect.Method.invoke(Method.java:597)
  257. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:123)
  258. at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:906)
  259. at org.apache.hadoop.hbase.regionserver.wal.HLog.append(HLog.java:857)
  260. at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1402)
  261. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1318)
  262. at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1803)
  263. at org.apache.hadoop.hbase.regionserver.HRegionServer.multiPut(HRegionServer.java:2464)
  264. at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
  265. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  266. at java.lang.reflect.Method.invoke(Method.java:597)
  267. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  268. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  269.  
  270. "IPC Server handler 44 on 60020" daemon prio=10 tid=0x00007f6f141a9000 nid=0x6546 runnable [0x00007f6f050ce000]
  271. java.lang.Thread.State: RUNNABLE
  272. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:74)
  273. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:62)
  274. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:561)
  275. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  276. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  277. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  278. - locked <0x000000050dec6978> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  279. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  280. - locked <0x000000050dec6978> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  281. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  282. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  283. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  284. - locked <0x000000050dec69e0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  285. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  286. - locked <0x000000050dec69e0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  287. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  288. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  289. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  290. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  291. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  292. at java.lang.reflect.Method.invoke(Method.java:597)
  293. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  294. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  295.  
  296. "IPC Server handler 43 on 60020" daemon prio=10 tid=0x00007f6f141a7000 nid=0x6545 runnable [0x00007f6f051d0000]
  297. java.lang.Thread.State: RUNNABLE
  298. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  299. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  300. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  301. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  302. - locked <0x0000000621e87a88> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  303. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  304. - locked <0x0000000621e87a88> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  305. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  306. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  307. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  308. - locked <0x0000000621d46ee8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  309. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  310. - locked <0x0000000621d46ee8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  311. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  312. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  313. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  314. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  315. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  316. at java.lang.reflect.Method.invoke(Method.java:597)
  317. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  318. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  319.  
  320. "IPC Server handler 42 on 60020" daemon prio=10 tid=0x00007f6f141a5000 nid=0x6544 runnable [0x00007f6f052d1000]
  321. java.lang.Thread.State: RUNNABLE
  322. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  323. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  324. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  325. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  326. - locked <0x000000050d028a68> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  327. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  328. - locked <0x000000050d028a68> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  329. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  330. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  331. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  332. - locked <0x000000050d028968> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  333. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  334. - locked <0x000000050d028968> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  335. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  336. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  337. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  338. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  339. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  340. at java.lang.reflect.Method.invoke(Method.java:597)
  341. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  342. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  343.  
  344. "IPC Server handler 41 on 60020" daemon prio=10 tid=0x00007f6f141a3000 nid=0x6543 runnable [0x00007f6f053d2000]
  345. java.lang.Thread.State: RUNNABLE
  346. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  347. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  348. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  349. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  350. - locked <0x000000050dec6af0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  351. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  352. - locked <0x000000050dec6af0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  353. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  354. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  355. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  356. - locked <0x000000050dec6b58> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  357. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  358. - locked <0x000000050dec6b58> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  359. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  360. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  361. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  362. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  363. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  364. at java.lang.reflect.Method.invoke(Method.java:597)
  365. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  366. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  367.  
  368. "IPC Server handler 40 on 60020" daemon prio=10 tid=0x00007f6f141a1000 nid=0x6542 runnable [0x00007f6f054d3000]
  369. java.lang.Thread.State: RUNNABLE
  370. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  371. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  372. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  373. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  374. - locked <0x0000000621ec3a78> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  375. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  376. - locked <0x0000000621ec3a78> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  377. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  378. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  379. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  380. - locked <0x0000000621eb60a0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  381. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  382. - locked <0x0000000621eb60a0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  383. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  384. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  385. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  386. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  387. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  388. at java.lang.reflect.Method.invoke(Method.java:597)
  389. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  390. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  391.  
  392. "IPC Server handler 39 on 60020" daemon prio=10 tid=0x00007f6f1419f000 nid=0x6541 runnable [0x00007f6f055d4000]
  393. java.lang.Thread.State: RUNNABLE
  394. at java.util.concurrent.ConcurrentSkipListMap$SubMap$SubMapEntryIterator.next(ConcurrentSkipListMap.java:3113)
  395. at java.util.concurrent.ConcurrentSkipListMap$SubMap$SubMapEntryIterator.next(ConcurrentSkipListMap.java:3108)
  396. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:74)
  397. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:62)
  398. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:561)
  399. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  400. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  401. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  402. - locked <0x000000050df135c0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  403. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  404. - locked <0x000000050df135c0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  405. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  406. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  407. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  408. - locked <0x000000050df13628> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  409. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  410. - locked <0x000000050df13628> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  411. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  412. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  413. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  414. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  415. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  416. at java.lang.reflect.Method.invoke(Method.java:597)
  417. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  418. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  419.  
  420. "IPC Server handler 38 on 60020" daemon prio=10 tid=0x00007f6f1419d000 nid=0x6540 in Object.wait() [0x00007f6f056d4000]
  421. java.lang.Thread.State: BLOCKED (on object monitor)
  422. at java.lang.Object.wait(Native Method)
  423. at java.lang.Object.wait(Object.java:485)
  424. at org.apache.hadoop.ipc.Client.call(Client.java:803)
  425. - locked <0x000000050df136d0> (a org.apache.hadoop.ipc.Client$Call)
  426. at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:221)
  427. at $Proxy1.getFileInfo(Unknown Source)
  428. at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
  429. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  430. at java.lang.reflect.Method.invoke(Method.java:597)
  431. at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
  432. at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
  433. at $Proxy1.getFileInfo(Unknown Source)
  434. at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:624)
  435. at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:461)
  436. at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:690)
  437. at org.apache.hadoop.hbase.util.FSUtils.checkFileSystemAvailable(FSUtils.java:119)
  438. at org.apache.hadoop.hbase.regionserver.HRegionServer.checkFileSystem(HRegionServer.java:953)
  439. at org.apache.hadoop.hbase.regionserver.HRegionServer.cleanup(HRegionServer.java:902)
  440. at org.apache.hadoop.hbase.regionserver.HRegionServer.cleanup(HRegionServer.java:880)
  441. at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1958)
  442. at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
  443. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  444. at java.lang.reflect.Method.invoke(Method.java:597)
  445. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  446. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  447.  
  448. "IPC Server handler 37 on 60020" daemon prio=10 tid=0x00007f6f1419b000 nid=0x653f runnable [0x00007f6f057d5000]
  449. java.lang.Thread.State: RUNNABLE
  450. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  451. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  452. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  453. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  454. - locked <0x000000050deed0d0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  455. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  456. - locked <0x000000050deed0d0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  457. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  458. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  459. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  460. - locked <0x000000050deed138> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  461. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  462. - locked <0x000000050deed138> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  463. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  464. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  465. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  466. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  467. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  468. at java.lang.reflect.Method.invoke(Method.java:597)
  469. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  470. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  471.  
  472. "IPC Server handler 36 on 60020" daemon prio=10 tid=0x00007f6f14199000 nid=0x653e runnable [0x00007f6f058d6000]
  473. java.lang.Thread.State: RUNNABLE
  474. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  475. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  476. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  477. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  478. - locked <0x000000050dea0368> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  479. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  480. - locked <0x000000050dea0368> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  481. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  482. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  483. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  484. - locked <0x000000050dea03d0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  485. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  486. - locked <0x000000050dea03d0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  487. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  488. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  489. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  490. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  491. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  492. at java.lang.reflect.Method.invoke(Method.java:597)
  493. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  494. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  495.  
  496. "IPC Server handler 35 on 60020" daemon prio=10 tid=0x00007f6f14197000 nid=0x653d runnable [0x00007f6f059d8000]
  497. java.lang.Thread.State: RUNNABLE
  498. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:74)
  499. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:62)
  500. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:561)
  501. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  502. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  503. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  504. - locked <0x000000050df13880> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  505. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  506. - locked <0x000000050df13880> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  507. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  508. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  509. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  510. - locked <0x000000050df138e8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  511. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  512. - locked <0x000000050df138e8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  513. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  514. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  515. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  516. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  517. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  518. at java.lang.reflect.Method.invoke(Method.java:597)
  519. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  520. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  521.  
  522. "IPC Server handler 34 on 60020" daemon prio=10 tid=0x00007f6f14195000 nid=0x653c runnable [0x00007f6f05ad9000]
  523. java.lang.Thread.State: RUNNABLE
  524. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  525. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  526. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  527. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  528. - locked <0x000000050deed248> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  529. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  530. - locked <0x000000050deed248> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  531. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  532. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  533. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  534. - locked <0x000000050deed2b0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  535. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  536. - locked <0x000000050deed2b0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  537. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  538. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  539. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  540. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  541. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  542. at java.lang.reflect.Method.invoke(Method.java:597)
  543. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  544. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  545.  
  546. "IPC Server handler 33 on 60020" daemon prio=10 tid=0x00007f6f14193000 nid=0x653b in Object.wait() [0x00007f6f05bda000]
  547. java.lang.Thread.State: BLOCKED (on object monitor)
  548. at java.lang.Object.wait(Native Method)
  549. at java.lang.Object.wait(Object.java:485)
  550. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.waitForAckedSeqno(DFSClient.java:3284)
  551. - locked <0x00000005db9b7800> (a java.util.LinkedList)
  552. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.sync(DFSClient.java:3201)
  553. at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:97)
  554. at org.apache.hadoop.io.SequenceFile$Writer.syncFs(SequenceFile.java:944)
  555. at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
  556. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  557. at java.lang.reflect.Method.invoke(Method.java:597)
  558. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:123)
  559. at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:906)
  560. at org.apache.hadoop.hbase.regionserver.wal.HLog.append(HLog.java:857)
  561. at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1402)
  562. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1318)
  563. at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1803)
  564. at org.apache.hadoop.hbase.regionserver.HRegionServer.multiPut(HRegionServer.java:2464)
  565. at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
  566. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  567. at java.lang.reflect.Method.invoke(Method.java:597)
  568. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  569. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  570.  
  571. "IPC Server handler 32 on 60020" daemon prio=10 tid=0x00007f6f14191000 nid=0x653a in Object.wait() [0x00007f6f05cdb000]
  572. java.lang.Thread.State: BLOCKED (on object monitor)
  573. at java.lang.Object.wait(Native Method)
  574. at java.lang.Object.wait(Object.java:485)
  575. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.waitForAckedSeqno(DFSClient.java:3284)
  576. - locked <0x00000005db9b7800> (a java.util.LinkedList)
  577. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.sync(DFSClient.java:3201)
  578. at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:97)
  579. at org.apache.hadoop.io.SequenceFile$Writer.syncFs(SequenceFile.java:944)
  580. at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
  581. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  582. at java.lang.reflect.Method.invoke(Method.java:597)
  583. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:123)
  584. at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:906)
  585. at org.apache.hadoop.hbase.regionserver.wal.HLog.append(HLog.java:857)
  586. at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1402)
  587. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1318)
  588. at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1803)
  589. at org.apache.hadoop.hbase.regionserver.HRegionServer.multiPut(HRegionServer.java:2464)
  590. at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
  591. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  592. at java.lang.reflect.Method.invoke(Method.java:597)
  593. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  594. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  595.  
  596. "IPC Server handler 31 on 60020" daemon prio=10 tid=0x00007f6f1418f000 nid=0x6539 runnable [0x00007f6f05ddc000]
  597. java.lang.Thread.State: RUNNABLE
  598. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  599. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  600. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  601. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  602. - locked <0x000000050d024d80> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  603. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  604. - locked <0x000000050d024d80> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  605. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  606. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  607. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  608. - locked <0x000000050d024c80> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  609. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  610. - locked <0x000000050d024c80> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  611. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  612. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  613. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  614. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  615. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  616. at java.lang.reflect.Method.invoke(Method.java:597)
  617. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  618. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  619.  
  620. "IPC Server handler 30 on 60020" daemon prio=10 tid=0x00007f6f1418d000 nid=0x6538 in Object.wait() [0x00007f6f05edc000]
  621. java.lang.Thread.State: WAITING (on object monitor)
  622. at java.lang.Object.wait(Native Method)
  623. at java.lang.Object.wait(Object.java:485)
  624. at org.apache.hadoop.ipc.Client.call(Client.java:803)
  625. - locked <0x000000050cf5e120> (a org.apache.hadoop.ipc.Client$Call)
  626. at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:221)
  627. at $Proxy1.getFileInfo(Unknown Source)
  628. at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
  629. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  630. at java.lang.reflect.Method.invoke(Method.java:597)
  631. at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
  632. at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
  633. at $Proxy1.getFileInfo(Unknown Source)
  634. at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:624)
  635. at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:461)
  636. at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:690)
  637. at org.apache.hadoop.hbase.util.FSUtils.checkFileSystemAvailable(FSUtils.java:119)
  638. at org.apache.hadoop.hbase.regionserver.HRegionServer.checkFileSystem(HRegionServer.java:953)
  639. at org.apache.hadoop.hbase.regionserver.HRegionServer.cleanup(HRegionServer.java:902)
  640. at org.apache.hadoop.hbase.regionserver.HRegionServer.cleanup(HRegionServer.java:880)
  641. at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1958)
  642. at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
  643. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  644. at java.lang.reflect.Method.invoke(Method.java:597)
  645. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  646. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  647.  
  648. "IPC Server handler 29 on 60020" daemon prio=10 tid=0x00007f6f1418b000 nid=0x6537 runnable [0x00007f6f05fde000]
  649. java.lang.Thread.State: RUNNABLE
  650. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:74)
  651. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:62)
  652. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:561)
  653. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  654. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  655. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  656. - locked <0x0000000621e376a0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  657. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  658. - locked <0x0000000621e376a0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  659. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  660. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  661. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  662. - locked <0x0000000621e56600> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  663. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  664. - locked <0x0000000621e56600> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  665. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  666. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  667. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  668. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  669. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  670. at java.lang.reflect.Method.invoke(Method.java:597)
  671. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  672. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  673.  
  674. "IPC Server handler 28 on 60020" daemon prio=10 tid=0x00007f6f14189000 nid=0x6536 runnable [0x00007f6f060de000]
  675. java.lang.Thread.State: RUNNABLE
  676. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:566)
  677. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  678. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  679. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  680. - locked <0x000000050d014338> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  681. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  682. - locked <0x000000050d014338> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  683. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  684. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  685. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  686. - locked <0x000000050d014238> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  687. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  688. - locked <0x000000050d014238> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  689. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  690. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  691. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  692. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  693. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  694. at java.lang.reflect.Method.invoke(Method.java:597)
  695. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  696. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  697.  
  698. "IPC Server handler 27 on 60020" daemon prio=10 tid=0x00007f6f14187000 nid=0x6535 runnable [0x00007f6f061e0000]
  699. java.lang.Thread.State: RUNNABLE
  700. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  701. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  702. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  703. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  704. - locked <0x0000000621eb8ab8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  705. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  706. - locked <0x0000000621eb8ab8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  707. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  708. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  709. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  710. - locked <0x0000000621ea5a20> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  711. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  712. - locked <0x0000000621ea5a20> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  713. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  714. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  715. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  716. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  717. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  718. at java.lang.reflect.Method.invoke(Method.java:597)
  719. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  720. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  721.  
  722. "IPC Server handler 26 on 60020" daemon prio=10 tid=0x00007f6f14185000 nid=0x6534 runnable [0x00007f6f062e1000]
  723. java.lang.Thread.State: RUNNABLE
  724. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:566)
  725. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  726. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  727. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  728. - locked <0x000000050dec7038> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  729. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  730. - locked <0x000000050dec7038> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  731. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  732. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  733. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  734. - locked <0x000000050dec74c8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  735. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  736. - locked <0x000000050dec74c8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  737. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  738. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  739. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  740. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  741. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  742. at java.lang.reflect.Method.invoke(Method.java:597)
  743. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  744. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  745.  
  746. "IPC Server handler 25 on 60020" daemon prio=10 tid=0x00007f6f14183000 nid=0x6533 runnable [0x00007f6f063e2000]
  747. java.lang.Thread.State: RUNNABLE
  748. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:74)
  749. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:62)
  750. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:561)
  751. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  752. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  753. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  754. - locked <0x0000000621eb7998> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  755. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  756. - locked <0x0000000621eb7998> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  757. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  758. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  759. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  760. - locked <0x0000000621d47348> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  761. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  762. - locked <0x0000000621d47348> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  763. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  764. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  765. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  766. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  767. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  768. at java.lang.reflect.Method.invoke(Method.java:597)
  769. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  770. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  771.  
  772. "IPC Server handler 24 on 60020" daemon prio=10 tid=0x00007f6f14181000 nid=0x6532 in Object.wait() [0x00007f6f064e3000]
  773. java.lang.Thread.State: BLOCKED (on object monitor)
  774. at java.lang.Object.wait(Native Method)
  775. at java.lang.Object.wait(Object.java:485)
  776. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.waitForAckedSeqno(DFSClient.java:3284)
  777. - locked <0x00000005db9b7800> (a java.util.LinkedList)
  778. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.sync(DFSClient.java:3201)
  779. at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:97)
  780. at org.apache.hadoop.io.SequenceFile$Writer.syncFs(SequenceFile.java:944)
  781. at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
  782. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  783. at java.lang.reflect.Method.invoke(Method.java:597)
  784. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:123)
  785. at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:906)
  786. at org.apache.hadoop.hbase.regionserver.wal.HLog.append(HLog.java:857)
  787. at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1402)
  788. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1318)
  789. at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1803)
  790. at org.apache.hadoop.hbase.regionserver.HRegionServer.multiPut(HRegionServer.java:2464)
  791. at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
  792. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  793. at java.lang.reflect.Method.invoke(Method.java:597)
  794. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  795. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  796.  
  797. "IPC Server handler 23 on 60020" daemon prio=10 tid=0x00007f6f1417f000 nid=0x6531 in Object.wait() [0x00007f6f065e4000]
  798. java.lang.Thread.State: BLOCKED (on object monitor)
  799. at java.lang.Object.wait(Native Method)
  800. at java.lang.Object.wait(Object.java:485)
  801. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.waitForAckedSeqno(DFSClient.java:3284)
  802. - locked <0x00000005db9b7800> (a java.util.LinkedList)
  803. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.sync(DFSClient.java:3201)
  804. at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:97)
  805. at org.apache.hadoop.io.SequenceFile$Writer.syncFs(SequenceFile.java:944)
  806. at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
  807. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  808. at java.lang.reflect.Method.invoke(Method.java:597)
  809. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:123)
  810. at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:906)
  811. at org.apache.hadoop.hbase.regionserver.wal.HLog.append(HLog.java:857)
  812. at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1402)
  813. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1318)
  814. at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1803)
  815. at org.apache.hadoop.hbase.regionserver.HRegionServer.multiPut(HRegionServer.java:2464)
  816. at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
  817. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  818. at java.lang.reflect.Method.invoke(Method.java:597)
  819. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  820. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  821.  
  822. "IPC Server handler 22 on 60020" daemon prio=10 tid=0x00007f6f1417d000 nid=0x6530 runnable [0x00007f6f066e5000]
  823. java.lang.Thread.State: RUNNABLE
  824. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  825. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  826. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  827. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  828. - locked <0x0000000621ec34d8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  829. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  830. - locked <0x0000000621ec34d8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  831. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  832. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  833. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  834. - locked <0x0000000621d47620> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  835. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  836. - locked <0x0000000621d47620> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  837. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  838. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  839. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  840. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  841. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  842. at java.lang.reflect.Method.invoke(Method.java:597)
  843. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  844. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  845.  
  846. "IPC Server handler 21 on 60020" daemon prio=10 tid=0x00007f6f1417b800 nid=0x652f runnable [0x00007f6f067e5000]
  847. java.lang.Thread.State: RUNNABLE
  848. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  849. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  850. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  851. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  852. - locked <0x0000000621e934e8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  853. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  854. - locked <0x0000000621e934e8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  855. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  856. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  857. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  858. - locked <0x0000000621ea6200> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  859. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  860. - locked <0x0000000621ea6200> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  861. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  862. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  863. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  864. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  865. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  866. at java.lang.reflect.Method.invoke(Method.java:597)
  867. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  868. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  869.  
  870. "IPC Server handler 20 on 60020" daemon prio=10 tid=0x00007f6f1416e800 nid=0x652e in Object.wait() [0x00007f6f068e7000]
  871. java.lang.Thread.State: BLOCKED (on object monitor)
  872. at java.lang.Object.wait(Native Method)
  873. at java.lang.Object.wait(Object.java:485)
  874. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.waitForAckedSeqno(DFSClient.java:3284)
  875. - locked <0x00000005db9b7800> (a java.util.LinkedList)
  876. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.sync(DFSClient.java:3201)
  877. at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:97)
  878. at org.apache.hadoop.io.SequenceFile$Writer.syncFs(SequenceFile.java:944)
  879. at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
  880. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  881. at java.lang.reflect.Method.invoke(Method.java:597)
  882. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:123)
  883. at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:906)
  884. at org.apache.hadoop.hbase.regionserver.wal.HLog.append(HLog.java:857)
  885. at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1402)
  886. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1318)
  887. at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1803)
  888. at org.apache.hadoop.hbase.regionserver.HRegionServer.multiPut(HRegionServer.java:2464)
  889. at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
  890. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  891. at java.lang.reflect.Method.invoke(Method.java:597)
  892. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  893. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  894.  
  895. "IPC Server handler 19 on 60020" daemon prio=10 tid=0x00007f6f1416c800 nid=0x652d in Object.wait() [0x00007f6f069e8000]
  896. java.lang.Thread.State: BLOCKED (on object monitor)
  897. at java.lang.Object.wait(Native Method)
  898. at java.lang.Object.wait(Object.java:485)
  899. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.waitForAckedSeqno(DFSClient.java:3284)
  900. - locked <0x00000005db9b7800> (a java.util.LinkedList)
  901. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.sync(DFSClient.java:3201)
  902. at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:97)
  903. at org.apache.hadoop.io.SequenceFile$Writer.syncFs(SequenceFile.java:944)
  904. at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
  905. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  906. at java.lang.reflect.Method.invoke(Method.java:597)
  907. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:123)
  908. at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:906)
  909. at org.apache.hadoop.hbase.regionserver.wal.HLog.append(HLog.java:857)
  910. at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1402)
  911. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1318)
  912. at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1803)
  913. at org.apache.hadoop.hbase.regionserver.HRegionServer.multiPut(HRegionServer.java:2464)
  914. at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
  915. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  916. at java.lang.reflect.Method.invoke(Method.java:597)
  917. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  918. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  919.  
  920. "IPC Server handler 18 on 60020" daemon prio=10 tid=0x00007f6f1416a800 nid=0x652c runnable [0x00007f6f06ae9000]
  921. java.lang.Thread.State: RUNNABLE
  922. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  923. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  924. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  925. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  926. - locked <0x000000050dec7be0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  927. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  928. - locked <0x000000050dec7be0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  929. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  930. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  931. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  932. - locked <0x000000050dec84d8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  933. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  934. - locked <0x000000050dec84d8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  935. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  936. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  937. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  938. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  939. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  940. at java.lang.reflect.Method.invoke(Method.java:597)
  941. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  942. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  943.  
  944. "IPC Server handler 17 on 60020" daemon prio=10 tid=0x00007f6f14168800 nid=0x652b runnable [0x00007f6f06bea000]
  945. java.lang.Thread.State: RUNNABLE
  946. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  947. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  948. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  949. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  950. - locked <0x000000050df144f0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  951. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  952. - locked <0x000000050df144f0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  953. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  954. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  955. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  956. - locked <0x000000050df14de8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  957. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  958. - locked <0x000000050df14de8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  959. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  960. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  961. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  962. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  963. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  964. at java.lang.reflect.Method.invoke(Method.java:597)
  965. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  966. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  967.  
  968. "IPC Server handler 16 on 60020" daemon prio=10 tid=0x00007f6f14166800 nid=0x652a in Object.wait() [0x00007f6f06ceb000]
  969. java.lang.Thread.State: BLOCKED (on object monitor)
  970. at java.lang.Object.wait(Native Method)
  971. at java.lang.Object.wait(Object.java:485)
  972. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.waitForAckedSeqno(DFSClient.java:3284)
  973. - locked <0x00000005db9b7800> (a java.util.LinkedList)
  974. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.sync(DFSClient.java:3201)
  975. at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:97)
  976. at org.apache.hadoop.io.SequenceFile$Writer.syncFs(SequenceFile.java:944)
  977. at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
  978. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  979. at java.lang.reflect.Method.invoke(Method.java:597)
  980. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:123)
  981. at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:906)
  982. at org.apache.hadoop.hbase.regionserver.wal.HLog.append(HLog.java:857)
  983. at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1402)
  984. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1318)
  985. at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1803)
  986. at org.apache.hadoop.hbase.regionserver.HRegionServer.multiPut(HRegionServer.java:2464)
  987. at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
  988. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  989. at java.lang.reflect.Method.invoke(Method.java:597)
  990. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  991. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  992.  
  993. "IPC Server handler 15 on 60020" daemon prio=10 tid=0x00007f6f14164000 nid=0x6529 runnable [0x00007f6f06dec000]
  994. java.lang.Thread.State: RUNNABLE
  995. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  996. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  997. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  998. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  999. - locked <0x000000050deede28> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  1000. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  1001. - locked <0x000000050deede28> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  1002. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  1003. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  1004. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  1005. - locked <0x000000050deee6f0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  1006. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  1007. - locked <0x000000050deee6f0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  1008. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  1009. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  1010. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  1011. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  1012. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  1013. at java.lang.reflect.Method.invoke(Method.java:597)
  1014. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  1015. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  1016.  
  1017. "IPC Server handler 14 on 60020" daemon prio=10 tid=0x00007f6f14162000 nid=0x6528 runnable [0x00007f6f06eed000]
  1018. java.lang.Thread.State: RUNNABLE
  1019. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  1020. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  1021. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  1022. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  1023. - locked <0x000000050ce33850> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  1024. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  1025. - locked <0x000000050ce33850> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  1026. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  1027. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  1028. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  1029. - locked <0x000000050ce33750> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  1030. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  1031. - locked <0x000000050ce33750> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  1032. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  1033. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  1034. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  1035. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  1036. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  1037. at java.lang.reflect.Method.invoke(Method.java:597)
  1038. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  1039. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  1040.  
  1041. "IPC Server handler 13 on 60020" daemon prio=10 tid=0x00007f6f14160000 nid=0x6527 runnable [0x00007f6f06fed000]
  1042. java.lang.Thread.State: RUNNABLE
  1043. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  1044. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  1045. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  1046. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  1047. - locked <0x0000000621ec36f8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  1048. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  1049. - locked <0x0000000621ec36f8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  1050. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  1051. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  1052. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  1053. - locked <0x0000000621d47850> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  1054. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  1055. - locked <0x0000000621d47850> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  1056. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  1057. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  1058. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  1059. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  1060. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  1061. at java.lang.reflect.Method.invoke(Method.java:597)
  1062. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  1063. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  1064.  
  1065. "IPC Server handler 12 on 60020" daemon prio=10 tid=0x00007f6f1415e800 nid=0x6526 runnable [0x00007f6f070ee000]
  1066. java.lang.Thread.State: RUNNABLE
  1067. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:566)
  1068. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  1069. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  1070. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  1071. - locked <0x0000000621e87b48> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  1072. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  1073. - locked <0x0000000621e87b48> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  1074. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  1075. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  1076. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  1077. - locked <0x0000000621d470e0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  1078. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  1079. - locked <0x0000000621d470e0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  1080. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  1081. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  1082. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  1083. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  1084. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  1085. at java.lang.reflect.Method.invoke(Method.java:597)
  1086. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  1087. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  1088.  
  1089. "IPC Server handler 11 on 60020" daemon prio=10 tid=0x00007f6f14152800 nid=0x6525 in Object.wait() [0x00007f6f071f0000]
  1090. java.lang.Thread.State: BLOCKED (on object monitor)
  1091. at java.lang.Object.wait(Native Method)
  1092. at java.lang.Object.wait(Object.java:485)
  1093. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.waitForAckedSeqno(DFSClient.java:3284)
  1094. - locked <0x00000005db9b7800> (a java.util.LinkedList)
  1095. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.sync(DFSClient.java:3201)
  1096. at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:97)
  1097. at org.apache.hadoop.io.SequenceFile$Writer.syncFs(SequenceFile.java:944)
  1098. at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
  1099. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  1100. at java.lang.reflect.Method.invoke(Method.java:597)
  1101. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:123)
  1102. at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:906)
  1103. at org.apache.hadoop.hbase.regionserver.wal.HLog.append(HLog.java:857)
  1104. at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1402)
  1105. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1318)
  1106. at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1803)
  1107. at org.apache.hadoop.hbase.regionserver.HRegionServer.multiPut(HRegionServer.java:2464)
  1108. at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
  1109. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  1110. at java.lang.reflect.Method.invoke(Method.java:597)
  1111. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  1112. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  1113.  
  1114. "IPC Server handler 10 on 60020" daemon prio=10 tid=0x00007f6f14150800 nid=0x6524 runnable [0x00007f6f072f1000]
  1115. java.lang.Thread.State: RUNNABLE
  1116. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  1117. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  1118. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  1119. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  1120. - locked <0x0000000621ec3158> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  1121. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  1122. - locked <0x0000000621ec3158> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  1123. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  1124. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  1125. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  1126. - locked <0x0000000621d47b28> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  1127. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  1128. - locked <0x0000000621d47b28> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  1129. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  1130. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  1131. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  1132. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  1133. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  1134. at java.lang.reflect.Method.invoke(Method.java:597)
  1135. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  1136. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  1137.  
  1138. "IPC Server handler 9 on 60020" daemon prio=10 tid=0x00007f6f14247800 nid=0x6523 runnable [0x00007f6f073f1000]
  1139. java.lang.Thread.State: RUNNABLE
  1140. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1141. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  1142. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  1143. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  1144. - locked <0x000000061b2d60c8> (a sun.nio.ch.Util$2)
  1145. - locked <0x000000061b2d60b0> (a java.util.Collections$UnmodifiableSet)
  1146. - locked <0x000000061b2c8f08> (a sun.nio.ch.EPollSelectorImpl)
  1147. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  1148. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:332)
  1149. at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:203)
  1150. at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:408)
  1151. at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.fetchBlockByteRange(DFSClient.java:1890)
  1152. at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1960)
  1153. at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:46)
  1154. at org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:101)
  1155. at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
  1156. at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
  1157. - locked <0x000000050d017b88> (a java.io.BufferedInputStream)
  1158. at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
  1159. at org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1085)
  1160. at org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1027)
  1161. - locked <0x00000005194ea7f0> (a [B)
  1162. at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.java:1433)
  1163. at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1290)
  1164. at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:136)
  1165. at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:96)
  1166. at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:75)
  1167. at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1202)
  1168. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.<init>(HRegion.java:2209)
  1169. at org.apache.hadoop.hbase.regionserver.HRegion.instantiateInternalScanner(HRegion.java:1063)
  1170. at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1055)
  1171. at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1039)
  1172. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2942)
  1173. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  1174. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  1175. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  1176. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  1177. at java.lang.reflect.Method.invoke(Method.java:597)
  1178. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  1179. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  1180.  
  1181. "IPC Server handler 8 on 60020" daemon prio=10 tid=0x00007f6f14245800 nid=0x6522 in Object.wait() [0x00007f6f074f3000]
  1182. java.lang.Thread.State: BLOCKED (on object monitor)
  1183. at java.lang.Object.wait(Native Method)
  1184. at java.lang.Object.wait(Object.java:485)
  1185. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.waitForAckedSeqno(DFSClient.java:3284)
  1186. - locked <0x00000005db9b7800> (a java.util.LinkedList)
  1187. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.sync(DFSClient.java:3201)
  1188. at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:97)
  1189. at org.apache.hadoop.io.SequenceFile$Writer.syncFs(SequenceFile.java:944)
  1190. at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
  1191. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  1192. at java.lang.reflect.Method.invoke(Method.java:597)
  1193. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:123)
  1194. at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:906)
  1195. at org.apache.hadoop.hbase.regionserver.wal.HLog.append(HLog.java:857)
  1196. at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1402)
  1197. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1318)
  1198. at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1803)
  1199. at org.apache.hadoop.hbase.regionserver.HRegionServer.multiPut(HRegionServer.java:2464)
  1200. at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
  1201. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  1202. at java.lang.reflect.Method.invoke(Method.java:597)
  1203. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  1204. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  1205.  
  1206. "IPC Server handler 7 on 60020" daemon prio=10 tid=0x00007f6f14243800 nid=0x6521 in Object.wait() [0x00007f6f075f4000]
  1207. java.lang.Thread.State: BLOCKED (on object monitor)
  1208. at java.lang.Object.wait(Native Method)
  1209. at java.lang.Object.wait(Object.java:485)
  1210. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.waitForAckedSeqno(DFSClient.java:3284)
  1211. - locked <0x00000005db9b7800> (a java.util.LinkedList)
  1212. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.sync(DFSClient.java:3201)
  1213. at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:97)
  1214. at org.apache.hadoop.io.SequenceFile$Writer.syncFs(SequenceFile.java:944)
  1215. at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
  1216. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  1217. at java.lang.reflect.Method.invoke(Method.java:597)
  1218. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:123)
  1219. at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:906)
  1220. at org.apache.hadoop.hbase.regionserver.wal.HLog.append(HLog.java:857)
  1221. at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1402)
  1222. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1318)
  1223. at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1803)
  1224. at org.apache.hadoop.hbase.regionserver.HRegionServer.multiPut(HRegionServer.java:2464)
  1225. at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
  1226. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  1227. at java.lang.reflect.Method.invoke(Method.java:597)
  1228. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  1229. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  1230.  
  1231. "IPC Server handler 6 on 60020" daemon prio=10 tid=0x00007f6f14241800 nid=0x6520 runnable [0x00007f6f076f4000]
  1232. java.lang.Thread.State: RUNNABLE
  1233. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1234. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  1235. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  1236. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  1237. - locked <0x000000061da31d48> (a sun.nio.ch.Util$2)
  1238. - locked <0x000000061da31d30> (a java.util.Collections$UnmodifiableSet)
  1239. - locked <0x000000061d8dd710> (a sun.nio.ch.EPollSelectorImpl)
  1240. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  1241. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:332)
  1242. at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:203)
  1243. at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:408)
  1244. at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1680)
  1245. - locked <0x0000000517451418> (a org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
  1246. at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1803)
  1247. - locked <0x0000000517451418> (a org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
  1248. at java.io.DataInputStream.read(DataInputStream.java:132)
  1249. at org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:105)
  1250. - locked <0x000000051745a108> (a org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
  1251. at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
  1252. at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
  1253. - locked <0x000000050d009f20> (a java.io.BufferedInputStream)
  1254. at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
  1255. at org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1085)
  1256. at org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1027)
  1257. - locked <0x00000005170863b0> (a [B)
  1258. at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.java:1433)
  1259. at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1290)
  1260. at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:136)
  1261. at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:96)
  1262. at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:75)
  1263. at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1202)
  1264. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.<init>(HRegion.java:2209)
  1265. at org.apache.hadoop.hbase.regionserver.HRegion.instantiateInternalScanner(HRegion.java:1063)
  1266. at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1055)
  1267. at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1039)
  1268. at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1887)
  1269. at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
  1270. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  1271. at java.lang.reflect.Method.invoke(Method.java:597)
  1272. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  1273. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  1274.  
  1275. "IPC Server handler 5 on 60020" daemon prio=10 tid=0x00007f6f1423f800 nid=0x651f runnable [0x00007f6f077f6000]
  1276. java.lang.Thread.State: RUNNABLE
  1277. at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.blockSeek(HFile.java:1336)
  1278. at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1291)
  1279. at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:136)
  1280. at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:96)
  1281. at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:75)
  1282. at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1202)
  1283. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.<init>(HRegion.java:2209)
  1284. at org.apache.hadoop.hbase.regionserver.HRegion.instantiateInternalScanner(HRegion.java:1063)
  1285. at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1055)
  1286. at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1039)
  1287. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2942)
  1288. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  1289. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  1290. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  1291. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  1292. at java.lang.reflect.Method.invoke(Method.java:597)
  1293. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  1294. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  1295.  
  1296. "IPC Server handler 4 on 60020" daemon prio=10 tid=0x00007f6f14222800 nid=0x651e runnable [0x00007f6f078f6000]
  1297. java.lang.Thread.State: RUNNABLE
  1298. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  1299. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  1300. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  1301. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  1302. - locked <0x0000000621ec2e98> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  1303. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  1304. - locked <0x0000000621ec2e98> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  1305. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  1306. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  1307. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  1308. - locked <0x0000000621d47d58> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  1309. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  1310. - locked <0x0000000621d47d58> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  1311. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  1312. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  1313. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  1314. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  1315. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  1316. at java.lang.reflect.Method.invoke(Method.java:597)
  1317. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  1318. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  1319.  
  1320. "IPC Server handler 3 on 60020" daemon prio=10 tid=0x00007f6f14221000 nid=0x651d runnable [0x00007f6f079f8000]
  1321. java.lang.Thread.State: RUNNABLE
  1322. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:74)
  1323. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:62)
  1324. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:561)
  1325. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  1326. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  1327. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  1328. - locked <0x000000050df15428> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  1329. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  1330. - locked <0x000000050df15428> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  1331. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  1332. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  1333. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  1334. - locked <0x000000050df15d20> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  1335. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  1336. - locked <0x000000050df15d20> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  1337. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  1338. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  1339. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  1340. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  1341. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  1342. at java.lang.reflect.Method.invoke(Method.java:597)
  1343. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  1344. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  1345.  
  1346. "IPC Server handler 2 on 60020" daemon prio=10 tid=0x00007f6f1421f800 nid=0x651c runnable [0x00007f6f07af9000]
  1347. java.lang.Thread.State: RUNNABLE
  1348. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:566)
  1349. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  1350. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  1351. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  1352. - locked <0x0000000621eb6b18> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  1353. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  1354. - locked <0x0000000621eb6b18> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  1355. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  1356. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  1357. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  1358. - locked <0x0000000621eb6c38> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  1359. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  1360. - locked <0x0000000621eb6c38> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  1361. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  1362. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  1363. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  1364. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  1365. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  1366. at java.lang.reflect.Method.invoke(Method.java:597)
  1367. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  1368. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  1369.  
  1370. "IPC Server handler 1 on 60020" daemon prio=10 tid=0x00007f6f1421b000 nid=0x651b runnable [0x00007f6f07bfa000]
  1371. java.lang.Thread.State: RUNNABLE
  1372. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  1373. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  1374. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  1375. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  1376. - locked <0x000000050dea10f0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  1377. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  1378. - locked <0x000000050dea10f0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  1379. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  1380. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  1381. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  1382. - locked <0x000000050dea19e8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  1383. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  1384. - locked <0x000000050dea19e8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  1385. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  1386. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  1387. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  1388. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  1389. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  1390. at java.lang.reflect.Method.invoke(Method.java:597)
  1391. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  1392. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  1393.  
  1394. "IPC Server handler 0 on 60020" daemon prio=10 tid=0x00007f6f14219800 nid=0x651a runnable [0x00007f6f07cfb000]
  1395. java.lang.Thread.State: RUNNABLE
  1396. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:566)
  1397. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  1398. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  1399. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  1400. - locked <0x0000000621ec32b8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  1401. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  1402. - locked <0x0000000621ec32b8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  1403. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  1404. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  1405. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  1406. - locked <0x0000000621d47fc0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  1407. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  1408. - locked <0x0000000621d47fc0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  1409. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  1410. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  1411. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  1412. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  1413. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  1414. at java.lang.reflect.Method.invoke(Method.java:597)
  1415. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  1416. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  1417.  
  1418. "IPC Server listener on 60020" daemon prio=10 tid=0x00007f6f14215000 nid=0x6519 runnable [0x00007f6f07dfc000]
  1419. java.lang.Thread.State: RUNNABLE
  1420. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1421. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  1422. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  1423. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  1424. - locked <0x0000000513554d80> (a sun.nio.ch.Util$2)
  1425. - locked <0x0000000513554d98> (a java.util.Collections$UnmodifiableSet)
  1426. - locked <0x0000000513554100> (a sun.nio.ch.EPollSelectorImpl)
  1427. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  1428. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
  1429. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:402)
  1430.  
  1431. "IPC Server Responder" daemon prio=10 tid=0x00007f6f1421c800 nid=0x6518 runnable [0x00007f6f07efd000]
  1432. java.lang.Thread.State: RUNNABLE
  1433. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1434. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  1435. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  1436. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  1437. - locked <0x0000000513554f00> (a sun.nio.ch.Util$2)
  1438. - locked <0x0000000513554f18> (a java.util.Collections$UnmodifiableSet)
  1439. - locked <0x0000000513554900> (a sun.nio.ch.EPollSelectorImpl)
  1440. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  1441. at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:576)
  1442.  
  1443. "Timer-0" daemon prio=10 tid=0x00007f6f14213000 nid=0x6517 in Object.wait() [0x00007f6f07ffe000]
  1444. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  1445. at java.lang.Object.wait(Native Method)
  1446. at java.util.TimerThread.mainLoop(Timer.java:509)
  1447. - locked <0x00000005137d7d70> (a java.util.TaskQueue)
  1448. at java.util.TimerThread.run(Timer.java:462)
  1449.  
  1450. "691789110@qtp-1373164447-1 - Acceptor0 SelectChannelConnector@0.0.0.0:60030" prio=10 tid=0x00007f6f14115800 nid=0x6516 runnable [0x00007f6f5c103000]
  1451. java.lang.Thread.State: RUNNABLE
  1452. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1453. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  1454. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  1455. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  1456. - locked <0x000000051386c608> (a sun.nio.ch.Util$2)
  1457. - locked <0x000000051386c5f0> (a java.util.Collections$UnmodifiableSet)
  1458. - locked <0x0000000513772568> (a sun.nio.ch.EPollSelectorImpl)
  1459. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  1460. at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:484)
  1461. at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:191)
  1462. at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
  1463. at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
  1464. at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
  1465.  
  1466. "589411738@qtp-1373164447-0" prio=10 tid=0x00007f6f1412a000 nid=0x6515 in Object.wait() [0x00007f6f5c204000]
  1467. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  1468. at java.lang.Object.wait(Native Method)
  1469. at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
  1470. - locked <0x0000000513826e10> (a org.mortbay.thread.QueuedThreadPool$PoolThread)
  1471.  
  1472. "regionserver60020.leaseChecker" prio=10 tid=0x00007f6f1410a800 nid=0x6514 runnable [0x00007f6f5c305000]
  1473. java.lang.Thread.State: TIMED_WAITING (parking)
  1474. at sun.misc.Unsafe.park(Native Method)
  1475. - parking to wait for <0x0000000513554f30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1476. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
  1477. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
  1478. at java.util.concurrent.DelayQueue.poll(DelayQueue.java:209)
  1479. at org.apache.hadoop.hbase.Leases.run(Leases.java:81)
  1480.  
  1481. "regionserver60020.majorCompactionChecker" daemon prio=10 tid=0x00007f6f14108800 nid=0x6513 in Object.wait() [0x00007f6f5c406000]
  1482. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  1483. at java.lang.Object.wait(Native Method)
  1484. - waiting on <0x0000000513554db0> (a java.lang.Object)
  1485. at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
  1486. - locked <0x0000000513554db0> (a java.lang.Object)
  1487. at org.apache.hadoop.hbase.Chore.run(Chore.java:76)
  1488.  
  1489. "regionserver60020.worker" daemon prio=10 tid=0x00007f6f14106800 nid=0x6512 waiting on condition [0x00007f6f5c507000]
  1490. java.lang.Thread.State: TIMED_WAITING (parking)
  1491. at sun.misc.Unsafe.park(Native Method)
  1492. - parking to wait for <0x0000000513554f90> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1493. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
  1494. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
  1495. at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424)
  1496. at org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1428)
  1497. at java.lang.Thread.run(Thread.java:662)
  1498.  
  1499. "regionserver60020.compactor" daemon prio=10 tid=0x00007f6f14104800 nid=0x6511 waiting on condition [0x00007f6f5c608000]
  1500. java.lang.Thread.State: TIMED_WAITING (parking)
  1501. at sun.misc.Unsafe.park(Native Method)
  1502. - parking to wait for <0x0000000513554dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1503. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
  1504. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
  1505. at java.util.concurrent.PriorityBlockingQueue.poll(PriorityBlockingQueue.java:245)
  1506. at org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue.poll(PriorityCompactionQueue.java:244)
  1507. at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:99)
  1508.  
  1509. "regionserver60020.cacheFlusher" daemon prio=10 tid=0x00007f6f14102800 nid=0x6510 waiting on condition [0x00007f6f5c709000]
  1510. java.lang.Thread.State: TIMED_WAITING (parking)
  1511. at sun.misc.Unsafe.park(Native Method)
  1512. - parking to wait for <0x0000000513554e28> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1513. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
  1514. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
  1515. at java.util.concurrent.DelayQueue.poll(DelayQueue.java:201)
  1516. at java.util.concurrent.DelayQueue.poll(DelayQueue.java:39)
  1517. at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:142)
  1518.  
  1519. "regionserver60020.logRoller" daemon prio=10 tid=0x00007f6f14101800 nid=0x650f in Object.wait() [0x00007f6f5c80a000]
  1520. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  1521. at java.lang.Object.wait(Native Method)
  1522. at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:67)
  1523. - locked <0x0000000513554e70> (a java.util.concurrent.atomic.AtomicBoolean)
  1524.  
  1525. "Timer thread for monitoring jvm" daemon prio=10 tid=0x00007f6f140c5800 nid=0x650e in Object.wait() [0x00007f6f5c90b000]
  1526. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  1527. at java.lang.Object.wait(Native Method)
  1528. at java.util.TimerThread.mainLoop(Timer.java:509)
  1529. - locked <0x0000000513822ea8> (a java.util.TaskQueue)
  1530. at java.util.TimerThread.run(Timer.java:462)
  1531.  
  1532. "Timer thread for monitoring hbase" daemon prio=10 tid=0x00007f6f140c3800 nid=0x650d in Object.wait() [0x00007f6f5ca0c000]
  1533. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  1534. at java.lang.Object.wait(Native Method)
  1535. at java.util.TimerThread.mainLoop(Timer.java:509)
  1536. - locked <0x0000000513822ed8> (a java.util.TaskQueue)
  1537. at java.util.TimerThread.run(Timer.java:462)
  1538.  
  1539. "regionserver60020.logSyncer" daemon prio=10 tid=0x00007f6f140b2000 nid=0x650c waiting on condition [0x00007f6f5cc3d000]
  1540. java.lang.Thread.State: TIMED_WAITING (sleeping)
  1541. at java.lang.Thread.sleep(Native Method)
  1542. at org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:882)
  1543.  
  1544. "LeaseChecker" daemon prio=10 tid=0x00007f6f140b1000 nid=0x650b waiting on condition [0x00007f6f5cb0d000]
  1545. java.lang.Thread.State: TIMED_WAITING (sleeping)
  1546. at java.lang.Thread.sleep(Native Method)
  1547. at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1071)
  1548. at java.lang.Thread.run(Thread.java:662)
  1549.  
  1550. "regionserver60020-EventThread" daemon prio=10 tid=0x00007f6f14082800 nid=0x6509 waiting on condition [0x00007f6f5ce3f000]
  1551. java.lang.Thread.State: WAITING (parking)
  1552. at sun.misc.Unsafe.park(Native Method)
  1553. - parking to wait for <0x00000005139bbd38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1554. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  1555. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  1556. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  1557. at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:477)
  1558.  
  1559. "regionserver60020-SendThread(mtag3:2181)" daemon prio=10 tid=0x00007f6f14080800 nid=0x6508 runnable [0x00007f6f5cf40000]
  1560. java.lang.Thread.State: RUNNABLE
  1561. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1562. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  1563. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  1564. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  1565. - locked <0x00000005139bbd20> (a sun.nio.ch.Util$2)
  1566. - locked <0x00000005139bbd08> (a java.util.Collections$UnmodifiableSet)
  1567. - locked <0x0000000513771ea8> (a sun.nio.ch.EPollSelectorImpl)
  1568. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  1569. at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1066)
  1570.  
  1571. "sendParams-0" daemon prio=10 tid=0x00007f6f14023800 nid=0x6506 waiting on condition [0x00007f6f5d142000]
  1572. java.lang.Thread.State: TIMED_WAITING (parking)
  1573. at sun.misc.Unsafe.park(Native Method)
  1574. - parking to wait for <0x0000000513a1c720> (a java.util.concurrent.SynchronousQueue$TransferStack)
  1575. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
  1576. at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
  1577. at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
  1578. at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
  1579. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
  1580. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  1581. at java.lang.Thread.run(Thread.java:662)
  1582.  
  1583. "IPC Client (47) connection to /10.103.7.2:60000 from an unknown user" daemon prio=10 tid=0x00007f6f14002800 nid=0x6501 in Object.wait() [0x00007f6f5d344000]
  1584. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  1585. at java.lang.Object.wait(Native Method)
  1586. at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.waitForWork(HBaseClient.java:408)
  1587. - locked <0x000000051360cd50> (a org.apache.hadoop.hbase.ipc.HBaseClient$Connection)
  1588. at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:453)
  1589.  
  1590. "DestroyJavaVM" prio=10 tid=0x00007f6f9c009000 nid=0x64dd waiting on condition [0x0000000000000000]
  1591. java.lang.Thread.State: RUNNABLE
  1592.  
  1593. "regionserver60020" prio=10 tid=0x00007f6f9c65c000 nid=0x64fa runnable [0x00007f6f5d445000]
  1594. java.lang.Thread.State: RUNNABLE
  1595. at org.apache.hadoop.hbase.regionserver.Store.getStorefilesIndexSize(Store.java:1241)
  1596. at org.apache.hadoop.hbase.regionserver.HRegionServer.createRegionLoad(HRegionServer.java:856)
  1597. - locked <0x000000052ec61878> (a java.util.concurrent.ConcurrentSkipListMap)
  1598. at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:556)
  1599. at java.lang.Thread.run(Thread.java:662)
  1600.  
  1601. "main-EventThread" daemon prio=10 tid=0x00007f6f9c65a000 nid=0x64f9 waiting on condition [0x00007f6f5d546000]
  1602. java.lang.Thread.State: WAITING (parking)
  1603. at sun.misc.Unsafe.park(Native Method)
  1604. - parking to wait for <0x0000000513557fd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1605. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  1606. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  1607. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  1608. at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:477)
  1609.  
  1610. "main-SendThread(mtag3:2181)" daemon prio=10 tid=0x00007f6f9c64f800 nid=0x64f8 runnable [0x00007f6f5d647000]
  1611. java.lang.Thread.State: RUNNABLE
  1612. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1613. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  1614. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  1615. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  1616. - locked <0x00000005135594b8> (a sun.nio.ch.Util$2)
  1617. - locked <0x00000005135594a0> (a java.util.Collections$UnmodifiableSet)
  1618. - locked <0x0000000513554148> (a sun.nio.ch.EPollSelectorImpl)
  1619. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  1620. at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1066)
  1621.  
  1622. "Timer thread for monitoring rpc" daemon prio=10 tid=0x00007f6f9c60d000 nid=0x64f6 in Object.wait() [0x00007f6f5db67000]
  1623. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  1624. at java.lang.Object.wait(Native Method)
  1625. at java.util.TimerThread.mainLoop(Timer.java:509)
  1626. - locked <0x0000000513551978> (a java.util.TaskQueue)
  1627. at java.util.TimerThread.run(Timer.java:462)
  1628.  
  1629. "pool-1-thread-10" prio=10 tid=0x00007f6f9c60a800 nid=0x64f5 waiting on condition [0x00007f6f5dc68000]
  1630. java.lang.Thread.State: WAITING (parking)
  1631. at sun.misc.Unsafe.park(Native Method)
  1632. - parking to wait for <0x000000051367fdd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1633. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  1634. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  1635. at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
  1636. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:973)
  1637. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:934)
  1638. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:510)
  1639. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:304)
  1640. - locked <0x0000000513551eb0> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  1641. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  1642. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  1643. at java.lang.Thread.run(Thread.java:662)
  1644.  
  1645. "pool-1-thread-9" prio=10 tid=0x00007f6f9c5f0000 nid=0x64f4 waiting on condition [0x00007f6f5dd69000]
  1646. java.lang.Thread.State: WAITING (parking)
  1647. at sun.misc.Unsafe.park(Native Method)
  1648. - parking to wait for <0x000000051367fdd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1649. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  1650. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  1651. at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
  1652. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:973)
  1653. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:934)
  1654. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:510)
  1655. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:304)
  1656. - locked <0x0000000513551f10> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  1657. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  1658. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  1659. at java.lang.Thread.run(Thread.java:662)
  1660.  
  1661. "pool-1-thread-8" prio=10 tid=0x00007f6f9c5d5800 nid=0x64f3 runnable [0x00007f6f5de6a000]
  1662. java.lang.Thread.State: WAITING (parking)
  1663. at sun.misc.Unsafe.park(Native Method)
  1664. - parking to wait for <0x000000051367fdd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1665. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  1666. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  1667. at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
  1668. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:973)
  1669. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:934)
  1670. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:510)
  1671. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:304)
  1672. - locked <0x0000000513551f70> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  1673. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  1674. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  1675. at java.lang.Thread.run(Thread.java:662)
  1676.  
  1677. "pool-1-thread-7" prio=10 tid=0x00007f6f9c5ba800 nid=0x64f2 waiting on condition [0x00007f6f5df6b000]
  1678. java.lang.Thread.State: WAITING (parking)
  1679. at sun.misc.Unsafe.park(Native Method)
  1680. - parking to wait for <0x000000051367fdd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1681. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  1682. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  1683. at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
  1684. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:973)
  1685. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:934)
  1686. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:510)
  1687. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:304)
  1688. - locked <0x0000000513579510> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  1689. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  1690. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  1691. at java.lang.Thread.run(Thread.java:662)
  1692.  
  1693. "pool-1-thread-6" prio=10 tid=0x00007f6f9c59f800 nid=0x64f1 waiting on condition [0x00007f6f5e06c000]
  1694. java.lang.Thread.State: WAITING (parking)
  1695. at sun.misc.Unsafe.park(Native Method)
  1696. - parking to wait for <0x000000051367fdd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1697. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  1698. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  1699. at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
  1700. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:973)
  1701. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:934)
  1702. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:510)
  1703. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:304)
  1704. - locked <0x0000000513579660> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  1705. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  1706. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  1707. at java.lang.Thread.run(Thread.java:662)
  1708.  
  1709. "pool-1-thread-5" prio=10 tid=0x00007f6f9c584800 nid=0x64f0 waiting on condition [0x00007f6f5e16d000]
  1710. java.lang.Thread.State: WAITING (parking)
  1711. at sun.misc.Unsafe.park(Native Method)
  1712. - parking to wait for <0x000000051367fdd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1713. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  1714. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  1715. at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
  1716. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:973)
  1717. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:934)
  1718. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:510)
  1719. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:304)
  1720. - locked <0x000000051357b630> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  1721. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  1722. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  1723. at java.lang.Thread.run(Thread.java:662)
  1724.  
  1725. "pool-1-thread-4" prio=10 tid=0x00007f6f9c569800 nid=0x64ef waiting on condition [0x00007f6f7817f000]
  1726. java.lang.Thread.State: WAITING (parking)
  1727. at sun.misc.Unsafe.park(Native Method)
  1728. - parking to wait for <0x000000051367fdd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1729. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  1730. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  1731. at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
  1732. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:973)
  1733. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:934)
  1734. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:510)
  1735. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:304)
  1736. - locked <0x000000051357b648> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  1737. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  1738. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  1739. at java.lang.Thread.run(Thread.java:662)
  1740.  
  1741. "pool-1-thread-3" prio=10 tid=0x00007f6f9c54f000 nid=0x64ee waiting on condition [0x00007f6f78280000]
  1742. java.lang.Thread.State: WAITING (parking)
  1743. at sun.misc.Unsafe.park(Native Method)
  1744. - parking to wait for <0x000000051367fdd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1745. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  1746. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  1747. at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
  1748. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:973)
  1749. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:934)
  1750. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:510)
  1751. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:304)
  1752. - locked <0x000000051357b660> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  1753. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  1754. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  1755. at java.lang.Thread.run(Thread.java:662)
  1756.  
  1757. "pool-1-thread-2" prio=10 tid=0x00007f6f9c4d9000 nid=0x64ed waiting on condition [0x00007f6f78381000]
  1758. java.lang.Thread.State: WAITING (parking)
  1759. at sun.misc.Unsafe.park(Native Method)
  1760. - parking to wait for <0x000000051367fdd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1761. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  1762. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  1763. at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
  1764. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:973)
  1765. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:934)
  1766. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:510)
  1767. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:304)
  1768. - locked <0x000000051357b678> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  1769. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  1770. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  1771. at java.lang.Thread.run(Thread.java:662)
  1772.  
  1773. "pool-1-thread-1" prio=10 tid=0x00007f6f9c4d8800 nid=0x64ec waiting on condition [0x00007f6f78482000]
  1774. java.lang.Thread.State: WAITING (parking)
  1775. at sun.misc.Unsafe.park(Native Method)
  1776. - parking to wait for <0x000000051367fdd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1777. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  1778. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  1779. at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
  1780. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:973)
  1781. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:934)
  1782. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:510)
  1783. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:304)
  1784. - locked <0x000000051357c5c0> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  1785. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  1786. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  1787. at java.lang.Thread.run(Thread.java:662)
  1788.  
  1789. "Low Memory Detector" daemon prio=10 tid=0x00007f6f9c15a800 nid=0x64ea runnable [0x0000000000000000]
  1790. java.lang.Thread.State: RUNNABLE
  1791.  
  1792. "CompilerThread1" daemon prio=10 tid=0x00007f6f9c158000 nid=0x64e9 waiting on condition [0x0000000000000000]
  1793. java.lang.Thread.State: RUNNABLE
  1794.  
  1795. "CompilerThread0" daemon prio=10 tid=0x00007f6f9c155800 nid=0x64e8 waiting on condition [0x0000000000000000]
  1796. java.lang.Thread.State: RUNNABLE
  1797.  
  1798. "Signal Dispatcher" daemon prio=10 tid=0x00007f6f9c153800 nid=0x64e7 waiting on condition [0x0000000000000000]
  1799. java.lang.Thread.State: RUNNABLE
  1800.  
  1801. "Surrogate Locker Thread (CMS)" daemon prio=10 tid=0x00007f6f9c151800 nid=0x64e6 waiting on condition [0x0000000000000000]
  1802. java.lang.Thread.State: RUNNABLE
  1803.  
  1804. "Finalizer" daemon prio=10 tid=0x00007f6f9c132800 nid=0x64e5 in Object.wait() [0x00007f6f98b64000]
  1805. java.lang.Thread.State: WAITING (on object monitor)
  1806. at java.lang.Object.wait(Native Method)
  1807. at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
  1808. - locked <0x000000051357c668> (a java.lang.ref.ReferenceQueue$Lock)
  1809. at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
  1810. at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
  1811.  
  1812. "Reference Handler" daemon prio=10 tid=0x00007f6f9c130800 nid=0x64e4 in Object.wait() [0x00007f6f98c65000]
  1813. java.lang.Thread.State: WAITING (on object monitor)
  1814. at java.lang.Object.wait(Native Method)
  1815. at java.lang.Object.wait(Object.java:485)
  1816. at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
  1817. - locked <0x0000000513551e50> (a java.lang.ref.Reference$Lock)
  1818.  
  1819. "VM Thread" prio=10 tid=0x00007f6f9c12c800 nid=0x64e3 runnable
  1820.  
  1821. "Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x00007f6f9c017800 nid=0x64de runnable
  1822.  
  1823. "Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x00007f6f9c019800 nid=0x64df runnable
  1824.  
  1825. "Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x00007f6f9c01b000 nid=0x64e0 runnable
  1826.  
  1827. "Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x00007f6f9c01d000 nid=0x64e1 runnable
  1828.  
  1829. "Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00007f6f9c0ac000 nid=0x64e2 runnable
  1830. "VM Periodic Task Thread" prio=10 tid=0x00007f6f9c165800 nid=0x64eb waiting on condition
  1831.  
  1832. JNI global references: 1230
  1833.  
  1834. Heap
  1835. par new generation total 19136K, used 3957K [0x000000050ce00000, 0x000000050e2c0000, 0x0000000512130000)
  1836. eden space 17024K, 16% used [0x000000050ce00000, 0x000000050d0b25f0, 0x000000050dea0000)
  1837. from space 2112K, 57% used [0x000000050dea0000, 0x000000050dfcd188, 0x000000050e0b0000)
  1838. to space 2112K, 0% used [0x000000050e0b0000, 0x000000050e0b0000, 0x000000050e2c0000)
  1839. concurrent mark-sweep generation total 5757924K, used 4453570K [0x0000000512130000, 0x0000000671829000, 0x00000007fae00000)
  1840. concurrent-mark-sweep perm gen total 30248K, used 18065K [0x00000007fae00000, 0x00000007fcb8a000, 0x0000000800000000)
  1841.  
  1842. 2011-05-14 15:51:20
  1843. Full thread dump Java HotSpot(TM) 64-Bit Server VM (19.0-b09 mixed mode):
  1844.  
  1845. "IPC Client (47) connection to mtag1/10.103.7.1:9000 from root" daemon prio=10 tid=0x00007f6f501ac000 nid=0xfa5 runnable [0x00007f6f5d041000]
  1846. java.lang.Thread.State: RUNNABLE
  1847. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1848. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  1849. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  1850. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  1851. - locked <0x000000061b2d60c8> (a sun.nio.ch.Util$2)
  1852. - locked <0x000000061b2d60b0> (a java.util.Collections$UnmodifiableSet)
  1853. - locked <0x000000061b2c8f08> (a sun.nio.ch.EPollSelectorImpl)
  1854. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  1855. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:332)
  1856. at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
  1857. at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
  1858. at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
  1859. at java.io.FilterInputStream.read(FilterInputStream.java:116)
  1860. at org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:304)
  1861. at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
  1862. at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
  1863. - locked <0x00000006187af310> (a java.io.BufferedInputStream)
  1864. at java.io.DataInputStream.readInt(DataInputStream.java:370)
  1865. at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:569)
  1866. at org.apache.hadoop.ipc.Client$Connection.run(Client.java:477)
  1867.  
  1868. "ResponseProcessor for block blk_1030279999998547108_1158319" daemon prio=10 tid=0x00007f6f5448f800 nid=0x24c0 runnable [0x00007f6f5cd3e000]
  1869. java.lang.Thread.State: RUNNABLE
  1870. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1871. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  1872. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  1873. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  1874. - locked <0x00000006173020d8> (a sun.nio.ch.Util$2)
  1875. - locked <0x00000006173020c0> (a java.util.Collections$UnmodifiableSet)
  1876. - locked <0x00000006173042d0> (a sun.nio.ch.EPollSelectorImpl)
  1877. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  1878. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:332)
  1879. at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
  1880. at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
  1881. at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
  1882. at java.io.DataInputStream.readFully(DataInputStream.java:178)
  1883. at java.io.DataInputStream.readLong(DataInputStream.java:399)
  1884. at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:119)
  1885. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2468)
  1886.  
  1887. "DataStreamer for file /hbase/.logs/mtag27.prod.imageshack.com,60020,1305408211142/10.103.7.27%3A60020.1305411438693 block blk_1030279999998547108_1158319" daemon prio=10 tid=0x00007f6f2c009000 nid=0x24bc in Object.wait() [0x00007f6f03dbc000]
  1888. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  1889. at java.lang.Object.wait(Native Method)
  1890. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2313)
  1891. - locked <0x00000005dbac2f88> (a java.util.LinkedList)
  1892.  
  1893. "pool-3-thread-1" prio=10 tid=0x00007f6f141cb000 nid=0x6557 waiting on condition [0x00007f6f03fbe000]
  1894. java.lang.Thread.State: TIMED_WAITING (parking)
  1895. at sun.misc.Unsafe.park(Native Method)
  1896. - parking to wait for <0x00000005138a8b30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1897. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
  1898. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
  1899. at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
  1900. at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
  1901. at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
  1902. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
  1903. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  1904. at java.lang.Thread.run(Thread.java:662)
  1905.  
  1906. "LruBlockCache.EvictionThread" daemon prio=10 tid=0x00007f6f141c9800 nid=0x6556 in Object.wait() [0x00007f6f040bf000]
  1907. java.lang.Thread.State: WAITING (on object monitor)
  1908. at java.lang.Object.wait(Native Method)
  1909. - waiting on <0x00000005138232e0> (a org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread)
  1910. at java.lang.Object.wait(Object.java:485)
  1911. at org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:514)
  1912. - locked <0x00000005138232e0> (a org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread)
  1913.  
  1914. "PRI IPC Server handler 9 on 60020" daemon prio=10 tid=0x00007f6f141c7800 nid=0x6555 waiting on condition [0x00007f6f041c0000]
  1915. java.lang.Thread.State: WAITING (parking)
  1916. at sun.misc.Unsafe.park(Native Method)
  1917. - parking to wait for <0x0000000513679408> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1918. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  1919. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  1920. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  1921. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1013)
  1922.  
  1923. "PRI IPC Server handler 8 on 60020" daemon prio=10 tid=0x00007f6f141c5800 nid=0x6554 waiting on condition [0x00007f6f042c1000]
  1924. java.lang.Thread.State: WAITING (parking)
  1925. at sun.misc.Unsafe.park(Native Method)
  1926. - parking to wait for <0x0000000513679408> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1927. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  1928. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  1929. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  1930. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1013)
  1931.  
  1932. "PRI IPC Server handler 7 on 60020" daemon prio=10 tid=0x00007f6f141c3800 nid=0x6553 waiting on condition [0x00007f6f043c2000]
  1933. java.lang.Thread.State: WAITING (parking)
  1934. at sun.misc.Unsafe.park(Native Method)
  1935. - parking to wait for <0x0000000513679408> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1936. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  1937. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  1938. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  1939. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1013)
  1940.  
  1941. "PRI IPC Server handler 6 on 60020" daemon prio=10 tid=0x00007f6f141c1800 nid=0x6552 waiting on condition [0x00007f6f044c3000]
  1942. java.lang.Thread.State: WAITING (parking)
  1943. at sun.misc.Unsafe.park(Native Method)
  1944. - parking to wait for <0x0000000513679408> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1945. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  1946. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  1947. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  1948. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1013)
  1949.  
  1950. "PRI IPC Server handler 5 on 60020" daemon prio=10 tid=0x00007f6f141bf800 nid=0x6551 waiting on condition [0x00007f6f045c4000]
  1951. java.lang.Thread.State: WAITING (parking)
  1952. at sun.misc.Unsafe.park(Native Method)
  1953. - parking to wait for <0x0000000513679408> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1954. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  1955. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  1956. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  1957. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1013)
  1958.  
  1959. "PRI IPC Server handler 4 on 60020" daemon prio=10 tid=0x00007f6f141bd800 nid=0x6550 waiting on condition [0x00007f6f046c5000]
  1960. java.lang.Thread.State: WAITING (parking)
  1961. at sun.misc.Unsafe.park(Native Method)
  1962. - parking to wait for <0x0000000513679408> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1963. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  1964. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  1965. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  1966. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1013)
  1967.  
  1968. "PRI IPC Server handler 3 on 60020" daemon prio=10 tid=0x00007f6f141bb800 nid=0x654f waiting on condition [0x00007f6f047c6000]
  1969. java.lang.Thread.State: WAITING (parking)
  1970. at sun.misc.Unsafe.park(Native Method)
  1971. - parking to wait for <0x0000000513679408> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1972. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  1973. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  1974. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  1975. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1013)
  1976.  
  1977. "PRI IPC Server handler 2 on 60020" daemon prio=10 tid=0x00007f6f141b9800 nid=0x654e waiting on condition [0x00007f6f048c7000]
  1978. java.lang.Thread.State: WAITING (parking)
  1979. at sun.misc.Unsafe.park(Native Method)
  1980. - parking to wait for <0x0000000513679408> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1981. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  1982. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  1983. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  1984. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1013)
  1985.  
  1986. "PRI IPC Server handler 1 on 60020" daemon prio=10 tid=0x00007f6f141b7800 nid=0x654d waiting on condition [0x00007f6f049c8000]
  1987. java.lang.Thread.State: WAITING (parking)
  1988. at sun.misc.Unsafe.park(Native Method)
  1989. - parking to wait for <0x0000000513679408> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1990. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  1991. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  1992. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  1993. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1013)
  1994.  
  1995. "PRI IPC Server handler 0 on 60020" daemon prio=10 tid=0x00007f6f141b5800 nid=0x654c waiting on condition [0x00007f6f04ac9000]
  1996. java.lang.Thread.State: WAITING (parking)
  1997. at sun.misc.Unsafe.park(Native Method)
  1998. - parking to wait for <0x0000000513679408> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1999. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  2000. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  2001. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  2002. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1013)
  2003.  
  2004. "IPC Server handler 49 on 60020" daemon prio=10 tid=0x00007f6f141b3800 nid=0x654b waiting for monitor entry [0x00007f6f04bca000]
  2005. java.lang.Thread.State: BLOCKED (on object monitor)
  2006. at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:507)
  2007. - waiting to lock <0x0000000618666ca8> (a java.lang.Object)
  2008. at org.apache.hadoop.ipc.Client.call(Client.java:790)
  2009. at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:221)
  2010. at $Proxy1.getFileInfo(Unknown Source)
  2011. at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
  2012. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2013. at java.lang.reflect.Method.invoke(Method.java:597)
  2014. at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
  2015. at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
  2016. at $Proxy1.getFileInfo(Unknown Source)
  2017. at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:624)
  2018. at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:461)
  2019. at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:690)
  2020. at org.apache.hadoop.hbase.util.FSUtils.checkFileSystemAvailable(FSUtils.java:119)
  2021. at org.apache.hadoop.hbase.regionserver.HRegionServer.checkFileSystem(HRegionServer.java:953)
  2022. at org.apache.hadoop.hbase.regionserver.HRegionServer.cleanup(HRegionServer.java:902)
  2023. at org.apache.hadoop.hbase.regionserver.HRegionServer.cleanup(HRegionServer.java:880)
  2024. at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1958)
  2025. at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
  2026. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2027. at java.lang.reflect.Method.invoke(Method.java:597)
  2028. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  2029. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  2030.  
  2031. "IPC Server handler 48 on 60020" daemon prio=10 tid=0x00007f6f141b1800 nid=0x654a runnable [0x00007f6f04ccb000]
  2032. java.lang.Thread.State: RUNNABLE
  2033. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  2034. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1344)
  2035. at java.util.concurrent.ConcurrentSkipListMap$ComparableUsingComparator.compareTo(ConcurrentSkipListMap.java:606)
  2036. at java.util.concurrent.ConcurrentSkipListMap.findPredecessor(ConcurrentSkipListMap.java:685)
  2037. at java.util.concurrent.ConcurrentSkipListMap.doPut(ConcurrentSkipListMap.java:864)
  2038. at java.util.concurrent.ConcurrentSkipListMap.put(ConcurrentSkipListMap.java:1659)
  2039. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet.add(KeyValueSkipListSet.java:158)
  2040. at org.apache.hadoop.hbase.regionserver.MemStore.add(MemStore.java:189)
  2041. at org.apache.hadoop.hbase.regionserver.Store.add(Store.java:272)
  2042. at org.apache.hadoop.hbase.regionserver.HRegion.applyFamilyMapToMemstore(HRegion.java:1687)
  2043. at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1423)
  2044. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1318)
  2045. at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1803)
  2046. at org.apache.hadoop.hbase.regionserver.HRegionServer.multiPut(HRegionServer.java:2464)
  2047. at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
  2048. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2049. at java.lang.reflect.Method.invoke(Method.java:597)
  2050. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  2051. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  2052.  
  2053. "IPC Server handler 47 on 60020" daemon prio=10 tid=0x00007f6f141af000 nid=0x6549 runnable [0x00007f6f04dcc000]
  2054. java.lang.Thread.State: RUNNABLE
  2055. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:74)
  2056. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:62)
  2057. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:561)
  2058. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  2059. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  2060. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  2061. - locked <0x000000050d8a10d8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2062. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  2063. - locked <0x000000050d8a10d8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2064. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  2065. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  2066. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  2067. - locked <0x000000050d8a0fd8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2068. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  2069. - locked <0x000000050d8a0fd8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2070. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  2071. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  2072. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  2073. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  2074. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2075. at java.lang.reflect.Method.invoke(Method.java:597)
  2076. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  2077. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  2078.  
  2079. "IPC Server handler 46 on 60020" daemon prio=10 tid=0x00007f6f141ad000 nid=0x6548 runnable [0x00007f6f04ecd000]
  2080. java.lang.Thread.State: RUNNABLE
  2081. at java.util.concurrent.ConcurrentSkipListMap$SubMap$SubMapEntryIterator.next(ConcurrentSkipListMap.java:3113)
  2082. at java.util.concurrent.ConcurrentSkipListMap$SubMap$SubMapEntryIterator.next(ConcurrentSkipListMap.java:3108)
  2083. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:74)
  2084. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:62)
  2085. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:561)
  2086. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  2087. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  2088. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  2089. - locked <0x000000050e0ebb78> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2090. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  2091. - locked <0x000000050e0ebb78> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2092. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  2093. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  2094. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  2095. - locked <0x000000050e0ebbe0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2096. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  2097. - locked <0x000000050e0ebbe0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2098. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  2099. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  2100. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  2101. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  2102. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2103. at java.lang.reflect.Method.invoke(Method.java:597)
  2104. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  2105. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  2106.  
  2107. "IPC Server handler 45 on 60020" daemon prio=10 tid=0x00007f6f141ab000 nid=0x6547 in Object.wait() [0x00007f6f04fce000]
  2108. java.lang.Thread.State: BLOCKED (on object monitor)
  2109. at java.lang.Object.wait(Native Method)
  2110. at java.lang.Object.wait(Object.java:485)
  2111. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.waitForAckedSeqno(DFSClient.java:3284)
  2112. - locked <0x00000005db9b7800> (a java.util.LinkedList)
  2113. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.sync(DFSClient.java:3201)
  2114. at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:97)
  2115. at org.apache.hadoop.io.SequenceFile$Writer.syncFs(SequenceFile.java:944)
  2116. at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
  2117. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2118. at java.lang.reflect.Method.invoke(Method.java:597)
  2119. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:123)
  2120. at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:906)
  2121. at org.apache.hadoop.hbase.regionserver.wal.HLog.append(HLog.java:857)
  2122. at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1402)
  2123. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1318)
  2124. at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1803)
  2125. at org.apache.hadoop.hbase.regionserver.HRegionServer.multiPut(HRegionServer.java:2464)
  2126. at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
  2127. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2128. at java.lang.reflect.Method.invoke(Method.java:597)
  2129. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  2130. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  2131.  
  2132. "IPC Server handler 44 on 60020" daemon prio=10 tid=0x00007f6f141a9000 nid=0x6546 runnable [0x00007f6f050ce000]
  2133. java.lang.Thread.State: RUNNABLE
  2134. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  2135. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  2136. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  2137. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  2138. - locked <0x000000050d08a9e8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2139. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  2140. - locked <0x000000050d08a9e8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2141. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  2142. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  2143. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  2144. - locked <0x000000050d08a8e8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2145. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  2146. - locked <0x000000050d08a8e8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2147. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  2148. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  2149. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  2150. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  2151. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2152. at java.lang.reflect.Method.invoke(Method.java:597)
  2153. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  2154. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  2155.  
  2156. "IPC Server handler 43 on 60020" daemon prio=10 tid=0x00007f6f141a7000 nid=0x6545 runnable [0x00007f6f051d0000]
  2157. java.lang.Thread.State: RUNNABLE
  2158. at java.util.concurrent.ConcurrentSkipListMap$SubMap$SubMapEntryIterator.next(ConcurrentSkipListMap.java:3113)
  2159. at java.util.concurrent.ConcurrentSkipListMap$SubMap$SubMapEntryIterator.next(ConcurrentSkipListMap.java:3108)
  2160. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:74)
  2161. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:62)
  2162. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:561)
  2163. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  2164. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  2165. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  2166. - locked <0x000000050e0ebd00> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2167. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  2168. - locked <0x000000050e0ebd00> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2169. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  2170. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  2171. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  2172. - locked <0x000000050e0ebd68> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2173. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  2174. - locked <0x000000050e0ebd68> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2175. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  2176. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  2177. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  2178. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  2179. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2180. at java.lang.reflect.Method.invoke(Method.java:597)
  2181. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  2182. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  2183.  
  2184. "IPC Server handler 42 on 60020" daemon prio=10 tid=0x00007f6f141a5000 nid=0x6544 runnable [0x00007f6f052d1000]
  2185. java.lang.Thread.State: RUNNABLE
  2186. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  2187. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  2188. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  2189. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  2190. - locked <0x000000050e0cdf68> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2191. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  2192. - locked <0x000000050e0cdf68> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2193. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  2194. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  2195. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  2196. - locked <0x000000050e0cdfd0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2197. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  2198. - locked <0x000000050e0cdfd0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2199. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  2200. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  2201. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  2202. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  2203. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2204. at java.lang.reflect.Method.invoke(Method.java:597)
  2205. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  2206. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  2207.  
  2208. "IPC Server handler 41 on 60020" daemon prio=10 tid=0x00007f6f141a3000 nid=0x6543 in Object.wait() [0x00007f6f053d2000]
  2209. java.lang.Thread.State: BLOCKED (on object monitor)
  2210. at java.lang.Object.wait(Native Method)
  2211. at java.lang.Object.wait(Object.java:485)
  2212. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.waitForAckedSeqno(DFSClient.java:3284)
  2213. - locked <0x00000005db9b7800> (a java.util.LinkedList)
  2214. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.sync(DFSClient.java:3201)
  2215. at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:97)
  2216. at org.apache.hadoop.io.SequenceFile$Writer.syncFs(SequenceFile.java:944)
  2217. at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
  2218. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2219. at java.lang.reflect.Method.invoke(Method.java:597)
  2220. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:123)
  2221. at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:906)
  2222. at org.apache.hadoop.hbase.regionserver.wal.HLog.append(HLog.java:857)
  2223. at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1402)
  2224. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1318)
  2225. at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1803)
  2226. at org.apache.hadoop.hbase.regionserver.HRegionServer.multiPut(HRegionServer.java:2464)
  2227. at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
  2228. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2229. at java.lang.reflect.Method.invoke(Method.java:597)
  2230. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  2231. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  2232.  
  2233. "IPC Server handler 40 on 60020" daemon prio=10 tid=0x00007f6f141a1000 nid=0x6542 runnable [0x00007f6f054d3000]
  2234. java.lang.Thread.State: RUNNABLE
  2235. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  2236. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  2237. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  2238. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  2239. - locked <0x000000050da216b8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2240. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  2241. - locked <0x000000050da216b8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2242. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  2243. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  2244. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  2245. - locked <0x000000050da215b8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2246. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  2247. - locked <0x000000050da215b8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2248. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  2249. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  2250. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  2251. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  2252. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2253. at java.lang.reflect.Method.invoke(Method.java:597)
  2254. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  2255. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  2256.  
  2257. "IPC Server handler 39 on 60020" daemon prio=10 tid=0x00007f6f1419f000 nid=0x6541 runnable [0x00007f6f055d4000]
  2258. java.lang.Thread.State: RUNNABLE
  2259. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  2260. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  2261. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  2262. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  2263. - locked <0x000000050e1099e0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2264. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  2265. - locked <0x000000050e1099e0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2266. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  2267. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  2268. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  2269. - locked <0x000000050e109a48> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2270. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  2271. - locked <0x000000050e109a48> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2272. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  2273. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  2274. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  2275. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  2276. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2277. at java.lang.reflect.Method.invoke(Method.java:597)
  2278. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  2279. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  2280.  
  2281. "IPC Server handler 38 on 60020" daemon prio=10 tid=0x00007f6f1419d000 nid=0x6540 runnable [0x00007f6f056d5000]
  2282. java.lang.Thread.State: RUNNABLE
  2283. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  2284. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  2285. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  2286. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  2287. - locked <0x000000050d5a9ea8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2288. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  2289. - locked <0x000000050d5a9ea8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2290. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  2291. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  2292. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  2293. - locked <0x000000050d5a9da8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2294. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  2295. - locked <0x000000050d5a9da8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2296. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  2297. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  2298. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  2299. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  2300. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2301. at java.lang.reflect.Method.invoke(Method.java:597)
  2302. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  2303. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  2304.  
  2305. "IPC Server handler 37 on 60020" daemon prio=10 tid=0x00007f6f1419b000 nid=0x653f runnable [0x00007f6f057d6000]
  2306. java.lang.Thread.State: RUNNABLE
  2307. at java.util.concurrent.ConcurrentSkipListMap$SubMap$SubMapEntryIterator.next(ConcurrentSkipListMap.java:3113)
  2308. at java.util.concurrent.ConcurrentSkipListMap$SubMap$SubMapEntryIterator.next(ConcurrentSkipListMap.java:3108)
  2309. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:74)
  2310. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:62)
  2311. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:561)
  2312. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  2313. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  2314. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  2315. - locked <0x000000050de5dab8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2316. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  2317. - locked <0x000000050de5dab8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2318. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  2319. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  2320. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  2321. - locked <0x000000050de5d9b8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2322. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  2323. - locked <0x000000050de5d9b8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2324. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  2325. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  2326. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  2327. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  2328. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2329. at java.lang.reflect.Method.invoke(Method.java:597)
  2330. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  2331. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  2332.  
  2333. "IPC Server handler 36 on 60020" daemon prio=10 tid=0x00007f6f14199000 nid=0x653e runnable [0x00007f6f058d6000]
  2334. java.lang.Thread.State: RUNNABLE
  2335. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  2336. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  2337. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  2338. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  2339. - locked <0x000000050d361238> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2340. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  2341. - locked <0x000000050d361238> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2342. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  2343. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  2344. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  2345. - locked <0x000000050d361138> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2346. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  2347. - locked <0x000000050d361138> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2348. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  2349. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  2350. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  2351. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  2352. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2353. at java.lang.reflect.Method.invoke(Method.java:597)
  2354. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  2355. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  2356.  
  2357. "IPC Server handler 35 on 60020" daemon prio=10 tid=0x00007f6f14197000 nid=0x653d in Object.wait() [0x00007f6f059d8000]
  2358. java.lang.Thread.State: BLOCKED (on object monitor)
  2359. at java.lang.Object.wait(Native Method)
  2360. at java.lang.Object.wait(Object.java:485)
  2361. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.waitForAckedSeqno(DFSClient.java:3284)
  2362. - locked <0x00000005db9b7800> (a java.util.LinkedList)
  2363. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.sync(DFSClient.java:3201)
  2364. at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:97)
  2365. at org.apache.hadoop.io.SequenceFile$Writer.syncFs(SequenceFile.java:944)
  2366. at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
  2367. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2368. at java.lang.reflect.Method.invoke(Method.java:597)
  2369. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:123)
  2370. at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:906)
  2371. at org.apache.hadoop.hbase.regionserver.wal.HLog.append(HLog.java:857)
  2372. at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1402)
  2373. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1318)
  2374. at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1803)
  2375. at org.apache.hadoop.hbase.regionserver.HRegionServer.multiPut(HRegionServer.java:2464)
  2376. at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
  2377. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2378. at java.lang.reflect.Method.invoke(Method.java:597)
  2379. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  2380. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  2381.  
  2382. "IPC Server handler 34 on 60020" daemon prio=10 tid=0x00007f6f14195000 nid=0x653c runnable [0x00007f6f05ad9000]
  2383. java.lang.Thread.State: RUNNABLE
  2384. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  2385. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  2386. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  2387. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  2388. - locked <0x000000050e0ec040> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2389. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  2390. - locked <0x000000050e0ec040> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2391. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  2392. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  2393. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  2394. - locked <0x000000050e0ec878> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2395. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  2396. - locked <0x000000050e0ec878> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2397. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  2398. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  2399. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  2400. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  2401. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2402. at java.lang.reflect.Method.invoke(Method.java:597)
  2403. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  2404. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  2405.  
  2406. "IPC Server handler 33 on 60020" daemon prio=10 tid=0x00007f6f14193000 nid=0x653b runnable [0x00007f6f05bda000]
  2407. java.lang.Thread.State: RUNNABLE
  2408. at java.util.concurrent.ConcurrentSkipListMap$SubMap$SubMapEntryIterator.next(ConcurrentSkipListMap.java:3113)
  2409. at java.util.concurrent.ConcurrentSkipListMap$SubMap$SubMapEntryIterator.next(ConcurrentSkipListMap.java:3108)
  2410. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:74)
  2411. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:62)
  2412. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:561)
  2413. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  2414. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  2415. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  2416. - locked <0x000000050d240410> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2417. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  2418. - locked <0x000000050d240410> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2419. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  2420. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  2421. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  2422. - locked <0x000000050d240310> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2423. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  2424. - locked <0x000000050d240310> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2425. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  2426. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  2427. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  2428. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  2429. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2430. at java.lang.reflect.Method.invoke(Method.java:597)
  2431. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  2432. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  2433.  
  2434. "IPC Server handler 32 on 60020" daemon prio=10 tid=0x00007f6f14191000 nid=0x653a runnable [0x00007f6f05cdb000]
  2435. java.lang.Thread.State: RUNNABLE
  2436. at org.apache.hadoop.hbase.KeyValue.createByteArray(KeyValue.java:461)
  2437. at org.apache.hadoop.hbase.KeyValue.<init>(KeyValue.java:399)
  2438. at org.apache.hadoop.hbase.KeyValue.<init>(KeyValue.java:368)
  2439. at org.apache.hadoop.hbase.KeyValue.<init>(KeyValue.java:347)
  2440. at org.apache.hadoop.hbase.KeyValue.<init>(KeyValue.java:317)
  2441. at org.apache.hadoop.hbase.KeyValue.createFirstOnRow(KeyValue.java:1584)
  2442. at org.apache.hadoop.hbase.KeyValue.createFirstOnRow(KeyValue.java:1572)
  2443. at org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.<init>(ScanQueryMatcher.java:80)
  2444. at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:64)
  2445. at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1202)
  2446. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.<init>(HRegion.java:2209)
  2447. at org.apache.hadoop.hbase.regionserver.HRegion.instantiateInternalScanner(HRegion.java:1063)
  2448. at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1055)
  2449. at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1039)
  2450. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2942)
  2451. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  2452. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  2453. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  2454. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2455. at java.lang.reflect.Method.invoke(Method.java:597)
  2456. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  2457. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  2458.  
  2459. "IPC Server handler 31 on 60020" daemon prio=10 tid=0x00007f6f1418f000 nid=0x6539 runnable [0x00007f6f05ddb000]
  2460. java.lang.Thread.State: RUNNABLE
  2461. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  2462. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  2463. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  2464. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  2465. - locked <0x000000061da31d48> (a sun.nio.ch.Util$2)
  2466. - locked <0x000000061da31d30> (a java.util.Collections$UnmodifiableSet)
  2467. - locked <0x000000061d8dd710> (a sun.nio.ch.EPollSelectorImpl)
  2468. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  2469. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:332)
  2470. at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
  2471. at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
  2472. at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
  2473. at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
  2474. at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
  2475. - locked <0x000000050d228888> (a java.io.BufferedInputStream)
  2476. at java.io.DataInputStream.readShort(DataInputStream.java:295)
  2477. at org.apache.hadoop.hdfs.DFSClient$BlockReader.newBlockReader(DFSClient.java:1379)
  2478. at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.fetchBlockByteRange(DFSClient.java:1895)
  2479. at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1960)
  2480. at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:46)
  2481. at org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:101)
  2482. at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
  2483. at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
  2484. - locked <0x000000050d0b26d0> (a java.io.BufferedInputStream)
  2485. at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
  2486. at org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1085)
  2487. at org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1027)
  2488. - locked <0x000000051ba0e370> (a [B)
  2489. at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.java:1433)
  2490. at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1290)
  2491. at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:136)
  2492. at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:96)
  2493. at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:75)
  2494. at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1202)
  2495. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.<init>(HRegion.java:2209)
  2496. at org.apache.hadoop.hbase.regionserver.HRegion.instantiateInternalScanner(HRegion.java:1063)
  2497. at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1055)
  2498. at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1039)
  2499. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2942)
  2500. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  2501. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  2502. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  2503. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2504. at java.lang.reflect.Method.invoke(Method.java:597)
  2505. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  2506. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  2507.  
  2508. "IPC Server handler 30 on 60020" daemon prio=10 tid=0x00007f6f1418d000 nid=0x6538 runnable [0x00007f6f05edd000]
  2509. java.lang.Thread.State: RUNNABLE
  2510. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:74)
  2511. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:62)
  2512. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:561)
  2513. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  2514. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  2515. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  2516. - locked <0x000000050e109bc8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2517. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  2518. - locked <0x000000050e109bc8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2519. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  2520. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  2521. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  2522. - locked <0x000000050e109c30> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2523. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  2524. - locked <0x000000050e109c30> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2525. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  2526. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  2527. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  2528. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  2529. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2530. at java.lang.reflect.Method.invoke(Method.java:597)
  2531. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  2532. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  2533.  
  2534. "IPC Server handler 29 on 60020" daemon prio=10 tid=0x00007f6f1418b000 nid=0x6537 runnable [0x00007f6f05fde000]
  2535. java.lang.Thread.State: RUNNABLE
  2536. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:566)
  2537. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  2538. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  2539. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  2540. - locked <0x000000050e0ce5b8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2541. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  2542. - locked <0x000000050e0ce5b8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2543. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  2544. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  2545. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  2546. - locked <0x000000050e0cef18> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2547. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  2548. - locked <0x000000050e0cef18> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2549. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  2550. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  2551. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  2552. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  2553. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2554. at java.lang.reflect.Method.invoke(Method.java:597)
  2555. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  2556. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  2557.  
  2558. "IPC Server handler 28 on 60020" daemon prio=10 tid=0x00007f6f14189000 nid=0x6536 runnable [0x00007f6f060df000]
  2559. java.lang.Thread.State: RUNNABLE
  2560. at java.util.Arrays.copyOf(Arrays.java:2882)
  2561. at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
  2562. at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
  2563. at java.lang.StringBuilder.append(StringBuilder.java:119)
  2564. at org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1004)
  2565. - locked <0x0000000530a681c8> (a [B)
  2566. at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.java:1433)
  2567. at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1290)
  2568. at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:136)
  2569. at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:96)
  2570. at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:75)
  2571. at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1202)
  2572. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.<init>(HRegion.java:2209)
  2573. at org.apache.hadoop.hbase.regionserver.HRegion.instantiateInternalScanner(HRegion.java:1063)
  2574. at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1055)
  2575. at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1039)
  2576. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2942)
  2577. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  2578. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  2579. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  2580. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2581. at java.lang.reflect.Method.invoke(Method.java:597)
  2582. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  2583. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  2584.  
  2585. "IPC Server handler 27 on 60020" daemon prio=10 tid=0x00007f6f14187000 nid=0x6535 waiting for monitor entry [0x00007f6f061e0000]
  2586. java.lang.Thread.State: BLOCKED (on object monitor)
  2587. at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:507)
  2588. - locked <0x0000000618666ca8> (a java.lang.Object)
  2589. at org.apache.hadoop.ipc.Client.call(Client.java:790)
  2590. at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:221)
  2591. at $Proxy1.getFileInfo(Unknown Source)
  2592. at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
  2593. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2594. at java.lang.reflect.Method.invoke(Method.java:597)
  2595. at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
  2596. at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
  2597. at $Proxy1.getFileInfo(Unknown Source)
  2598. at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:624)
  2599. at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:461)
  2600. at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:690)
  2601. at org.apache.hadoop.hbase.util.FSUtils.checkFileSystemAvailable(FSUtils.java:119)
  2602. at org.apache.hadoop.hbase.regionserver.HRegionServer.checkFileSystem(HRegionServer.java:953)
  2603. at org.apache.hadoop.hbase.regionserver.HRegionServer.cleanup(HRegionServer.java:902)
  2604. at org.apache.hadoop.hbase.regionserver.HRegionServer.cleanup(HRegionServer.java:880)
  2605. at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1958)
  2606. at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
  2607. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2608. at java.lang.reflect.Method.invoke(Method.java:597)
  2609. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  2610. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  2611.  
  2612. "IPC Server handler 26 on 60020" daemon prio=10 tid=0x00007f6f14185000 nid=0x6534 runnable [0x00007f6f062e1000]
  2613. java.lang.Thread.State: RUNNABLE
  2614. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:566)
  2615. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  2616. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  2617. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  2618. - locked <0x000000050e109d40> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2619. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  2620. - locked <0x000000050e109d40> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2621. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  2622. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  2623. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  2624. - locked <0x000000050e109da8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2625. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  2626. - locked <0x000000050e109da8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2627. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  2628. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  2629. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  2630. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  2631. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2632. at java.lang.reflect.Method.invoke(Method.java:597)
  2633. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  2634. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  2635.  
  2636. "IPC Server handler 25 on 60020" daemon prio=10 tid=0x00007f6f14183000 nid=0x6533 waiting for monitor entry [0x00007f6f063e2000]
  2637. java.lang.Thread.State: BLOCKED (on object monitor)
  2638. at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:507)
  2639. - waiting to lock <0x0000000618666ca8> (a java.lang.Object)
  2640. at org.apache.hadoop.ipc.Client.call(Client.java:790)
  2641. at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:221)
  2642. at $Proxy1.getFileInfo(Unknown Source)
  2643. at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
  2644. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2645. at java.lang.reflect.Method.invoke(Method.java:597)
  2646. at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
  2647. at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
  2648. at $Proxy1.getFileInfo(Unknown Source)
  2649. at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:624)
  2650. at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:461)
  2651. at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:690)
  2652. at org.apache.hadoop.hbase.util.FSUtils.checkFileSystemAvailable(FSUtils.java:119)
  2653. at org.apache.hadoop.hbase.regionserver.HRegionServer.checkFileSystem(HRegionServer.java:953)
  2654. at org.apache.hadoop.hbase.regionserver.HRegionServer.cleanup(HRegionServer.java:902)
  2655. at org.apache.hadoop.hbase.regionserver.HRegionServer.cleanup(HRegionServer.java:880)
  2656. at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1958)
  2657. at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
  2658. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2659. at java.lang.reflect.Method.invoke(Method.java:597)
  2660. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  2661. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  2662.  
  2663. "IPC Server handler 24 on 60020" daemon prio=10 tid=0x00007f6f14181000 nid=0x6532 runnable [0x00007f6f064e3000]
  2664. java.lang.Thread.State: RUNNABLE
  2665. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  2666. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  2667. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  2668. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  2669. - locked <0x000000050d6d9df8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2670. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  2671. - locked <0x000000050d6d9df8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2672. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  2673. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  2674. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  2675. - locked <0x000000050d6d5000> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2676. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  2677. - locked <0x000000050d6d5000> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2678. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  2679. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  2680. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  2681. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  2682. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2683. at java.lang.reflect.Method.invoke(Method.java:597)
  2684. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  2685. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  2686.  
  2687. "IPC Server handler 23 on 60020" daemon prio=10 tid=0x00007f6f1417f000 nid=0x6531 runnable [0x00007f6f065e4000]
  2688. java.lang.Thread.State: RUNNABLE
  2689. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:74)
  2690. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:62)
  2691. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:561)
  2692. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  2693. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  2694. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  2695. - locked <0x00000006236a5938> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2696. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  2697. - locked <0x00000006236a5938> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2698. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  2699. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  2700. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  2701. - locked <0x00000006236a00a8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2702. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  2703. - locked <0x00000006236a00a8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2704. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  2705. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  2706. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  2707. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  2708. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2709. at java.lang.reflect.Method.invoke(Method.java:597)
  2710. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  2711. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  2712.  
  2713. "IPC Server handler 22 on 60020" daemon prio=10 tid=0x00007f6f1417d000 nid=0x6530 runnable [0x00007f6f066e5000]
  2714. java.lang.Thread.State: RUNNABLE
  2715. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:74)
  2716. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:62)
  2717. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:561)
  2718. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  2719. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  2720. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  2721. - locked <0x000000050db3e9a8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2722. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  2723. - locked <0x000000050db3e9a8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2724. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  2725. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  2726. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  2727. - locked <0x000000050db3e8a8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2728. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  2729. - locked <0x000000050db3e8a8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2730. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  2731. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  2732. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  2733. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  2734. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2735. at java.lang.reflect.Method.invoke(Method.java:597)
  2736. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  2737. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  2738.  
  2739. "IPC Server handler 21 on 60020" daemon prio=10 tid=0x00007f6f1417b800 nid=0x652f runnable [0x00007f6f067e6000]
  2740. java.lang.Thread.State: RUNNABLE
  2741. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:74)
  2742. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:62)
  2743. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:561)
  2744. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  2745. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  2746. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  2747. - locked <0x000000050e0cf418> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2748. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  2749. - locked <0x000000050e0cf418> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2750. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  2751. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  2752. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  2753. - locked <0x000000050e0cfd10> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2754. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  2755. - locked <0x000000050e0cfd10> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2756. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  2757. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  2758. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  2759. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  2760. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2761. at java.lang.reflect.Method.invoke(Method.java:597)
  2762. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  2763. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  2764.  
  2765. "IPC Server handler 20 on 60020" daemon prio=10 tid=0x00007f6f1416e800 nid=0x652e runnable [0x00007f6f068e7000]
  2766. java.lang.Thread.State: RUNNABLE
  2767. at org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1000)
  2768. - locked <0x0000000515e9de50> (a [B)
  2769. at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.java:1438)
  2770. at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.reseekTo(HFile.java:1316)
  2771. at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:152)
  2772. at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:110)
  2773. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  2774. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  2775. - locked <0x000000050e10a2e0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2776. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  2777. - locked <0x000000050e10a2e0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2778. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  2779. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  2780. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  2781. - locked <0x000000050e10abd8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2782. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  2783. - locked <0x000000050e10abd8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2784. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  2785. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  2786. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  2787. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  2788. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2789. at java.lang.reflect.Method.invoke(Method.java:597)
  2790. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  2791. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  2792.  
  2793. "IPC Server handler 19 on 60020" daemon prio=10 tid=0x00007f6f1416c800 nid=0x652d runnable [0x00007f6f069e8000]
  2794. java.lang.Thread.State: RUNNABLE
  2795. at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.blockSeek(HFile.java:1336)
  2796. at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1291)
  2797. at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:136)
  2798. at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:96)
  2799. at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:75)
  2800. at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1202)
  2801. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.<init>(HRegion.java:2209)
  2802. at org.apache.hadoop.hbase.regionserver.HRegion.instantiateInternalScanner(HRegion.java:1063)
  2803. at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1055)
  2804. at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1039)
  2805. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2942)
  2806. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  2807. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  2808. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  2809. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2810. at java.lang.reflect.Method.invoke(Method.java:597)
  2811. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  2812. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  2813.  
  2814. "IPC Server handler 18 on 60020" daemon prio=10 tid=0x00007f6f1416a800 nid=0x652c waiting on condition [0x00007f6f06ae9000]
  2815. java.lang.Thread.State: RUNNABLE
  2816. at java.util.concurrent.CopyOnWriteArrayList.addIfAbsent(CopyOnWriteArrayList.java:559)
  2817. at java.util.concurrent.CopyOnWriteArraySet.add(CopyOnWriteArraySet.java:221)
  2818. at org.apache.hadoop.hbase.regionserver.Store.addChangedReaderObserver(Store.java:540)
  2819. at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:81)
  2820. at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1202)
  2821. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.<init>(HRegion.java:2209)
  2822. at org.apache.hadoop.hbase.regionserver.HRegion.instantiateInternalScanner(HRegion.java:1063)
  2823. at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1055)
  2824. at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1039)
  2825. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2942)
  2826. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  2827. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  2828. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  2829. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2830. at java.lang.reflect.Method.invoke(Method.java:597)
  2831. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  2832. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  2833.  
  2834. "IPC Server handler 17 on 60020" daemon prio=10 tid=0x00007f6f14168800 nid=0x652b in Object.wait() [0x00007f6f06bea000]
  2835. java.lang.Thread.State: BLOCKED (on object monitor)
  2836. at java.lang.Object.wait(Native Method)
  2837. at java.lang.Object.wait(Object.java:485)
  2838. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.waitForAckedSeqno(DFSClient.java:3284)
  2839. - locked <0x00000005db9b7800> (a java.util.LinkedList)
  2840. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.sync(DFSClient.java:3201)
  2841. at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:97)
  2842. at org.apache.hadoop.io.SequenceFile$Writer.syncFs(SequenceFile.java:944)
  2843. at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
  2844. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2845. at java.lang.reflect.Method.invoke(Method.java:597)
  2846. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:123)
  2847. at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:906)
  2848. at org.apache.hadoop.hbase.regionserver.wal.HLog.append(HLog.java:857)
  2849. at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1402)
  2850. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1318)
  2851. at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1803)
  2852. at org.apache.hadoop.hbase.regionserver.HRegionServer.multiPut(HRegionServer.java:2464)
  2853. at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
  2854. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2855. at java.lang.reflect.Method.invoke(Method.java:597)
  2856. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  2857. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  2858.  
  2859. "IPC Server handler 16 on 60020" daemon prio=10 tid=0x00007f6f14166800 nid=0x652a runnable [0x00007f6f06ceb000]
  2860. java.lang.Thread.State: RUNNABLE
  2861. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:74)
  2862. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:62)
  2863. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:561)
  2864. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  2865. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  2866. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  2867. - locked <0x000000050e0b08a8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2868. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  2869. - locked <0x000000050e0b08a8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2870. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  2871. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  2872. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  2873. - locked <0x000000050e0b11a0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2874. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  2875. - locked <0x000000050e0b11a0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2876. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  2877. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  2878. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  2879. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  2880. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2881. at java.lang.reflect.Method.invoke(Method.java:597)
  2882. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  2883. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  2884.  
  2885. "IPC Server handler 15 on 60020" daemon prio=10 tid=0x00007f6f14164000 nid=0x6529 runnable [0x00007f6f06dec000]
  2886. java.lang.Thread.State: RUNNABLE
  2887. at java.util.Arrays.copyOf(Arrays.java:2882)
  2888. at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
  2889. at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
  2890. at java.lang.StringBuilder.append(StringBuilder.java:119)
  2891. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1029)
  2892.  
  2893. "IPC Server handler 14 on 60020" daemon prio=10 tid=0x00007f6f14162000 nid=0x6528 runnable [0x00007f6f06eed000]
  2894. java.lang.Thread.State: RUNNABLE
  2895. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  2896. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  2897. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  2898. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  2899. - locked <0x000000050dca8e10> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2900. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  2901. - locked <0x000000050dca8e10> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2902. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  2903. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  2904. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  2905. - locked <0x000000050dca8d10> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2906. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  2907. - locked <0x000000050dca8d10> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2908. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  2909. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  2910. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  2911. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  2912. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2913. at java.lang.reflect.Method.invoke(Method.java:597)
  2914. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  2915. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  2916.  
  2917. "IPC Server handler 13 on 60020" daemon prio=10 tid=0x00007f6f14160000 nid=0x6527 runnable [0x00007f6f06fed000]
  2918. java.lang.Thread.State: RUNNABLE
  2919. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  2920. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  2921. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  2922. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  2923. - locked <0x000000050e0d0208> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2924. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  2925. - locked <0x000000050e0d0208> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  2926. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  2927. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  2928. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  2929. - locked <0x000000050e0d0b00> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2930. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  2931. - locked <0x000000050e0d0b00> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  2932. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  2933. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  2934. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  2935. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  2936. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2937. at java.lang.reflect.Method.invoke(Method.java:597)
  2938. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  2939. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  2940.  
  2941. "IPC Server handler 12 on 60020" daemon prio=10 tid=0x00007f6f1415e800 nid=0x6526 runnable [0x00007f6f070ef000]
  2942. java.lang.Thread.State: RUNNABLE
  2943. at org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.<init>(ScanQueryMatcher.java:90)
  2944. at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:64)
  2945. at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1202)
  2946. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.<init>(HRegion.java:2209)
  2947. at org.apache.hadoop.hbase.regionserver.HRegion.instantiateInternalScanner(HRegion.java:1063)
  2948. at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1055)
  2949. at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1039)
  2950. at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1887)
  2951. at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
  2952. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2953. at java.lang.reflect.Method.invoke(Method.java:597)
  2954. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  2955. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  2956.  
  2957. "IPC Server handler 11 on 60020" daemon prio=10 tid=0x00007f6f14152800 nid=0x6525 in Object.wait() [0x00007f6f071f0000]
  2958. java.lang.Thread.State: BLOCKED (on object monitor)
  2959. at java.lang.Object.wait(Native Method)
  2960. at java.lang.Object.wait(Object.java:485)
  2961. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.waitForAckedSeqno(DFSClient.java:3284)
  2962. - locked <0x00000005db9b7800> (a java.util.LinkedList)
  2963. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.sync(DFSClient.java:3201)
  2964. at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:97)
  2965. at org.apache.hadoop.io.SequenceFile$Writer.syncFs(SequenceFile.java:944)
  2966. at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
  2967. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2968. at java.lang.reflect.Method.invoke(Method.java:597)
  2969. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:123)
  2970. at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:906)
  2971. at org.apache.hadoop.hbase.regionserver.wal.HLog.append(HLog.java:857)
  2972. at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1402)
  2973. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1318)
  2974. at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1803)
  2975. at org.apache.hadoop.hbase.regionserver.HRegionServer.multiPut(HRegionServer.java:2464)
  2976. at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
  2977. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  2978. at java.lang.reflect.Method.invoke(Method.java:597)
  2979. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  2980. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  2981.  
  2982. "IPC Server handler 10 on 60020" daemon prio=10 tid=0x00007f6f14150800 nid=0x6524 runnable [0x00007f6f072f0000]
  2983. java.lang.Thread.State: RUNNABLE
  2984. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  2985. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  2986. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  2987. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  2988. - locked <0x00000006233859a0> (a sun.nio.ch.Util$2)
  2989. - locked <0x0000000623385988> (a java.util.Collections$UnmodifiableSet)
  2990. - locked <0x0000000623414c18> (a sun.nio.ch.EPollSelectorImpl)
  2991. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  2992. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:332)
  2993. at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
  2994. at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
  2995. at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
  2996. at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
  2997. at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
  2998. at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
  2999. - locked <0x000000050dc42ac0> (a java.io.BufferedInputStream)
  3000. at java.io.DataInputStream.read(DataInputStream.java:132)
  3001. at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
  3002. at org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:1293)
  3003. - locked <0x000000050dc42ec0> (a org.apache.hadoop.hdfs.DFSClient$BlockReader)
  3004. at org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:237)
  3005. at org.apache.hadoop.fs.FSInputChecker.fill(FSInputChecker.java:176)
  3006. at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:193)
  3007. at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
  3008. - locked <0x000000050dc42ec0> (a org.apache.hadoop.hdfs.DFSClient$BlockReader)
  3009. at org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1147)
  3010. - locked <0x000000050dc42ec0> (a org.apache.hadoop.hdfs.DFSClient$BlockReader)
  3011. at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.java:1754)
  3012. - locked <0x0000000529e13cb8> (a org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
  3013. at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1806)
  3014. - locked <0x0000000529e13cb8> (a org.apache.hadoop.hdfs.DFSClient$DFSInputStream)
  3015. at java.io.DataInputStream.read(DataInputStream.java:132)
  3016. at org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:105)
  3017. - locked <0x00000005238b04c0> (a org.apache.hadoop.hdfs.DFSClient$DFSDataInputStream)
  3018. at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
  3019. at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
  3020. - locked <0x000000050dc41f18> (a java.io.BufferedInputStream)
  3021. at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
  3022. at org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1085)
  3023. at org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1027)
  3024. - locked <0x000000052b752100> (a [B)
  3025. at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.java:1433)
  3026. at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1290)
  3027. at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:136)
  3028. at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:96)
  3029. at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:75)
  3030. at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1202)
  3031. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.<init>(HRegion.java:2209)
  3032. at org.apache.hadoop.hbase.regionserver.HRegion.instantiateInternalScanner(HRegion.java:1063)
  3033. at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1055)
  3034. at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1039)
  3035. at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1887)
  3036. at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
  3037. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  3038. at java.lang.reflect.Method.invoke(Method.java:597)
  3039. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  3040. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  3041.  
  3042. "IPC Server handler 9 on 60020" daemon prio=10 tid=0x00007f6f14247800 nid=0x6523 in Object.wait() [0x00007f6f073f2000]
  3043. java.lang.Thread.State: BLOCKED (on object monitor)
  3044. at java.lang.Object.wait(Native Method)
  3045. at java.lang.Object.wait(Object.java:485)
  3046. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.waitForAckedSeqno(DFSClient.java:3284)
  3047. - locked <0x00000005db9b7800> (a java.util.LinkedList)
  3048. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.sync(DFSClient.java:3201)
  3049. at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:97)
  3050. at org.apache.hadoop.io.SequenceFile$Writer.syncFs(SequenceFile.java:944)
  3051. at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
  3052. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  3053. at java.lang.reflect.Method.invoke(Method.java:597)
  3054. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:123)
  3055. at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:906)
  3056. at org.apache.hadoop.hbase.regionserver.wal.HLog.append(HLog.java:857)
  3057. at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1402)
  3058. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1318)
  3059. at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1803)
  3060. at org.apache.hadoop.hbase.regionserver.HRegionServer.multiPut(HRegionServer.java:2464)
  3061. at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
  3062. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  3063. at java.lang.reflect.Method.invoke(Method.java:597)
  3064. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  3065. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  3066.  
  3067. "IPC Server handler 8 on 60020" daemon prio=10 tid=0x00007f6f14245800 nid=0x6522 runnable [0x00007f6f074f3000]
  3068. java.lang.Thread.State: RUNNABLE
  3069. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:566)
  3070. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  3071. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  3072. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  3073. - locked <0x000000050e10b2a0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  3074. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  3075. - locked <0x000000050e10b2a0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  3076. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  3077. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  3078. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  3079. - locked <0x000000050e10bb68> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  3080. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  3081. - locked <0x000000050e10bb68> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  3082. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  3083. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  3084. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  3085. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  3086. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  3087. at java.lang.reflect.Method.invoke(Method.java:597)
  3088. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  3089. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  3090.  
  3091. "IPC Server handler 7 on 60020" daemon prio=10 tid=0x00007f6f14243800 nid=0x6521 runnable [0x00007f6f075f4000]
  3092. java.lang.Thread.State: RUNNABLE
  3093. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:566)
  3094. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  3095. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  3096. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  3097. - locked <0x000000050d31e1a0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  3098. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  3099. - locked <0x000000050d31e1a0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  3100. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  3101. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  3102. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  3103. - locked <0x000000050d31e0a0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  3104. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  3105. - locked <0x000000050d31e0a0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  3106. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  3107. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  3108. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  3109. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  3110. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  3111. at java.lang.reflect.Method.invoke(Method.java:597)
  3112. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  3113. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  3114.  
  3115. "IPC Server handler 6 on 60020" daemon prio=10 tid=0x00007f6f14241800 nid=0x6520 waiting for monitor entry [0x00007f6f076f4000]
  3116. java.lang.Thread.State: BLOCKED (on object monitor)
  3117. at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:507)
  3118. - waiting to lock <0x0000000618666ca8> (a java.lang.Object)
  3119. at org.apache.hadoop.ipc.Client.call(Client.java:790)
  3120. at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:221)
  3121. at $Proxy1.getFileInfo(Unknown Source)
  3122. at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
  3123. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  3124. at java.lang.reflect.Method.invoke(Method.java:597)
  3125. at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
  3126. at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
  3127. at $Proxy1.getFileInfo(Unknown Source)
  3128. at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:624)
  3129. at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:461)
  3130. at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:690)
  3131. at org.apache.hadoop.hbase.util.FSUtils.checkFileSystemAvailable(FSUtils.java:119)
  3132. at org.apache.hadoop.hbase.regionserver.HRegionServer.checkFileSystem(HRegionServer.java:953)
  3133. at org.apache.hadoop.hbase.regionserver.HRegionServer.cleanup(HRegionServer.java:902)
  3134. at org.apache.hadoop.hbase.regionserver.HRegionServer.cleanup(HRegionServer.java:880)
  3135. at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1958)
  3136. at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
  3137. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  3138. at java.lang.reflect.Method.invoke(Method.java:597)
  3139. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  3140. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  3141.  
  3142. "IPC Server handler 5 on 60020" daemon prio=10 tid=0x00007f6f1423f800 nid=0x651f runnable [0x00007f6f077f5000]
  3143. java.lang.Thread.State: RUNNABLE
  3144. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  3145. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  3146. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  3147. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  3148. - locked <0x0000000623594ec0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  3149. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  3150. - locked <0x0000000623594ec0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  3151. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  3152. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  3153. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  3154. - locked <0x0000000623613170> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  3155. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  3156. - locked <0x0000000623613170> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  3157. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  3158. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  3159. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  3160. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  3161. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  3162. at java.lang.reflect.Method.invoke(Method.java:597)
  3163. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  3164. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  3165.  
  3166. "IPC Server handler 4 on 60020" daemon prio=10 tid=0x00007f6f14222800 nid=0x651e runnable [0x00007f6f078f6000]
  3167. java.lang.Thread.State: RUNNABLE
  3168. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:74)
  3169. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:62)
  3170. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:561)
  3171. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  3172. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  3173. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  3174. - locked <0x000000050e0d10a0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  3175. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  3176. - locked <0x000000050e0d10a0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  3177. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  3178. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  3179. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  3180. - locked <0x000000050e0d1998> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  3181. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  3182. - locked <0x000000050e0d1998> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  3183. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  3184. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  3185. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  3186. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  3187. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  3188. at java.lang.reflect.Method.invoke(Method.java:597)
  3189. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  3190. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  3191.  
  3192. "IPC Server handler 3 on 60020" daemon prio=10 tid=0x00007f6f14221000 nid=0x651d runnable [0x00007f6f079f8000]
  3193. java.lang.Thread.State: RUNNABLE
  3194. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  3195. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  3196. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  3197. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  3198. - locked <0x000000050e0ed088> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  3199. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  3200. - locked <0x000000050e0ed088> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  3201. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  3202. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  3203. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  3204. - locked <0x000000050e0ed980> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  3205. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  3206. - locked <0x000000050e0ed980> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  3207. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  3208. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  3209. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  3210. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  3211. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  3212. at java.lang.reflect.Method.invoke(Method.java:597)
  3213. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  3214. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  3215.  
  3216. "IPC Server handler 2 on 60020" daemon prio=10 tid=0x00007f6f1421f800 nid=0x651c waiting for monitor entry [0x00007f6f07af9000]
  3217. java.lang.Thread.State: BLOCKED (on object monitor)
  3218. at org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1000)
  3219. - waiting to lock <0x0000000515e9de50> (a [B)
  3220. at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.java:1438)
  3221. at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.reseekTo(HFile.java:1316)
  3222. at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:152)
  3223. at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:110)
  3224. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  3225. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  3226. - locked <0x000000050e0b2460> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  3227. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  3228. - locked <0x000000050e0b2460> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  3229. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  3230. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  3231. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  3232. - locked <0x000000050e0b2d58> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  3233. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  3234. - locked <0x000000050e0b2d58> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  3235. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  3236. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  3237. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  3238. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  3239. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  3240. at java.lang.reflect.Method.invoke(Method.java:597)
  3241. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  3242. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  3243.  
  3244. "IPC Server handler 1 on 60020" daemon prio=10 tid=0x00007f6f1421b000 nid=0x651b runnable [0x00007f6f07bfa000]
  3245. java.lang.Thread.State: RUNNABLE
  3246. at java.util.concurrent.ConcurrentSkipListMap$SubMap$SubMapEntryIterator.next(ConcurrentSkipListMap.java:3113)
  3247. at java.util.concurrent.ConcurrentSkipListMap$SubMap$SubMapEntryIterator.next(ConcurrentSkipListMap.java:3108)
  3248. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:74)
  3249. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:62)
  3250. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:561)
  3251. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  3252. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  3253. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  3254. - locked <0x000000050e10c068> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  3255. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  3256. - locked <0x000000050e10c068> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  3257. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  3258. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  3259. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  3260. - locked <0x000000050e10c960> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  3261. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  3262. - locked <0x000000050e10c960> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  3263. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  3264. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  3265. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  3266. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  3267. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  3268. at java.lang.reflect.Method.invoke(Method.java:597)
  3269. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  3270. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  3271.  
  3272. "IPC Server handler 0 on 60020" daemon prio=10 tid=0x00007f6f14219800 nid=0x651a runnable [0x00007f6f07cfb000]
  3273. java.lang.Thread.State: RUNNABLE
  3274. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:74)
  3275. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:62)
  3276. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:561)
  3277. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  3278. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  3279. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  3280. - locked <0x000000050e0d1e98> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  3281. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  3282. - locked <0x000000050e0d1e98> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  3283. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  3284. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  3285. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  3286. - locked <0x000000050e0d2790> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  3287. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  3288. - locked <0x000000050e0d2790> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  3289. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  3290. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  3291. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  3292. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  3293. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  3294. at java.lang.reflect.Method.invoke(Method.java:597)
  3295. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  3296. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  3297.  
  3298. "IPC Server listener on 60020" daemon prio=10 tid=0x00007f6f14215000 nid=0x6519 runnable [0x00007f6f07dfc000]
  3299. java.lang.Thread.State: RUNNABLE
  3300. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  3301. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  3302. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  3303. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  3304. - locked <0x0000000513554d80> (a sun.nio.ch.Util$2)
  3305. - locked <0x0000000513554d98> (a java.util.Collections$UnmodifiableSet)
  3306. - locked <0x0000000513554100> (a sun.nio.ch.EPollSelectorImpl)
  3307. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  3308. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
  3309. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:402)
  3310.  
  3311. "IPC Server Responder" daemon prio=10 tid=0x00007f6f1421c800 nid=0x6518 runnable [0x00007f6f07efd000]
  3312. java.lang.Thread.State: RUNNABLE
  3313. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  3314. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  3315. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  3316. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  3317. - locked <0x0000000513554f00> (a sun.nio.ch.Util$2)
  3318. - locked <0x0000000513554f18> (a java.util.Collections$UnmodifiableSet)
  3319. - locked <0x0000000513554900> (a sun.nio.ch.EPollSelectorImpl)
  3320. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  3321. at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:576)
  3322.  
  3323. "Timer-0" daemon prio=10 tid=0x00007f6f14213000 nid=0x6517 in Object.wait() [0x00007f6f07ffe000]
  3324. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  3325. at java.lang.Object.wait(Native Method)
  3326. at java.util.TimerThread.mainLoop(Timer.java:509)
  3327. - locked <0x00000005137d7d70> (a java.util.TaskQueue)
  3328. at java.util.TimerThread.run(Timer.java:462)
  3329.  
  3330. "691789110@qtp-1373164447-1 - Acceptor0 SelectChannelConnector@0.0.0.0:60030" prio=10 tid=0x00007f6f14115800 nid=0x6516 runnable [0x00007f6f5c103000]
  3331. java.lang.Thread.State: RUNNABLE
  3332. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  3333. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  3334. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  3335. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  3336. - locked <0x000000051386c608> (a sun.nio.ch.Util$2)
  3337. - locked <0x000000051386c5f0> (a java.util.Collections$UnmodifiableSet)
  3338. - locked <0x0000000513772568> (a sun.nio.ch.EPollSelectorImpl)
  3339. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  3340. at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:484)
  3341. at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:191)
  3342. at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
  3343. at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
  3344. at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
  3345.  
  3346. "589411738@qtp-1373164447-0" prio=10 tid=0x00007f6f1412a000 nid=0x6515 in Object.wait() [0x00007f6f5c204000]
  3347. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  3348. at java.lang.Object.wait(Native Method)
  3349. at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
  3350. - locked <0x0000000513826e10> (a org.mortbay.thread.QueuedThreadPool$PoolThread)
  3351.  
  3352. "regionserver60020.leaseChecker" prio=10 tid=0x00007f6f1410a800 nid=0x6514 waiting on condition [0x00007f6f5c305000]
  3353. java.lang.Thread.State: TIMED_WAITING (parking)
  3354. at sun.misc.Unsafe.park(Native Method)
  3355. - parking to wait for <0x0000000513554f30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  3356. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
  3357. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
  3358. at java.util.concurrent.DelayQueue.poll(DelayQueue.java:209)
  3359. at org.apache.hadoop.hbase.Leases.run(Leases.java:81)
  3360.  
  3361. "regionserver60020.majorCompactionChecker" daemon prio=10 tid=0x00007f6f14108800 nid=0x6513 in Object.wait() [0x00007f6f5c406000]
  3362. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  3363. at java.lang.Object.wait(Native Method)
  3364. - waiting on <0x0000000513554db0> (a java.lang.Object)
  3365. at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
  3366. - locked <0x0000000513554db0> (a java.lang.Object)
  3367. at org.apache.hadoop.hbase.Chore.run(Chore.java:76)
  3368.  
  3369. "regionserver60020.worker" daemon prio=10 tid=0x00007f6f14106800 nid=0x6512 waiting on condition [0x00007f6f5c507000]
  3370. java.lang.Thread.State: TIMED_WAITING (parking)
  3371. at sun.misc.Unsafe.park(Native Method)
  3372. - parking to wait for <0x0000000513554f90> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  3373. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
  3374. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
  3375. at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424)
  3376. at org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1428)
  3377. at java.lang.Thread.run(Thread.java:662)
  3378.  
  3379. "regionserver60020.compactor" daemon prio=10 tid=0x00007f6f14104800 nid=0x6511 waiting on condition [0x00007f6f5c608000]
  3380. java.lang.Thread.State: TIMED_WAITING (parking)
  3381. at sun.misc.Unsafe.park(Native Method)
  3382. - parking to wait for <0x0000000513554dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  3383. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
  3384. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
  3385. at java.util.concurrent.PriorityBlockingQueue.poll(PriorityBlockingQueue.java:245)
  3386. at org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue.poll(PriorityCompactionQueue.java:244)
  3387. at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:99)
  3388.  
  3389. "regionserver60020.cacheFlusher" daemon prio=10 tid=0x00007f6f14102800 nid=0x6510 waiting on condition [0x00007f6f5c709000]
  3390. java.lang.Thread.State: TIMED_WAITING (parking)
  3391. at sun.misc.Unsafe.park(Native Method)
  3392. - parking to wait for <0x0000000513554e28> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  3393. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
  3394. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
  3395. at java.util.concurrent.DelayQueue.poll(DelayQueue.java:201)
  3396. at java.util.concurrent.DelayQueue.poll(DelayQueue.java:39)
  3397. at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:142)
  3398.  
  3399. "regionserver60020.logRoller" daemon prio=10 tid=0x00007f6f14101800 nid=0x650f in Object.wait() [0x00007f6f5c80a000]
  3400. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  3401. at java.lang.Object.wait(Native Method)
  3402. at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:67)
  3403. - locked <0x0000000513554e70> (a java.util.concurrent.atomic.AtomicBoolean)
  3404.  
  3405. "Timer thread for monitoring jvm" daemon prio=10 tid=0x00007f6f140c5800 nid=0x650e in Object.wait() [0x00007f6f5c90b000]
  3406. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  3407. at java.lang.Object.wait(Native Method)
  3408. at java.util.TimerThread.mainLoop(Timer.java:509)
  3409. - locked <0x0000000513822ea8> (a java.util.TaskQueue)
  3410. at java.util.TimerThread.run(Timer.java:462)
  3411.  
  3412. "Timer thread for monitoring hbase" daemon prio=10 tid=0x00007f6f140c3800 nid=0x650d in Object.wait() [0x00007f6f5ca0c000]
  3413. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  3414. at java.lang.Object.wait(Native Method)
  3415. at java.util.TimerThread.mainLoop(Timer.java:509)
  3416. - locked <0x0000000513822ed8> (a java.util.TaskQueue)
  3417. at java.util.TimerThread.run(Timer.java:462)
  3418.  
  3419. "regionserver60020.logSyncer" daemon prio=10 tid=0x00007f6f140b2000 nid=0x650c waiting on condition [0x00007f6f5cc3d000]
  3420. java.lang.Thread.State: TIMED_WAITING (sleeping)
  3421. at java.lang.Thread.sleep(Native Method)
  3422. at org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:882)
  3423.  
  3424. "LeaseChecker" daemon prio=10 tid=0x00007f6f140b1000 nid=0x650b waiting on condition [0x00007f6f5cb0d000]
  3425. java.lang.Thread.State: TIMED_WAITING (sleeping)
  3426. at java.lang.Thread.sleep(Native Method)
  3427. at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1071)
  3428. at java.lang.Thread.run(Thread.java:662)
  3429.  
  3430. "regionserver60020-EventThread" daemon prio=10 tid=0x00007f6f14082800 nid=0x6509 waiting on condition [0x00007f6f5ce3f000]
  3431. java.lang.Thread.State: WAITING (parking)
  3432. at sun.misc.Unsafe.park(Native Method)
  3433. - parking to wait for <0x00000005139bbd38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  3434. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  3435. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  3436. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  3437. at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:477)
  3438.  
  3439. "regionserver60020-SendThread(mtag3:2181)" daemon prio=10 tid=0x00007f6f14080800 nid=0x6508 runnable [0x00007f6f5cf40000]
  3440. java.lang.Thread.State: RUNNABLE
  3441. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  3442. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  3443. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  3444. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  3445. - locked <0x00000005139bbd20> (a sun.nio.ch.Util$2)
  3446. - locked <0x00000005139bbd08> (a java.util.Collections$UnmodifiableSet)
  3447. - locked <0x0000000513771ea8> (a sun.nio.ch.EPollSelectorImpl)
  3448. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  3449. at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1066)
  3450.  
  3451. "sendParams-0" daemon prio=10 tid=0x00007f6f14023800 nid=0x6506 waiting on condition [0x00007f6f5d142000]
  3452. java.lang.Thread.State: TIMED_WAITING (parking)
  3453. at sun.misc.Unsafe.park(Native Method)
  3454. - parking to wait for <0x0000000513a1c720> (a java.util.concurrent.SynchronousQueue$TransferStack)
  3455. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
  3456. at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
  3457. at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
  3458. at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
  3459. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
  3460. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  3461. at java.lang.Thread.run(Thread.java:662)
  3462.  
  3463. "IPC Client (47) connection to /10.103.7.2:60000 from an unknown user" daemon prio=10 tid=0x00007f6f14002800 nid=0x6501 in Object.wait() [0x00007f6f5d344000]
  3464. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  3465. at java.lang.Object.wait(Native Method)
  3466. at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.waitForWork(HBaseClient.java:408)
  3467. - locked <0x000000051360cd50> (a org.apache.hadoop.hbase.ipc.HBaseClient$Connection)
  3468. at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:453)
  3469.  
  3470. "DestroyJavaVM" prio=10 tid=0x00007f6f9c009000 nid=0x64dd waiting on condition [0x0000000000000000]
  3471. java.lang.Thread.State: RUNNABLE
  3472.  
  3473. "regionserver60020" prio=10 tid=0x00007f6f9c65c000 nid=0x64fa runnable [0x00007f6f5d445000]
  3474. java.lang.Thread.State: RUNNABLE
  3475. at com.google.common.collect.Iterators.forArray(Iterators.java:967)
  3476. at com.google.common.collect.RegularImmutableList.iterator(RegularImmutableList.java:65)
  3477. at com.google.common.collect.RegularImmutableList.iterator(RegularImmutableList.java:33)
  3478. at org.apache.hadoop.hbase.regionserver.Store.getStorefilesIndexSize(Store.java:1241)
  3479. at org.apache.hadoop.hbase.regionserver.HRegionServer.createRegionLoad(HRegionServer.java:856)
  3480. - locked <0x00000005220a7f48> (a java.util.concurrent.ConcurrentSkipListMap)
  3481. at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:556)
  3482. at java.lang.Thread.run(Thread.java:662)
  3483.  
  3484. "main-EventThread" daemon prio=10 tid=0x00007f6f9c65a000 nid=0x64f9 waiting on condition [0x00007f6f5d546000]
  3485. java.lang.Thread.State: WAITING (parking)
  3486. at sun.misc.Unsafe.park(Native Method)
  3487. - parking to wait for <0x0000000513557fd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  3488. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  3489. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  3490. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  3491. at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:477)
  3492.  
  3493. "main-SendThread(mtag3:2181)" daemon prio=10 tid=0x00007f6f9c64f800 nid=0x64f8 runnable [0x00007f6f5d647000]
  3494. java.lang.Thread.State: RUNNABLE
  3495. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  3496. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  3497. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  3498. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  3499. - locked <0x00000005135594b8> (a sun.nio.ch.Util$2)
  3500. - locked <0x00000005135594a0> (a java.util.Collections$UnmodifiableSet)
  3501. - locked <0x0000000513554148> (a sun.nio.ch.EPollSelectorImpl)
  3502. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  3503. at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1066)
  3504.  
  3505. "Timer thread for monitoring rpc" daemon prio=10 tid=0x00007f6f9c60d000 nid=0x64f6 in Object.wait() [0x00007f6f5db67000]
  3506. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  3507. at java.lang.Object.wait(Native Method)
  3508. at java.util.TimerThread.mainLoop(Timer.java:509)
  3509. - locked <0x0000000513551978> (a java.util.TaskQueue)
  3510. at java.util.TimerThread.run(Timer.java:462)
  3511.  
  3512. "pool-1-thread-10" prio=10 tid=0x00007f6f9c60a800 nid=0x64f5 waiting on condition [0x00007f6f5dc68000]
  3513. java.lang.Thread.State: WAITING (parking)
  3514. at sun.misc.Unsafe.park(Native Method)
  3515. - parking to wait for <0x000000051367fdd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  3516. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  3517. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  3518. at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
  3519. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:973)
  3520. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:934)
  3521. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:510)
  3522. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:304)
  3523. - locked <0x0000000513551eb0> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  3524. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  3525. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  3526. at java.lang.Thread.run(Thread.java:662)
  3527.  
  3528. "pool-1-thread-9" prio=10 tid=0x00007f6f9c5f0000 nid=0x64f4 waiting on condition [0x00007f6f5dd69000]
  3529. java.lang.Thread.State: WAITING (parking)
  3530. at sun.misc.Unsafe.park(Native Method)
  3531. - parking to wait for <0x000000051367fdd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  3532. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  3533. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  3534. at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
  3535. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:973)
  3536. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:934)
  3537. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:510)
  3538. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:304)
  3539. - locked <0x0000000513551f10> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  3540. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  3541. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  3542. at java.lang.Thread.run(Thread.java:662)
  3543.  
  3544. "pool-1-thread-8" prio=10 tid=0x00007f6f9c5d5800 nid=0x64f3 waiting on condition [0x00007f6f5de6a000]
  3545. java.lang.Thread.State: WAITING (parking)
  3546. at sun.misc.Unsafe.park(Native Method)
  3547. - parking to wait for <0x000000051367fdd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  3548. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  3549. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  3550. at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
  3551. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:973)
  3552. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:934)
  3553. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:510)
  3554. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:304)
  3555. - locked <0x0000000513551f70> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  3556. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  3557. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  3558. at java.lang.Thread.run(Thread.java:662)
  3559.  
  3560. "pool-1-thread-7" prio=10 tid=0x00007f6f9c5ba800 nid=0x64f2 waiting on condition [0x00007f6f5df6b000]
  3561. java.lang.Thread.State: WAITING (parking)
  3562. at sun.misc.Unsafe.park(Native Method)
  3563. - parking to wait for <0x000000051367fdd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  3564. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  3565. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  3566. at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
  3567. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:973)
  3568. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:934)
  3569. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:510)
  3570. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:304)
  3571. - locked <0x0000000513579510> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  3572. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  3573. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  3574. at java.lang.Thread.run(Thread.java:662)
  3575.  
  3576. "pool-1-thread-6" prio=10 tid=0x00007f6f9c59f800 nid=0x64f1 waiting on condition [0x00007f6f5e06c000]
  3577. java.lang.Thread.State: WAITING (parking)
  3578. at sun.misc.Unsafe.park(Native Method)
  3579. - parking to wait for <0x000000051367fdd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  3580. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  3581. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  3582. at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
  3583. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:973)
  3584. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:934)
  3585. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:510)
  3586. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:304)
  3587. - locked <0x0000000513579660> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  3588. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  3589. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  3590. at java.lang.Thread.run(Thread.java:662)
  3591.  
  3592. "pool-1-thread-5" prio=10 tid=0x00007f6f9c584800 nid=0x64f0 waiting on condition [0x00007f6f5e16d000]
  3593. java.lang.Thread.State: WAITING (parking)
  3594. at sun.misc.Unsafe.park(Native Method)
  3595. - parking to wait for <0x000000051367fdd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  3596. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  3597. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  3598. at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
  3599. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:973)
  3600. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:934)
  3601. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:510)
  3602. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:304)
  3603. - locked <0x000000051357b630> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  3604. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  3605. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  3606. at java.lang.Thread.run(Thread.java:662)
  3607.  
  3608. "pool-1-thread-4" prio=10 tid=0x00007f6f9c569800 nid=0x64ef waiting on condition [0x00007f6f7817f000]
  3609. java.lang.Thread.State: WAITING (parking)
  3610. at sun.misc.Unsafe.park(Native Method)
  3611. - parking to wait for <0x000000051367fdd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  3612. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  3613. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  3614. at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
  3615. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:973)
  3616. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:934)
  3617. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:510)
  3618. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:304)
  3619. - locked <0x000000051357b648> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  3620. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  3621. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  3622. at java.lang.Thread.run(Thread.java:662)
  3623.  
  3624. "pool-1-thread-3" prio=10 tid=0x00007f6f9c54f000 nid=0x64ee waiting on condition [0x00007f6f78280000]
  3625. java.lang.Thread.State: WAITING (parking)
  3626. at sun.misc.Unsafe.park(Native Method)
  3627. - parking to wait for <0x000000051367fdd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  3628. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  3629. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  3630. at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
  3631. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:973)
  3632. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:934)
  3633. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:510)
  3634. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:304)
  3635. - locked <0x000000051357b660> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  3636. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  3637. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  3638. at java.lang.Thread.run(Thread.java:662)
  3639.  
  3640. "pool-1-thread-2" prio=10 tid=0x00007f6f9c4d9000 nid=0x64ed waiting on condition [0x00007f6f78381000]
  3641. java.lang.Thread.State: WAITING (parking)
  3642. at sun.misc.Unsafe.park(Native Method)
  3643. - parking to wait for <0x000000051367fdd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  3644. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  3645. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  3646. at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
  3647. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:973)
  3648. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:934)
  3649. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:510)
  3650. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:304)
  3651. - locked <0x000000051357b678> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  3652. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  3653. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  3654. at java.lang.Thread.run(Thread.java:662)
  3655.  
  3656. "pool-1-thread-1" prio=10 tid=0x00007f6f9c4d8800 nid=0x64ec runnable [0x00007f6f78482000]
  3657. java.lang.Thread.State: WAITING (parking)
  3658. at sun.misc.Unsafe.park(Native Method)
  3659. - parking to wait for <0x000000051367fdd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  3660. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  3661. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  3662. at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
  3663. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:973)
  3664. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:934)
  3665. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:510)
  3666. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:304)
  3667. - locked <0x000000051357c5c0> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  3668. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  3669. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  3670. at java.lang.Thread.run(Thread.java:662)
  3671.  
  3672. "Low Memory Detector" daemon prio=10 tid=0x00007f6f9c15a800 nid=0x64ea runnable [0x0000000000000000]
  3673. java.lang.Thread.State: RUNNABLE
  3674.  
  3675. "CompilerThread1" daemon prio=10 tid=0x00007f6f9c158000 nid=0x64e9 waiting on condition [0x0000000000000000]
  3676. java.lang.Thread.State: RUNNABLE
  3677.  
  3678. "CompilerThread0" daemon prio=10 tid=0x00007f6f9c155800 nid=0x64e8 waiting on condition [0x0000000000000000]
  3679. java.lang.Thread.State: RUNNABLE
  3680.  
  3681. "Signal Dispatcher" daemon prio=10 tid=0x00007f6f9c153800 nid=0x64e7 waiting on condition [0x0000000000000000]
  3682. java.lang.Thread.State: RUNNABLE
  3683.  
  3684. "Surrogate Locker Thread (CMS)" daemon prio=10 tid=0x00007f6f9c151800 nid=0x64e6 waiting on condition [0x0000000000000000]
  3685. java.lang.Thread.State: RUNNABLE
  3686.  
  3687. "Finalizer" daemon prio=10 tid=0x00007f6f9c132800 nid=0x64e5 in Object.wait() [0x00007f6f98b64000]
  3688. java.lang.Thread.State: WAITING (on object monitor)
  3689. at java.lang.Object.wait(Native Method)
  3690. at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
  3691. - locked <0x000000051357c668> (a java.lang.ref.ReferenceQueue$Lock)
  3692. at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
  3693. at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
  3694.  
  3695. "Reference Handler" daemon prio=10 tid=0x00007f6f9c130800 nid=0x64e4 in Object.wait() [0x00007f6f98c65000]
  3696. java.lang.Thread.State: WAITING (on object monitor)
  3697. at java.lang.Object.wait(Native Method)
  3698. at java.lang.Object.wait(Object.java:485)
  3699. at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
  3700. - locked <0x0000000513551e50> (a java.lang.ref.Reference$Lock)
  3701.  
  3702. "VM Thread" prio=10 tid=0x00007f6f9c12c800 nid=0x64e3 runnable
  3703.  
  3704. "Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x00007f6f9c017800 nid=0x64de runnable
  3705.  
  3706. "Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x00007f6f9c019800 nid=0x64df runnable
  3707.  
  3708. "Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x00007f6f9c01b000 nid=0x64e0 runnable
  3709.  
  3710. "Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x00007f6f9c01d000 nid=0x64e1 runnable
  3711.  
  3712. "Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00007f6f9c0ac000 nid=0x64e2 runnable
  3713. "VM Periodic Task Thread" prio=10 tid=0x00007f6f9c165800 nid=0x64eb waiting on condition
  3714.  
  3715. JNI global references: 1230
  3716.  
  3717. Heap
  3718. par new generation total 19136K, used 10935K [0x000000050ce00000, 0x000000050e2c0000, 0x0000000512130000)
  3719. eden space 17024K, 58% used [0x000000050ce00000, 0x000000050d7ac190, 0x000000050dea0000)
  3720. from space 2112K, 48% used [0x000000050dea0000, 0x000000050dfa1d50, 0x000000050e0b0000)
  3721. to space 2112K, 0% used [0x000000050e0b0000, 0x000000050e0b0000, 0x000000050e2c0000)
  3722. concurrent mark-sweep generation total 5757924K, used 4478114K [0x0000000512130000, 0x0000000671829000, 0x00000007fae00000)
  3723. concurrent-mark-sweep perm gen total 30248K, used 18065K [0x00000007fae00000, 0x00000007fcb8a000, 0x0000000800000000)
  3724.  
  3725. 2011-05-14 15:51:41
  3726. Full thread dump Java HotSpot(TM) 64-Bit Server VM (19.0-b09 mixed mode):
  3727.  
  3728. "IPC Client (47) connection to mtag1/10.103.7.1:9000 from root" daemon prio=10 tid=0x00007f6f501ac000 nid=0xfa5 in Object.wait() [0x00007f6f5d041000]
  3729. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  3730. at java.lang.Object.wait(Native Method)
  3731. at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:433)
  3732. - locked <0x000000061874eee8> (a org.apache.hadoop.ipc.Client$Connection)
  3733. at org.apache.hadoop.ipc.Client$Connection.run(Client.java:476)
  3734.  
  3735. "ResponseProcessor for block blk_1030279999998547108_1158319" daemon prio=10 tid=0x00007f6f5448f800 nid=0x24c0 runnable [0x00007f6f5cd3e000]
  3736. java.lang.Thread.State: RUNNABLE
  3737. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  3738. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  3739. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  3740. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  3741. - locked <0x00000006233859a0> (a sun.nio.ch.Util$2)
  3742. - locked <0x0000000623385988> (a java.util.Collections$UnmodifiableSet)
  3743. - locked <0x0000000623414c18> (a sun.nio.ch.EPollSelectorImpl)
  3744. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  3745. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:332)
  3746. at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
  3747. at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
  3748. at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
  3749. at java.io.DataInputStream.readFully(DataInputStream.java:178)
  3750. at java.io.DataInputStream.readLong(DataInputStream.java:399)
  3751. at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:119)
  3752. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2468)
  3753.  
  3754. "DataStreamer for file /hbase/.logs/mtag27.prod.imageshack.com,60020,1305408211142/10.103.7.27%3A60020.1305411438693 block blk_1030279999998547108_1158319" daemon prio=10 tid=0x00007f6f2c009000 nid=0x24bc in Object.wait() [0x00007f6f03dbc000]
  3755. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  3756. at java.lang.Object.wait(Native Method)
  3757. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2313)
  3758. - locked <0x00000005dbac2f88> (a java.util.LinkedList)
  3759.  
  3760. "pool-3-thread-1" prio=10 tid=0x00007f6f141cb000 nid=0x6557 waiting on condition [0x00007f6f03fbe000]
  3761. java.lang.Thread.State: TIMED_WAITING (parking)
  3762. at sun.misc.Unsafe.park(Native Method)
  3763. - parking to wait for <0x00000005138a8b30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  3764. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
  3765. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
  3766. at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
  3767. at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
  3768. at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
  3769. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
  3770. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  3771. at java.lang.Thread.run(Thread.java:662)
  3772.  
  3773. "LruBlockCache.EvictionThread" daemon prio=10 tid=0x00007f6f141c9800 nid=0x6556 in Object.wait() [0x00007f6f040bf000]
  3774. java.lang.Thread.State: WAITING (on object monitor)
  3775. at java.lang.Object.wait(Native Method)
  3776. - waiting on <0x00000005138232e0> (a org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread)
  3777. at java.lang.Object.wait(Object.java:485)
  3778. at org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:514)
  3779. - locked <0x00000005138232e0> (a org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread)
  3780.  
  3781. "PRI IPC Server handler 9 on 60020" daemon prio=10 tid=0x00007f6f141c7800 nid=0x6555 waiting on condition [0x00007f6f041c0000]
  3782. java.lang.Thread.State: WAITING (parking)
  3783. at sun.misc.Unsafe.park(Native Method)
  3784. - parking to wait for <0x0000000513679408> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  3785. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  3786. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  3787. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  3788. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1013)
  3789.  
  3790. "PRI IPC Server handler 8 on 60020" daemon prio=10 tid=0x00007f6f141c5800 nid=0x6554 waiting on condition [0x00007f6f042c1000]
  3791. java.lang.Thread.State: WAITING (parking)
  3792. at sun.misc.Unsafe.park(Native Method)
  3793. - parking to wait for <0x0000000513679408> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  3794. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  3795. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  3796. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  3797. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1013)
  3798.  
  3799. "PRI IPC Server handler 7 on 60020" daemon prio=10 tid=0x00007f6f141c3800 nid=0x6553 waiting on condition [0x00007f6f043c2000]
  3800. java.lang.Thread.State: WAITING (parking)
  3801. at sun.misc.Unsafe.park(Native Method)
  3802. - parking to wait for <0x0000000513679408> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  3803. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  3804. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  3805. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  3806. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1013)
  3807.  
  3808. "PRI IPC Server handler 6 on 60020" daemon prio=10 tid=0x00007f6f141c1800 nid=0x6552 waiting on condition [0x00007f6f044c3000]
  3809. java.lang.Thread.State: WAITING (parking)
  3810. at sun.misc.Unsafe.park(Native Method)
  3811. - parking to wait for <0x0000000513679408> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  3812. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  3813. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  3814. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  3815. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1013)
  3816.  
  3817. "PRI IPC Server handler 5 on 60020" daemon prio=10 tid=0x00007f6f141bf800 nid=0x6551 waiting on condition [0x00007f6f045c4000]
  3818. java.lang.Thread.State: WAITING (parking)
  3819. at sun.misc.Unsafe.park(Native Method)
  3820. - parking to wait for <0x0000000513679408> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  3821. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  3822. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  3823. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  3824. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1013)
  3825.  
  3826. "PRI IPC Server handler 4 on 60020" daemon prio=10 tid=0x00007f6f141bd800 nid=0x6550 waiting on condition [0x00007f6f046c5000]
  3827. java.lang.Thread.State: WAITING (parking)
  3828. at sun.misc.Unsafe.park(Native Method)
  3829. - parking to wait for <0x0000000513679408> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  3830. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  3831. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  3832. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  3833. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1013)
  3834.  
  3835. "PRI IPC Server handler 3 on 60020" daemon prio=10 tid=0x00007f6f141bb800 nid=0x654f waiting on condition [0x00007f6f047c6000]
  3836. java.lang.Thread.State: WAITING (parking)
  3837. at sun.misc.Unsafe.park(Native Method)
  3838. - parking to wait for <0x0000000513679408> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  3839. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  3840. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  3841. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  3842. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1013)
  3843.  
  3844. "PRI IPC Server handler 2 on 60020" daemon prio=10 tid=0x00007f6f141b9800 nid=0x654e waiting on condition [0x00007f6f048c7000]
  3845. java.lang.Thread.State: WAITING (parking)
  3846. at sun.misc.Unsafe.park(Native Method)
  3847. - parking to wait for <0x0000000513679408> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  3848. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  3849. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  3850. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  3851. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1013)
  3852.  
  3853. "PRI IPC Server handler 1 on 60020" daemon prio=10 tid=0x00007f6f141b7800 nid=0x654d waiting on condition [0x00007f6f049c8000]
  3854. java.lang.Thread.State: WAITING (parking)
  3855. at sun.misc.Unsafe.park(Native Method)
  3856. - parking to wait for <0x0000000513679408> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  3857. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  3858. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  3859. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  3860. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1013)
  3861.  
  3862. "PRI IPC Server handler 0 on 60020" daemon prio=10 tid=0x00007f6f141b5800 nid=0x654c waiting on condition [0x00007f6f04ac9000]
  3863. java.lang.Thread.State: WAITING (parking)
  3864. at sun.misc.Unsafe.park(Native Method)
  3865. - parking to wait for <0x0000000513679408> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  3866. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  3867. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  3868. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  3869. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1013)
  3870.  
  3871. "IPC Server handler 49 on 60020" daemon prio=10 tid=0x00007f6f141b3800 nid=0x654b runnable [0x00007f6f04bca000]
  3872. java.lang.Thread.State: RUNNABLE
  3873. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:566)
  3874. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  3875. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  3876. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  3877. - locked <0x000000050e0db300> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  3878. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  3879. - locked <0x000000050e0db300> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  3880. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  3881. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  3882. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  3883. - locked <0x000000050e0db368> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  3884. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  3885. - locked <0x000000050e0db368> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  3886. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  3887. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  3888. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  3889. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  3890. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  3891. at java.lang.reflect.Method.invoke(Method.java:597)
  3892. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  3893. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  3894.  
  3895. "IPC Server handler 48 on 60020" daemon prio=10 tid=0x00007f6f141b1800 nid=0x654a runnable [0x00007f6f04ccb000]
  3896. java.lang.Thread.State: RUNNABLE
  3897. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  3898. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  3899. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  3900. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  3901. - locked <0x000000050e106578> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  3902. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  3903. - locked <0x000000050e106578> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  3904. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  3905. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  3906. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  3907. - locked <0x000000050e1065e0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  3908. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  3909. - locked <0x000000050e1065e0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  3910. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  3911. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  3912. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  3913. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  3914. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  3915. at java.lang.reflect.Method.invoke(Method.java:597)
  3916. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  3917. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  3918.  
  3919. "IPC Server handler 47 on 60020" daemon prio=10 tid=0x00007f6f141af000 nid=0x6549 runnable [0x00007f6f04dcc000]
  3920. java.lang.Thread.State: RUNNABLE
  3921. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  3922. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  3923. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  3924. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  3925. - locked <0x0000000625000e00> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  3926. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  3927. - locked <0x0000000625000e00> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  3928. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  3929. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  3930. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  3931. - locked <0x00000006250aae80> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  3932. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  3933. - locked <0x00000006250aae80> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  3934. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  3935. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  3936. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  3937. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  3938. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  3939. at java.lang.reflect.Method.invoke(Method.java:597)
  3940. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  3941. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  3942.  
  3943. "IPC Server handler 46 on 60020" daemon prio=10 tid=0x00007f6f141ad000 nid=0x6548 runnable [0x00007f6f04ecd000]
  3944. java.lang.Thread.State: RUNNABLE
  3945. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:566)
  3946. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  3947. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  3948. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  3949. - locked <0x000000050e0b00c0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  3950. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  3951. - locked <0x000000050e0b00c0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  3952. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  3953. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  3954. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  3955. - locked <0x000000050e0b0128> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  3956. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  3957. - locked <0x000000050e0b0128> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  3958. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  3959. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  3960. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  3961. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  3962. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  3963. at java.lang.reflect.Method.invoke(Method.java:597)
  3964. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  3965. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  3966.  
  3967. "IPC Server handler 45 on 60020" daemon prio=10 tid=0x00007f6f141ab000 nid=0x6547 runnable [0x00007f6f04fcd000]
  3968. java.lang.Thread.State: RUNNABLE
  3969. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  3970. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  3971. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  3972. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  3973. - locked <0x000000050e0db478> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  3974. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  3975. - locked <0x000000050e0db478> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  3976. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  3977. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  3978. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  3979. - locked <0x000000050e0db4e0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  3980. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  3981. - locked <0x000000050e0db4e0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  3982. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  3983. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  3984. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  3985. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  3986. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  3987. at java.lang.reflect.Method.invoke(Method.java:597)
  3988. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  3989. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  3990.  
  3991. "IPC Server handler 44 on 60020" daemon prio=10 tid=0x00007f6f141a9000 nid=0x6546 runnable [0x00007f6f050ce000]
  3992. java.lang.Thread.State: RUNNABLE
  3993. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  3994. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  3995. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  3996. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  3997. - locked <0x000000050e131830> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  3998. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  3999. - locked <0x000000050e131830> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4000. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4001. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4002. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4003. - locked <0x000000050e131898> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4004. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4005. - locked <0x000000050e131898> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4006. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4007. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4008. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4009. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4010. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4011. at java.lang.reflect.Method.invoke(Method.java:597)
  4012. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4013. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4014.  
  4015. "IPC Server handler 43 on 60020" daemon prio=10 tid=0x00007f6f141a7000 nid=0x6545 runnable [0x00007f6f051d0000]
  4016. java.lang.Thread.State: RUNNABLE
  4017. at java.util.concurrent.ConcurrentSkipListMap$SubMap$SubMapEntryIterator.next(ConcurrentSkipListMap.java:3113)
  4018. at java.util.concurrent.ConcurrentSkipListMap$SubMap$SubMapEntryIterator.next(ConcurrentSkipListMap.java:3108)
  4019. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:74)
  4020. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:62)
  4021. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:561)
  4022. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  4023. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  4024. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  4025. - locked <0x000000050e106700> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4026. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  4027. - locked <0x000000050e106700> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4028. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4029. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4030. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4031. - locked <0x000000050e106768> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4032. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4033. - locked <0x000000050e106768> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4034. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4035. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4036. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4037. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4038. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4039. at java.lang.reflect.Method.invoke(Method.java:597)
  4040. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4041. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4042.  
  4043. "IPC Server handler 42 on 60020" daemon prio=10 tid=0x00007f6f141a5000 nid=0x6544 runnable [0x00007f6f052d1000]
  4044. java.lang.Thread.State: RUNNABLE
  4045. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  4046. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  4047. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  4048. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  4049. - locked <0x0000000625000780> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4050. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  4051. - locked <0x0000000625000780> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4052. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4053. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4054. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4055. - locked <0x0000000624e9fbe8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4056. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4057. - locked <0x0000000624e9fbe8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4058. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4059. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4060. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4061. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4062. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4063. at java.lang.reflect.Method.invoke(Method.java:597)
  4064. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4065. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4066.  
  4067. "IPC Server handler 41 on 60020" daemon prio=10 tid=0x00007f6f141a3000 nid=0x6543 runnable [0x00007f6f053d2000]
  4068. java.lang.Thread.State: RUNNABLE
  4069. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:74)
  4070. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:62)
  4071. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:561)
  4072. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  4073. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  4074. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  4075. - locked <0x000000050e0db5f0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4076. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  4077. - locked <0x000000050e0db5f0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4078. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4079. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4080. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4081. - locked <0x000000050e0db658> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4082. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4083. - locked <0x000000050e0db658> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4084. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4085. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4086. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4087. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4088. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4089. at java.lang.reflect.Method.invoke(Method.java:597)
  4090. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4091. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4092.  
  4093. "IPC Server handler 40 on 60020" daemon prio=10 tid=0x00007f6f141a1000 nid=0x6542 runnable [0x00007f6f054d3000]
  4094. java.lang.Thread.State: RUNNABLE
  4095. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:74)
  4096. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:62)
  4097. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:561)
  4098. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  4099. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  4100. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  4101. - locked <0x000000050e1068a0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4102. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  4103. - locked <0x000000050e1068a0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4104. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4105. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4106. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4107. - locked <0x000000050e106908> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4108. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4109. - locked <0x000000050e106908> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4110. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4111. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4112. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4113. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4114. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4115. at java.lang.reflect.Method.invoke(Method.java:597)
  4116. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4117. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4118.  
  4119. "IPC Server handler 39 on 60020" daemon prio=10 tid=0x00007f6f1419f000 nid=0x6541 runnable [0x00007f6f055d4000]
  4120. java.lang.Thread.State: RUNNABLE
  4121. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:74)
  4122. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:62)
  4123. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:561)
  4124. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  4125. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  4126. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  4127. - locked <0x000000050e0b0260> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4128. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  4129. - locked <0x000000050e0b0260> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4130. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4131. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4132. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4133. - locked <0x000000050e0b02c8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4134. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4135. - locked <0x000000050e0b02c8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4136. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4137. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4138. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4139. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4140. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4141. at java.lang.reflect.Method.invoke(Method.java:597)
  4142. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4143. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4144.  
  4145. "IPC Server handler 38 on 60020" daemon prio=10 tid=0x00007f6f1419d000 nid=0x6540 runnable [0x00007f6f056d5000]
  4146. java.lang.Thread.State: RUNNABLE
  4147. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:74)
  4148. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:62)
  4149. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:561)
  4150. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  4151. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  4152. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  4153. - locked <0x000000050e131a00> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4154. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  4155. - locked <0x000000050e131a00> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4156. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4157. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4158. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4159. - locked <0x000000050e131a68> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4160. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4161. - locked <0x000000050e131a68> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4162. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4163. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4164. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4165. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4166. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4167. at java.lang.reflect.Method.invoke(Method.java:597)
  4168. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4169. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4170.  
  4171. "IPC Server handler 37 on 60020" daemon prio=10 tid=0x00007f6f1419b000 nid=0x653f runnable [0x00007f6f057d5000]
  4172. java.lang.Thread.State: RUNNABLE
  4173. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  4174. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  4175. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  4176. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  4177. - locked <0x000000050cec9228> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4178. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  4179. - locked <0x000000050cec9228> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4180. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4181. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4182. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4183. - locked <0x000000050cec9128> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4184. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4185. - locked <0x000000050cec9128> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4186. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4187. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4188. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4189. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4190. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4191. at java.lang.reflect.Method.invoke(Method.java:597)
  4192. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4193. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4194.  
  4195. "IPC Server handler 36 on 60020" daemon prio=10 tid=0x00007f6f14199000 nid=0x653e runnable [0x00007f6f058d6000]
  4196. java.lang.Thread.State: RUNNABLE
  4197. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  4198. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  4199. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  4200. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  4201. - locked <0x0000000625001080> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4202. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  4203. - locked <0x0000000625001080> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4204. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4205. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4206. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4207. - locked <0x0000000624e941f0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4208. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4209. - locked <0x0000000624e941f0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4210. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4211. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4212. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4213. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4214. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4215. at java.lang.reflect.Method.invoke(Method.java:597)
  4216. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4217. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4218.  
  4219. "IPC Server handler 35 on 60020" daemon prio=10 tid=0x00007f6f14197000 nid=0x653d runnable [0x00007f6f059d8000]
  4220. java.lang.Thread.State: RUNNABLE
  4221. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:566)
  4222. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  4223. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  4224. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  4225. - locked <0x000000050e0b03e8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4226. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  4227. - locked <0x000000050e0b03e8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4228. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4229. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4230. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4231. - locked <0x000000050e0b0450> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4232. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4233. - locked <0x000000050e0b0450> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4234. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4235. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4236. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4237. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4238. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4239. at java.lang.reflect.Method.invoke(Method.java:597)
  4240. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4241. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4242.  
  4243. "IPC Server handler 34 on 60020" daemon prio=10 tid=0x00007f6f14195000 nid=0x653c runnable [0x00007f6f05ad9000]
  4244. java.lang.Thread.State: RUNNABLE
  4245. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  4246. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  4247. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  4248. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  4249. - locked <0x000000050e131b78> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4250. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  4251. - locked <0x000000050e131b78> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4252. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4253. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4254. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4255. - locked <0x000000050e132348> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4256. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4257. - locked <0x000000050e132348> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4258. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4259. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4260. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4261. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4262. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4263. at java.lang.reflect.Method.invoke(Method.java:597)
  4264. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4265. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4266.  
  4267. "IPC Server handler 33 on 60020" daemon prio=10 tid=0x00007f6f14193000 nid=0x653b runnable [0x00007f6f05bda000]
  4268. java.lang.Thread.State: RUNNABLE
  4269. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:566)
  4270. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  4271. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  4272. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  4273. - locked <0x000000050e0b0570> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4274. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  4275. - locked <0x000000050e0b0570> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4276. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4277. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4278. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4279. - locked <0x000000050e0b0de0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4280. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4281. - locked <0x000000050e0b0de0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4282. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4283. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4284. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4285. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4286. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4287. at java.lang.reflect.Method.invoke(Method.java:597)
  4288. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4289. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4290.  
  4291. "IPC Server handler 32 on 60020" daemon prio=10 tid=0x00007f6f14191000 nid=0x653a runnable [0x00007f6f05cdb000]
  4292. java.lang.Thread.State: RUNNABLE
  4293. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:566)
  4294. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  4295. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  4296. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  4297. - locked <0x0000000625001140> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4298. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  4299. - locked <0x0000000625001140> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4300. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4301. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4302. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4303. - locked <0x00000006250d6d80> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4304. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4305. - locked <0x00000006250d6d80> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4306. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4307. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4308. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4309. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4310. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4311. at java.lang.reflect.Method.invoke(Method.java:597)
  4312. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4313. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4314.  
  4315. "IPC Server handler 31 on 60020" daemon prio=10 tid=0x00007f6f1418f000 nid=0x6539 runnable [0x00007f6f05ddc000]
  4316. java.lang.Thread.State: RUNNABLE
  4317. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  4318. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  4319. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  4320. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  4321. - locked <0x000000050e0db830> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4322. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  4323. - locked <0x000000050e0db830> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4324. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4325. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4326. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4327. - locked <0x000000050e0dc190> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4328. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4329. - locked <0x000000050e0dc190> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4330. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4331. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4332. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4333. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4334. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4335. at java.lang.reflect.Method.invoke(Method.java:597)
  4336. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4337. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4338.  
  4339. "IPC Server handler 30 on 60020" daemon prio=10 tid=0x00007f6f1418d000 nid=0x6538 runnable [0x00007f6f05edd000]
  4340. java.lang.Thread.State: RUNNABLE
  4341. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  4342. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  4343. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  4344. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  4345. - locked <0x000000050e132848> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4346. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  4347. - locked <0x000000050e132848> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4348. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4349. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4350. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4351. - locked <0x000000050e133140> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4352. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4353. - locked <0x000000050e133140> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4354. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4355. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4356. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4357. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4358. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4359. at java.lang.reflect.Method.invoke(Method.java:597)
  4360. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4361. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4362.  
  4363. "IPC Server handler 29 on 60020" daemon prio=10 tid=0x00007f6f1418b000 nid=0x6537 runnable [0x00007f6f05fdd000]
  4364. java.lang.Thread.State: RUNNABLE
  4365. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  4366. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  4367. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  4368. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  4369. - locked <0x000000050e0b12d8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4370. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  4371. - locked <0x000000050e0b12d8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4372. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4373. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4374. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4375. - locked <0x000000050e0b1bd0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4376. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4377. - locked <0x000000050e0b1bd0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4378. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4379. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4380. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4381. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4382. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4383. at java.lang.reflect.Method.invoke(Method.java:597)
  4384. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4385. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4386.  
  4387. "IPC Server handler 28 on 60020" daemon prio=10 tid=0x00007f6f14189000 nid=0x6536 runnable [0x00007f6f060de000]
  4388. java.lang.Thread.State: RUNNABLE
  4389. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:74)
  4390. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:62)
  4391. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:561)
  4392. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  4393. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  4394. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  4395. - locked <0x0000000625001400> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4396. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  4397. - locked <0x0000000625001400> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4398. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4399. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4400. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4401. - locked <0x00000006250d6958> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4402. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4403. - locked <0x00000006250d6958> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4404. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4405. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4406. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4407. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4408. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4409. at java.lang.reflect.Method.invoke(Method.java:597)
  4410. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4411. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4412.  
  4413. "IPC Server handler 27 on 60020" daemon prio=10 tid=0x00007f6f14187000 nid=0x6535 runnable [0x00007f6f061e0000]
  4414. java.lang.Thread.State: RUNNABLE
  4415. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:74)
  4416. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:62)
  4417. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:561)
  4418. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  4419. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  4420. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  4421. - locked <0x000000050e106f78> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4422. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  4423. - locked <0x000000050e106f78> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4424. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4425. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4426. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4427. - locked <0x000000050e107870> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4428. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4429. - locked <0x000000050e107870> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4430. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4431. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4432. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4433. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4434. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4435. at java.lang.reflect.Method.invoke(Method.java:597)
  4436. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4437. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4438.  
  4439. "IPC Server handler 26 on 60020" daemon prio=10 tid=0x00007f6f14185000 nid=0x6534 runnable [0x00007f6f062e1000]
  4440. java.lang.Thread.State: RUNNABLE
  4441. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:566)
  4442. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  4443. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  4444. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  4445. - locked <0x0000000624fdd760> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4446. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  4447. - locked <0x0000000624fdd760> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4448. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4449. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4450. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4451. - locked <0x00000006250368f8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4452. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4453. - locked <0x00000006250368f8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4454. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4455. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4456. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4457. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4458. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4459. at java.lang.reflect.Method.invoke(Method.java:597)
  4460. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4461. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4462.  
  4463. "IPC Server handler 25 on 60020" daemon prio=10 tid=0x00007f6f14183000 nid=0x6533 runnable [0x00007f6f063e2000]
  4464. java.lang.Thread.State: RUNNABLE
  4465. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  4466. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  4467. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  4468. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  4469. - locked <0x000000050e0b20d0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4470. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  4471. - locked <0x000000050e0b20d0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4472. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4473. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4474. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4475. - locked <0x000000050e0b29c8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4476. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4477. - locked <0x000000050e0b29c8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4478. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4479. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4480. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4481. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4482. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4483. at java.lang.reflect.Method.invoke(Method.java:597)
  4484. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4485. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4486.  
  4487. "IPC Server handler 24 on 60020" daemon prio=10 tid=0x00007f6f14181000 nid=0x6532 runnable [0x00007f6f064e3000]
  4488. java.lang.Thread.State: RUNNABLE
  4489. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  4490. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  4491. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  4492. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  4493. - locked <0x000000050e1336a8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4494. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  4495. - locked <0x000000050e1336a8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4496. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4497. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4498. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4499. - locked <0x000000050e133fa0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4500. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4501. - locked <0x000000050e133fa0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4502. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4503. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4504. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4505. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4506. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4507. at java.lang.reflect.Method.invoke(Method.java:597)
  4508. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4509. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4510.  
  4511. "IPC Server handler 23 on 60020" daemon prio=10 tid=0x00007f6f1417f000 nid=0x6531 runnable [0x00007f6f065e4000]
  4512. java.lang.Thread.State: RUNNABLE
  4513. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:74)
  4514. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:62)
  4515. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:561)
  4516. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  4517. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  4518. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  4519. - locked <0x00000006250d2b68> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4520. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  4521. - locked <0x00000006250d2b68> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4522. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4523. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4524. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4525. - locked <0x00000006250e0278> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4526. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4527. - locked <0x00000006250e0278> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4528. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4529. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4530. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4531. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4532. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4533. at java.lang.reflect.Method.invoke(Method.java:597)
  4534. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4535. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4536.  
  4537. "IPC Server handler 22 on 60020" daemon prio=10 tid=0x00007f6f1417d000 nid=0x6530 runnable [0x00007f6f066e5000]
  4538. java.lang.Thread.State: RUNNABLE
  4539. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:74)
  4540. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:62)
  4541. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:561)
  4542. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  4543. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  4544. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  4545. - locked <0x00000006250019c0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4546. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  4547. - locked <0x00000006250019c0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4548. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4549. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4550. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4551. - locked <0x00000006250e0198> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4552. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4553. - locked <0x00000006250e0198> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4554. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4555. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4556. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4557. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4558. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4559. at java.lang.reflect.Method.invoke(Method.java:597)
  4560. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4561. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4562.  
  4563. "IPC Server handler 21 on 60020" daemon prio=10 tid=0x00007f6f1417b800 nid=0x652f runnable [0x00007f6f067e6000]
  4564. java.lang.Thread.State: RUNNABLE
  4565. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:566)
  4566. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  4567. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  4568. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  4569. - locked <0x000000050e0dc7d0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4570. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  4571. - locked <0x000000050e0dc7d0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4572. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4573. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4574. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4575. - locked <0x000000050e0dd0c8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4576. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4577. - locked <0x000000050e0dd0c8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4578. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4579. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4580. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4581. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4582. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4583. at java.lang.reflect.Method.invoke(Method.java:597)
  4584. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4585. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4586.  
  4587. "IPC Server handler 20 on 60020" daemon prio=10 tid=0x00007f6f1416e800 nid=0x652e runnable [0x00007f6f068e6000]
  4588. java.lang.Thread.State: RUNNABLE
  4589. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:74)
  4590. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:62)
  4591. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:561)
  4592. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  4593. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  4594. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  4595. - locked <0x000000050e1344a0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4596. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  4597. - locked <0x000000050e1344a0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4598. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4599. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4600. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4601. - locked <0x000000050e134d98> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4602. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4603. - locked <0x000000050e134d98> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4604. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4605. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4606. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4607. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4608. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4609. at java.lang.reflect.Method.invoke(Method.java:597)
  4610. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4611. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4612.  
  4613. "IPC Server handler 19 on 60020" daemon prio=10 tid=0x00007f6f1416c800 nid=0x652d runnable [0x00007f6f069e8000]
  4614. java.lang.Thread.State: RUNNABLE
  4615. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:566)
  4616. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  4617. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  4618. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  4619. - locked <0x000000050e107eb8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4620. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  4621. - locked <0x000000050e107eb8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4622. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4623. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4624. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4625. - locked <0x000000050e1087b0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4626. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4627. - locked <0x000000050e1087b0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4628. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4629. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4630. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4631. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4632. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4633. at java.lang.reflect.Method.invoke(Method.java:597)
  4634. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4635. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4636.  
  4637. "IPC Server handler 18 on 60020" daemon prio=10 tid=0x00007f6f1416a800 nid=0x652c runnable [0x00007f6f06ae9000]
  4638. java.lang.Thread.State: RUNNABLE
  4639. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  4640. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  4641. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  4642. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  4643. - locked <0x000000050e0dd5c8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4644. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  4645. - locked <0x000000050e0dd5c8> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4646. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4647. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4648. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4649. - locked <0x000000050e0ddec0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4650. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4651. - locked <0x000000050e0ddec0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4652. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4653. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4654. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4655. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4656. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4657. at java.lang.reflect.Method.invoke(Method.java:597)
  4658. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4659. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4660.  
  4661. "IPC Server handler 17 on 60020" daemon prio=10 tid=0x00007f6f14168800 nid=0x652b runnable [0x00007f6f06bea000]
  4662. java.lang.Thread.State: RUNNABLE
  4663. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  4664. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  4665. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  4666. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  4667. - locked <0x000000050cf85638> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4668. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  4669. - locked <0x000000050cf85638> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4670. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4671. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4672. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4673. - locked <0x000000050cf85538> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4674. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4675. - locked <0x000000050cf85538> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4676. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4677. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4678. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4679. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4680. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4681. at java.lang.reflect.Method.invoke(Method.java:597)
  4682. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4683. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4684.  
  4685. "IPC Server handler 16 on 60020" daemon prio=10 tid=0x00007f6f14166800 nid=0x652a runnable [0x00007f6f06ceb000]
  4686. java.lang.Thread.State: RUNNABLE
  4687. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  4688. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  4689. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  4690. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  4691. - locked <0x000000050e0b3038> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4692. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  4693. - locked <0x000000050e0b3038> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4694. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4695. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4696. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4697. - locked <0x000000050e0b3498> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4698. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4699. - locked <0x000000050e0b3498> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4700. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4701. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4702. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4703. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4704. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4705. at java.lang.reflect.Method.invoke(Method.java:597)
  4706. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4707. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4708.  
  4709. "IPC Server handler 15 on 60020" daemon prio=10 tid=0x00007f6f14164000 nid=0x6529 runnable [0x00007f6f06dec000]
  4710. java.lang.Thread.State: RUNNABLE
  4711. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:74)
  4712. at org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet$MapEntryIterator.next(KeyValueSkipListSet.java:62)
  4713. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:561)
  4714. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  4715. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  4716. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  4717. - locked <0x000000050e135458> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4718. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  4719. - locked <0x000000050e135458> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4720. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4721. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4722. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4723. - locked <0x000000050e135d50> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4724. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4725. - locked <0x000000050e135d50> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4726. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4727. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4728. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4729. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4730. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4731. at java.lang.reflect.Method.invoke(Method.java:597)
  4732. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4733. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4734.  
  4735. "IPC Server handler 14 on 60020" daemon prio=10 tid=0x00007f6f14162000 nid=0x6528 runnable [0x00007f6f06eed000]
  4736. java.lang.Thread.State: RUNNABLE
  4737. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  4738. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  4739. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  4740. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  4741. - locked <0x000000050e108cb0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4742. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  4743. - locked <0x000000050e108cb0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4744. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4745. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4746. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4747. - locked <0x000000050e1095a8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4748. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4749. - locked <0x000000050e1095a8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4750. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4751. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4752. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4753. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4754. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4755. at java.lang.reflect.Method.invoke(Method.java:597)
  4756. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4757. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4758.  
  4759. "IPC Server handler 13 on 60020" daemon prio=10 tid=0x00007f6f14160000 nid=0x6527 runnable [0x00007f6f06fee000]
  4760. java.lang.Thread.State: RUNNABLE
  4761. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:566)
  4762. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  4763. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  4764. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  4765. - locked <0x000000050e0de3d0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4766. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  4767. - locked <0x000000050e0de3d0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4768. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4769. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4770. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4771. - locked <0x000000050e0decc8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4772. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4773. - locked <0x000000050e0decc8> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4774. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4775. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4776. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4777. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4778. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4779. at java.lang.reflect.Method.invoke(Method.java:597)
  4780. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4781. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4782.  
  4783. "IPC Server handler 12 on 60020" daemon prio=10 tid=0x00007f6f1415e800 nid=0x6526 runnable [0x00007f6f070ee000]
  4784. java.lang.Thread.State: RUNNABLE
  4785. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  4786. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  4787. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  4788. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  4789. - locked <0x000000050e0b3998> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4790. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  4791. - locked <0x000000050e0b3998> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4792. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4793. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4794. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4795. - locked <0x000000050e0b4290> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4796. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4797. - locked <0x000000050e0b4290> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4798. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4799. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4800. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4801. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4802. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4803. at java.lang.reflect.Method.invoke(Method.java:597)
  4804. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4805. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4806.  
  4807. "IPC Server handler 11 on 60020" daemon prio=10 tid=0x00007f6f14152800 nid=0x6525 runnable [0x00007f6f071f0000]
  4808. java.lang.Thread.State: RUNNABLE
  4809. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  4810. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  4811. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  4812. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  4813. - locked <0x000000050e136250> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4814. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  4815. - locked <0x000000050e136250> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4816. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4817. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4818. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4819. - locked <0x000000050e136b48> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4820. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4821. - locked <0x000000050e136b48> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4822. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4823. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4824. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4825. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4826. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4827. at java.lang.reflect.Method.invoke(Method.java:597)
  4828. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4829. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4830.  
  4831. "IPC Server handler 10 on 60020" daemon prio=10 tid=0x00007f6f14150800 nid=0x6524 in Object.wait() [0x00007f6f072f1000]
  4832. java.lang.Thread.State: BLOCKED (on object monitor)
  4833. at java.lang.Object.wait(Native Method)
  4834. at java.lang.Object.wait(Object.java:485)
  4835. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.waitForAckedSeqno(DFSClient.java:3284)
  4836. - locked <0x00000005db9b7800> (a java.util.LinkedList)
  4837. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.sync(DFSClient.java:3201)
  4838. at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:97)
  4839. at org.apache.hadoop.io.SequenceFile$Writer.syncFs(SequenceFile.java:944)
  4840. at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
  4841. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4842. at java.lang.reflect.Method.invoke(Method.java:597)
  4843. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:123)
  4844. at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:906)
  4845. at org.apache.hadoop.hbase.regionserver.wal.HLog.append(HLog.java:857)
  4846. at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1402)
  4847. at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1318)
  4848. at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1803)
  4849. at org.apache.hadoop.hbase.regionserver.HRegionServer.multiPut(HRegionServer.java:2464)
  4850. at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
  4851. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4852. at java.lang.reflect.Method.invoke(Method.java:597)
  4853. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4854. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4855.  
  4856. "IPC Server handler 9 on 60020" daemon prio=10 tid=0x00007f6f14247800 nid=0x6523 runnable [0x00007f6f073f2000]
  4857. java.lang.Thread.State: RUNNABLE
  4858. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  4859. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  4860. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  4861. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  4862. - locked <0x00000006250aa660> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4863. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  4864. - locked <0x00000006250aa660> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4865. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4866. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4867. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4868. - locked <0x000000062508da78> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4869. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4870. - locked <0x000000062508da78> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4871. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4872. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4873. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4874. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4875. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4876. at java.lang.reflect.Method.invoke(Method.java:597)
  4877. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4878. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4879.  
  4880. "IPC Server handler 8 on 60020" daemon prio=10 tid=0x00007f6f14245800 nid=0x6522 runnable [0x00007f6f074f3000]
  4881. java.lang.Thread.State: RUNNABLE
  4882. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:566)
  4883. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  4884. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  4885. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  4886. - locked <0x000000050e0b4790> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4887. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  4888. - locked <0x000000050e0b4790> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4889. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4890. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4891. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4892. - locked <0x000000050e0b5058> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4893. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4894. - locked <0x000000050e0b5058> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4895. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4896. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4897. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4898. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4899. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4900. at java.lang.reflect.Method.invoke(Method.java:597)
  4901. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4902. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4903.  
  4904. "IPC Server handler 7 on 60020" daemon prio=10 tid=0x00007f6f14243800 nid=0x6521 runnable [0x00007f6f075f4000]
  4905. java.lang.Thread.State: RUNNABLE
  4906. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  4907. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  4908. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  4909. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  4910. - locked <0x00000006250aac80> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4911. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  4912. - locked <0x00000006250aac80> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4913. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4914. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4915. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4916. - locked <0x00000006250aada0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4917. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4918. - locked <0x00000006250aada0> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4919. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4920. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4921. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4922. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4923. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4924. at java.lang.reflect.Method.invoke(Method.java:597)
  4925. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4926. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4927.  
  4928. "IPC Server handler 6 on 60020" daemon prio=10 tid=0x00007f6f14241800 nid=0x6520 runnable [0x00007f6f076f5000]
  4929. java.lang.Thread.State: RUNNABLE
  4930. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.getNext(MemStore.java:566)
  4931. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:605)
  4932. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  4933. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  4934. - locked <0x0000000625001c20> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4935. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  4936. - locked <0x0000000625001c20> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4937. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4938. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4939. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4940. - locked <0x00000006250e0a20> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4941. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4942. - locked <0x00000006250e0a20> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4943. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4944. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4945. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4946. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4947. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4948. at java.lang.reflect.Method.invoke(Method.java:597)
  4949. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4950. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4951.  
  4952. "IPC Server handler 5 on 60020" daemon prio=10 tid=0x00007f6f1423f800 nid=0x651f runnable [0x00007f6f077f5000]
  4953. java.lang.Thread.State: RUNNABLE
  4954. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  4955. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  4956. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  4957. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  4958. - locked <0x000000050e137048> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4959. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  4960. - locked <0x000000050e137048> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4961. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4962. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4963. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4964. - locked <0x000000050e137940> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4965. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4966. - locked <0x000000050e137940> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4967. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4968. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4969. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4970. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4971. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4972. at java.lang.reflect.Method.invoke(Method.java:597)
  4973. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4974. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4975.  
  4976. "IPC Server handler 4 on 60020" daemon prio=10 tid=0x00007f6f14222800 nid=0x651e runnable [0x00007f6f078f6000]
  4977. java.lang.Thread.State: RUNNABLE
  4978. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  4979. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  4980. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  4981. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  4982. - locked <0x00000006250ab500> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4983. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  4984. - locked <0x00000006250ab500> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  4985. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  4986. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  4987. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  4988. - locked <0x00000006250ab120> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4989. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  4990. - locked <0x00000006250ab120> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  4991. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  4992. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  4993. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  4994. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  4995. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  4996. at java.lang.reflect.Method.invoke(Method.java:597)
  4997. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  4998. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  4999.  
  5000. "IPC Server handler 3 on 60020" daemon prio=10 tid=0x00007f6f14221000 nid=0x651d runnable [0x00007f6f079f8000]
  5001. java.lang.Thread.State: RUNNABLE
  5002. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  5003. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  5004. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  5005. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  5006. - locked <0x0000000625074790> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  5007. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  5008. - locked <0x0000000625074790> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  5009. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  5010. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  5011. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  5012. - locked <0x000000062508df10> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  5013. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  5014. - locked <0x000000062508df10> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  5015. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  5016. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  5017. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  5018. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  5019. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  5020. at java.lang.reflect.Method.invoke(Method.java:597)
  5021. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  5022. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  5023.  
  5024. "IPC Server handler 2 on 60020" daemon prio=10 tid=0x00007f6f1421f800 nid=0x651c runnable [0x00007f6f07af9000]
  5025. java.lang.Thread.State: RUNNABLE
  5026. at sun.misc.Unsafe.unpark(Native Method)
  5027. at java.util.concurrent.locks.LockSupport.unpark(LockSupport.java:124)
  5028. at java.util.concurrent.locks.AbstractQueuedSynchronizer.unparkSuccessor(AbstractQueuedSynchronizer.java:640)
  5029. at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1242)
  5030. at java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:431)
  5031. at java.util.concurrent.LinkedBlockingQueue.signalNotFull(LinkedBlockingQueue.java:147)
  5032. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:409)
  5033. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1013)
  5034.  
  5035. "IPC Server handler 1 on 60020" daemon prio=10 tid=0x00007f6f1421b000 nid=0x651b runnable [0x00007f6f07bfa000]
  5036. java.lang.Thread.State: RUNNABLE
  5037. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  5038. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  5039. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  5040. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  5041. - locked <0x000000050ce10d10> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  5042. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  5043. - locked <0x000000050ce10d10> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  5044. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  5045. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  5046. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  5047. - locked <0x000000050ce10c10> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  5048. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  5049. - locked <0x000000050ce10c10> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  5050. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  5051. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  5052. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  5053. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  5054. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  5055. at java.lang.reflect.Method.invoke(Method.java:597)
  5056. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  5057. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  5058.  
  5059. "IPC Server handler 0 on 60020" daemon prio=10 tid=0x00007f6f14219800 nid=0x651a runnable [0x00007f6f07cfb000]
  5060. java.lang.Thread.State: RUNNABLE
  5061. at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1356)
  5062. at org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:604)
  5063. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:240)
  5064. at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:382)
  5065. - locked <0x000000050e109da0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  5066. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:288)
  5067. - locked <0x000000050e109da0> (a org.apache.hadoop.hbase.regionserver.StoreScanner)
  5068. at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
  5069. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2284)
  5070. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2241)
  5071. - locked <0x000000050e10a698> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  5072. at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2257)
  5073. - locked <0x000000050e10a698> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
  5074. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2943)
  5075. at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2847)
  5076. at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1746)
  5077. at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
  5078. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  5079. at java.lang.reflect.Method.invoke(Method.java:597)
  5080. at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
  5081. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
  5082.  
  5083. "IPC Server listener on 60020" daemon prio=10 tid=0x00007f6f14215000 nid=0x6519 runnable [0x00007f6f07dfc000]
  5084. java.lang.Thread.State: RUNNABLE
  5085. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  5086. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  5087. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  5088. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  5089. - locked <0x0000000513554d80> (a sun.nio.ch.Util$2)
  5090. - locked <0x0000000513554d98> (a java.util.Collections$UnmodifiableSet)
  5091. - locked <0x0000000513554100> (a sun.nio.ch.EPollSelectorImpl)
  5092. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  5093. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
  5094. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:402)
  5095.  
  5096. "IPC Server Responder" daemon prio=10 tid=0x00007f6f1421c800 nid=0x6518 runnable [0x00007f6f07efd000]
  5097. java.lang.Thread.State: RUNNABLE
  5098. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  5099. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  5100. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  5101. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  5102. - locked <0x0000000513554f00> (a sun.nio.ch.Util$2)
  5103. - locked <0x0000000513554f18> (a java.util.Collections$UnmodifiableSet)
  5104. - locked <0x0000000513554900> (a sun.nio.ch.EPollSelectorImpl)
  5105. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  5106. at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:576)
  5107.  
  5108. "Timer-0" daemon prio=10 tid=0x00007f6f14213000 nid=0x6517 in Object.wait() [0x00007f6f07ffe000]
  5109. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  5110. at java.lang.Object.wait(Native Method)
  5111. at java.util.TimerThread.mainLoop(Timer.java:509)
  5112. - locked <0x00000005137d7d70> (a java.util.TaskQueue)
  5113. at java.util.TimerThread.run(Timer.java:462)
  5114.  
  5115. "691789110@qtp-1373164447-1 - Acceptor0 SelectChannelConnector@0.0.0.0:60030" prio=10 tid=0x00007f6f14115800 nid=0x6516 runnable [0x00007f6f5c103000]
  5116. java.lang.Thread.State: RUNNABLE
  5117. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  5118. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  5119. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  5120. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  5121. - locked <0x000000051386c608> (a sun.nio.ch.Util$2)
  5122. - locked <0x000000051386c5f0> (a java.util.Collections$UnmodifiableSet)
  5123. - locked <0x0000000513772568> (a sun.nio.ch.EPollSelectorImpl)
  5124. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  5125. at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:484)
  5126. at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:191)
  5127. at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
  5128. at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
  5129. at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
  5130.  
  5131. "589411738@qtp-1373164447-0" prio=10 tid=0x00007f6f1412a000 nid=0x6515 in Object.wait() [0x00007f6f5c204000]
  5132. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  5133. at java.lang.Object.wait(Native Method)
  5134. at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
  5135. - locked <0x0000000513826e10> (a org.mortbay.thread.QueuedThreadPool$PoolThread)
  5136.  
  5137. "regionserver60020.leaseChecker" prio=10 tid=0x00007f6f1410a800 nid=0x6514 waiting on condition [0x00007f6f5c305000]
  5138. java.lang.Thread.State: TIMED_WAITING (parking)
  5139. at sun.misc.Unsafe.park(Native Method)
  5140. - parking to wait for <0x0000000513554f30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  5141. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
  5142. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
  5143. at java.util.concurrent.DelayQueue.poll(DelayQueue.java:209)
  5144. at org.apache.hadoop.hbase.Leases.run(Leases.java:81)
  5145.  
  5146. "regionserver60020.majorCompactionChecker" daemon prio=10 tid=0x00007f6f14108800 nid=0x6513 in Object.wait() [0x00007f6f5c406000]
  5147. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  5148. at java.lang.Object.wait(Native Method)
  5149. - waiting on <0x0000000513554db0> (a java.lang.Object)
  5150. at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
  5151. - locked <0x0000000513554db0> (a java.lang.Object)
  5152. at org.apache.hadoop.hbase.Chore.run(Chore.java:76)
  5153.  
  5154. "regionserver60020.worker" daemon prio=10 tid=0x00007f6f14106800 nid=0x6512 waiting on condition [0x00007f6f5c507000]
  5155. java.lang.Thread.State: TIMED_WAITING (parking)
  5156. at sun.misc.Unsafe.park(Native Method)
  5157. - parking to wait for <0x0000000513554f90> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  5158. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
  5159. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
  5160. at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424)
  5161. at org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1428)
  5162. at java.lang.Thread.run(Thread.java:662)
  5163.  
  5164. "regionserver60020.compactor" daemon prio=10 tid=0x00007f6f14104800 nid=0x6511 waiting on condition [0x00007f6f5c608000]
  5165. java.lang.Thread.State: TIMED_WAITING (parking)
  5166. at sun.misc.Unsafe.park(Native Method)
  5167. - parking to wait for <0x0000000513554dc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  5168. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
  5169. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
  5170. at java.util.concurrent.PriorityBlockingQueue.poll(PriorityBlockingQueue.java:245)
  5171. at org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue.poll(PriorityCompactionQueue.java:244)
  5172. at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:99)
  5173.  
  5174. "regionserver60020.cacheFlusher" daemon prio=10 tid=0x00007f6f14102800 nid=0x6510 waiting on condition [0x00007f6f5c709000]
  5175. java.lang.Thread.State: TIMED_WAITING (parking)
  5176. at sun.misc.Unsafe.park(Native Method)
  5177. - parking to wait for <0x0000000513554e28> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  5178. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
  5179. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
  5180. at java.util.concurrent.DelayQueue.poll(DelayQueue.java:201)
  5181. at java.util.concurrent.DelayQueue.poll(DelayQueue.java:39)
  5182. at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:142)
  5183.  
  5184. "regionserver60020.logRoller" daemon prio=10 tid=0x00007f6f14101800 nid=0x650f in Object.wait() [0x00007f6f5c80a000]
  5185. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  5186. at java.lang.Object.wait(Native Method)
  5187. at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:67)
  5188. - locked <0x0000000513554e70> (a java.util.concurrent.atomic.AtomicBoolean)
  5189.  
  5190. "Timer thread for monitoring jvm" daemon prio=10 tid=0x00007f6f140c5800 nid=0x650e in Object.wait() [0x00007f6f5c90b000]
  5191. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  5192. at java.lang.Object.wait(Native Method)
  5193. at java.util.TimerThread.mainLoop(Timer.java:509)
  5194. - locked <0x0000000513822ea8> (a java.util.TaskQueue)
  5195. at java.util.TimerThread.run(Timer.java:462)
  5196.  
  5197. "Timer thread for monitoring hbase" daemon prio=10 tid=0x00007f6f140c3800 nid=0x650d in Object.wait() [0x00007f6f5ca0c000]
  5198. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  5199. at java.lang.Object.wait(Native Method)
  5200. at java.util.TimerThread.mainLoop(Timer.java:509)
  5201. - locked <0x0000000513822ed8> (a java.util.TaskQueue)
  5202. at java.util.TimerThread.run(Timer.java:462)
  5203.  
  5204. "regionserver60020.logSyncer" daemon prio=10 tid=0x00007f6f140b2000 nid=0x650c waiting on condition [0x00007f6f5cc3d000]
  5205. java.lang.Thread.State: TIMED_WAITING (sleeping)
  5206. at java.lang.Thread.sleep(Native Method)
  5207. at org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:882)
  5208.  
  5209. "LeaseChecker" daemon prio=10 tid=0x00007f6f140b1000 nid=0x650b waiting on condition [0x00007f6f5cb0d000]
  5210. java.lang.Thread.State: TIMED_WAITING (sleeping)
  5211. at java.lang.Thread.sleep(Native Method)
  5212. at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1071)
  5213. at java.lang.Thread.run(Thread.java:662)
  5214.  
  5215. "regionserver60020-EventThread" daemon prio=10 tid=0x00007f6f14082800 nid=0x6509 waiting on condition [0x00007f6f5ce3f000]
  5216. java.lang.Thread.State: WAITING (parking)
  5217. at sun.misc.Unsafe.park(Native Method)
  5218. - parking to wait for <0x00000005139bbd38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  5219. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  5220. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  5221. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  5222. at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:477)
  5223.  
  5224. "regionserver60020-SendThread(mtag3:2181)" daemon prio=10 tid=0x00007f6f14080800 nid=0x6508 runnable [0x00007f6f5cf40000]
  5225. java.lang.Thread.State: RUNNABLE
  5226. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  5227. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  5228. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  5229. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  5230. - locked <0x00000005139bbd20> (a sun.nio.ch.Util$2)
  5231. - locked <0x00000005139bbd08> (a java.util.Collections$UnmodifiableSet)
  5232. - locked <0x0000000513771ea8> (a sun.nio.ch.EPollSelectorImpl)
  5233. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  5234. at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1066)
  5235.  
  5236. "sendParams-0" daemon prio=10 tid=0x00007f6f14023800 nid=0x6506 waiting on condition [0x00007f6f5d142000]
  5237. java.lang.Thread.State: TIMED_WAITING (parking)
  5238. at sun.misc.Unsafe.park(Native Method)
  5239. - parking to wait for <0x0000000513a1c720> (a java.util.concurrent.SynchronousQueue$TransferStack)
  5240. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
  5241. at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
  5242. at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
  5243. at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
  5244. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
  5245. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  5246. at java.lang.Thread.run(Thread.java:662)
  5247.  
  5248. "IPC Client (47) connection to /10.103.7.2:60000 from an unknown user" daemon prio=10 tid=0x00007f6f14002800 nid=0x6501 in Object.wait() [0x00007f6f5d344000]
  5249. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  5250. at java.lang.Object.wait(Native Method)
  5251. at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.waitForWork(HBaseClient.java:408)
  5252. - locked <0x000000051360cd50> (a org.apache.hadoop.hbase.ipc.HBaseClient$Connection)
  5253. at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:453)
  5254.  
  5255. "DestroyJavaVM" prio=10 tid=0x00007f6f9c009000 nid=0x64dd waiting on condition [0x0000000000000000]
  5256. java.lang.Thread.State: RUNNABLE
  5257.  
  5258. "regionserver60020" prio=10 tid=0x00007f6f9c65c000 nid=0x64fa waiting on condition [0x00007f6f5d445000]
  5259. java.lang.Thread.State: TIMED_WAITING (parking)
  5260. at sun.misc.Unsafe.park(Native Method)
  5261. - parking to wait for <0x00000005135da010> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  5262. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
  5263. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
  5264. at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424)
  5265. at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:647)
  5266. at java.lang.Thread.run(Thread.java:662)
  5267.  
  5268. "main-EventThread" daemon prio=10 tid=0x00007f6f9c65a000 nid=0x64f9 waiting on condition [0x00007f6f5d546000]
  5269. java.lang.Thread.State: WAITING (parking)
  5270. at sun.misc.Unsafe.park(Native Method)
  5271. - parking to wait for <0x0000000513557fd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  5272. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  5273. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  5274. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  5275. at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:477)
  5276.  
  5277. "main-SendThread(mtag3:2181)" daemon prio=10 tid=0x00007f6f9c64f800 nid=0x64f8 runnable [0x00007f6f5d647000]
  5278. java.lang.Thread.State: RUNNABLE
  5279. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  5280. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
  5281. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
  5282. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  5283. - locked <0x00000005135594b8> (a sun.nio.ch.Util$2)
  5284. - locked <0x00000005135594a0> (a java.util.Collections$UnmodifiableSet)
  5285. - locked <0x0000000513554148> (a sun.nio.ch.EPollSelectorImpl)
  5286. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  5287. at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1066)
  5288.  
  5289. "Timer thread for monitoring rpc" daemon prio=10 tid=0x00007f6f9c60d000 nid=0x64f6 in Object.wait() [0x00007f6f5db67000]
  5290. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  5291. at java.lang.Object.wait(Native Method)
  5292. at java.util.TimerThread.mainLoop(Timer.java:509)
  5293. - locked <0x0000000513551978> (a java.util.TaskQueue)
  5294. at java.util.TimerThread.run(Timer.java:462)
  5295.  
  5296. "pool-1-thread-10" prio=10 tid=0x00007f6f9c60a800 nid=0x64f5 runnable [0x00007f6f5dc68000]
  5297. java.lang.Thread.State: WAITING (parking)
  5298. at sun.misc.Unsafe.park(Native Method)
  5299. - parking to wait for <0x000000051367fdd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  5300. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  5301. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  5302. at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
  5303. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:973)
  5304. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:934)
  5305. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:510)
  5306. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:304)
  5307. - locked <0x0000000513551eb0> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  5308. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  5309. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  5310. at java.lang.Thread.run(Thread.java:662)
  5311.  
  5312. "pool-1-thread-9" prio=10 tid=0x00007f6f9c5f0000 nid=0x64f4 waiting on condition [0x00007f6f5dd69000]
  5313. java.lang.Thread.State: WAITING (parking)
  5314. at sun.misc.Unsafe.park(Native Method)
  5315. - parking to wait for <0x000000051367fdd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  5316. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  5317. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  5318. at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
  5319. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:973)
  5320. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:934)
  5321. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:510)
  5322. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:304)
  5323. - locked <0x0000000513551f10> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  5324. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  5325. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  5326. at java.lang.Thread.run(Thread.java:662)
  5327.  
  5328. "pool-1-thread-8" prio=10 tid=0x00007f6f9c5d5800 nid=0x64f3 waiting on condition [0x00007f6f5de6a000]
  5329. java.lang.Thread.State: WAITING (parking)
  5330. at sun.misc.Unsafe.park(Native Method)
  5331. - parking to wait for <0x000000051367fdd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  5332. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  5333. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  5334. at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
  5335. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:973)
  5336. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:934)
  5337. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:510)
  5338. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:304)
  5339. - locked <0x0000000513551f70> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  5340. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  5341. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  5342. at java.lang.Thread.run(Thread.java:662)
  5343.  
  5344. "pool-1-thread-7" prio=10 tid=0x00007f6f9c5ba800 nid=0x64f2 waiting on condition [0x00007f6f5df6b000]
  5345. java.lang.Thread.State: WAITING (parking)
  5346. at sun.misc.Unsafe.park(Native Method)
  5347. - parking to wait for <0x000000051367fdd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  5348. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  5349. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  5350. at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
  5351. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:973)
  5352. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:934)
  5353. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:510)
  5354. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:304)
  5355. - locked <0x0000000513579510> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  5356. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  5357. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  5358. at java.lang.Thread.run(Thread.java:662)
  5359.  
  5360. "pool-1-thread-6" prio=10 tid=0x00007f6f9c59f800 nid=0x64f1 waiting on condition [0x00007f6f5e06c000]
  5361. java.lang.Thread.State: WAITING (parking)
  5362. at sun.misc.Unsafe.park(Native Method)
  5363. - parking to wait for <0x000000051367fdd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  5364. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  5365. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  5366. at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
  5367. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:973)
  5368. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:934)
  5369. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:510)
  5370. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:304)
  5371. - locked <0x0000000513579660> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  5372. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  5373. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  5374. at java.lang.Thread.run(Thread.java:662)
  5375.  
  5376. "pool-1-thread-5" prio=10 tid=0x00007f6f9c584800 nid=0x64f0 waiting on condition [0x00007f6f5e16d000]
  5377. java.lang.Thread.State: WAITING (parking)
  5378. at sun.misc.Unsafe.park(Native Method)
  5379. - parking to wait for <0x000000051367fdd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  5380. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  5381. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  5382. at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
  5383. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:973)
  5384. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:934)
  5385. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:510)
  5386. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:304)
  5387. - locked <0x000000051357b630> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  5388. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  5389. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  5390. at java.lang.Thread.run(Thread.java:662)
  5391.  
  5392. "pool-1-thread-4" prio=10 tid=0x00007f6f9c569800 nid=0x64ef waiting on condition [0x00007f6f7817f000]
  5393. java.lang.Thread.State: WAITING (parking)
  5394. at sun.misc.Unsafe.park(Native Method)
  5395. - parking to wait for <0x000000051367fdd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  5396. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  5397. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  5398. at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
  5399. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:973)
  5400. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:934)
  5401. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:510)
  5402. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:304)
  5403. - locked <0x000000051357b648> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  5404. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  5405. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  5406. at java.lang.Thread.run(Thread.java:662)
  5407.  
  5408. "pool-1-thread-3" prio=10 tid=0x00007f6f9c54f000 nid=0x64ee waiting on condition [0x00007f6f78280000]
  5409. java.lang.Thread.State: WAITING (parking)
  5410. at sun.misc.Unsafe.park(Native Method)
  5411. - parking to wait for <0x000000051367fdd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  5412. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  5413. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  5414. at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
  5415. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:973)
  5416. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:934)
  5417. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:510)
  5418. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:304)
  5419. - locked <0x000000051357b660> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  5420. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  5421. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  5422. at java.lang.Thread.run(Thread.java:662)
  5423.  
  5424. "pool-1-thread-2" prio=10 tid=0x00007f6f9c4d9000 nid=0x64ed waiting on condition [0x00007f6f78381000]
  5425. java.lang.Thread.State: WAITING (parking)
  5426. at sun.misc.Unsafe.park(Native Method)
  5427. - parking to wait for <0x000000051367fdd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  5428. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  5429. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  5430. at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
  5431. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:973)
  5432. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:934)
  5433. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:510)
  5434. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:304)
  5435. - locked <0x000000051357b678> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  5436. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  5437. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  5438. at java.lang.Thread.run(Thread.java:662)
  5439.  
  5440. "pool-1-thread-1" prio=10 tid=0x00007f6f9c4d8800 nid=0x64ec waiting on condition [0x00007f6f78482000]
  5441. java.lang.Thread.State: WAITING (parking)
  5442. at sun.misc.Unsafe.park(Native Method)
  5443. - parking to wait for <0x000000051367fdd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  5444. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  5445. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  5446. at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
  5447. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:973)
  5448. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:934)
  5449. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:510)
  5450. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:304)
  5451. - locked <0x000000051357c5c0> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
  5452. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  5453. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  5454. at java.lang.Thread.run(Thread.java:662)
  5455.  
  5456. "Low Memory Detector" daemon prio=10 tid=0x00007f6f9c15a800 nid=0x64ea runnable [0x0000000000000000]
  5457. java.lang.Thread.State: RUNNABLE
  5458.  
  5459. "CompilerThread1" daemon prio=10 tid=0x00007f6f9c158000 nid=0x64e9 waiting on condition [0x0000000000000000]
  5460. java.lang.Thread.State: RUNNABLE
  5461.  
  5462. "CompilerThread0" daemon prio=10 tid=0x00007f6f9c155800 nid=0x64e8 waiting on condition [0x0000000000000000]
  5463. java.lang.Thread.State: RUNNABLE
  5464.  
  5465. "Signal Dispatcher" daemon prio=10 tid=0x00007f6f9c153800 nid=0x64e7 waiting on condition [0x0000000000000000]
  5466. java.lang.Thread.State: RUNNABLE
  5467.  
  5468. "Surrogate Locker Thread (CMS)" daemon prio=10 tid=0x00007f6f9c151800 nid=0x64e6 waiting on condition [0x0000000000000000]
  5469. java.lang.Thread.State: RUNNABLE
  5470.  
  5471. "Finalizer" daemon prio=10 tid=0x00007f6f9c132800 nid=0x64e5 in Object.wait() [0x00007f6f98b64000]
  5472. java.lang.Thread.State: WAITING (on object monitor)
  5473. at java.lang.Object.wait(Native Method)
  5474. at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
  5475. - locked <0x000000051357c668> (a java.lang.ref.ReferenceQueue$Lock)
  5476. at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
  5477. at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
  5478.  
  5479. "Reference Handler" daemon prio=10 tid=0x00007f6f9c130800 nid=0x64e4 in Object.wait() [0x00007f6f98c65000]
  5480. java.lang.Thread.State: WAITING (on object monitor)
  5481. at java.lang.Object.wait(Native Method)
  5482. at java.lang.Object.wait(Object.java:485)
  5483. at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
  5484. - locked <0x0000000513551e50> (a java.lang.ref.Reference$Lock)
  5485.  
  5486. "VM Thread" prio=10 tid=0x00007f6f9c12c800 nid=0x64e3 runnable
  5487.  
  5488. "Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x00007f6f9c017800 nid=0x64de runnable
  5489.  
  5490. "Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x00007f6f9c019800 nid=0x64df runnable
  5491.  
  5492. "Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x00007f6f9c01b000 nid=0x64e0 runnable
  5493.  
  5494. "Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x00007f6f9c01d000 nid=0x64e1 runnable
  5495.  
  5496. "Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00007f6f9c0ac000 nid=0x64e2 runnable
  5497. "VM Periodic Task Thread" prio=10 tid=0x00007f6f9c165800 nid=0x64eb waiting on condition
  5498.  
  5499. JNI global references: 1230
  5500.  
  5501. Heap
  5502. par new generation total 19136K, used 6038K [0x000000050ce00000, 0x000000050e2c0000, 0x0000000512130000)
  5503. eden space 17024K, 29% used [0x000000050ce00000, 0x000000050d2e2bc8, 0x000000050dea0000)
  5504. from space 2112K, 49% used [0x000000050e0b0000, 0x000000050e1b2f30, 0x000000050e2c0000)
  5505. to space 2112K, 0% used [0x000000050dea0000, 0x000000050dea0000, 0x000000050e0b0000)
  5506. concurrent mark-sweep generation total 5757924K, used 4504846K [0x0000000512130000, 0x0000000671829000, 0x00000007fae00000)
  5507. concurrent-mark-sweep perm gen total 30248K, used 18065K [0x00000007fae00000, 0x00000007fcb8a000, 0x0000000800000000)
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!