API tools faq
paste
dhutchis

jstack of Accumulo stuck BatchWriter

dhutchis
Apr 18th, 2016
193
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 86.75 KB | None | 0 0
raw download clone embed print report
  1. 2016-04-18 13:21:14
  2. Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.77-b03 mixed mode):
  3.  
  4. "org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter 35539" #55838 daemon prio=5 os_prio=0 tid=0x00000000096ee000 nid=0x774c waiting on condition [0x00007f7e389f9000]
  5. java.lang.Thread.State: TIMED_WAITING (parking)
  6. at sun.misc.Unsafe.park(Native Method)
  7. - parking to wait for <0x00000000ecce8030> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  8. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  9. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  10. at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
  11. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
  12. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  13. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  14. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  15. at java.lang.Thread.run(Thread.java:745)
  16.  
  17. "org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter 35538" #55837 daemon prio=5 os_prio=0 tid=0x0000000004673000 nid=0x774b waiting on condition [0x00007f7e377e9000]
  18. java.lang.Thread.State: TIMED_WAITING (parking)
  19. at sun.misc.Unsafe.park(Native Method)
  20. - parking to wait for <0x00000000ecce8030> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  21. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  22. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  23. at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
  24. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
  25. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  26. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  27. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  28. at java.lang.Thread.run(Thread.java:745)
  29.  
  30. "IPC Client (1307157548) connection to localhost/127.0.0.1:9000 from dhutchis" #55836 daemon prio=5 os_prio=0 tid=0x000000000662b800 nid=0x774a in Object.wait() [0x00007f7e378ea000]
  31. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  32. at java.lang.Object.wait(Native Method)
  33. at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:903)
  34. - locked <0x00000000e9993840> (a org.apache.hadoop.ipc.Client$Connection)
  35. at org.apache.hadoop.ipc.Client$Connection.run(Client.java:948)
  36.  
  37. "org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter 35537" #55835 daemon prio=5 os_prio=0 tid=0x0000000003ea1000 nid=0x7749 waiting on condition [0x00007f7e3b6af000]
  38. java.lang.Thread.State: TIMED_WAITING (parking)
  39. at sun.misc.Unsafe.park(Native Method)
  40. - parking to wait for <0x00000000ecce8030> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  41. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  42. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  43. at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
  44. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
  45. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  46. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  47. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  48. at java.lang.Thread.run(Thread.java:745)
  49.  
  50. "Attach Listener" #55827 daemon prio=9 os_prio=0 tid=0x0000000002810800 nid=0x7739 waiting on condition [0x0000000000000000]
  51. java.lang.Thread.State: RUNNABLE
  52.  
  53. "md root major compactor 184" #55784 daemon prio=5 os_prio=0 tid=0x0000000007954800 nid=0x7680 waiting on condition [0x00007f7e3bab3000]
  54. java.lang.Thread.State: TIMED_WAITING (parking)
  55. at sun.misc.Unsafe.park(Native Method)
  56. - parking to wait for <0x00000000eb8d4ec0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  57. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  58. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  59. at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
  60. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
  61. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  62. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  63. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  64. at java.lang.Thread.run(Thread.java:745)
  65.  
  66. "md major compactor 180" #55779 daemon prio=5 os_prio=0 tid=0x000000000c720000 nid=0x7672 waiting on condition [0x00007f7e3aca5000]
  67. java.lang.Thread.State: TIMED_WAITING (parking)
  68. at sun.misc.Unsafe.park(Native Method)
  69. - parking to wait for <0x00000000eaaf29b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  70. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  71. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  72. at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
  73. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
  74. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  75. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  76. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  77. at java.lang.Thread.run(Thread.java:745)
  78.  
  79. "ResponseProcessor for block BP-1039915091-127.0.1.1-1443951085871:blk_1073754983_14166" #16547 daemon prio=5 os_prio=0 tid=0x00000000059da800 nid=0x1cec runnable [0x00007f7e3a8a1000]
  80. java.lang.Thread.State: RUNNABLE
  81. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  82. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
  83. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
  84. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
  85. - locked <0x00000000f6da3348> (a sun.nio.ch.Util$2)
  86. - locked <0x00000000f6da3330> (a java.util.Collections$UnmodifiableSet)
  87. - locked <0x00000000f6d02c98> (a sun.nio.ch.EPollSelectorImpl)
  88. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
  89. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
  90. at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
  91. at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
  92. at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
  93. at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
  94. at java.io.FilterInputStream.read(FilterInputStream.java:83)
  95. at java.io.FilterInputStream.read(FilterInputStream.java:83)
  96. at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1984)
  97. at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
  98. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:796)
  99.  
  100. "Accumulo WALog thread localhost:9997/hdfs://localhost:9000/accumulo/wal/localhost+9997/bf84b87f-5954-4c9d-9bb9-1e726e64080b" #16546 daemon prio=5 os_prio=0 tid=0x0000000006542000 nid=0x1ce9 waiting on condition [0x00007f7e3b3ac000]
  101. java.lang.Thread.State: WAITING (parking)
  102. at sun.misc.Unsafe.park(Native Method)
  103. - parking to wait for <0x00000000ed2fba28> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  104. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  105. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  106. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  107. at org.apache.accumulo.tserver.log.DfsLogger$LogSyncingTask.run(DfsLogger.java:154)
  108. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  109. at java.lang.Thread.run(Thread.java:745)
  110.  
  111. "DataStreamer for file /accumulo/wal/localhost+9997/bf84b87f-5954-4c9d-9bb9-1e726e64080b block BP-1039915091-127.0.1.1-1443951085871:blk_1073754983_14166" #16545 daemon prio=5 os_prio=0 tid=0x000000000c6a1000 nid=0x1ce8 in Object.wait() [0x00007f7e3c3f9000]
  112. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  113. at java.lang.Object.wait(Native Method)
  114. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:502)
  115. - locked <0x00000000edb46598> (a java.util.LinkedList)
  116.  
  117. "ClientPool 2389" #15777 daemon prio=5 os_prio=0 tid=0x000000000448b000 nid=0x154d waiting on condition [0x00007f7e384f6000]
  118. java.lang.Thread.State: TIMED_WAITING (parking)
  119. at sun.misc.Unsafe.park(Native Method)
  120. - parking to wait for <0x00000000eb8d6680> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  121. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  122. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  123. at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
  124. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
  125. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  126. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  127. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  128. at java.lang.Thread.run(Thread.java:745)
  129.  
  130. "ClientPool 2388" #15776 daemon prio=5 os_prio=0 tid=0x00000000045e0000 nid=0x154c waiting on condition [0x00007f7e3c4fa000]
  131. java.lang.Thread.State: TIMED_WAITING (parking)
  132. at sun.misc.Unsafe.park(Native Method)
  133. - parking to wait for <0x00000000eb8d6680> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  134. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  135. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  136. at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
  137. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
  138. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  139. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  140. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  141. at java.lang.Thread.run(Thread.java:745)
  142.  
  143. "ClientPool 2387" #15775 daemon prio=5 os_prio=0 tid=0x0000000002abc800 nid=0x154b waiting on condition [0x00007f7e371e3000]
  144. java.lang.Thread.State: TIMED_WAITING (parking)
  145. at sun.misc.Unsafe.park(Native Method)
  146. - parking to wait for <0x00000000eb8d6680> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  147. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  148. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  149. at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
  150. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
  151. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  152. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  153. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  154. at java.lang.Thread.run(Thread.java:745)
  155.  
  156. "ClientPool 2386" #15774 daemon prio=5 os_prio=0 tid=0x0000000007a44000 nid=0x154a waiting on condition [0x00007f7e3e69d000]
  157. java.lang.Thread.State: TIMED_WAITING (parking)
  158. at sun.misc.Unsafe.park(Native Method)
  159. - parking to wait for <0x00000000eb8d6680> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  160. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  161. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  162. at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
  163. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
  164. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  165. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  166. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  167. at java.lang.Thread.run(Thread.java:745)
  168.  
  169. "ClientPool 2385" #15773 daemon prio=5 os_prio=0 tid=0x0000000008af4800 nid=0x1549 waiting on condition [0x00007f7e37ff1000]
  170. java.lang.Thread.State: TIMED_WAITING (parking)
  171. at sun.misc.Unsafe.park(Native Method)
  172. - parking to wait for <0x00000000eccddfc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  173. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  174. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  175. at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
  176. at org.apache.accumulo.tserver.scan.ScanTask.get(ScanTask.java:108)
  177. at org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.continueMultiScan(TabletServer.java:667)
  178. at org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.continueMultiScan(TabletServer.java:653)
  179. at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
  180. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  181. at java.lang.reflect.Method.invoke(Method.java:498)
  182. at org.apache.accumulo.core.trace.wrappers.RpcServerInvocationHandler.invoke(RpcServerInvocationHandler.java:46)
  183. at org.apache.accumulo.server.rpc.RpcWrapper$1.invoke(RpcWrapper.java:74)
  184. at com.sun.proxy.$Proxy19.continueMultiScan(Unknown Source)
  185. at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$continueMultiScan.getResult(TabletClientService.java:2354)
  186. at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$continueMultiScan.getResult(TabletClientService.java:2338)
  187. at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
  188. at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
  189. at org.apache.accumulo.server.rpc.TimedProcessor.process(TimedProcessor.java:63)
  190. at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:516)
  191. at org.apache.accumulo.server.rpc.CustomNonBlockingServer$1.run(CustomNonBlockingServer.java:78)
  192. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  193. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  194. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  195. at java.lang.Thread.run(Thread.java:745)
  196.  
  197. "ClientPool 2384" #15772 daemon prio=5 os_prio=0 tid=0x0000000003947000 nid=0x1548 waiting on condition [0x00007f7e3a5a0000]
  198. java.lang.Thread.State: TIMED_WAITING (parking)
  199. at sun.misc.Unsafe.park(Native Method)
  200. - parking to wait for <0x00000000eb8d6680> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  201. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  202. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  203. at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
  204. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
  205. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  206. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  207. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  208. at java.lang.Thread.run(Thread.java:745)
  209.  
  210. "ClientPool 2383" #15771 daemon prio=5 os_prio=0 tid=0x0000000008f18800 nid=0x1547 waiting on condition [0x00007f7e3b9b2000]
  211. java.lang.Thread.State: TIMED_WAITING (parking)
  212. at sun.misc.Unsafe.park(Native Method)
  213. - parking to wait for <0x00000000eb8d6680> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  214. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  215. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  216. at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
  217. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
  218. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  219. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  220. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  221. at java.lang.Thread.run(Thread.java:745)
  222.  
  223. "ClientPool 2382" #15770 daemon prio=5 os_prio=0 tid=0x0000000008a9e800 nid=0x1546 waiting on condition [0x00007f7e376e8000]
  224. java.lang.Thread.State: TIMED_WAITING (parking)
  225. at sun.misc.Unsafe.park(Native Method)
  226. - parking to wait for <0x00000000eb8d6680> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  227. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  228. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  229. at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
  230. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
  231. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  232. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  233. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  234. at java.lang.Thread.run(Thread.java:745)
  235.  
  236. "ClientPool 2381" #15769 daemon prio=5 os_prio=0 tid=0x0000000001f00800 nid=0x1545 waiting on condition [0x00007f7e3a9a2000]
  237. java.lang.Thread.State: TIMED_WAITING (parking)
  238. at sun.misc.Unsafe.park(Native Method)
  239. - parking to wait for <0x00000000eb8d6680> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  240. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  241. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  242. at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
  243. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
  244. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  245. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  246. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  247. at java.lang.Thread.run(Thread.java:745)
  248.  
  249. "ClientPool 2380" #15768 daemon prio=5 os_prio=0 tid=0x00000000096eb000 nid=0x1542 waiting on condition [0x00007f7e379eb000]
  250. java.lang.Thread.State: TIMED_WAITING (parking)
  251. at sun.misc.Unsafe.park(Native Method)
  252. - parking to wait for <0x00000000eb8d6680> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  253. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  254. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  255. at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
  256. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
  257. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  258. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  259. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  260. at java.lang.Thread.run(Thread.java:745)
  261.  
  262. "ClientPool 2379" #15767 daemon prio=5 os_prio=0 tid=0x0000000004568800 nid=0x1541 waiting on condition [0x00007f7e3aaa3000]
  263. java.lang.Thread.State: TIMED_WAITING (parking)
  264. at sun.misc.Unsafe.park(Native Method)
  265. - parking to wait for <0x00000000eb8d6680> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  266. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  267. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  268. at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
  269. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
  270. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  271. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  272. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  273. at java.lang.Thread.run(Thread.java:745)
  274.  
  275. "ClientPool 2378" #15766 daemon prio=5 os_prio=0 tid=0x000000000318c800 nid=0x153f waiting on condition [0x00007f7e3c1f7000]
  276. java.lang.Thread.State: TIMED_WAITING (parking)
  277. at sun.misc.Unsafe.park(Native Method)
  278. - parking to wait for <0x00000000eb8d6680> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  279. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  280. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  281. at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
  282. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
  283. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  284. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  285. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  286. at java.lang.Thread.run(Thread.java:745)
  287.  
  288. "ClientPool 2377" #15765 daemon prio=5 os_prio=0 tid=0x0000000003c59800 nid=0x153e waiting on condition [0x00007f7e385f7000]
  289. java.lang.Thread.State: TIMED_WAITING (parking)
  290. at sun.misc.Unsafe.park(Native Method)
  291. - parking to wait for <0x00000000eb8d6680> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  292. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  293. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  294. at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
  295. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
  296. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  297. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  298. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  299. at java.lang.Thread.run(Thread.java:745)
  300.  
  301. "ClientPool 2376" #15764 daemon prio=5 os_prio=0 tid=0x00000000096ec800 nid=0x153d waiting on condition [0x00007f7e38afa000]
  302. java.lang.Thread.State: TIMED_WAITING (parking)
  303. at sun.misc.Unsafe.park(Native Method)
  304. - parking to wait for <0x00000000eb8d6680> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  305. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  306. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  307. at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
  308. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
  309. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  310. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  311. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  312. at java.lang.Thread.run(Thread.java:745)
  313.  
  314. "ClientPool 2375" #15763 daemon prio=5 os_prio=0 tid=0x000000000d19f800 nid=0x153a waiting on condition [0x00007f7e372e4000]
  315. java.lang.Thread.State: TIMED_WAITING (parking)
  316. at sun.misc.Unsafe.park(Native Method)
  317. - parking to wait for <0x00000000eb8d6680> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  318. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  319. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  320. at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
  321. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
  322. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  323. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  324. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  325. at java.lang.Thread.run(Thread.java:745)
  326.  
  327. "ClientPool 2374" #15762 daemon prio=5 os_prio=0 tid=0x000000000a8b9800 nid=0x1539 waiting on condition [0x00007f7e3b7b0000]
  328. java.lang.Thread.State: TIMED_WAITING (parking)
  329. at sun.misc.Unsafe.park(Native Method)
  330. - parking to wait for <0x00000000eb8d6680> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  331. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  332. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  333. at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
  334. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
  335. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  336. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  337. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  338. at java.lang.Thread.run(Thread.java:745)
  339.  
  340. "ClientPool 2373" #15761 daemon prio=5 os_prio=0 tid=0x0000000004247000 nid=0x1538 waiting on condition [0x00007f7e3b8b1000]
  341. java.lang.Thread.State: TIMED_WAITING (parking)
  342. at sun.misc.Unsafe.park(Native Method)
  343. - parking to wait for <0x00000000eb8d6680> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  344. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  345. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  346. at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
  347. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
  348. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  349. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  350. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  351. at java.lang.Thread.run(Thread.java:745)
  352.  
  353. "ClientPool 2372" #15760 daemon prio=5 os_prio=0 tid=0x0000000008a84000 nid=0x1537 waiting on condition [0x00007f7e3fced000]
  354. java.lang.Thread.State: TIMED_WAITING (parking)
  355. at sun.misc.Unsafe.park(Native Method)
  356. - parking to wait for <0x00000000eb8d6680> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  357. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  358. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  359. at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
  360. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
  361. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  362. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  363. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  364. at java.lang.Thread.run(Thread.java:745)
  365.  
  366. "ClientPool 2369" #15757 daemon prio=5 os_prio=0 tid=0x0000000002118800 nid=0x1533 waiting on condition [0x00007f7e374e6000]
  367. java.lang.Thread.State: TIMED_WAITING (parking)
  368. at sun.misc.Unsafe.park(Native Method)
  369. - parking to wait for <0x00000000eb8d6680> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  370. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  371. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  372. at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
  373. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
  374. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  375. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  376. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  377. at java.lang.Thread.run(Thread.java:745)
  378.  
  379. "ClientPool 2350" #15699 daemon prio=5 os_prio=0 tid=0x00000000048dc800 nid=0x14c6 waiting on condition [0x00007f7e373e5000]
  380. java.lang.Thread.State: TIMED_WAITING (parking)
  381. at sun.misc.Unsafe.park(Native Method)
  382. - parking to wait for <0x00000000eb8d6680> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  383. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  384. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  385. at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
  386. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
  387. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  388. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  389. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  390. at java.lang.Thread.run(Thread.java:745)
  391.  
  392. "BatchWriterLatencyTimer" #9472 daemon prio=5 os_prio=0 tid=0x0000000011060000 nid=0x5997 in Object.wait() [0x00007f7e381f3000]
  393. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  394. at java.lang.Object.wait(Native Method)
  395. at java.util.TimerThread.mainLoop(Timer.java:552)
  396. - locked <0x00000000eccddf98> (a java.util.TaskQueue)
  397. at java.util.TimerThread.run(Timer.java:505)
  398.  
  399. "splitter 1" #9425 daemon prio=5 os_prio=0 tid=0x0000000004524000 nid=0x591a waiting on condition [0x00007f7e382f4000]
  400. java.lang.Thread.State: WAITING (parking)
  401. at sun.misc.Unsafe.park(Native Method)
  402. - parking to wait for <0x00000000eb9df3f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  403. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  404. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  405. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  406. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  407. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  408. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  409. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  410. at java.lang.Thread.run(Thread.java:745)
  411.  
  412. "tablet read ahead 16" #1752 daemon prio=5 os_prio=0 tid=0x0000000004916800 nid=0x5a10 waiting on condition [0x00007f7e37bed000]
  413. java.lang.Thread.State: WAITING (parking)
  414. at sun.misc.Unsafe.park(Native Method)
  415. - parking to wait for <0x00000000eb9e5398> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  416. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  417. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  418. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  419. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  420. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  421. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  422. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  423. at java.lang.Thread.run(Thread.java:745)
  424.  
  425. "tablet read ahead 15" #1751 daemon prio=5 os_prio=0 tid=0x0000000005aab000 nid=0x5a0f waiting on condition [0x00007f7e37cee000]
  426. java.lang.Thread.State: WAITING (parking)
  427. at sun.misc.Unsafe.park(Native Method)
  428. - parking to wait for <0x00000000eb9e5398> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  429. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  430. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  431. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  432. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  433. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  434. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  435. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  436. at java.lang.Thread.run(Thread.java:745)
  437.  
  438. "tablet read ahead 14" #1750 daemon prio=5 os_prio=0 tid=0x000000000c0f4800 nid=0x5a0e waiting on condition [0x00007f7e37ef0000]
  439. java.lang.Thread.State: WAITING (parking)
  440. at sun.misc.Unsafe.park(Native Method)
  441. - parking to wait for <0x00000000eb9e5398> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  442. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  443. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  444. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  445. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  446. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  447. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  448. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  449. at java.lang.Thread.run(Thread.java:745)
  450.  
  451. "tablet read ahead 13" #1749 daemon prio=5 os_prio=0 tid=0x0000000004525800 nid=0x5a0d waiting on condition [0x00007f7e3aba4000]
  452. java.lang.Thread.State: WAITING (parking)
  453. at sun.misc.Unsafe.park(Native Method)
  454. - parking to wait for <0x00000000eb9e5398> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  455. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  456. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  457. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  458. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  459. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  460. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  461. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  462. at java.lang.Thread.run(Thread.java:745)
  463.  
  464. "tablet read ahead 12" #1748 daemon prio=5 os_prio=0 tid=0x0000000004525000 nid=0x5a0c waiting on condition [0x00007f7e380f2000]
  465. java.lang.Thread.State: WAITING (parking)
  466. at sun.misc.Unsafe.park(Native Method)
  467. - parking to wait for <0x00000000eb9e5398> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  468. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  469. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  470. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  471. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  472. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  473. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  474. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  475. at java.lang.Thread.run(Thread.java:745)
  476.  
  477. "tablet read ahead 11" #1747 daemon prio=5 os_prio=0 tid=0x0000000008afe800 nid=0x5a0b waiting on condition [0x00007f7e39392000]
  478. java.lang.Thread.State: WAITING (parking)
  479. at sun.misc.Unsafe.park(Native Method)
  480. - parking to wait for <0x00000000eb9e5398> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  481. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  482. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  483. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  484. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  485. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  486. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  487. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  488. at java.lang.Thread.run(Thread.java:745)
  489.  
  490. "tablet read ahead 10" #1745 daemon prio=5 os_prio=0 tid=0x000000000a93e000 nid=0x5a09 waiting on condition [0x00007f7e3c0f6000]
  491. java.lang.Thread.State: WAITING (parking)
  492. at sun.misc.Unsafe.park(Native Method)
  493. - parking to wait for <0x00000000eb9e5398> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  494. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  495. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  496. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  497. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  498. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  499. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  500. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  501. at java.lang.Thread.run(Thread.java:745)
  502.  
  503. "tablet read ahead 9" #1744 daemon prio=5 os_prio=0 tid=0x000000000556e800 nid=0x5a08 waiting on condition [0x00007f7e383f5000]
  504. java.lang.Thread.State: WAITING (parking)
  505. at sun.misc.Unsafe.park(Native Method)
  506. - parking to wait for <0x00000000eb9e5398> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  507. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  508. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  509. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  510. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  511. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  512. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  513. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  514. at java.lang.Thread.run(Thread.java:745)
  515.  
  516. "tablet read ahead 8" #1743 daemon prio=5 os_prio=0 tid=0x000000000a339000 nid=0x5a07 waiting on condition [0x00007f7e3aea7000]
  517. java.lang.Thread.State: WAITING (parking)
  518. at sun.misc.Unsafe.park(Native Method)
  519. - parking to wait for <0x00000000eb9e5398> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  520. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  521. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  522. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  523. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  524. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  525. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  526. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  527. at java.lang.Thread.run(Thread.java:745)
  528.  
  529. "tablet read ahead 7" #1742 daemon prio=5 os_prio=0 tid=0x000000000aac7000 nid=0x5a06 waiting on condition [0x00007f7e38bfb000]
  530. java.lang.Thread.State: WAITING (parking)
  531. at sun.misc.Unsafe.park(Native Method)
  532. - parking to wait for <0x00000000eb9e5398> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  533. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  534. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  535. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  536. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  537. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  538. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  539. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  540. at java.lang.Thread.run(Thread.java:745)
  541.  
  542. "tablet read ahead 6" #1741 daemon prio=5 os_prio=0 tid=0x0000000006b71000 nid=0x5a05 waiting on condition [0x00007f7e3b0a9000]
  543. java.lang.Thread.State: WAITING (parking)
  544. at sun.misc.Unsafe.park(Native Method)
  545. - parking to wait for <0x00000000eb9e5398> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  546. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  547. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  548. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  549. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  550. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  551. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  552. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  553. at java.lang.Thread.run(Thread.java:745)
  554.  
  555. "tablet read ahead 5" #804 daemon prio=5 os_prio=0 tid=0x00000000033d4000 nid=0x4e67 waiting on condition [0x00007f7e3b5ae000]
  556. java.lang.Thread.State: WAITING (parking)
  557. at sun.misc.Unsafe.park(Native Method)
  558. - parking to wait for <0x00000000eb9e5398> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  559. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  560. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  561. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  562. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  563. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  564. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  565. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  566. at java.lang.Thread.run(Thread.java:745)
  567.  
  568. "tablet read ahead 4" #756 daemon prio=5 os_prio=0 tid=0x000000000496e800 nid=0x4db2 waiting on condition [0x00007f7e388f8000]
  569. java.lang.Thread.State: WAITING (parking)
  570. at sun.misc.Unsafe.park(Native Method)
  571. - parking to wait for <0x00000000eb9e5398> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  572. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  573. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  574. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  575. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  576. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  577. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  578. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  579. at java.lang.Thread.run(Thread.java:745)
  580.  
  581. "Client: 127.0.0.1:35056 User: root Start: 1460955404777 Tablet: v3<<" #641 daemon prio=5 os_prio=0 tid=0x0000000006c5b800 nid=0x4834 in Object.wait() [0x00007f7e38cfb000]
  582. java.lang.Thread.State: WAITING (on object monitor)
  583. at java.lang.Object.wait(Native Method)
  584. at java.lang.Object.wait(Object.java:502)
  585. at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.waitRTE(TabletServerBatchWriter.java:459)
  586. at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.addMutation(TabletServerBatchWriter.java:244)
  587. - locked <0x00000000ecd006d0> (a org.apache.accumulo.core.client.impl.TabletServerBatchWriter)
  588. at org.apache.accumulo.core.client.impl.BatchWriterImpl.addMutation(BatchWriterImpl.java:43)
  589. at edu.mit.ll.graphulo.skvi.RemoteWriteIterator.writeUntilSafeOrFinish(RemoteWriteIterator.java:456)
  590. at edu.mit.ll.graphulo.skvi.RemoteWriteIterator.writeWrapper(RemoteWriteIterator.java:405)
  591. at edu.mit.ll.graphulo.skvi.RemoteWriteIterator.seek(RemoteWriteIterator.java:383)
  592. at org.apache.accumulo.core.iterators.WrappingIterator.seek(WrappingIterator.java:101)
  593. at org.apache.accumulo.core.iterators.WrappingIterator.seek(WrappingIterator.java:101)
  594. at org.apache.accumulo.core.iterators.user.VersioningIterator.seek(VersioningIterator.java:81)
  595. at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:135)
  596. at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.seek(SourceSwitchingIterator.java:182)
  597. - locked <0x00000000eccde4a8> (a java.util.ArrayList)
  598. at org.apache.accumulo.tserver.tablet.Tablet.lookup(Tablet.java:735)
  599. at org.apache.accumulo.tserver.tablet.Tablet.lookup(Tablet.java:848)
  600. at org.apache.accumulo.tserver.scan.LookupTask.run(LookupTask.java:113)
  601. at org.apache.htrace.wrappers.TraceRunnable.run(TraceRunnable.java:57)
  602. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  603. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  604. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  605. at java.lang.Thread.run(Thread.java:745)
  606.  
  607. "tablet read ahead 2" #571 daemon prio=5 os_prio=0 tid=0x0000000001ec9000 nid=0x4526 waiting on condition [0x00007f7e39100000]
  608. java.lang.Thread.State: WAITING (parking)
  609. at sun.misc.Unsafe.park(Native Method)
  610. - parking to wait for <0x00000000eb9e5398> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  611. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  612. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  613. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  614. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  615. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  616. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  617. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  618. at java.lang.Thread.run(Thread.java:745)
  619.  
  620. "major compactor 3" #568 daemon prio=5 os_prio=0 tid=0x00000000032ed000 nid=0x4512 in Object.wait() [0x00007f7e38dfd000]
  621. java.lang.Thread.State: WAITING (on object monitor)
  622. at java.lang.Object.wait(Native Method)
  623. at java.lang.Object.wait(Object.java:502)
  624. at org.apache.accumulo.tserver.CompactionQueue.take(CompactionQueue.java:94)
  625. - locked <0x00000000eb9d27b0> (a org.apache.accumulo.tserver.CompactionQueue)
  626. at org.apache.accumulo.tserver.CompactionQueue.take(CompactionQueue.java:34)
  627. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  628. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  629. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  630. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  631. at java.lang.Thread.run(Thread.java:745)
  632.  
  633. "minor compactor 4" #562 daemon prio=5 os_prio=0 tid=0x0000000002d3c000 nid=0x44da waiting on condition [0x00007f7e38efe000]
  634. java.lang.Thread.State: WAITING (parking)
  635. at sun.misc.Unsafe.park(Native Method)
  636. - parking to wait for <0x00000000eb8d4c20> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  637. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  638. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  639. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  640. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  641. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  642. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  643. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  644. at java.lang.Thread.run(Thread.java:745)
  645.  
  646. "major compactor 2" #531 daemon prio=5 os_prio=0 tid=0x000000000542b000 nid=0x439f in Object.wait() [0x00007f7e38fff000]
  647. java.lang.Thread.State: WAITING (on object monitor)
  648. at java.lang.Object.wait(Native Method)
  649. at java.lang.Object.wait(Object.java:502)
  650. at org.apache.accumulo.tserver.CompactionQueue.take(CompactionQueue.java:94)
  651. - locked <0x00000000eb9d27b0> (a org.apache.accumulo.tserver.CompactionQueue)
  652. at org.apache.accumulo.tserver.CompactionQueue.take(CompactionQueue.java:34)
  653. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  654. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  655. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  656. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  657. at java.lang.Thread.run(Thread.java:745)
  658.  
  659. "tablet read ahead 1" #528 daemon prio=5 os_prio=0 tid=0x0000000004025800 nid=0x4392 waiting on condition [0x00007f7e39201000]
  660. java.lang.Thread.State: WAITING (parking)
  661. at sun.misc.Unsafe.park(Native Method)
  662. - parking to wait for <0x00000000eb9e5398> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  663. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  664. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  665. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  666. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  667. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  668. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  669. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  670. at java.lang.Thread.run(Thread.java:745)
  671.  
  672. "major compactor 1" #523 daemon prio=5 os_prio=0 tid=0x000000000a3e9000 nid=0x4379 in Object.wait() [0x00007f7e39693000]
  673. java.lang.Thread.State: WAITING (on object monitor)
  674. at java.lang.Object.wait(Native Method)
  675. at java.lang.Object.wait(Object.java:502)
  676. at org.apache.accumulo.tserver.CompactionQueue.take(CompactionQueue.java:94)
  677. - locked <0x00000000eb9d27b0> (a org.apache.accumulo.tserver.CompactionQueue)
  678. at org.apache.accumulo.tserver.CompactionQueue.take(CompactionQueue.java:34)
  679. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  680. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  681. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  682. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  683. at java.lang.Thread.run(Thread.java:745)
  684.  
  685. "minor compactor 3" #520 daemon prio=5 os_prio=0 tid=0x0000000004970800 nid=0x436e waiting on condition [0x00007f7e3ada6000]
  686. java.lang.Thread.State: WAITING (parking)
  687. at sun.misc.Unsafe.park(Native Method)
  688. - parking to wait for <0x00000000eb8d4c20> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  689. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  690. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  691. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  692. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  693. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  694. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  695. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  696. at java.lang.Thread.run(Thread.java:745)
  697.  
  698. "minor compactor 2" #270 daemon prio=5 os_prio=0 tid=0x00000000040bb800 nid=0x40e8 waiting on condition [0x00007f7e39b96000]
  699. java.lang.Thread.State: WAITING (parking)
  700. at sun.misc.Unsafe.park(Native Method)
  701. - parking to wait for <0x00000000eb8d4c20> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  702. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  703. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  704. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  705. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  706. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  707. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  708. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  709. at java.lang.Thread.run(Thread.java:745)
  710.  
  711. "minor compactor 1" #263 daemon prio=5 os_prio=0 tid=0x0000000002d29000 nid=0x40d3 waiting on condition [0x00007f7e39994000]
  712. java.lang.Thread.State: WAITING (parking)
  713. at sun.misc.Unsafe.park(Native Method)
  714. - parking to wait for <0x00000000eb8d4c20> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  715. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  716. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  717. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  718. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  719. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  720. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  721. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  722. at java.lang.Thread.run(Thread.java:745)
  723.  
  724. "FileWatchdog" #204 daemon prio=5 os_prio=0 tid=0x00000000033e1800 nid=0x405b waiting on condition [0x00007f7e3eda4000]
  725. java.lang.Thread.State: TIMED_WAITING (sleeping)
  726. at java.lang.Thread.sleep(Native Method)
  727. at org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:104)
  728.  
  729. "AsyncAppender-Dispatcher-Thread-82" #203 daemon prio=5 os_prio=0 tid=0x000000000312a000 nid=0x4057 in Object.wait() [0x00007f7e3fdee000]
  730. java.lang.Thread.State: WAITING (on object monitor)
  731. at java.lang.Object.wait(Native Method)
  732. at java.lang.Object.wait(Object.java:502)
  733. at org.apache.log4j.AsyncAppender$Dispatcher.run(AsyncAppender.java:548)
  734. - locked <0x00000000ebc02be8> (a java.util.ArrayList)
  735. at java.lang.Thread.run(Thread.java:745)
  736.  
  737. "metadata tablets read ahead 8" #104 daemon prio=5 os_prio=0 tid=0x0000000002aad800 nid=0x3fab waiting on condition [0x00007f7e39c97000]
  738. java.lang.Thread.State: WAITING (parking)
  739. at sun.misc.Unsafe.park(Native Method)
  740. - parking to wait for <0x00000000eb8d4c68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  741. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  742. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  743. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  744. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  745. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  746. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  747. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  748. at java.lang.Thread.run(Thread.java:745)
  749.  
  750. "metadata tablets read ahead 7" #103 daemon prio=5 os_prio=0 tid=0x00000000033e0000 nid=0x3faa waiting on condition [0x00007f7e39a95000]
  751. java.lang.Thread.State: WAITING (parking)
  752. at sun.misc.Unsafe.park(Native Method)
  753. - parking to wait for <0x00000000eb8d4c68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  754. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  755. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  756. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  757. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  758. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  759. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  760. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  761. at java.lang.Thread.run(Thread.java:745)
  762.  
  763. "metadata tablets read ahead 6" #100 daemon prio=5 os_prio=0 tid=0x0000000002d97800 nid=0x3fa6 waiting on condition [0x00007f7e39d98000]
  764. java.lang.Thread.State: WAITING (parking)
  765. at sun.misc.Unsafe.park(Native Method)
  766. - parking to wait for <0x00000000eb8d4c68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  767. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  768. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  769. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  770. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  771. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  772. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  773. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  774. at java.lang.Thread.run(Thread.java:745)
  775.  
  776. "metadata tablets read ahead 5" #97 daemon prio=5 os_prio=0 tid=0x00000000036f2800 nid=0x3fa3 waiting on condition [0x00007f7e39e99000]
  777. java.lang.Thread.State: WAITING (parking)
  778. at sun.misc.Unsafe.park(Native Method)
  779. - parking to wait for <0x00000000eb8d4c68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  780. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  781. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  782. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  783. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  784. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  785. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  786. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  787. at java.lang.Thread.run(Thread.java:745)
  788.  
  789. "metadata tablets read ahead 4" #94 daemon prio=5 os_prio=0 tid=0x000000000473c800 nid=0x3fa0 waiting on condition [0x00007f7e39f9a000]
  790. java.lang.Thread.State: WAITING (parking)
  791. at sun.misc.Unsafe.park(Native Method)
  792. - parking to wait for <0x00000000eb8d4c68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  793. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  794. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  795. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  796. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  797. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  798. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  799. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  800. at java.lang.Thread.run(Thread.java:745)
  801.  
  802. "metadata tablets read ahead 3" #91 daemon prio=5 os_prio=0 tid=0x00000000045b2000 nid=0x3f9c waiting on condition [0x00007f7e3a09b000]
  803. java.lang.Thread.State: WAITING (parking)
  804. at sun.misc.Unsafe.park(Native Method)
  805. - parking to wait for <0x00000000eb8d4c68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  806. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  807. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  808. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  809. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  810. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  811. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  812. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  813. at java.lang.Thread.run(Thread.java:745)
  814.  
  815. "metadata tablets read ahead 2" #90 daemon prio=5 os_prio=0 tid=0x0000000003dd0800 nid=0x3f9b waiting on condition [0x00007f7e3a19c000]
  816. java.lang.Thread.State: WAITING (parking)
  817. at sun.misc.Unsafe.park(Native Method)
  818. - parking to wait for <0x00000000eb8d4c68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  819. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  820. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  821. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  822. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  823. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  824. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  825. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  826. at java.lang.Thread.run(Thread.java:745)
  827.  
  828. "tablet assignment 2" #89 daemon prio=5 os_prio=0 tid=0x0000000003dcd800 nid=0x3f9a waiting on condition [0x00007f7e3a29d000]
  829. java.lang.Thread.State: WAITING (parking)
  830. at sun.misc.Unsafe.park(Native Method)
  831. - parking to wait for <0x00000000eb8d4f38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  832. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  833. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  834. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  835. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  836. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  837. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  838. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  839. at java.lang.Thread.run(Thread.java:745)
  840.  
  841. "tablet assignment 1" #88 daemon prio=5 os_prio=0 tid=0x0000000002f7f000 nid=0x3f99 waiting on condition [0x00007f7e3a39e000]
  842. java.lang.Thread.State: WAITING (parking)
  843. at sun.misc.Unsafe.park(Native Method)
  844. - parking to wait for <0x00000000eb8d4f38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  845. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  846. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  847. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  848. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  849. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  850. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  851. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  852. at java.lang.Thread.run(Thread.java:745)
  853.  
  854. "metadata tablets read ahead 1" #85 daemon prio=5 os_prio=0 tid=0x000000000380a800 nid=0x3f91 waiting on condition [0x00007f7e3a49f000]
  855. java.lang.Thread.State: WAITING (parking)
  856. at sun.misc.Unsafe.park(Native Method)
  857. - parking to wait for <0x00000000eb8d4c68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  858. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  859. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  860. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  861. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  862. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  863. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  864. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  865. at java.lang.Thread.run(Thread.java:745)
  866.  
  867. "LeaseRenewer:dhutchis@localhost:9000" #74 daemon prio=5 os_prio=0 tid=0x000000000430f800 nid=0x3f83 waiting on condition [0x00007f7e3afa8000]
  868. java.lang.Thread.State: TIMED_WAITING (sleeping)
  869. at java.lang.Thread.sleep(Native Method)
  870. at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438)
  871. at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
  872. at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
  873. at java.lang.Thread.run(Thread.java:745)
  874.  
  875. "Thrift Connection Pool Checker" #62 daemon prio=5 os_prio=0 tid=0x000000000558f800 nid=0x3f70 waiting on condition [0x00007f7e3c2f8000]
  876. java.lang.Thread.State: TIMED_WAITING (sleeping)
  877. at java.lang.Thread.sleep(Native Method)
  878. at org.apache.accumulo.core.client.impl.ThriftTransportPool$Closer.closeConnections(ThriftTransportPool.java:136)
  879. at org.apache.accumulo.core.client.impl.ThriftTransportPool$Closer.run(ThriftTransportPool.java:146)
  880. at java.lang.Thread.run(Thread.java:745)
  881.  
  882. "org.apache.hadoop.hdfs.PeerCache@3d4e061d" #60 daemon prio=5 os_prio=0 tid=0x0000000002d0c800 nid=0x3f6f waiting on condition [0x00007f7e3bff5000]
  883. java.lang.Thread.State: TIMED_WAITING (sleeping)
  884. at java.lang.Thread.sleep(Native Method)
  885. at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
  886. at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
  887. at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
  888. at java.lang.Thread.run(Thread.java:745)
  889.  
  890. "Thread-22" #53 daemon prio=5 os_prio=0 tid=0x0000000003f88800 nid=0x3edb runnable [0x00007f7e3c6fc000]
  891. java.lang.Thread.State: RUNNABLE
  892. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  893. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
  894. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
  895. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
  896. - locked <0x00000000eb8d4cf8> (a sun.nio.ch.Util$2)
  897. - locked <0x00000000eb8d4d10> (a java.util.Collections$UnmodifiableSet)
  898. - locked <0x00000000eb8e05a0> (a sun.nio.ch.EPollSelectorImpl)
  899. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
  900. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
  901. at org.apache.accumulo.server.rpc.CustomNonBlockingServer$SelectAcceptThread.select(CustomNonBlockingServer.java:208)
  902. at org.apache.accumulo.server.rpc.CustomNonBlockingServer$SelectAcceptThread.run(CustomNonBlockingServer.java:184)
  903.  
  904. "Replication Servicer" #52 daemon prio=5 os_prio=0 tid=0x000000000432d000 nid=0x3eda in Object.wait() [0x00007f7e3c7fd000]
  905. java.lang.Thread.State: WAITING (on object monitor)
  906. at java.lang.Object.wait(Native Method)
  907. at java.lang.Thread.join(Thread.java:1245)
  908. - locked <0x00000000eab088c0> (a org.apache.accumulo.server.rpc.CustomNonBlockingServer$SelectAcceptThread)
  909. at java.lang.Thread.join(Thread.java:1319)
  910. at org.apache.accumulo.server.rpc.CustomNonBlockingServer.joinSelector(CustomNonBlockingServer.java:113)
  911. at org.apache.thrift.server.THsHaServer.waitForShutdown(THsHaServer.java:106)
  912. at org.apache.thrift.server.AbstractNonblockingServer.serve(AbstractNonblockingServer.java:95)
  913. at org.apache.accumulo.server.rpc.TServerUtils$2.run(TServerUtils.java:518)
  914. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  915. at java.lang.Thread.run(Thread.java:745)
  916.  
  917. "Thread-21" #51 daemon prio=5 os_prio=0 tid=0x0000000003e9a000 nid=0x3eaf runnable [0x00007f7e3cafe000]
  918. java.lang.Thread.State: RUNNABLE
  919. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  920. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
  921. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
  922. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
  923. - locked <0x00000000eb8d4f98> (a sun.nio.ch.Util$2)
  924. - locked <0x00000000eb8d4f80> (a java.util.Collections$UnmodifiableSet)
  925. - locked <0x00000000eb8e8fd8> (a sun.nio.ch.EPollSelectorImpl)
  926. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
  927. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
  928. at org.apache.accumulo.server.rpc.CustomNonBlockingServer$SelectAcceptThread.select(CustomNonBlockingServer.java:208)
  929. at org.apache.accumulo.server.rpc.CustomNonBlockingServer$SelectAcceptThread.run(CustomNonBlockingServer.java:184)
  930.  
  931. "Thrift Client Server" #50 daemon prio=5 os_prio=0 tid=0x0000000003f8a000 nid=0x3eab in Object.wait() [0x00007f7e3cbff000]
  932. java.lang.Thread.State: WAITING (on object monitor)
  933. at java.lang.Object.wait(Native Method)
  934. at java.lang.Thread.join(Thread.java:1245)
  935. - locked <0x00000000eb8fbde0> (a org.apache.accumulo.server.rpc.CustomNonBlockingServer$SelectAcceptThread)
  936. at java.lang.Thread.join(Thread.java:1319)
  937. at org.apache.accumulo.server.rpc.CustomNonBlockingServer.joinSelector(CustomNonBlockingServer.java:113)
  938. at org.apache.thrift.server.THsHaServer.waitForShutdown(THsHaServer.java:106)
  939. at org.apache.thrift.server.AbstractNonblockingServer.serve(AbstractNonblockingServer.java:95)
  940. at org.apache.accumulo.server.rpc.TServerUtils$2.run(TServerUtils.java:518)
  941. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  942. at java.lang.Thread.run(Thread.java:745)
  943.  
  944. "SpanSender" #49 daemon prio=5 os_prio=0 tid=0x0000000003f8d000 nid=0x3e2b in Object.wait() [0x00007f7e3cd00000]
  945. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  946. at java.lang.Object.wait(Native Method)
  947. at java.util.TimerThread.mainLoop(Timer.java:552)
  948. - locked <0x00000000eb8f5a28> (a java.util.TaskQueue)
  949. at java.util.TimerThread.run(Timer.java:505)
  950.  
  951. "filesystem monitor timer" #48 daemon prio=5 os_prio=0 tid=0x0000000003e13000 nid=0x3e18 in Object.wait() [0x00007f7e3ce01000]
  952. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  953. at java.lang.Object.wait(Native Method)
  954. at java.util.TimerThread.mainLoop(Timer.java:552)
  955. - locked <0x00000000eb8d52f0> (a java.util.TaskQueue)
  956. at java.util.TimerThread.run(Timer.java:505)
  957.  
  958. "Split/MajC initiator" #47 daemon prio=5 os_prio=0 tid=0x0000000003e0e800 nid=0x3e17 waiting on condition [0x00007f7e3cf02000]
  959. java.lang.Thread.State: TIMED_WAITING (sleeping)
  960. at java.lang.Thread.sleep(Native Method)
  961. at org.apache.accumulo.core.util.UtilWaitThread.sleep(UtilWaitThread.java:27)
  962. at org.apache.accumulo.tserver.TabletServer$MajorCompactor.run(TabletServer.java:1820)
  963. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  964. at java.lang.Thread.run(Thread.java:745)
  965.  
  966. "Timer for 'Accumulo' metrics system" #46 daemon prio=5 os_prio=0 tid=0x00000000039ed800 nid=0x3dfa in Object.wait() [0x00007f7e3d44a000]
  967. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  968. at java.lang.Object.wait(Native Method)
  969. at java.util.TimerThread.mainLoop(Timer.java:552)
  970. - locked <0x00000000eb8d7210> (a java.util.TaskQueue)
  971. at java.util.TimerThread.run(Timer.java:505)
  972.  
  973. "Accumulo Minor Compaction Initiator" #45 daemon prio=5 os_prio=0 tid=0x00000000031c0800 nid=0x3df9 waiting on condition [0x00007f7e3d54b000]
  974. java.lang.Thread.State: TIMED_WAITING (sleeping)
  975. at java.lang.Thread.sleep(Native Method)
  976. at org.apache.accumulo.core.util.UtilWaitThread.sleep(UtilWaitThread.java:27)
  977. at org.apache.accumulo.tserver.TabletServerResourceManager$MemoryManagementFramework.manageMemory(TabletServerResourceManager.java:457)
  978. at org.apache.accumulo.tserver.TabletServerResourceManager$MemoryManagementFramework.access$400(TabletServerResourceManager.java:324)
  979. at org.apache.accumulo.tserver.TabletServerResourceManager$MemoryManagementFramework$2.run(TabletServerResourceManager.java:352)
  980. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  981. at java.lang.Thread.run(Thread.java:745)
  982.  
  983. "Accumulo Memory Guard" #44 daemon prio=6 os_prio=0 tid=0x00000000031bf000 nid=0x3df8 waiting on condition [0x00007f7e3d64c000]
  984. java.lang.Thread.State: WAITING (parking)
  985. at sun.misc.Unsafe.park(Native Method)
  986. - parking to wait for <0x00000000eb8d5320> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  987. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  988. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  989. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  990. at org.apache.accumulo.tserver.TabletServerResourceManager$MemoryManagementFramework.processTabletMemStats(TabletServerResourceManager.java:369)
  991. at org.apache.accumulo.tserver.TabletServerResourceManager$MemoryManagementFramework.access$300(TabletServerResourceManager.java:324)
  992. at org.apache.accumulo.tserver.TabletServerResourceManager$MemoryManagementFramework$1.run(TabletServerResourceManager.java:341)
  993. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  994. at java.lang.Thread.run(Thread.java:745)
  995.  
  996. "LRUBlockCacheStats 1" #43 daemon prio=5 os_prio=0 tid=0x00000000035af800 nid=0x3df7 waiting on condition [0x00007f7e3d74d000]
  997. java.lang.Thread.State: TIMED_WAITING (parking)
  998. at sun.misc.Unsafe.park(Native Method)
  999. - parking to wait for <0x00000000eaba5eb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1000. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  1001. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  1002. at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
  1003. at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
  1004. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  1005. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  1006. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  1007. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  1008. at java.lang.Thread.run(Thread.java:745)
  1009.  
  1010. "LruBlockCache.EvictionThread" #41 daemon prio=5 os_prio=0 tid=0x00000000035ad800 nid=0x3df6 in Object.wait() [0x00007f7e3d84e000]
  1011. java.lang.Thread.State: WAITING (on object monitor)
  1012. at java.lang.Object.wait(Native Method)
  1013. - waiting on <0x00000000eaba60e0> (a org.apache.accumulo.core.file.blockfile.cache.LruBlockCache$EvictionThread)
  1014. at java.lang.Object.wait(Object.java:502)
  1015. at org.apache.accumulo.core.file.blockfile.cache.LruBlockCache$EvictionThread.run(LruBlockCache.java:537)
  1016. - locked <0x00000000eaba60e0> (a org.apache.accumulo.core.file.blockfile.cache.LruBlockCache$EvictionThread)
  1017.  
  1018. "LRUBlockCacheStats 1" #40 daemon prio=5 os_prio=0 tid=0x00000000035ac800 nid=0x3df5 waiting on condition [0x00007f7e3d94f000]
  1019. java.lang.Thread.State: TIMED_WAITING (parking)
  1020. at sun.misc.Unsafe.park(Native Method)
  1021. - parking to wait for <0x00000000eaba4920> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1022. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  1023. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  1024. at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
  1025. at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
  1026. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  1027. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  1028. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  1029. at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
  1030. at java.lang.Thread.run(Thread.java:745)
  1031.  
  1032. "LruBlockCache.EvictionThread" #38 daemon prio=5 os_prio=0 tid=0x00000000035a9000 nid=0x3df4 in Object.wait() [0x00007f7e3da50000]
  1033. java.lang.Thread.State: WAITING (on object monitor)
  1034. at java.lang.Object.wait(Native Method)
  1035. - waiting on <0x00000000eaba4f30> (a org.apache.accumulo.core.file.blockfile.cache.LruBlockCache$EvictionThread)
  1036. at java.lang.Object.wait(Object.java:502)
  1037. at org.apache.accumulo.core.file.blockfile.cache.LruBlockCache$EvictionThread.run(LruBlockCache.java:537)
  1038. - locked <0x00000000eaba4f30> (a org.apache.accumulo.core.file.blockfile.cache.LruBlockCache$EvictionThread)
  1039.  
  1040. "SimpleTimer-0" #37 daemon prio=5 os_prio=0 tid=0x000000000359c800 nid=0x3df3 waiting on condition [0x00007f7e3e49b000]
  1041. java.lang.Thread.State: TIMED_WAITING (parking)
  1042. at sun.misc.Unsafe.park(Native Method)
  1043. - parking to wait for <0x00000000eab928c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1044. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  1045. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  1046. at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
  1047. at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
  1048. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  1049. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  1050. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  1051. at java.lang.Thread.run(Thread.java:745)
  1052.  
  1053. "FileWatchdog" #36 daemon prio=5 os_prio=0 tid=0x0000000003273000 nid=0x3df2 waiting on condition [0x00007f7e3e59c000]
  1054. java.lang.Thread.State: TIMED_WAITING (sleeping)
  1055. at java.lang.Thread.sleep(Native Method)
  1056. at org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:104)
  1057.  
  1058. "FileWatchdog-EventThread" #33 daemon prio=5 os_prio=0 tid=0x00000000034f6000 nid=0x3de6 waiting on condition [0x00007f7e3e79e000]
  1059. java.lang.Thread.State: WAITING (parking)
  1060. at sun.misc.Unsafe.park(Native Method)
  1061. - parking to wait for <0x00000000eab830a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1062. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  1063. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  1064. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  1065. at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
  1066.  
  1067. "FileWatchdog-SendThread(localhost:2181)" #32 daemon prio=5 os_prio=0 tid=0x000000000391f800 nid=0x3de5 runnable [0x00007f7e3e89f000]
  1068. java.lang.Thread.State: RUNNABLE
  1069. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  1070. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
  1071. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
  1072. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
  1073. - locked <0x00000000eab792c8> (a sun.nio.ch.Util$2)
  1074. - locked <0x00000000eab792b0> (a java.util.Collections$UnmodifiableSet)
  1075. - locked <0x00000000eab79178> (a sun.nio.ch.EPollSelectorImpl)
  1076. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
  1077. at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
  1078. at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
  1079.  
  1080. "FileWatchdog" #28 daemon prio=5 os_prio=0 tid=0x0000000003192000 nid=0x3de4 waiting on condition [0x00007f7e3e9a0000]
  1081. java.lang.Thread.State: TIMED_WAITING (sleeping)
  1082. at java.lang.Thread.sleep(Native Method)
  1083. at org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:104)
  1084. at org.apache.accumulo.server.watcher.MonitorLog4jWatcher.run(MonitorLog4jWatcher.java:75)
  1085.  
  1086. "FileWatchdog" #31 daemon prio=5 os_prio=0 tid=0x000000000318e000 nid=0x3de3 waiting on condition [0x00007f7e3eaa1000]
  1087. java.lang.Thread.State: TIMED_WAITING (sleeping)
  1088. at java.lang.Thread.sleep(Native Method)
  1089. at org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:104)
  1090.  
  1091. "IPC Parameter Sending Thread #0" #27 daemon prio=5 os_prio=0 tid=0x0000000003128000 nid=0x3dda waiting on condition [0x00007f7e3eca3000]
  1092. java.lang.Thread.State: TIMED_WAITING (parking)
  1093. at sun.misc.Unsafe.park(Native Method)
  1094. - parking to wait for <0x00000000eaaecde0> (a java.util.concurrent.SynchronousQueue$TransferStack)
  1095. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  1096. at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
  1097. at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
  1098. at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
  1099. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
  1100. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  1101. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  1102. at java.lang.Thread.run(Thread.java:745)
  1103.  
  1104. "Thread-8" #24 daemon prio=5 os_prio=0 tid=0x00000000031ef000 nid=0x3dd8 runnable [0x00007f7e3f8be000]
  1105. java.lang.Thread.State: RUNNABLE
  1106. at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
  1107. at org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
  1108. at org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
  1109. at java.lang.Thread.run(Thread.java:745)
  1110.  
  1111. "FileWatchdog" #22 daemon prio=5 os_prio=0 tid=0x0000000002f26800 nid=0x3dcb waiting on condition [0x00007f7e3fbda000]
  1112. java.lang.Thread.State: TIMED_WAITING (sleeping)
  1113. at java.lang.Thread.sleep(Native Method)
  1114. at org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:104)
  1115.  
  1116. "DestroyJavaVM" #19 prio=5 os_prio=0 tid=0x0000000001c9b000 nid=0x3d93 waiting on condition [0x0000000000000000]
  1117. java.lang.Thread.State: RUNNABLE
  1118.  
  1119. "tserver" #18 prio=5 os_prio=0 tid=0x0000000002cd7000 nid=0x3dbe waiting on condition [0x00007f7e400ef000]
  1120. java.lang.Thread.State: TIMED_WAITING (parking)
  1121. at sun.misc.Unsafe.park(Native Method)
  1122. - parking to wait for <0x00000000eab9b648> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  1123. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  1124. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  1125. at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
  1126. at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
  1127. at org.apache.accumulo.tserver.TabletServer.run(TabletServer.java:2500)
  1128. at org.apache.accumulo.tserver.TabletServer.main(TabletServer.java:2959)
  1129. at org.apache.accumulo.tserver.TServerExecutable.execute(TServerExecutable.java:33)
  1130. at org.apache.accumulo.start.Main$1.run(Main.java:93)
  1131. at java.lang.Thread.run(Thread.java:745)
  1132.  
  1133. "NonBlockingInputStreamThread" #17 daemon prio=5 os_prio=0 tid=0x0000000002cf1000 nid=0x3dbd in Object.wait() [0x00007f7e401f0000]
  1134. java.lang.Thread.State: WAITING (on object monitor)
  1135. at java.lang.Object.wait(Native Method)
  1136. - waiting on <0x00000000eae9cdd8> (a jline.internal.NonBlockingInputStream)
  1137. at jline.internal.NonBlockingInputStream.run(NonBlockingInputStream.java:278)
  1138. - locked <0x00000000eae9cdd8> (a jline.internal.NonBlockingInputStream)
  1139. at java.lang.Thread.run(Thread.java:745)
  1140.  
  1141. "Thread-2" #13 daemon prio=1 os_prio=0 tid=0x0000000002daf000 nid=0x3daa waiting on condition [0x00007f7e4052a000]
  1142. java.lang.Thread.State: TIMED_WAITING (sleeping)
  1143. at java.lang.Thread.sleep(Native Method)
  1144. at org.apache.commons.vfs2.impl.DefaultFileMonitor.run(DefaultFileMonitor.java:416)
  1145. at java.lang.Thread.run(Thread.java:745)
  1146.  
  1147. "Service Thread" #10 daemon prio=9 os_prio=0 tid=0x0000000001ea6000 nid=0x3da8 runnable [0x0000000000000000]
  1148. java.lang.Thread.State: RUNNABLE
  1149.  
  1150. "C1 CompilerThread3" #9 daemon prio=9 os_prio=0 tid=0x0000000001e68000 nid=0x3da7 waiting on condition [0x0000000000000000]
  1151. java.lang.Thread.State: RUNNABLE
  1152.  
  1153. "C2 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x0000000001e65800 nid=0x3da6 waiting on condition [0x0000000000000000]
  1154. java.lang.Thread.State: RUNNABLE
  1155.  
  1156. "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x0000000001e63800 nid=0x3da5 waiting on condition [0x0000000000000000]
  1157. java.lang.Thread.State: RUNNABLE
  1158.  
  1159. "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x0000000001e60800 nid=0x3da4 waiting on condition [0x0000000000000000]
  1160. java.lang.Thread.State: RUNNABLE
  1161.  
  1162. "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x0000000001e5e800 nid=0x3da3 runnable [0x0000000000000000]
  1163. java.lang.Thread.State: RUNNABLE
  1164.  
  1165. "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0 tid=0x0000000001e5d000 nid=0x3da2 waiting on condition [0x0000000000000000]
  1166. java.lang.Thread.State: RUNNABLE
  1167.  
  1168. "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x0000000001e29800 nid=0x3da1 in Object.wait() [0x00007f7e4197a000]
  1169. java.lang.Thread.State: WAITING (on object monitor)
  1170. at java.lang.Object.wait(Native Method)
  1171. at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
  1172. - locked <0x00000000eaea3b30> (a java.lang.ref.ReferenceQueue$Lock)
  1173. at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
  1174. at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
  1175.  
  1176. "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x0000000001e1f000 nid=0x3da0 in Object.wait() [0x00007f7e41a7b000]
  1177. java.lang.Thread.State: WAITING (on object monitor)
  1178. at java.lang.Object.wait(Native Method)
  1179. at java.lang.Object.wait(Object.java:502)
  1180. at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
  1181. - locked <0x00000000eaea3cf8> (a java.lang.ref.Reference$Lock)
  1182. at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
  1183.  
  1184. "VM Thread" os_prio=0 tid=0x0000000001e17800 nid=0x3d9f runnable
  1185.  
  1186. "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x0000000001cac800 nid=0x3d94 runnable
  1187.  
  1188. "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x0000000001cae000 nid=0x3d95 runnable
  1189.  
  1190. "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x0000000001cb0000 nid=0x3d96 runnable
  1191.  
  1192. "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x0000000001cb1800 nid=0x3d97 runnable
  1193.  
  1194. "Gang worker#4 (Parallel GC Threads)" os_prio=0 tid=0x0000000001cb3800 nid=0x3d98 runnable
  1195.  
  1196. "Gang worker#5 (Parallel GC Threads)" os_prio=0 tid=0x0000000001cb5000 nid=0x3d99 runnable
  1197.  
  1198. "Gang worker#6 (Parallel GC Threads)" os_prio=0 tid=0x0000000001cb7000 nid=0x3d9a runnable
  1199.  
  1200. "Gang worker#7 (Parallel GC Threads)" os_prio=0 tid=0x0000000001cb8800 nid=0x3d9b runnable
  1201.  
  1202. "Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x0000000001d29800 nid=0x3d9e runnable
  1203.  
  1204. "Gang worker#0 (Parallel CMS Threads)" os_prio=0 tid=0x0000000001d26000 nid=0x3d9c runnable
  1205.  
  1206. "Gang worker#1 (Parallel CMS Threads)" os_prio=0 tid=0x0000000001d27800 nid=0x3d9d runnable
  1207.  
  1208. "VM Periodic Task Thread" os_prio=0 tid=0x0000000001ea9800 nid=0x3da9 waiting on condition
  1209.  
  1210. JNI global references: 266
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!