API tools faq
paste
Advertisement
ronbuckley

RegionServerHungDuringShutdown JStack

ronbuckley
May 30th, 2014
229
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 26.19 KB | None | 0 0
raw download clone embed print report
  1. hadoop@wchddb07txdu:/prod/hadoop/prod/hbase/logs$ cat /home/hadoop/35943.jstack
  2. 2014-05-30 10:33:40
  3. Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.45-b08 mixed mode):
  4.  
  5. "Attach Listener" daemon prio=10 tid=0x00007f42bc058000 nid=0xd29d waiting on condition [0x0000000000000000]
  6. java.lang.Thread.State: RUNNABLE
  7.  
  8. "ResponseProcessor for block BP-1985911218-10.192.215.171-1352732338449:blk_1438989233018680029_51000934" daemon prio=10 tid=0x00007f42c4e25000 nid=0xabcd runnable [0x00007f428a82c000]
  9. java.lang.Thread.State: RUNNABLE
  10. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  11. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
  12. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
  13. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
  14. - locked <0x00000007dd11bf68> (a sun.nio.ch.Util$2)
  15. - locked <0x00000007dd11bf58> (a java.util.Collections$UnmodifiableSet)
  16. - locked <0x00000007dd11bd30> (a sun.nio.ch.EPollSelectorImpl)
  17. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
  18. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:336)
  19. at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:158)
  20. at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:156)
  21. at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:129)
  22. at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:117)
  23. at java.io.FilterInputStream.read(FilterInputStream.java:83)
  24. at java.io.FilterInputStream.read(FilterInputStream.java:83)
  25. at org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:169)
  26. at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:114)
  27. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:695)
  28.  
  29. "DataStreamer for file /hbase/.logs/wchddb07txdu.dev.oclc.org,9009,1401300454962/wchddb07txdu.dev.oclc.org%2C9009%2C1401300454962.1401404228252 block BP-1985911218-10.192.215.171-1352732338449:blk_1438989233018680029_51000934" daemon prio=10 tid=0x00007f42c52d1000 nid=0xabca in Object.wait() [0x00007f428ab2f000]
  30. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  31. at java.lang.Object.wait(Native Method)
  32. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:465)
  33. - locked <0x00000007c2919a98> (a java.util.LinkedList)
  34.  
  35. "RS_CLOSE_REGION-wchddb07txdu.dev.oclc.org,9009,1401300454962-2" prio=10 tid=0x00007f42bedf3800 nid=0xba08 waiting on condition [0x00007f428b337000]
  36. java.lang.Thread.State: WAITING (parking)
  37. at sun.misc.Unsafe.park(Native Method)
  38. - parking to wait for <0x00000005fb5748a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  39. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
  40. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
  41. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  42. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
  43. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
  44. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  45. at java.lang.Thread.run(Thread.java:744)
  46.  
  47. "RMI Scheduler(0)" daemon prio=10 tid=0x00007f42c45bf000 nid=0xb396 waiting on condition [0x00007f428b539000]
  48. java.lang.Thread.State: TIMED_WAITING (parking)
  49. at sun.misc.Unsafe.park(Native Method)
  50. - parking to wait for <0x00000005fb67e778> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  51. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
  52. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
  53. at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
  54. at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
  55. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
  56. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
  57. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  58. at java.lang.Thread.run(Thread.java:744)
  59.  
  60. "RS_CLOSE_REGION-wchddb07txdu.dev.oclc.org,9009,1401300454962-1" prio=10 tid=0x00007f42bccaa000 nid=0xa908 waiting on condition [0x00007f428ae32000]
  61. java.lang.Thread.State: WAITING (parking)
  62. at sun.misc.Unsafe.park(Native Method)
  63. - parking to wait for <0x00000005fc175118> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
  64. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
  65. at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
  66. at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
  67. at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
  68. at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:945)
  69. at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1022)
  70. at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:980)
  71. - locked <0x00000005fc1754d0> (a java.lang.Object)
  72. at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:119)
  73. at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
  74. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  75. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  76. at java.lang.Thread.run(Thread.java:744)
  77.  
  78. "RS_CLOSE_REGION-wchddb07txdu.dev.oclc.org,9009,1401300454962-0" prio=10 tid=0x00007f42c49b2800 nid=0xa8be waiting on condition [0x00007f428af33000]
  79. java.lang.Thread.State: WAITING (parking)
  80. at sun.misc.Unsafe.park(Native Method)
  81. - parking to wait for <0x00000005fb5748a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  82. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
  83. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
  84. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  85. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
  86. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
  87. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  88. at java.lang.Thread.run(Thread.java:744)
  89.  
  90. "ReplicationExecutor-0" prio=10 tid=0x00007f42c49a1800 nid=0x97e2 waiting on condition [0x00007f429a530000]
  91. java.lang.Thread.State: WAITING (parking)
  92. at sun.misc.Unsafe.park(Native Method)
  93. - parking to wait for <0x00000005fb424e68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  94. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
  95. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
  96. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  97. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
  98. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
  99. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  100. at java.lang.Thread.run(Thread.java:744)
  101.  
  102. "FileInputStreamCache Cleaner" daemon prio=10 tid=0x00007f42bc2b5800 nid=0x942d waiting on condition [0x00007f429a732000]
  103. java.lang.Thread.State: TIMED_WAITING (parking)
  104. at sun.misc.Unsafe.park(Native Method)
  105. - parking to wait for <0x00000005fb1ce908> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  106. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
  107. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
  108. at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
  109. at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
  110. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
  111. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
  112. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  113. at java.lang.Thread.run(Thread.java:744)
  114.  
  115. "RS_OPEN_REGION-wchddb07txdu.dev.oclc.org,9009,1401300454962-2" prio=10 tid=0x0000000001397000 nid=0x91c7 waiting on condition [0x00007f428b438000]
  116. java.lang.Thread.State: WAITING (parking)
  117. at sun.misc.Unsafe.park(Native Method)
  118. - parking to wait for <0x00000005fb1cf358> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  119. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
  120. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
  121. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  122. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
  123. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
  124. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  125. at java.lang.Thread.run(Thread.java:744)
  126.  
  127. "RS_OPEN_REGION-wchddb07txdu.dev.oclc.org,9009,1401300454962-1" prio=10 tid=0x00007f42cd587000 nid=0x91af waiting on condition [0x00007f428b83c000]
  128. java.lang.Thread.State: WAITING (parking)
  129. at sun.misc.Unsafe.park(Native Method)
  130. - parking to wait for <0x00000005fb1cf358> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  131. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
  132. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
  133. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  134. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
  135. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
  136. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  137. at java.lang.Thread.run(Thread.java:744)
  138.  
  139. "RS_OPEN_REGION-wchddb07txdu.dev.oclc.org,9009,1401300454962-0" prio=10 tid=0x00007f42c452d000 nid=0x919c waiting on condition [0x00007f428bb43000]
  140. java.lang.Thread.State: WAITING (parking)
  141. at sun.misc.Unsafe.park(Native Method)
  142. - parking to wait for <0x00000005fb1cf358> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  143. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
  144. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
  145. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  146. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
  147. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
  148. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  149. at java.lang.Thread.run(Thread.java:744)
  150.  
  151. "org.apache.hadoop.hdfs.PeerCache@574e6595" daemon prio=10 tid=0x00007f42c460a000 nid=0x919b waiting on condition [0x00007f428bc44000]
  152. java.lang.Thread.State: TIMED_WAITING (sleeping)
  153. at java.lang.Thread.sleep(Native Method)
  154. at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:252)
  155. at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:39)
  156. at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:135)
  157. at java.lang.Thread.run(Thread.java:744)
  158.  
  159. "wchddb07txdu.dev.oclc.org,9009,1401300454962Replication Statistics #0" daemon prio=10 tid=0x0000000000d2f000 nid=0x8ebb waiting on condition [0x00007f429961d000]
  160. java.lang.Thread.State: TIMED_WAITING (parking)
  161. at sun.misc.Unsafe.park(Native Method)
  162. - parking to wait for <0x00000005fb1d41b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  163. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
  164. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
  165. at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
  166. at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
  167. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
  168. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
  169. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  170. at java.lang.Thread.run(Thread.java:744)
  171.  
  172. "Timer thread for monitoring jvm" daemon prio=10 tid=0x0000000000c41800 nid=0x8ea4 in Object.wait() [0x00007f4299f26000]
  173. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  174. at java.lang.Object.wait(Native Method)
  175. at java.util.TimerThread.mainLoop(Timer.java:552)
  176. - locked <0x00000005fb1d58a8> (a java.util.TaskQueue)
  177. at java.util.TimerThread.run(Timer.java:505)
  178.  
  179. "Timer thread for monitoring hbase" daemon prio=10 tid=0x0000000000b40800 nid=0x8ea2 in Object.wait() [0x00007f429a027000]
  180. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  181. at java.lang.Object.wait(Native Method)
  182. at java.util.TimerThread.mainLoop(Timer.java:552)
  183. - locked <0x00000005fb1d6800> (a java.util.TaskQueue)
  184. at java.util.TimerThread.run(Timer.java:505)
  185.  
  186. "regionserver9009.logSyncer" daemon prio=10 tid=0x0000000000b36000 nid=0x8ea1 in Object.wait() [0x00007f429a128000]
  187. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  188. at java.lang.Object.wait(Native Method)
  189. at org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:1295)
  190. - locked <0x00000005fb1d5960> (a java.util.concurrent.atomic.AtomicBoolean)
  191. at java.lang.Thread.run(Thread.java:744)
  192.  
  193. "LeaseRenewer:hadoop@cluster" daemon prio=10 tid=0x0000000000ae6800 nid=0x8e9a waiting on condition [0x00007f429a32a000]
  194. java.lang.Thread.State: TIMED_WAITING (sleeping)
  195. at java.lang.Thread.sleep(Native Method)
  196. at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438)
  197. at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
  198. at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
  199. at java.lang.Thread.run(Thread.java:744)
  200.  
  201. "IPC Parameter Sending Thread #0" daemon prio=10 tid=0x0000000000b75800 nid=0x8e89 waiting on condition [0x00007f429a631000]
  202. java.lang.Thread.State: TIMED_WAITING (parking)
  203. at sun.misc.Unsafe.park(Native Method)
  204. - parking to wait for <0x00000005fb1d7cb8> (a java.util.concurrent.SynchronousQueue$TransferStack)
  205. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
  206. at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
  207. at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
  208. at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
  209. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
  210. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
  211. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  212. at java.lang.Thread.run(Thread.java:744)
  213.  
  214. "IncidentDistributionPoint-0/ProcessSyncQueue" daemon prio=10 tid=0x0000000000e01800 nid=0x8d3e in Object.wait() [0x00007f429a833000]
  215. java.lang.Thread.State: WAITING (on object monitor)
  216. at java.lang.Object.wait(Native Method)
  217. at java.lang.Object.wait(Object.java:503)
  218. at ORG.oclc.infra.util.SyncQueue.waitForEntries(SyncQueue.java:276)
  219. - locked <0x00000005fb1d5a00> (a ORG.oclc.infra.util.SyncQueue)
  220. at ORG.oclc.infra.drax.DestinationList.execute(DestinationList.java:123)
  221. at ORG.oclc.infra.drax.DistributionPoint.run(DistributionPoint.java:160)
  222. at java.lang.Thread.run(Thread.java:744)
  223.  
  224. "regionserver9009-EventThread" daemon prio=10 tid=0x0000000000b87000 nid=0x8d3b waiting on condition [0x00007f429aa35000]
  225. java.lang.Thread.State: WAITING (parking)
  226. at sun.misc.Unsafe.park(Native Method)
  227. - parking to wait for <0x00000005fb1d5be8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  228. at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
  229. at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
  230. at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  231. at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
  232.  
  233. "regionserver9009-SendThread(wchddb02txdu.dev.oclc.org:9011)" daemon prio=10 tid=0x0000000000cf8000 nid=0x8d3a runnable [0x00007f429ab36000]
  234. java.lang.Thread.State: RUNNABLE
  235. at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  236. at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
  237. at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
  238. at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
  239. - locked <0x00000005fb1d6978> (a sun.nio.ch.Util$2)
  240. - locked <0x00000005fb1d6988> (a java.util.Collections$UnmodifiableSet)
  241. - locked <0x00000005fb1d6930> (a sun.nio.ch.EPollSelectorImpl)
  242. at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
  243. at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
  244. at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1075)
  245.  
  246. "regionserver9009" prio=10 tid=0x00007f42cd3df000 nid=0x8d39 waiting on condition [0x00007f429ae3a000]
  247. java.lang.Thread.State: TIMED_WAITING (sleeping)
  248. at java.lang.Thread.sleep(Native Method)
  249. at org.apache.hadoop.hbase.util.Threads.sleep(Threads.java:131)
  250. at org.apache.hadoop.hbase.regionserver.HRegionServer.waitOnAllRegionsToClose(HRegionServer.java:1013)
  251. at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:877)
  252. at java.lang.Thread.run(Thread.java:744)
  253.  
  254. "Timer thread for monitoring rpc" daemon prio=10 tid=0x00007f42cd359800 nid=0x8d2f in Object.wait() [0x00007f429b350000]
  255. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  256. at java.lang.Object.wait(Native Method)
  257. at java.util.TimerThread.mainLoop(Timer.java:552)
  258. - locked <0x00000005fb1d5d58> (a java.util.TaskQueue)
  259. at java.util.TimerThread.run(Timer.java:505)
  260.  
  261. "RMI TCP Accept-0" daemon prio=10 tid=0x00007f42cce98800 nid=0x8d1b runnable [0x00007f429bf9f000]
  262. java.lang.Thread.State: RUNNABLE
  263. at java.net.PlainSocketImpl.socketAccept(Native Method)
  264. at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
  265. at java.net.ServerSocket.implAccept(ServerSocket.java:530)
  266. at java.net.ServerSocket.accept(ServerSocket.java:498)
  267. at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)
  268. at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:388)
  269. at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:360)
  270. at java.lang.Thread.run(Thread.java:744)
  271.  
  272. "RMI TCP Accept-9021" daemon prio=10 tid=0x00007f42cce8d800 nid=0x8d1a runnable [0x00007f429c0a0000]
  273. java.lang.Thread.State: RUNNABLE
  274. at java.net.PlainSocketImpl.socketAccept(Native Method)
  275. at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
  276. at java.net.ServerSocket.implAccept(ServerSocket.java:530)
  277. at java.net.ServerSocket.accept(ServerSocket.java:498)
  278. at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:388)
  279. at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:360)
  280. at java.lang.Thread.run(Thread.java:744)
  281.  
  282. "RMI TCP Accept-0" daemon prio=10 tid=0x00007f42cce7f800 nid=0x8d19 runnable [0x00007f429c1a1000]
  283. java.lang.Thread.State: RUNNABLE
  284. at java.net.PlainSocketImpl.socketAccept(Native Method)
  285. at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
  286. at java.net.ServerSocket.implAccept(ServerSocket.java:530)
  287. at java.net.ServerSocket.accept(ServerSocket.java:498)
  288. at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:388)
  289. at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:360)
  290. at java.lang.Thread.run(Thread.java:744)
  291.  
  292. "Service Thread" daemon prio=10 tid=0x00007f42cc632800 nid=0x8d18 runnable [0x0000000000000000]
  293. java.lang.Thread.State: RUNNABLE
  294.  
  295. "C2 CompilerThread1" daemon prio=10 tid=0x00007f42cc630000 nid=0x8d17 waiting on condition [0x0000000000000000]
  296. java.lang.Thread.State: RUNNABLE
  297.  
  298. "C2 CompilerThread0" daemon prio=10 tid=0x00007f42cc62d800 nid=0x8d16 waiting on condition [0x0000000000000000]
  299. java.lang.Thread.State: RUNNABLE
  300.  
  301. "Signal Dispatcher" daemon prio=10 tid=0x00007f42cc62b800 nid=0x8d15 runnable [0x0000000000000000]
  302. java.lang.Thread.State: RUNNABLE
  303.  
  304. "Surrogate Locker Thread (Concurrent GC)" daemon prio=10 tid=0x00007f42cc62a000 nid=0x8d14 waiting on condition [0x0000000000000000]
  305. java.lang.Thread.State: RUNNABLE
  306.  
  307. "Finalizer" daemon prio=10 tid=0x00007f42cc608000 nid=0x8d00 in Object.wait() [0x00007f42a37ab000]
  308. java.lang.Thread.State: WAITING (on object monitor)
  309. at java.lang.Object.wait(Native Method)
  310. at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
  311. - locked <0x00000005fb1d7400> (a java.lang.ref.ReferenceQueue$Lock)
  312. at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
  313. at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:189)
  314.  
  315. "Reference Handler" daemon prio=10 tid=0x00007f42cc604000 nid=0x8cff in Object.wait() [0x00007f42a38ac000]
  316. java.lang.Thread.State: WAITING (on object monitor)
  317. at java.lang.Object.wait(Native Method)
  318. at java.lang.Object.wait(Object.java:503)
  319. at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
  320. - locked <0x00000005fb1d4cf8> (a java.lang.ref.Reference$Lock)
  321.  
  322. "main" prio=10 tid=0x00007f42cc019800 nid=0x8ce9 in Object.wait() [0x00007f42d1f13000]
  323. java.lang.Thread.State: WAITING (on object monitor)
  324. at java.lang.Object.wait(Native Method)
  325. - waiting on <0x00000005fb1d5c50> (a java.lang.Thread)
  326. at java.lang.Thread.join(Thread.java:1280)
  327. - locked <0x00000005fb1d5c50> (a java.lang.Thread)
  328. at java.lang.Thread.join(Thread.java:1354)
  329. at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:63)
  330. at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:84)
  331. at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
  332. at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
  333. at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:4105)
  334.  
  335. "VM Thread" prio=10 tid=0x00007f42cc601800 nid=0x8cfe runnable
  336.  
  337. "Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x00007f42cc02e000 nid=0x8cea runnable
  338.  
  339. "Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x00007f42cc030000 nid=0x8ceb runnable
  340.  
  341. "Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x00007f42cc032000 nid=0x8cec runnable
  342.  
  343. "Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x00007f42cc033800 nid=0x8ced runnable
  344.  
  345. "Gang worker#4 (Parallel GC Threads)" prio=10 tid=0x00007f42cc035800 nid=0x8cee runnable
  346.  
  347. "Gang worker#5 (Parallel GC Threads)" prio=10 tid=0x00007f42cc037800 nid=0x8cef runnable
  348.  
  349. "Gang worker#6 (Parallel GC Threads)" prio=10 tid=0x00007f42cc039000 nid=0x8cf0 runnable
  350.  
  351. "Gang worker#7 (Parallel GC Threads)" prio=10 tid=0x00007f42cc03b000 nid=0x8cf1 runnable
  352.  
  353. "G1 Main Concurrent Mark GC Thread" prio=10 tid=0x00007f42cc065000 nid=0x8cfb runnable
  354.  
  355. "Gang worker#0 (G1 Parallel Marking Threads)" prio=10 tid=0x00007f42cc0e8000 nid=0x8cfc runnable
  356.  
  357. "Gang worker#1 (G1 Parallel Marking Threads)" prio=10 tid=0x00007f42cc0ea000 nid=0x8cfd runnable
  358.  
  359. "G1 Concurrent Refinement Thread#0" prio=10 tid=0x00007f42cc04d000 nid=0x8cfa runnable
  360.  
  361. "G1 Concurrent Refinement Thread#1" prio=10 tid=0x00007f42cc04b800 nid=0x8cf9 runnable
  362.  
  363. "G1 Concurrent Refinement Thread#2" prio=10 tid=0x00007f42cc049800 nid=0x8cf8 runnable
  364.  
  365. "G1 Concurrent Refinement Thread#3" prio=10 tid=0x00007f42cc047800 nid=0x8cf7 runnable
  366.  
  367. "G1 Concurrent Refinement Thread#4" prio=10 tid=0x00007f42cc045800 nid=0x8cf6 runnable
  368.  
  369. "G1 Concurrent Refinement Thread#5" prio=10 tid=0x00007f42cc043800 nid=0x8cf5 runnable
  370.  
  371. "G1 Concurrent Refinement Thread#6" prio=10 tid=0x00007f42cc042000 nid=0x8cf4 runnable
  372.  
  373. "G1 Concurrent Refinement Thread#7" prio=10 tid=0x00007f42cc040000 nid=0x8cf3 runnable
  374.  
  375. "G1 Concurrent Refinement Thread#8" prio=10 tid=0x00007f42cc03e000 nid=0x8cf2 runnable
  376.  
  377.  
  378. "VM Periodic Task Thread" prio=10 tid=0x00007f42cce9a800 nid=0x8d1c waiting on condition
  379.  
  380. JNI global references: 287
  381.  
  382. hadoop@wchddb07txdu:/prod/hadoop/prod/hbase/logs$
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!