API tools faq
paste
Advertisement
Guest User

RS_log

a guest
Aug 13th, 2012
105
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 55.74 KB | None | 0 0
raw download clone embed print report
  1. 2012-08-07 10:02:57,845 DEBUG org.apache.hadoop.hbase.io.hfile.HFileWriterV2: Initialized with CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
  2. 2012-08-07 10:02:58,506 DEBUG org.apache.hadoop.hbase.regionserver.Store: Renaming flushed file at hdfs://ihubcluster/hbase/transactions_str_as_dbl/eb1f6886364054bc96f1f190f24b6eed/.tmp/4a556e5cbc01432187048c50c6c8d030 to hdfs://ihubcluster/hbase/transactions_str_as_dbl/eb1f6886364054bc96f1f190f24b6eed/t/4a556e5cbc01432187048c50c6c8d030
  3. 2012-08-07 10:02:58,562 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://ihubcluster/hbase/transactions_str_as_dbl/eb1f6886364054bc96f1f190f24b6eed/t/4a556e5cbc01432187048c50c6c8d030, entries=594386, sequenceid=1441817, memsize=135.9m, filesize=34.7m
  4. 2012-08-07 10:02:58,565 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~136.3m/142962624, currentsize=5.2m/5457392 for region transactions_str_as_dbl,17761522899263569193018364,1344358522026.eb1f6886364054bc96f1f190f24b6eed. in 1700ms, sequenceid=1441817, compaction requested=false
  5. 2012-08-07 10:02:58,565 DEBUG org.apache.hadoop.hbase.regionserver.Store: hdfs://ihubcluster/hbase/transactions_str_as_dbl/eb1f6886364054bc96f1f190f24b6eed/t/c3d8ea5156ab49afb5f2e70d2c2f5b02.189f8d09c53dc72b08660ec1f9b50646-hdfs://ihubcluster/hbase/transactions_str_as_dbl/189f8d09c53dc72b08660ec1f9b50646/t/c3d8ea5156ab49afb5f2e70d2c2f5b02-bottom is not splittable
  6. 2012-08-07 10:03:50,094 WARN org.apache.hadoop.hbase.util.Sleeper: We slept 41838ms instead of 3000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
  7. 2012-08-07 10:03:50,111 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 9 on 60020 took 41382 ms appending an edit to hlog; editcount=1132, len~=18.3k
  8. 2012-08-07 10:03:50,114 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":41575,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@423c5ad9), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"172.31.192.54:33894","starttimems":1344358988536,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  9. 2012-08-07 10:03:50,231 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":41592,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@2e27d67e), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"172.31.192.153:42073","starttimems":1344358988638,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  10. 2012-08-07 10:03:50,235 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":41685,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@2d7fb349), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"172.31.192.53:40319","starttimems":1344358988549,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  11. 2012-08-07 10:03:50,233 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":41631,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@397301de), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"172.31.192.153:42052","starttimems":1344358988601,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  12. 2012-08-07 10:03:50,308 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":41741,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@2731a9e7), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"172.31.192.102:52815","starttimems":1344358988545,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  13. 2012-08-07 10:03:50,319 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 46074ms for sessionid 0x238e96bb4e00011, closing socket connection and attempting reconnect
  14. 2012-08-07 10:03:50,319 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 46522ms for sessionid 0x338e96bc1ae000c, closing socket connection and attempting reconnect
  15. 2012-08-07 10:03:50,413 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":41673,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@26f68905), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"172.31.192.104:47766","starttimems":1344358988728,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  16. 2012-08-07 10:03:50,680 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server ihub-an-l1/172.31.192.151:2181
  17. 2012-08-07 10:03:50,965 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
  18. 2012-08-07 10:03:50,965 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
  19. 2012-08-07 10:03:50,967 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to ihub-an-l1/172.31.192.151:2181, initiating session
  20. 2012-08-07 10:03:50,974 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x238e96bb4e00011 has expired, closing socket connection
  21. 2012-08-07 10:03:50,974 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server ihub-dn-l4,60020,1344317192178: Unhandled exception: org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing ihub-dn-l4,60020,1344317192178 as dead server
  22. at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:222)
  23. at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:148)
  24. at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:851)
  25. at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
  26. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  27. at java.lang.reflect.Method.invoke(Method.java:597)
  28. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  29. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336)
  30.  
  31. org.apache.hadoop.hbase.YouAreDeadException: org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing ihub-dn-l4,60020,1344317192178 as dead server
  32. at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:222)
  33. at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:148)
  34. at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:851)
  35. at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
  36. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  37. at java.lang.reflect.Method.invoke(Method.java:597)
  38. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  39. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336)
  40.  
  41. at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  42. at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
  43. at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
  44. at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
  45. at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90)
  46. at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
  47. at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:791)
  48. at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:682)
  49. at java.lang.Thread.run(Thread.java:662)
  50. Caused by: org.apache.hadoop.hbase.YouAreDeadException: org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing ihub-dn-l4,60020,1344317192178 as dead server
  51. at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:222)
  52. at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:148)
  53. at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:851)
  54. at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
  55. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  56. at java.lang.reflect.Method.invoke(Method.java:597)
  57. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  58. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336)
  59.  
  60. at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:918)
  61. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
  62. at $Proxy10.regionServerReport(Unknown Source)
  63. at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:788)
  64. ... 2 more
  65. 2012-08-07 10:03:51,043 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.AggregateImplementation]
  66. 2012-08-07 10:03:51,013 INFO org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: This client just lost it's session with ZooKeeper, trying to reconnect.
  67. 2012-08-07 10:03:51,074 INFO org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Trying to reconnect to zookeeper
  68. 2012-08-07 10:03:51,225 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server ihub-an-l1/172.31.192.151:2181
  69. 2012-08-07 10:03:51,225 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
  70. 2012-08-07 10:03:51,225 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
  71. 2012-08-07 10:03:51,226 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to ihub-an-l1/172.31.192.151:2181, initiating session
  72. 2012-08-07 10:03:51,227 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server ihub-dn-l4,60020,1344317192178: regionserver:60020-0x338e96bc1ae000c-0x338e96bc1ae000c regionserver:60020-0x338e96bc1ae000c-0x338e96bc1ae000c received expired from ZooKeeper, aborting
  73. org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired
  74. at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:374)
  75. at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
  76. at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
  77. at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
  78. 2012-08-07 10:03:51,523 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.AggregateImplementation]
  79. 2012-08-07 10:03:51,524 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: requestsPerSecond=23, numberOfOnlineRegions=25, numberOfStores=25, numberOfStorefiles=42, storefileIndexSizeMB=0, rootIndexSizeKB=594, totalStaticIndexSizeKB=32130, totalStaticBloomSizeKB=0, memstoreSizeMB=278, mbInMemoryWithoutWAL=0, numberOfPutsWithoutWAL=0, readRequestsCount=95323, writeRequestsCount=43379, compactionQueueSize=3, flushQueueSize=0, usedHeapMB=569, maxHeapMB=1197, blockCacheSizeMB=3.12, blockCacheFreeMB=296.37, blockCacheCount=33, blockCacheHitCount=20247, blockCacheMissCount=95108, blockCacheEvictedCount=34, blockCacheHitRatio=17%, blockCacheHitCachingRatio=99%, hdfsBlocksLocalityIndex=16, deleteRequestLatencyMean=497337129.82, deleteRequestLatencyCount=40, deleteRequestLatencyMedian=2348868.5, deleteRequestLatency75th=6156771.5, deleteRequestLatency95th=4972486679.9, deleteRequestLatency99th=9093879552, deleteRequestLatency999th=9093879552, getRequestLatencyMean=0, getRequestLatencyCount=0, getRequestLatencyMedian=0, getRequestLatency75th=0, getRequestLatency95th=0, getRequestLatency99th=0, getRequestLatency999th=0, putRequestLatencyMean=0, putRequestLatencyCount=0, putRequestLatencyMedian=0, putRequestLatency75th=0, putRequestLatency95th=0, putRequestLatency99th=0, putRequestLatency999th=0, fsReadLatencyHistogramMean=0, fsReadLatencyHistogramCount=0, fsReadLatencyHistogramMedian=0, fsReadLatencyHistogram75th=0, fsReadLatencyHistogram95th=0, fsReadLatencyHistogram99th=0, fsReadLatencyHistogram999th=0, fsWriteLatencyHistogramMean=0, fsWriteLatencyHistogramCount=0, fsWriteLatencyHistogramMedian=0, fsWriteLatencyHistogram75th=0, fsWriteLatencyHistogram95th=0, fsWriteLatencyHistogram99th=0, fsWriteLatencyHistogram999th=0
  80. 2012-08-07 10:03:51,524 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: requestsPerSecond=23, numberOfOnlineRegions=25, numberOfStores=25, numberOfStorefiles=42, storefileIndexSizeMB=0, rootIndexSizeKB=594, totalStaticIndexSizeKB=32130, totalStaticBloomSizeKB=0, memstoreSizeMB=278, mbInMemoryWithoutWAL=0, numberOfPutsWithoutWAL=0, readRequestsCount=95323, writeRequestsCount=43379, compactionQueueSize=3, flushQueueSize=0, usedHeapMB=559, maxHeapMB=1197, blockCacheSizeMB=3.12, blockCacheFreeMB=296.37, blockCacheCount=33, blockCacheHitCount=20247, blockCacheMissCount=95108, blockCacheEvictedCount=34, blockCacheHitRatio=17%, blockCacheHitCachingRatio=99%, hdfsBlocksLocalityIndex=16, deleteRequestLatencyMean=497337129.82, deleteRequestLatencyCount=40, deleteRequestLatencyMedian=2348868.5, deleteRequestLatency75th=6156771.5, deleteRequestLatency95th=4972486679.9, deleteRequestLatency99th=9093879552, deleteRequestLatency999th=9093879552, getRequestLatencyMean=0, getRequestLatencyCount=0, getRequestLatencyMedian=0, getRequestLatency75th=0, getRequestLatency95th=0, getRequestLatency99th=0, getRequestLatency999th=0, putRequestLatencyMean=0, putRequestLatencyCount=0, putRequestLatencyMedian=0, putRequestLatency75th=0, putRequestLatency95th=0, putRequestLatency99th=0, putRequestLatency999th=0, fsReadLatencyHistogramMean=0, fsReadLatencyHistogramCount=0, fsReadLatencyHistogramMedian=0, fsReadLatencyHistogram75th=0, fsReadLatencyHistogram95th=0, fsReadLatencyHistogram99th=0, fsReadLatencyHistogram999th=0, fsWriteLatencyHistogramMean=0, fsWriteLatencyHistogramCount=0, fsWriteLatencyHistogramMedian=0, fsWriteLatencyHistogram75th=0, fsWriteLatencyHistogram95th=0, fsWriteLatencyHistogram99th=0, fsWriteLatencyHistogram999th=0
  81. 2012-08-07 10:03:51,517 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=ihub-an-l1:2181,ihub-an-g1:2181,ihub-nn-a2:2181 sessionTimeout=180000 watcher=hconnection
  82. 2012-08-07 10:03:51,227 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x338e96bc1ae000c has expired, closing socket connection
  83. 2012-08-07 10:03:51,685 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The identifier of this process is 22543@ihub-dn-l4
  84. 2012-08-07 10:03:51,682 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /172.31.192.151:2181
  85. 2012-08-07 10:03:51,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Unhandled exception: org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing ihub-dn-l4,60020,1344317192178 as dead server
  86. at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:222)
  87. at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:148)
  88. at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:851)
  89. at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
  90. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  91. at java.lang.reflect.Method.invoke(Method.java:597)
  92. at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  93. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336)
  94.  
  95. 2012-08-07 10:03:51,695 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: regionserver:60020-0x338e96bc1ae000c-0x338e96bc1ae000c regionserver:60020-0x338e96bc1ae000c-0x338e96bc1ae000c received expired from ZooKeeper, aborting
  96. 2012-08-07 10:03:51,695 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60020
  97. 2012-08-07 10:03:51,799 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 0 on 60020: exiting
  98. 2012-08-07 10:03:51,799 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 1 on 60020: exiting
  99. 2012-08-07 10:03:51,799 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 2 on 60020: exiting
  100. 2012-08-07 10:03:51,799 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 3 on 60020: exiting
  101. 2012-08-07 10:03:51,799 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 4 on 60020: exiting
  102. 2012-08-07 10:03:51,800 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 5 on 60020: exiting
  103. 2012-08-07 10:03:51,800 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 6 on 60020: exiting
  104. 2012-08-07 10:03:51,800 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 7 on 60020: exiting
  105. 2012-08-07 10:03:51,800 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 8 on 60020: exiting
  106. 2012-08-07 10:03:51,800 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 9 on 60020: exiting
  107. 2012-08-07 10:03:51,800 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 0 on 60020: exiting
  108. 2012-08-07 10:03:51,801 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 1 on 60020: exiting
  109. 2012-08-07 10:03:51,801 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 2 on 60020: exiting
  110. 2012-08-07 10:03:51,801 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 3 on 60020: exiting
  111. 2012-08-07 10:03:51,801 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 4 on 60020: exiting
  112. 2012-08-07 10:03:51,801 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 5 on 60020: exiting
  113. 2012-08-07 10:03:51,801 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 6 on 60020: exiting
  114. 2012-08-07 10:03:51,801 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 7 on 60020: exiting
  115. 2012-08-07 10:03:51,802 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 8 on 60020: exiting
  116. 2012-08-07 10:03:51,802 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 9 on 60020: exiting
  117. 2012-08-07 10:03:51,802 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server listener on 60020
  118. 2012-08-07 10:03:51,695 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
  119. 2012-08-07 10:03:51,804 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: Sending interrupt to stop the worker thread
  120. 2012-08-07 10:03:51,805 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server Responder
  121. 2012-08-07 10:03:51,923 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server Responder
  122. 2012-08-07 10:03:51,860 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
  123. 2012-08-07 10:03:51,860 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
  124. 2012-08-07 10:03:51,924 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
  125. 2012-08-07 10:03:51,979 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: SplitLogWorker interrupted while waiting for task, exiting: java.lang.InterruptedException
  126. 2012-08-07 10:03:51,979 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: SplitLogWorker ihub-dn-l4,60020,1344317192178 exiting
  127. 2012-08-07 10:03:51,987 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to ihub-an-l1/172.31.192.151:2181, initiating session
  128. 2012-08-07 10:03:51,989 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server ihub-an-l1/172.31.192.151:2181, sessionid = 0x238fea720cb0033, negotiated timeout = 40000
  129. 2012-08-07 10:03:51,993 INFO org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Reconnected successfully. This disconnect could have been caused by a network partition or a long-running GC pause, either way it's recommended that you verify your environment.
  130. 2012-08-07 10:03:51,993 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
  131. 2012-08-07 10:03:52,514 INFO org.mortbay.log: Stopped SelectChannelConnector@0.0.0.0:60030
  132. 2012-08-07 10:03:53,275 INFO org.apache.hadoop.hbase.regionserver.MemStoreFlusher: regionserver60020.cacheFlusher exiting
  133. 2012-08-07 10:03:53,276 INFO org.apache.hadoop.hbase.regionserver.LogRoller: LogRoller exiting.
  134. 2012-08-07 10:03:53,276 INFO org.apache.hadoop.hbase.regionserver.HRegionServer$CompactionChecker: regionserver60020.compactionChecker exiting
  135. 2012-08-07 10:03:53,355 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Block cache LRU eviction started; Attempting to free -226813.98 KB of total=3.12 MB
  136. 2012-08-07 10:03:53,361 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_snappy,20756742902739464179096085,1343852492741.b8b5b4750c31a20ce55ece01d4e5d1fe.
  137. 2012-08-07 10:03:53,362 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_snappy,20756742902739464179096085,1343852492741.b8b5b4750c31a20ce55ece01d4e5d1fe.: disabling compactions & flushes
  138. 2012-08-07 10:03:53,362 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_snappy,20756742902739464179096085,1343852492741.b8b5b4750c31a20ce55ece01d4e5d1fe.
  139. 2012-08-07 10:03:53,362 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server ihub-dn-l4,60020,1344317192178
  140. 2012-08-07 10:03:53,362 DEBUG org.apache.hadoop.hbase.catalog.CatalogTracker: Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@53dafbaf
  141. 2012-08-07 10:03:53,363 INFO org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Closed zookeeper sessionid=0x238fea720cb0033
  142. 2012-08-07 10:03:53,361 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_old,19885532901713190000183147464,1343804251322.f7a055786a4bc2f71709f405657f1c42.
  143. 2012-08-07 10:03:53,369 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
  144. 2012-08-07 10:03:53,369 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_snappy,20756742902739464179096085,1343852492741.b8b5b4750c31a20ce55ece01d4e5d1fe.
  145. 2012-08-07 10:03:53,369 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_snappy,20756742902739464179096085,1343852492741.b8b5b4750c31a20ce55ece01d4e5d1fe.
  146. 2012-08-07 10:03:53,369 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_old,20357432903087031000177634367,1343804251322.ac272787bc1553b48d4fff87237591a6.
  147. 2012-08-07 10:03:53,362 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_old,19706122901502318000183811736,1343804251322.db69bf9d8b60a8904e6ded8866e8a8fc.
  148. 2012-08-07 10:03:53,369 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_old,19885532901713190000183147464,1343804251322.f7a055786a4bc2f71709f405657f1c42.: disabling compactions & flushes
  149. 2012-08-07 10:03:53,369 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_old,19885532901713190000183147464,1343804251322.f7a055786a4bc2f71709f405657f1c42.
  150. 2012-08-07 10:03:53,369 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_old,20357432903087031000177634367,1343804251322.ac272787bc1553b48d4fff87237591a6.: disabling compactions & flushes
  151. 2012-08-07 10:03:53,370 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_old,20357432903087031000177634367,1343804251322.ac272787bc1553b48d4fff87237591a6.
  152. 2012-08-07 10:03:53,370 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_old,19706122901502318000183811736,1343804251322.db69bf9d8b60a8904e6ded8866e8a8fc.: disabling compactions & flushes
  153. 2012-08-07 10:03:53,370 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_old,19706122901502318000183811736,1343804251322.db69bf9d8b60a8904e6ded8866e8a8fc.
  154. 2012-08-07 10:03:53,370 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of .META.,,1.1028785192
  155. 2012-08-07 10:03:53,370 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing .META.,,1.1028785192: disabling compactions & flushes
  156. 2012-08-07 10:03:53,370 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region .META.,,1.1028785192
  157. 2012-08-07 10:03:53,370 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed info
  158. 2012-08-07 10:03:53,370 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed .META.,,1.1028785192
  159. 2012-08-07 10:03:53,370 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region .META.,,1.1028785192
  160. 2012-08-07 10:03:53,370 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
  161. 2012-08-07 10:03:53,370 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_old,19885532901713190000183147464,1343804251322.f7a055786a4bc2f71709f405657f1c42.
  162. 2012-08-07 10:03:53,370 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_old,19885532901713190000183147464,1343804251322.f7a055786a4bc2f71709f405657f1c42.
  163. 2012-08-07 10:03:53,370 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_snappy,20431902900489359187744368,1343852492741.d83c9fc7663c1170fe9cee2fdaf9c005.
  164. 2012-08-07 10:03:53,370 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
  165. 2012-08-07 10:03:53,370 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_old,20357432903087031000177634367,1343804251322.ac272787bc1553b48d4fff87237591a6.
  166. 2012-08-07 10:03:53,370 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_old,20357432903087031000177634367,1343804251322.ac272787bc1553b48d4fff87237591a6.
  167. 2012-08-07 10:03:53,370 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_old,21487562897552179000200620405,1343804251325.030a2dcbc1c2ec67df982c0819cab85b.
  168. 2012-08-07 10:03:53,370 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
  169. 2012-08-07 10:03:53,370 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_old,19706122901502318000183811736,1343804251322.db69bf9d8b60a8904e6ded8866e8a8fc.
  170. 2012-08-07 10:03:53,370 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_old,19706122901502318000183811736,1343804251322.db69bf9d8b60a8904e6ded8866e8a8fc.
  171. 2012-08-07 10:03:53,370 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_old,17245962901699611000183411982,1343804209781.ba9b6482d51cd38ac12d17f97a2ac7d7.
  172. 2012-08-07 10:03:53,371 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_snappy,20431902900489359187744368,1343852492741.d83c9fc7663c1170fe9cee2fdaf9c005.: disabling compactions & flushes
  173. 2012-08-07 10:03:53,371 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_snappy,20431902900489359187744368,1343852492741.d83c9fc7663c1170fe9cee2fdaf9c005.
  174. 2012-08-07 10:03:53,371 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_old,21487562897552179000200620405,1343804251325.030a2dcbc1c2ec67df982c0819cab85b.: disabling compactions & flushes
  175. 2012-08-07 10:03:53,371 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_old,21487562897552179000200620405,1343804251325.030a2dcbc1c2ec67df982c0819cab85b.
  176. 2012-08-07 10:03:53,371 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
  177. 2012-08-07 10:03:53,371 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_snappy,20431902900489359187744368,1343852492741.d83c9fc7663c1170fe9cee2fdaf9c005.
  178. 2012-08-07 10:03:53,371 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_snappy,20431902900489359187744368,1343852492741.d83c9fc7663c1170fe9cee2fdaf9c005.
  179. 2012-08-07 10:03:53,371 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_snappy,17769202898836285194987066,1343851946957.16a9e7abda75159f1e20e7b11bcce066.
  180. 2012-08-07 10:03:53,371 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_old,17245962901699611000183411982,1343804209781.ba9b6482d51cd38ac12d17f97a2ac7d7.: disabling compactions & flushes
  181. 2012-08-07 10:03:53,371 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_old,17245962901699611000183411982,1343804209781.ba9b6482d51cd38ac12d17f97a2ac7d7.
  182. 2012-08-07 10:03:53,371 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_snappy,17769202898836285194987066,1343851946957.16a9e7abda75159f1e20e7b11bcce066.: disabling compactions & flushes
  183. 2012-08-07 10:03:53,371 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_snappy,17769202898836285194987066,1343851946957.16a9e7abda75159f1e20e7b11bcce066.
  184. 2012-08-07 10:03:53,372 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
  185. 2012-08-07 10:03:53,372 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_old,21487562897552179000200620405,1343804251325.030a2dcbc1c2ec67df982c0819cab85b.
  186. 2012-08-07 10:03:53,372 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_old,21487562897552179000200620405,1343804251325.030a2dcbc1c2ec67df982c0819cab85b.
  187. 2012-08-07 10:03:53,372 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_str_as_dbl,18102762899727366191186601,1344358659571.a02f822737d799ef9efd2ae2e7b02ac0.
  188. 2012-08-07 10:03:53,372 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_str_as_dbl,18102762899727366191186601,1344358659571.a02f822737d799ef9efd2ae2e7b02ac0.: disabling compactions & flushes
  189. 2012-08-07 10:03:53,372 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_str_as_dbl,18102762899727366191186601,1344358659571.a02f822737d799ef9efd2ae2e7b02ac0.
  190. 2012-08-07 10:03:53,372 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
  191. 2012-08-07 10:03:53,372 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_old,17245962901699611000183411982,1343804209781.ba9b6482d51cd38ac12d17f97a2ac7d7.
  192. 2012-08-07 10:03:53,372 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_old,17245962901699611000183411982,1343804209781.ba9b6482d51cd38ac12d17f97a2ac7d7.
  193. 2012-08-07 10:03:53,372 INFO org.apache.zookeeper.ZooKeeper: Session: 0x238fea720cb0033 closed
  194. 2012-08-07 10:03:53,372 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Waiting on 17 regions to close
  195. 2012-08-07 10:03:53,372 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
  196. 2012-08-07 10:03:53,372 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions,20618372902310925180472143,1343763705439.a02efc6f59615cc37e8945a2b8607dad.
  197. 2012-08-07 10:03:53,372 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
  198. 2012-08-07 10:03:53,372 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_snappy,17769202898836285194987066,1343851946957.16a9e7abda75159f1e20e7b11bcce066.
  199. 2012-08-07 10:03:53,372 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_snappy,17769202898836285194987066,1343851946957.16a9e7abda75159f1e20e7b11bcce066.
  200. 2012-08-07 10:03:53,372 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_snappy,19149572902570637179441354,1343852583791.d40ef82d421cc772aa1e8b12e5e4b63d.
  201. 2012-08-07 10:03:53,373 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions,20618372902310925180472143,1343763705439.a02efc6f59615cc37e8945a2b8607dad.: disabling compactions & flushes
  202. 2012-08-07 10:03:53,373 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions,20618372902310925180472143,1343763705439.a02efc6f59615cc37e8945a2b8607dad.
  203. 2012-08-07 10:03:53,373 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_snappy,19149572902570637179441354,1343852583791.d40ef82d421cc772aa1e8b12e5e4b63d.: disabling compactions & flushes
  204. 2012-08-07 10:03:53,373 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_snappy,19149572902570637179441354,1343852583791.d40ef82d421cc772aa1e8b12e5e4b63d.
  205. 2012-08-07 10:03:53,373 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
  206. 2012-08-07 10:03:53,373 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions,20618372902310925180472143,1343763705439.a02efc6f59615cc37e8945a2b8607dad.
  207. 2012-08-07 10:03:53,373 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions,20618372902310925180472143,1343763705439.a02efc6f59615cc37e8945a2b8607dad.
  208. 2012-08-07 10:03:53,373 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_old,17068542902003200000182390457,1343804209781.ac1cc94fb91fed0fba464b7731b0a622.
  209. 2012-08-07 10:03:53,373 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
  210. 2012-08-07 10:03:53,373 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_snappy,19149572902570637179441354,1343852583791.d40ef82d421cc772aa1e8b12e5e4b63d.
  211. 2012-08-07 10:03:53,373 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_snappy,19149572902570637179441354,1343852583791.d40ef82d421cc772aa1e8b12e5e4b63d.
  212. 2012-08-07 10:03:53,373 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_lzo,20728472897865225199286692,1343890502802.f64965f713fd71679feacd8f0652f856.
  213. 2012-08-07 10:03:53,373 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_old,17068542902003200000182390457,1343804209781.ac1cc94fb91fed0fba464b7731b0a622.: disabling compactions & flushes
  214. 2012-08-07 10:03:53,374 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_old,17068542902003200000182390457,1343804209781.ac1cc94fb91fed0fba464b7731b0a622.
  215. 2012-08-07 10:03:53,374 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_lzo,20728472897865225199286692,1343890502802.f64965f713fd71679feacd8f0652f856.: disabling compactions & flushes
  216. 2012-08-07 10:03:53,374 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_lzo,20728472897865225199286692,1343890502802.f64965f713fd71679feacd8f0652f856.
  217. 2012-08-07 10:03:53,374 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
  218. 2012-08-07 10:03:53,374 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_str_as_dbl,18102762899727366191186601,1344358659571.a02f822737d799ef9efd2ae2e7b02ac0.
  219. 2012-08-07 10:03:53,374 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_str_as_dbl,18102762899727366191186601,1344358659571.a02f822737d799ef9efd2ae2e7b02ac0.
  220. 2012-08-07 10:03:53,374 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_old,19938982902616316000179367339,1343804251322.e11e34ce344c7b5ca516ce2874d88878.
  221. 2012-08-07 10:03:53,374 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_old,19938982902616316000179367339,1343804251322.e11e34ce344c7b5ca516ce2874d88878.: disabling compactions & flushes
  222. 2012-08-07 10:03:53,374 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_old,19938982902616316000179367339,1343804251322.e11e34ce344c7b5ca516ce2874d88878.
  223. 2012-08-07 10:03:53,374 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
  224. 2012-08-07 10:03:53,374 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_lzo,20728472897865225199286692,1343890502802.f64965f713fd71679feacd8f0652f856.
  225. 2012-08-07 10:03:53,374 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_lzo,20728472897865225199286692,1343890502802.f64965f713fd71679feacd8f0652f856.
  226. 2012-08-07 10:03:53,374 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_lzo,17419812896783007204292703,1343861353204.9991b5b6d39ac0185a15d01d611466f1.
  227. 2012-08-07 10:03:53,374 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_lzo,17419812896783007204292703,1343861353204.9991b5b6d39ac0185a15d01d611466f1.: disabling compactions & flushes
  228. 2012-08-07 10:03:53,374 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_lzo,17419812896783007204292703,1343861353204.9991b5b6d39ac0185a15d01d611466f1.
  229. 2012-08-07 10:03:53,511 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
  230. 2012-08-07 10:03:53,511 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_lzo,17419812896783007204292703,1343861353204.9991b5b6d39ac0185a15d01d611466f1.
  231. 2012-08-07 10:03:53,511 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_lzo,17419812896783007204292703,1343861353204.9991b5b6d39ac0185a15d01d611466f1.
  232. 2012-08-07 10:03:53,511 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_str_as_dbl,17929912900560606187890916,1344358522026.74def75a6e7b1f9d7003988f4cb9b4b4.
  233. 2012-08-07 10:03:53,512 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_str_as_dbl,17929912900560606187890916,1344358522026.74def75a6e7b1f9d7003988f4cb9b4b4.: disabling compactions & flushes
  234. 2012-08-07 10:03:53,512 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_str_as_dbl,17929912900560606187890916,1344358522026.74def75a6e7b1f9d7003988f4cb9b4b4.
  235. 2012-08-07 10:03:53,512 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
  236. 2012-08-07 10:03:53,512 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_old,19938982902616316000179367339,1343804251322.e11e34ce344c7b5ca516ce2874d88878.
  237. 2012-08-07 10:03:53,512 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_old,19938982902616316000179367339,1343804251322.e11e34ce344c7b5ca516ce2874d88878.
  238. 2012-08-07 10:03:53,512 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_lzo,19701672896085190206718344,1343890676286.0cc8b4f9bf91f609efa45f90ba799eeb.
  239. 2012-08-07 10:03:53,512 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_lzo,19701672896085190206718344,1343890676286.0cc8b4f9bf91f609efa45f90ba799eeb.: disabling compactions & flushes
  240. 2012-08-07 10:03:53,512 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_lzo,19701672896085190206718344,1343890676286.0cc8b4f9bf91f609efa45f90ba799eeb.
  241. 2012-08-07 10:03:53,513 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
  242. 2012-08-07 10:03:53,513 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_lzo,19701672896085190206718344,1343890676286.0cc8b4f9bf91f609efa45f90ba799eeb.
  243. 2012-08-07 10:03:53,513 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_lzo,19701672896085190206718344,1343890676286.0cc8b4f9bf91f609efa45f90ba799eeb.
  244. 2012-08-07 10:03:53,513 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_snappy,20994522896624231204509343,1343852298510.5e66dd1d3d5f8bf88714d35b3487aaa6.
  245. 2012-08-07 10:03:53,513 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_snappy,20994522896624231204509343,1343852298510.5e66dd1d3d5f8bf88714d35b3487aaa6.: disabling compactions & flushes
  246. 2012-08-07 10:03:53,513 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_snappy,20994522896624231204509343,1343852298510.5e66dd1d3d5f8bf88714d35b3487aaa6.
  247. 2012-08-07 10:03:53,513 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
  248. 2012-08-07 10:03:53,513 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_snappy,20994522896624231204509343,1343852298510.5e66dd1d3d5f8bf88714d35b3487aaa6.
  249. 2012-08-07 10:03:53,513 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_snappy,20994522896624231204509343,1343852298510.5e66dd1d3d5f8bf88714d35b3487aaa6.
  250. 2012-08-07 10:03:53,513 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_snappy,17084522903332660176976987,1343852003271.7819916c078ba9e7fbc6ed349fa3b49e.
  251. 2012-08-07 10:03:53,513 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_snappy,17084522903332660176976987,1343852003271.7819916c078ba9e7fbc6ed349fa3b49e.: disabling compactions & flushes
  252. 2012-08-07 10:03:53,514 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_snappy,17084522903332660176976987,1343852003271.7819916c078ba9e7fbc6ed349fa3b49e.
  253. 2012-08-07 10:03:53,514 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
  254. 2012-08-07 10:03:53,514 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_old,17068542902003200000182390457,1343804209781.ac1cc94fb91fed0fba464b7731b0a622.
  255. 2012-08-07 10:03:53,514 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_old,17068542902003200000182390457,1343804209781.ac1cc94fb91fed0fba464b7731b0a622.
  256. 2012-08-07 10:03:53,514 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_old,19441462902893156000178907118,1343804251322.d148fc9ee74c2e486d57257ff3229c42.
  257. 2012-08-07 10:03:53,514 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_old,19441462902893156000178907118,1343804251322.d148fc9ee74c2e486d57257ff3229c42.: disabling compactions & flushes
  258. 2012-08-07 10:03:53,514 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_old,19441462902893156000178907118,1343804251322.d148fc9ee74c2e486d57257ff3229c42.
  259. 2012-08-07 10:03:53,514 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
  260. 2012-08-07 10:03:53,514 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_snappy,17084522903332660176976987,1343852003271.7819916c078ba9e7fbc6ed349fa3b49e.
  261. 2012-08-07 10:03:53,514 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_snappy,17084522903332660176976987,1343852003271.7819916c078ba9e7fbc6ed349fa3b49e.
  262. 2012-08-07 10:03:53,514 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_old,17924342900647286000187101637,1343810650642.f00debc67cd020cf7f9f5778d1a8760b.
  263. 2012-08-07 10:03:53,515 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_old,17924342900647286000187101637,1343810650642.f00debc67cd020cf7f9f5778d1a8760b.: disabling compactions & flushes
  264. 2012-08-07 10:03:53,515 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_old,17924342900647286000187101637,1343810650642.f00debc67cd020cf7f9f5778d1a8760b.
  265. 2012-08-07 10:03:53,515 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
  266. 2012-08-07 10:03:53,515 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_str_as_dbl,17929912900560606187890916,1344358522026.74def75a6e7b1f9d7003988f4cb9b4b4.
  267. 2012-08-07 10:03:53,515 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_str_as_dbl,17929912900560606187890916,1344358522026.74def75a6e7b1f9d7003988f4cb9b4b4.
  268. 2012-08-07 10:03:53,515 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_old,21324392899982106000190083590,1343804251323.c5fdceeeab7cd5d3dcbdef81054326e4.
  269. 2012-08-07 10:03:53,515 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_old,21324392899982106000190083590,1343804251323.c5fdceeeab7cd5d3dcbdef81054326e4.: disabling compactions & flushes
  270. 2012-08-07 10:03:53,515 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_old,21324392899982106000190083590,1343804251323.c5fdceeeab7cd5d3dcbdef81054326e4.
  271. 2012-08-07 10:03:53,516 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
  272. 2012-08-07 10:03:53,516 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_old,17924342900647286000187101637,1343810650642.f00debc67cd020cf7f9f5778d1a8760b.
  273. 2012-08-07 10:03:53,516 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_old,17924342900647286000187101637,1343810650642.f00debc67cd020cf7f9f5778d1a8760b.
  274. 2012-08-07 10:03:53,516 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_str_as_dbl,18276072903006942178391606,1344358659571.a98079c3cae6ea5c819f06c652a107f4.
  275. 2012-08-07 10:03:53,516 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_str_as_dbl,18276072903006942178391606,1344358659571.a98079c3cae6ea5c819f06c652a107f4.: disabling compactions & flushes
  276. 2012-08-07 10:03:53,516 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_str_as_dbl,18276072903006942178391606,1344358659571.a98079c3cae6ea5c819f06c652a107f4.
  277. 2012-08-07 10:03:53,518 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
  278. 2012-08-07 10:03:53,518 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_str_as_dbl,18276072903006942178391606,1344358659571.a98079c3cae6ea5c819f06c652a107f4.
  279. 2012-08-07 10:03:53,518 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_str_as_dbl,18276072903006942178391606,1344358659571.a98079c3cae6ea5c819f06c652a107f4.
  280. 2012-08-07 10:03:53,518 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_str_as_dbl,17761522899263569193018364,1344358522026.eb1f6886364054bc96f1f190f24b6eed.
  281. 2012-08-07 10:03:53,518 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_str_as_dbl,17761522899263569193018364,1344358522026.eb1f6886364054bc96f1f190f24b6eed.: disabling compactions & flushes
  282. 2012-08-07 10:03:53,518 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: waiting for 1 compactions to complete for region transactions_str_as_dbl,17761522899263569193018364,1344358522026.eb1f6886364054bc96f1f190f24b6eed.
  283. 2012-08-07 10:03:53,518 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
  284. 2012-08-07 10:03:53,518 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_old,19441462902893156000178907118,1343804251322.d148fc9ee74c2e486d57257ff3229c42.
  285. 2012-08-07 10:03:53,518 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_old,19441462902893156000178907118,1343804251322.d148fc9ee74c2e486d57257ff3229c42.
  286. 2012-08-07 10:03:53,518 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
  287. 2012-08-07 10:03:53,518 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_old,21324392899982106000190083590,1343804251323.c5fdceeeab7cd5d3dcbdef81054326e4.
  288. 2012-08-07 10:03:53,518 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_old,21324392899982106000190083590,1343804251323.c5fdceeeab7cd5d3dcbdef81054326e4.
  289. 2012-08-07 10:03:54,372 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Waiting on 1 regions to close
  290. 2012-08-07 10:03:54,372 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: {eb1f6886364054bc96f1f190f24b6eed=transactions_str_as_dbl,17761522899263569193018364,1344358522026.eb1f6886364054bc96f1f190f24b6eed.}
  291. 2012-08-07 10:03:58,714 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction interrupted
  292. java.io.InterruptedIOException: Aborting compaction of store t in region transactions_str_as_dbl,17761522899263569193018364,1344358522026.eb1f6886364054bc96f1f190f24b6eed. because it was interrupted.
  293. at org.apache.hadoop.hbase.regionserver.Store.compactStore(Store.java:1285)
  294. at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:767)
  295. at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1039)
  296. at org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest.run(CompactionRequest.java:177)
  297. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  298. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  299. at java.lang.Thread.run(Thread.java:662)
  300. 2012-08-07 10:03:58,714 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_str_as_dbl,17761522899263569193018364,1344358522026.eb1f6886364054bc96f1f190f24b6eed.
  301. 2012-08-07 10:03:58,716 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
  302. 2012-08-07 10:03:58,716 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_str_as_dbl,17761522899263569193018364,1344358522026.eb1f6886364054bc96f1f190f24b6eed.
  303. 2012-08-07 10:03:58,716 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_str_as_dbl,17761522899263569193018364,1344358522026.eb1f6886364054bc96f1f190f24b6eed.
  304. 2012-08-07 10:03:58,716 INFO org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: aborted compaction: regionName=transactions_str_as_dbl,17761522899263569193018364,1344358522026.eb1f6886364054bc96f1f190f24b6eed., storeName=t, fileCount=7, fileSize=1.0g, priority=0, time=398634508588496; duration=2mins, 18sec
  305. 2012-08-07 10:03:58,716 DEBUG org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: CompactSplitThread status: compaction_queue=(3:0), split_queue=0
  306. 2012-08-07 10:03:59,375 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: stopping server ihub-dn-l4,60020,1344317192178; all regions closed.
  307. 2012-08-07 10:03:59,375 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: regionserver60020.logSyncer interrupted while waiting for sync requests
  308. 2012-08-07 10:03:59,375 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: regionserver60020.logSyncer exiting
  309. 2012-08-07 10:03:59,375 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: closing hlog writer in hdfs://ihubcluster/hbase/.logs/ihub-dn-l4,60020,1344317192178
  310. 2012-08-07 10:03:59,750 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: Close and delete failed
  311. org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /hbase/.logs/ihub-dn-l4,60020,1344317192178/ihub-dn-l4%2C60020%2C1344317192178.1344358960566 File does not exist. [Lease. Holder: DFSClient_hb_rs_ihub-dn-l4,60020,1344317192178_1366946297_1, pendingcreates: 1]
  312. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2115)
  313. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2106)
  314. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:2173)
  315. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:2150)
  316. at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:525)
  317. at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:330)
  318. at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:42606)
  319. at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:427)
  320. at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:916)
  321. at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1692)
  322. at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1688)
  323. at java.security.AccessController.doPrivileged(Native Method)
  324. at javax.security.auth.Subject.doAs(Subject.java:396)
  325. at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
  326. at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1686)
  327.  
  328. at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  329. at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
  330. at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
  331. at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
  332. at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:96)
  333. at org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
  334. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeWAL(HRegionServer.java:869)
  335. at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:746)
  336. at java.lang.Thread.run(Thread.java:662)
  337. 2012-08-07 10:03:59,751 INFO org.apache.hadoop.hbase.regionserver.Leases: regionserver60020 closing leases
  338. 2012-08-07 10:03:59,751 INFO org.apache.hadoop.hbase.regionserver.Leases: regionserver60020 closed leases
  339. 2012-08-07 10:03:59,787 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Failed deleting my ephemeral node
  340. org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/rs/ihub-dn-l4,60020,1344317192178
  341. at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
  342. at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
  343. at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:868)
  344. at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:107)
  345. at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:967)
  346. at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:956)
  347. at org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:958)
  348. at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:756)
  349. at java.lang.Thread.run(Thread.java:662)
  350. 2012-08-07 10:03:59,787 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: stopping server ihub-dn-l4,60020,1344317192178; zookeeper connection closed.
  351. 2012-08-07 10:03:59,787 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Split Thread to finish...
  352. 2012-08-07 10:03:59,788 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large Compaction Thread to finish...
  353. 2012-08-07 10:03:59,788 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Small Compaction Thread to finish...
  354. 2012-08-07 10:03:59,788 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020 exiting
  355. 2012-08-07 10:04:00,095 INFO org.apache.hadoop.hbase.regionserver.Leases: regionserver60020.leaseChecker closing leases
  356. 2012-08-07 10:04:00,107 INFO org.apache.hadoop.hbase.regionserver.Leases: regionserver60020.leaseChecker closed leases
  357. 2012-08-07 10:04:00,324 INFO org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@5374a6e2
  358. 2012-08-07 10:04:00,324 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown hook
  359. 2012-08-07 10:04:00,324 INFO org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs shutdown hook thread.
  360. 2012-08-07 10:04:00,325 INFO org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook finished.
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!