2012-08-07 10:02:57,845 DEBUG org.apache.hadoop.hbase.io.hfile.HFileWriterV2: Initialized with CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
2012-08-07 10:02:58,506 DEBUG org.apache.hadoop.hbase.regionserver.Store: Renaming flushed file at hdfs://ihubcluster/hbase/transactions_str_as_dbl/eb1f6886364054bc96f1f190f24b6eed/.tmp/4a556e5cbc01432187048c50c6c8d030 to hdfs://ihubcluster/hbase/transactions_str_as_dbl/eb1f6886364054bc96f1f190f24b6eed/t/4a556e5cbc01432187048c50c6c8d030
2012-08-07 10:02:58,562 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://ihubcluster/hbase/transactions_str_as_dbl/eb1f6886364054bc96f1f190f24b6eed/t/4a556e5cbc01432187048c50c6c8d030, entries=594386, sequenceid=1441817, memsize=135.9m, filesize=34.7m
2012-08-07 10:02:58,565 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~136.3m/142962624, currentsize=5.2m/5457392 for region transactions_str_as_dbl,17761522899263569193018364,1344358522026.eb1f6886364054bc96f1f190f24b6eed. in 1700ms, sequenceid=1441817, compaction requested=false
2012-08-07 10:02:58,565 DEBUG org.apache.hadoop.hbase.regionserver.Store: hdfs://ihubcluster/hbase/transactions_str_as_dbl/eb1f6886364054bc96f1f190f24b6eed/t/c3d8ea5156ab49afb5f2e70d2c2f5b02.189f8d09c53dc72b08660ec1f9b50646-hdfs://ihubcluster/hbase/transactions_str_as_dbl/189f8d09c53dc72b08660ec1f9b50646/t/c3d8ea5156ab49afb5f2e70d2c2f5b02-bottom is not splittable
2012-08-07 10:03:50,094 WARN org.apache.hadoop.hbase.util.Sleeper: We slept 41838ms instead of 3000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2012-08-07 10:03:50,111 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 9 on 60020 took 41382 ms appending an edit to hlog; editcount=1132, len~=18.3k
2012-08-07 10:03:50,114 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":41575,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@423c5ad9), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"172.31.192.54:33894","starttimems":1344358988536,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
2012-08-07 10:03:50,231 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":41592,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@2e27d67e), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"172.31.192.153:42073","starttimems":1344358988638,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
2012-08-07 10:03:50,235 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":41685,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@2d7fb349), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"172.31.192.53:40319","starttimems":1344358988549,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
2012-08-07 10:03:50,233 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":41631,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@397301de), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"172.31.192.153:42052","starttimems":1344358988601,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
2012-08-07 10:03:50,308 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":41741,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@2731a9e7), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"172.31.192.102:52815","starttimems":1344358988545,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
2012-08-07 10:03:50,319 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 46074ms for sessionid 0x238e96bb4e00011, closing socket connection and attempting reconnect
2012-08-07 10:03:50,319 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 46522ms for sessionid 0x338e96bc1ae000c, closing socket connection and attempting reconnect
2012-08-07 10:03:50,413 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":41673,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@26f68905), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"172.31.192.104:47766","starttimems":1344358988728,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
2012-08-07 10:03:50,680 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server ihub-an-l1/172.31.192.151:2181
2012-08-07 10:03:50,965 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-08-07 10:03:50,965 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-08-07 10:03:50,967 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to ihub-an-l1/172.31.192.151:2181, initiating session
2012-08-07 10:03:50,974 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x238e96bb4e00011 has expired, closing socket connection
2012-08-07 10:03:50,974 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server ihub-dn-l4,60020,1344317192178: Unhandled exception: org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing ihub-dn-l4,60020,1344317192178 as dead server
	at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:222)
	at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:148)
	at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:851)
	at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
	at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336)

org.apache.hadoop.hbase.YouAreDeadException: org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing ihub-dn-l4,60020,1344317192178 as dead server
	at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:222)
	at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:148)
	at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:851)
	at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
	at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336)

	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
	at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90)
	at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:791)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:682)
	at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.hadoop.hbase.YouAreDeadException: org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing ihub-dn-l4,60020,1344317192178 as dead server
	at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:222)
	at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:148)
	at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:851)
	at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
	at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336)

	at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:918)
	at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
	at $Proxy10.regionServerReport(Unknown Source)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:788)
	... 2 more
2012-08-07 10:03:51,043 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.AggregateImplementation]
2012-08-07 10:03:51,013 INFO org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: This client just lost it's session with ZooKeeper, trying to reconnect.
2012-08-07 10:03:51,074 INFO org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Trying to reconnect to zookeeper
2012-08-07 10:03:51,225 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server ihub-an-l1/172.31.192.151:2181
2012-08-07 10:03:51,225 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-08-07 10:03:51,225 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-08-07 10:03:51,226 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to ihub-an-l1/172.31.192.151:2181, initiating session
2012-08-07 10:03:51,227 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server ihub-dn-l4,60020,1344317192178: regionserver:60020-0x338e96bc1ae000c-0x338e96bc1ae000c regionserver:60020-0x338e96bc1ae000c-0x338e96bc1ae000c received expired from ZooKeeper, aborting
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired
	at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:374)
	at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
2012-08-07 10:03:51,523 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.AggregateImplementation]
2012-08-07 10:03:51,524 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: requestsPerSecond=23, numberOfOnlineRegions=25, numberOfStores=25, numberOfStorefiles=42, storefileIndexSizeMB=0, rootIndexSizeKB=594, totalStaticIndexSizeKB=32130, totalStaticBloomSizeKB=0, memstoreSizeMB=278, mbInMemoryWithoutWAL=0, numberOfPutsWithoutWAL=0, readRequestsCount=95323, writeRequestsCount=43379, compactionQueueSize=3, flushQueueSize=0, usedHeapMB=569, maxHeapMB=1197, blockCacheSizeMB=3.12, blockCacheFreeMB=296.37, blockCacheCount=33, blockCacheHitCount=20247, blockCacheMissCount=95108, blockCacheEvictedCount=34, blockCacheHitRatio=17%, blockCacheHitCachingRatio=99%, hdfsBlocksLocalityIndex=16, deleteRequestLatencyMean=497337129.82, deleteRequestLatencyCount=40, deleteRequestLatencyMedian=2348868.5, deleteRequestLatency75th=6156771.5, deleteRequestLatency95th=4972486679.9, deleteRequestLatency99th=9093879552, deleteRequestLatency999th=9093879552, getRequestLatencyMean=0, getRequestLatencyCount=0, getRequestLatencyMedian=0, getRequestLatency75th=0, getRequestLatency95th=0, getRequestLatency99th=0, getRequestLatency999th=0, putRequestLatencyMean=0, putRequestLatencyCount=0, putRequestLatencyMedian=0, putRequestLatency75th=0, putRequestLatency95th=0, putRequestLatency99th=0, putRequestLatency999th=0, fsReadLatencyHistogramMean=0, fsReadLatencyHistogramCount=0, fsReadLatencyHistogramMedian=0, fsReadLatencyHistogram75th=0, fsReadLatencyHistogram95th=0, fsReadLatencyHistogram99th=0, fsReadLatencyHistogram999th=0, fsWriteLatencyHistogramMean=0, fsWriteLatencyHistogramCount=0, fsWriteLatencyHistogramMedian=0, fsWriteLatencyHistogram75th=0, fsWriteLatencyHistogram95th=0, fsWriteLatencyHistogram99th=0, fsWriteLatencyHistogram999th=0
2012-08-07 10:03:51,524 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: requestsPerSecond=23, numberOfOnlineRegions=25, numberOfStores=25, numberOfStorefiles=42, storefileIndexSizeMB=0, rootIndexSizeKB=594, totalStaticIndexSizeKB=32130, totalStaticBloomSizeKB=0, memstoreSizeMB=278, mbInMemoryWithoutWAL=0, numberOfPutsWithoutWAL=0, readRequestsCount=95323, writeRequestsCount=43379, compactionQueueSize=3, flushQueueSize=0, usedHeapMB=559, maxHeapMB=1197, blockCacheSizeMB=3.12, blockCacheFreeMB=296.37, blockCacheCount=33, blockCacheHitCount=20247, blockCacheMissCount=95108, blockCacheEvictedCount=34, blockCacheHitRatio=17%, blockCacheHitCachingRatio=99%, hdfsBlocksLocalityIndex=16, deleteRequestLatencyMean=497337129.82, deleteRequestLatencyCount=40, deleteRequestLatencyMedian=2348868.5, deleteRequestLatency75th=6156771.5, deleteRequestLatency95th=4972486679.9, deleteRequestLatency99th=9093879552, deleteRequestLatency999th=9093879552, getRequestLatencyMean=0, getRequestLatencyCount=0, getRequestLatencyMedian=0, getRequestLatency75th=0, getRequestLatency95th=0, getRequestLatency99th=0, getRequestLatency999th=0, putRequestLatencyMean=0, putRequestLatencyCount=0, putRequestLatencyMedian=0, putRequestLatency75th=0, putRequestLatency95th=0, putRequestLatency99th=0, putRequestLatency999th=0, fsReadLatencyHistogramMean=0, fsReadLatencyHistogramCount=0, fsReadLatencyHistogramMedian=0, fsReadLatencyHistogram75th=0, fsReadLatencyHistogram95th=0, fsReadLatencyHistogram99th=0, fsReadLatencyHistogram999th=0, fsWriteLatencyHistogramMean=0, fsWriteLatencyHistogramCount=0, fsWriteLatencyHistogramMedian=0, fsWriteLatencyHistogram75th=0, fsWriteLatencyHistogram95th=0, fsWriteLatencyHistogram99th=0, fsWriteLatencyHistogram999th=0
2012-08-07 10:03:51,517 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=ihub-an-l1:2181,ihub-an-g1:2181,ihub-nn-a2:2181 sessionTimeout=180000 watcher=hconnection
2012-08-07 10:03:51,227 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x338e96bc1ae000c has expired, closing socket connection
2012-08-07 10:03:51,685 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The identifier of this process is 22543@ihub-dn-l4
2012-08-07 10:03:51,682 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /172.31.192.151:2181
2012-08-07 10:03:51,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Unhandled exception: org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing ihub-dn-l4,60020,1344317192178 as dead server
	at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:222)
	at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:148)
	at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:851)
	at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
	at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336)

2012-08-07 10:03:51,695 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: regionserver:60020-0x338e96bc1ae000c-0x338e96bc1ae000c regionserver:60020-0x338e96bc1ae000c-0x338e96bc1ae000c received expired from ZooKeeper, aborting
2012-08-07 10:03:51,695 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60020
2012-08-07 10:03:51,799 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 0 on 60020: exiting
2012-08-07 10:03:51,799 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 1 on 60020: exiting
2012-08-07 10:03:51,799 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 2 on 60020: exiting
2012-08-07 10:03:51,799 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 3 on 60020: exiting
2012-08-07 10:03:51,799 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 4 on 60020: exiting
2012-08-07 10:03:51,800 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 5 on 60020: exiting
2012-08-07 10:03:51,800 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 6 on 60020: exiting
2012-08-07 10:03:51,800 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 7 on 60020: exiting
2012-08-07 10:03:51,800 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 8 on 60020: exiting
2012-08-07 10:03:51,800 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 9 on 60020: exiting
2012-08-07 10:03:51,800 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 0 on 60020: exiting
2012-08-07 10:03:51,801 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 1 on 60020: exiting
2012-08-07 10:03:51,801 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 2 on 60020: exiting
2012-08-07 10:03:51,801 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 3 on 60020: exiting
2012-08-07 10:03:51,801 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 4 on 60020: exiting
2012-08-07 10:03:51,801 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 5 on 60020: exiting
2012-08-07 10:03:51,801 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 6 on 60020: exiting
2012-08-07 10:03:51,801 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 7 on 60020: exiting
2012-08-07 10:03:51,802 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 8 on 60020: exiting
2012-08-07 10:03:51,802 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 9 on 60020: exiting
2012-08-07 10:03:51,802 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server listener on 60020
2012-08-07 10:03:51,695 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2012-08-07 10:03:51,804 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: Sending interrupt to stop the worker thread
2012-08-07 10:03:51,805 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server Responder
2012-08-07 10:03:51,923 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server Responder
2012-08-07 10:03:51,860 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
2012-08-07 10:03:51,860 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-08-07 10:03:51,924 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-08-07 10:03:51,979 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: SplitLogWorker interrupted while waiting for task, exiting: java.lang.InterruptedException
2012-08-07 10:03:51,979 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: SplitLogWorker ihub-dn-l4,60020,1344317192178 exiting
2012-08-07 10:03:51,987 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to ihub-an-l1/172.31.192.151:2181, initiating session
2012-08-07 10:03:51,989 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server ihub-an-l1/172.31.192.151:2181, sessionid = 0x238fea720cb0033, negotiated timeout = 40000
2012-08-07 10:03:51,993 INFO org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Reconnected successfully. This disconnect could have been caused by a network partition or a long-running GC pause, either way it's recommended that you verify your environment.
2012-08-07 10:03:51,993 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2012-08-07 10:03:52,514 INFO org.mortbay.log: Stopped SelectChannelConnector@0.0.0.0:60030
2012-08-07 10:03:53,275 INFO org.apache.hadoop.hbase.regionserver.MemStoreFlusher: regionserver60020.cacheFlusher exiting
2012-08-07 10:03:53,276 INFO org.apache.hadoop.hbase.regionserver.LogRoller: LogRoller exiting.
2012-08-07 10:03:53,276 INFO org.apache.hadoop.hbase.regionserver.HRegionServer$CompactionChecker: regionserver60020.compactionChecker exiting
2012-08-07 10:03:53,355 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Block cache LRU eviction started; Attempting to free -226813.98 KB of total=3.12 MB
2012-08-07 10:03:53,361 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_snappy,20756742902739464179096085,1343852492741.b8b5b4750c31a20ce55ece01d4e5d1fe.
2012-08-07 10:03:53,362 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_snappy,20756742902739464179096085,1343852492741.b8b5b4750c31a20ce55ece01d4e5d1fe.: disabling compactions & flushes
2012-08-07 10:03:53,362 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_snappy,20756742902739464179096085,1343852492741.b8b5b4750c31a20ce55ece01d4e5d1fe.
2012-08-07 10:03:53,362 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server ihub-dn-l4,60020,1344317192178
2012-08-07 10:03:53,362 DEBUG org.apache.hadoop.hbase.catalog.CatalogTracker: Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@53dafbaf
2012-08-07 10:03:53,363 INFO org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Closed zookeeper sessionid=0x238fea720cb0033
2012-08-07 10:03:53,361 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_old,19885532901713190000183147464,1343804251322.f7a055786a4bc2f71709f405657f1c42.
2012-08-07 10:03:53,369 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
2012-08-07 10:03:53,369 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_snappy,20756742902739464179096085,1343852492741.b8b5b4750c31a20ce55ece01d4e5d1fe.
2012-08-07 10:03:53,369 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_snappy,20756742902739464179096085,1343852492741.b8b5b4750c31a20ce55ece01d4e5d1fe.
2012-08-07 10:03:53,369 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_old,20357432903087031000177634367,1343804251322.ac272787bc1553b48d4fff87237591a6.
2012-08-07 10:03:53,362 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_old,19706122901502318000183811736,1343804251322.db69bf9d8b60a8904e6ded8866e8a8fc.
2012-08-07 10:03:53,369 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_old,19885532901713190000183147464,1343804251322.f7a055786a4bc2f71709f405657f1c42.: disabling compactions & flushes
2012-08-07 10:03:53,369 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_old,19885532901713190000183147464,1343804251322.f7a055786a4bc2f71709f405657f1c42.
2012-08-07 10:03:53,369 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_old,20357432903087031000177634367,1343804251322.ac272787bc1553b48d4fff87237591a6.: disabling compactions & flushes
2012-08-07 10:03:53,370 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_old,20357432903087031000177634367,1343804251322.ac272787bc1553b48d4fff87237591a6.
2012-08-07 10:03:53,370 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_old,19706122901502318000183811736,1343804251322.db69bf9d8b60a8904e6ded8866e8a8fc.: disabling compactions & flushes
2012-08-07 10:03:53,370 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_old,19706122901502318000183811736,1343804251322.db69bf9d8b60a8904e6ded8866e8a8fc.
2012-08-07 10:03:53,370 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of .META.,,1.1028785192
2012-08-07 10:03:53,370 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing .META.,,1.1028785192: disabling compactions & flushes
2012-08-07 10:03:53,370 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region .META.,,1.1028785192
2012-08-07 10:03:53,370 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed info
2012-08-07 10:03:53,370 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed .META.,,1.1028785192
2012-08-07 10:03:53,370 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region .META.,,1.1028785192
2012-08-07 10:03:53,370 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
2012-08-07 10:03:53,370 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_old,19885532901713190000183147464,1343804251322.f7a055786a4bc2f71709f405657f1c42.
2012-08-07 10:03:53,370 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_old,19885532901713190000183147464,1343804251322.f7a055786a4bc2f71709f405657f1c42.
2012-08-07 10:03:53,370 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_snappy,20431902900489359187744368,1343852492741.d83c9fc7663c1170fe9cee2fdaf9c005.
2012-08-07 10:03:53,370 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
2012-08-07 10:03:53,370 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_old,20357432903087031000177634367,1343804251322.ac272787bc1553b48d4fff87237591a6.
2012-08-07 10:03:53,370 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_old,20357432903087031000177634367,1343804251322.ac272787bc1553b48d4fff87237591a6.
2012-08-07 10:03:53,370 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_old,21487562897552179000200620405,1343804251325.030a2dcbc1c2ec67df982c0819cab85b.
2012-08-07 10:03:53,370 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
2012-08-07 10:03:53,370 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_old,19706122901502318000183811736,1343804251322.db69bf9d8b60a8904e6ded8866e8a8fc.
2012-08-07 10:03:53,370 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_old,19706122901502318000183811736,1343804251322.db69bf9d8b60a8904e6ded8866e8a8fc.
2012-08-07 10:03:53,370 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_old,17245962901699611000183411982,1343804209781.ba9b6482d51cd38ac12d17f97a2ac7d7.
2012-08-07 10:03:53,371 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_snappy,20431902900489359187744368,1343852492741.d83c9fc7663c1170fe9cee2fdaf9c005.: disabling compactions & flushes
2012-08-07 10:03:53,371 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_snappy,20431902900489359187744368,1343852492741.d83c9fc7663c1170fe9cee2fdaf9c005.
2012-08-07 10:03:53,371 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_old,21487562897552179000200620405,1343804251325.030a2dcbc1c2ec67df982c0819cab85b.: disabling compactions & flushes
2012-08-07 10:03:53,371 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_old,21487562897552179000200620405,1343804251325.030a2dcbc1c2ec67df982c0819cab85b.
2012-08-07 10:03:53,371 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
2012-08-07 10:03:53,371 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_snappy,20431902900489359187744368,1343852492741.d83c9fc7663c1170fe9cee2fdaf9c005.
2012-08-07 10:03:53,371 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_snappy,20431902900489359187744368,1343852492741.d83c9fc7663c1170fe9cee2fdaf9c005.
2012-08-07 10:03:53,371 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_snappy,17769202898836285194987066,1343851946957.16a9e7abda75159f1e20e7b11bcce066.
2012-08-07 10:03:53,371 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_old,17245962901699611000183411982,1343804209781.ba9b6482d51cd38ac12d17f97a2ac7d7.: disabling compactions & flushes
2012-08-07 10:03:53,371 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_old,17245962901699611000183411982,1343804209781.ba9b6482d51cd38ac12d17f97a2ac7d7.
2012-08-07 10:03:53,371 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_snappy,17769202898836285194987066,1343851946957.16a9e7abda75159f1e20e7b11bcce066.: disabling compactions & flushes
2012-08-07 10:03:53,371 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_snappy,17769202898836285194987066,1343851946957.16a9e7abda75159f1e20e7b11bcce066.
2012-08-07 10:03:53,372 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
2012-08-07 10:03:53,372 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_old,21487562897552179000200620405,1343804251325.030a2dcbc1c2ec67df982c0819cab85b.
2012-08-07 10:03:53,372 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_old,21487562897552179000200620405,1343804251325.030a2dcbc1c2ec67df982c0819cab85b.
2012-08-07 10:03:53,372 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_str_as_dbl,18102762899727366191186601,1344358659571.a02f822737d799ef9efd2ae2e7b02ac0.
2012-08-07 10:03:53,372 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_str_as_dbl,18102762899727366191186601,1344358659571.a02f822737d799ef9efd2ae2e7b02ac0.: disabling compactions & flushes
2012-08-07 10:03:53,372 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_str_as_dbl,18102762899727366191186601,1344358659571.a02f822737d799ef9efd2ae2e7b02ac0.
2012-08-07 10:03:53,372 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
2012-08-07 10:03:53,372 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_old,17245962901699611000183411982,1343804209781.ba9b6482d51cd38ac12d17f97a2ac7d7.
2012-08-07 10:03:53,372 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_old,17245962901699611000183411982,1343804209781.ba9b6482d51cd38ac12d17f97a2ac7d7.
2012-08-07 10:03:53,372 INFO org.apache.zookeeper.ZooKeeper: Session: 0x238fea720cb0033 closed
2012-08-07 10:03:53,372 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Waiting on 17 regions to close
2012-08-07 10:03:53,372 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2012-08-07 10:03:53,372 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions,20618372902310925180472143,1343763705439.a02efc6f59615cc37e8945a2b8607dad.
2012-08-07 10:03:53,372 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
2012-08-07 10:03:53,372 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_snappy,17769202898836285194987066,1343851946957.16a9e7abda75159f1e20e7b11bcce066.
2012-08-07 10:03:53,372 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_snappy,17769202898836285194987066,1343851946957.16a9e7abda75159f1e20e7b11bcce066.
2012-08-07 10:03:53,372 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_snappy,19149572902570637179441354,1343852583791.d40ef82d421cc772aa1e8b12e5e4b63d.
2012-08-07 10:03:53,373 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions,20618372902310925180472143,1343763705439.a02efc6f59615cc37e8945a2b8607dad.: disabling compactions & flushes
2012-08-07 10:03:53,373 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions,20618372902310925180472143,1343763705439.a02efc6f59615cc37e8945a2b8607dad.
2012-08-07 10:03:53,373 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_snappy,19149572902570637179441354,1343852583791.d40ef82d421cc772aa1e8b12e5e4b63d.: disabling compactions & flushes
2012-08-07 10:03:53,373 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_snappy,19149572902570637179441354,1343852583791.d40ef82d421cc772aa1e8b12e5e4b63d.
2012-08-07 10:03:53,373 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
2012-08-07 10:03:53,373 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions,20618372902310925180472143,1343763705439.a02efc6f59615cc37e8945a2b8607dad.
2012-08-07 10:03:53,373 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions,20618372902310925180472143,1343763705439.a02efc6f59615cc37e8945a2b8607dad.
2012-08-07 10:03:53,373 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_old,17068542902003200000182390457,1343804209781.ac1cc94fb91fed0fba464b7731b0a622.
2012-08-07 10:03:53,373 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
2012-08-07 10:03:53,373 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_snappy,19149572902570637179441354,1343852583791.d40ef82d421cc772aa1e8b12e5e4b63d.
2012-08-07 10:03:53,373 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_snappy,19149572902570637179441354,1343852583791.d40ef82d421cc772aa1e8b12e5e4b63d.
2012-08-07 10:03:53,373 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_lzo,20728472897865225199286692,1343890502802.f64965f713fd71679feacd8f0652f856.
2012-08-07 10:03:53,373 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_old,17068542902003200000182390457,1343804209781.ac1cc94fb91fed0fba464b7731b0a622.: disabling compactions & flushes
2012-08-07 10:03:53,374 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_old,17068542902003200000182390457,1343804209781.ac1cc94fb91fed0fba464b7731b0a622.
2012-08-07 10:03:53,374 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_lzo,20728472897865225199286692,1343890502802.f64965f713fd71679feacd8f0652f856.: disabling compactions & flushes
2012-08-07 10:03:53,374 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_lzo,20728472897865225199286692,1343890502802.f64965f713fd71679feacd8f0652f856.
2012-08-07 10:03:53,374 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
2012-08-07 10:03:53,374 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_str_as_dbl,18102762899727366191186601,1344358659571.a02f822737d799ef9efd2ae2e7b02ac0.
2012-08-07 10:03:53,374 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_str_as_dbl,18102762899727366191186601,1344358659571.a02f822737d799ef9efd2ae2e7b02ac0.
2012-08-07 10:03:53,374 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_old,19938982902616316000179367339,1343804251322.e11e34ce344c7b5ca516ce2874d88878.
2012-08-07 10:03:53,374 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_old,19938982902616316000179367339,1343804251322.e11e34ce344c7b5ca516ce2874d88878.: disabling compactions & flushes
2012-08-07 10:03:53,374 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_old,19938982902616316000179367339,1343804251322.e11e34ce344c7b5ca516ce2874d88878.
2012-08-07 10:03:53,374 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
2012-08-07 10:03:53,374 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_lzo,20728472897865225199286692,1343890502802.f64965f713fd71679feacd8f0652f856.
2012-08-07 10:03:53,374 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_lzo,20728472897865225199286692,1343890502802.f64965f713fd71679feacd8f0652f856.
2012-08-07 10:03:53,374 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_lzo,17419812896783007204292703,1343861353204.9991b5b6d39ac0185a15d01d611466f1.
2012-08-07 10:03:53,374 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_lzo,17419812896783007204292703,1343861353204.9991b5b6d39ac0185a15d01d611466f1.: disabling compactions & flushes
2012-08-07 10:03:53,374 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_lzo,17419812896783007204292703,1343861353204.9991b5b6d39ac0185a15d01d611466f1.
2012-08-07 10:03:53,511 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
2012-08-07 10:03:53,511 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_lzo,17419812896783007204292703,1343861353204.9991b5b6d39ac0185a15d01d611466f1.
2012-08-07 10:03:53,511 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_lzo,17419812896783007204292703,1343861353204.9991b5b6d39ac0185a15d01d611466f1.
2012-08-07 10:03:53,511 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_str_as_dbl,17929912900560606187890916,1344358522026.74def75a6e7b1f9d7003988f4cb9b4b4.
2012-08-07 10:03:53,512 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_str_as_dbl,17929912900560606187890916,1344358522026.74def75a6e7b1f9d7003988f4cb9b4b4.: disabling compactions & flushes
2012-08-07 10:03:53,512 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_str_as_dbl,17929912900560606187890916,1344358522026.74def75a6e7b1f9d7003988f4cb9b4b4.
2012-08-07 10:03:53,512 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
2012-08-07 10:03:53,512 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_old,19938982902616316000179367339,1343804251322.e11e34ce344c7b5ca516ce2874d88878.
2012-08-07 10:03:53,512 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_old,19938982902616316000179367339,1343804251322.e11e34ce344c7b5ca516ce2874d88878.
2012-08-07 10:03:53,512 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_lzo,19701672896085190206718344,1343890676286.0cc8b4f9bf91f609efa45f90ba799eeb.
2012-08-07 10:03:53,512 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_lzo,19701672896085190206718344,1343890676286.0cc8b4f9bf91f609efa45f90ba799eeb.: disabling compactions & flushes
2012-08-07 10:03:53,512 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_lzo,19701672896085190206718344,1343890676286.0cc8b4f9bf91f609efa45f90ba799eeb.
2012-08-07 10:03:53,513 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
2012-08-07 10:03:53,513 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_lzo,19701672896085190206718344,1343890676286.0cc8b4f9bf91f609efa45f90ba799eeb.
2012-08-07 10:03:53,513 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_lzo,19701672896085190206718344,1343890676286.0cc8b4f9bf91f609efa45f90ba799eeb.
2012-08-07 10:03:53,513 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_snappy,20994522896624231204509343,1343852298510.5e66dd1d3d5f8bf88714d35b3487aaa6.
2012-08-07 10:03:53,513 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_snappy,20994522896624231204509343,1343852298510.5e66dd1d3d5f8bf88714d35b3487aaa6.: disabling compactions & flushes
2012-08-07 10:03:53,513 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_snappy,20994522896624231204509343,1343852298510.5e66dd1d3d5f8bf88714d35b3487aaa6.
2012-08-07 10:03:53,513 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
2012-08-07 10:03:53,513 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_snappy,20994522896624231204509343,1343852298510.5e66dd1d3d5f8bf88714d35b3487aaa6.
2012-08-07 10:03:53,513 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_snappy,20994522896624231204509343,1343852298510.5e66dd1d3d5f8bf88714d35b3487aaa6.
2012-08-07 10:03:53,513 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_snappy,17084522903332660176976987,1343852003271.7819916c078ba9e7fbc6ed349fa3b49e.
2012-08-07 10:03:53,513 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_snappy,17084522903332660176976987,1343852003271.7819916c078ba9e7fbc6ed349fa3b49e.: disabling compactions & flushes
2012-08-07 10:03:53,514 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_snappy,17084522903332660176976987,1343852003271.7819916c078ba9e7fbc6ed349fa3b49e.
2012-08-07 10:03:53,514 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
2012-08-07 10:03:53,514 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_old,17068542902003200000182390457,1343804209781.ac1cc94fb91fed0fba464b7731b0a622.
2012-08-07 10:03:53,514 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_old,17068542902003200000182390457,1343804209781.ac1cc94fb91fed0fba464b7731b0a622.
2012-08-07 10:03:53,514 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_old,19441462902893156000178907118,1343804251322.d148fc9ee74c2e486d57257ff3229c42.
2012-08-07 10:03:53,514 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_old,19441462902893156000178907118,1343804251322.d148fc9ee74c2e486d57257ff3229c42.: disabling compactions & flushes
2012-08-07 10:03:53,514 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_old,19441462902893156000178907118,1343804251322.d148fc9ee74c2e486d57257ff3229c42.
2012-08-07 10:03:53,514 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
2012-08-07 10:03:53,514 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_snappy,17084522903332660176976987,1343852003271.7819916c078ba9e7fbc6ed349fa3b49e.
2012-08-07 10:03:53,514 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_snappy,17084522903332660176976987,1343852003271.7819916c078ba9e7fbc6ed349fa3b49e.
2012-08-07 10:03:53,514 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_old,17924342900647286000187101637,1343810650642.f00debc67cd020cf7f9f5778d1a8760b.
2012-08-07 10:03:53,515 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_old,17924342900647286000187101637,1343810650642.f00debc67cd020cf7f9f5778d1a8760b.: disabling compactions & flushes
2012-08-07 10:03:53,515 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_old,17924342900647286000187101637,1343810650642.f00debc67cd020cf7f9f5778d1a8760b.
2012-08-07 10:03:53,515 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
2012-08-07 10:03:53,515 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_str_as_dbl,17929912900560606187890916,1344358522026.74def75a6e7b1f9d7003988f4cb9b4b4.
2012-08-07 10:03:53,515 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_str_as_dbl,17929912900560606187890916,1344358522026.74def75a6e7b1f9d7003988f4cb9b4b4.
2012-08-07 10:03:53,515 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_old,21324392899982106000190083590,1343804251323.c5fdceeeab7cd5d3dcbdef81054326e4.
2012-08-07 10:03:53,515 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_old,21324392899982106000190083590,1343804251323.c5fdceeeab7cd5d3dcbdef81054326e4.: disabling compactions & flushes
2012-08-07 10:03:53,515 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_old,21324392899982106000190083590,1343804251323.c5fdceeeab7cd5d3dcbdef81054326e4.
2012-08-07 10:03:53,516 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
2012-08-07 10:03:53,516 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_old,17924342900647286000187101637,1343810650642.f00debc67cd020cf7f9f5778d1a8760b.
2012-08-07 10:03:53,516 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_old,17924342900647286000187101637,1343810650642.f00debc67cd020cf7f9f5778d1a8760b.
2012-08-07 10:03:53,516 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_str_as_dbl,18276072903006942178391606,1344358659571.a98079c3cae6ea5c819f06c652a107f4.
2012-08-07 10:03:53,516 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_str_as_dbl,18276072903006942178391606,1344358659571.a98079c3cae6ea5c819f06c652a107f4.: disabling compactions & flushes
2012-08-07 10:03:53,516 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_str_as_dbl,18276072903006942178391606,1344358659571.a98079c3cae6ea5c819f06c652a107f4.
2012-08-07 10:03:53,518 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
2012-08-07 10:03:53,518 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_str_as_dbl,18276072903006942178391606,1344358659571.a98079c3cae6ea5c819f06c652a107f4.
2012-08-07 10:03:53,518 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_str_as_dbl,18276072903006942178391606,1344358659571.a98079c3cae6ea5c819f06c652a107f4.
2012-08-07 10:03:53,518 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of transactions_str_as_dbl,17761522899263569193018364,1344358522026.eb1f6886364054bc96f1f190f24b6eed.
2012-08-07 10:03:53,518 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing transactions_str_as_dbl,17761522899263569193018364,1344358522026.eb1f6886364054bc96f1f190f24b6eed.: disabling compactions & flushes
2012-08-07 10:03:53,518 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: waiting for 1 compactions to complete for region transactions_str_as_dbl,17761522899263569193018364,1344358522026.eb1f6886364054bc96f1f190f24b6eed.
2012-08-07 10:03:53,518 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
2012-08-07 10:03:53,518 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_old,19441462902893156000178907118,1343804251322.d148fc9ee74c2e486d57257ff3229c42.
2012-08-07 10:03:53,518 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_old,19441462902893156000178907118,1343804251322.d148fc9ee74c2e486d57257ff3229c42.
2012-08-07 10:03:53,518 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
2012-08-07 10:03:53,518 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_old,21324392899982106000190083590,1343804251323.c5fdceeeab7cd5d3dcbdef81054326e4.
2012-08-07 10:03:53,518 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_old,21324392899982106000190083590,1343804251323.c5fdceeeab7cd5d3dcbdef81054326e4.
2012-08-07 10:03:54,372 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Waiting on 1 regions to close
2012-08-07 10:03:54,372 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: {eb1f6886364054bc96f1f190f24b6eed=transactions_str_as_dbl,17761522899263569193018364,1344358522026.eb1f6886364054bc96f1f190f24b6eed.}
2012-08-07 10:03:58,714 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction interrupted
java.io.InterruptedIOException: Aborting compaction of store t in region transactions_str_as_dbl,17761522899263569193018364,1344358522026.eb1f6886364054bc96f1f190f24b6eed. because it was interrupted.
	at org.apache.hadoop.hbase.regionserver.Store.compactStore(Store.java:1285)
	at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:767)
	at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1039)
	at org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest.run(CompactionRequest.java:177)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)
2012-08-07 10:03:58,714 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region transactions_str_as_dbl,17761522899263569193018364,1344358522026.eb1f6886364054bc96f1f190f24b6eed.
2012-08-07 10:03:58,716 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed t
2012-08-07 10:03:58,716 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed transactions_str_as_dbl,17761522899263569193018364,1344358522026.eb1f6886364054bc96f1f190f24b6eed.
2012-08-07 10:03:58,716 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed region transactions_str_as_dbl,17761522899263569193018364,1344358522026.eb1f6886364054bc96f1f190f24b6eed.
2012-08-07 10:03:58,716 INFO org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: aborted compaction: regionName=transactions_str_as_dbl,17761522899263569193018364,1344358522026.eb1f6886364054bc96f1f190f24b6eed., storeName=t, fileCount=7, fileSize=1.0g, priority=0, time=398634508588496; duration=2mins, 18sec
2012-08-07 10:03:58,716 DEBUG org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: CompactSplitThread status: compaction_queue=(3:0), split_queue=0
2012-08-07 10:03:59,375 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: stopping server ihub-dn-l4,60020,1344317192178; all regions closed.
2012-08-07 10:03:59,375 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: regionserver60020.logSyncer interrupted while waiting for sync requests
2012-08-07 10:03:59,375 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: regionserver60020.logSyncer exiting
2012-08-07 10:03:59,375 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: closing hlog writer in hdfs://ihubcluster/hbase/.logs/ihub-dn-l4,60020,1344317192178
2012-08-07 10:03:59,750 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: Close and delete failed
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /hbase/.logs/ihub-dn-l4,60020,1344317192178/ihub-dn-l4%2C60020%2C1344317192178.1344358960566 File does not exist. [Lease.  Holder: DFSClient_hb_rs_ihub-dn-l4,60020,1344317192178_1366946297_1, pendingcreates: 1]
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2115)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2106)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:2173)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:2150)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:525)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:330)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:42606)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:427)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:916)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1692)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1688)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1686)

	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
	at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:96)
	at org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.closeWAL(HRegionServer.java:869)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:746)
	at java.lang.Thread.run(Thread.java:662)
2012-08-07 10:03:59,751 INFO org.apache.hadoop.hbase.regionserver.Leases: regionserver60020 closing leases
2012-08-07 10:03:59,751 INFO org.apache.hadoop.hbase.regionserver.Leases: regionserver60020 closed leases
2012-08-07 10:03:59,787 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Failed deleting my ephemeral node
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/rs/ihub-dn-l4,60020,1344317192178
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:868)
	at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:107)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:967)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:956)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:958)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:756)
	at java.lang.Thread.run(Thread.java:662)
2012-08-07 10:03:59,787 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: stopping server ihub-dn-l4,60020,1344317192178; zookeeper connection closed.
2012-08-07 10:03:59,787 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Split Thread to finish...
2012-08-07 10:03:59,788 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large Compaction Thread to finish...
2012-08-07 10:03:59,788 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Small Compaction Thread to finish...
2012-08-07 10:03:59,788 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020 exiting
2012-08-07 10:04:00,095 INFO org.apache.hadoop.hbase.regionserver.Leases: regionserver60020.leaseChecker closing leases
2012-08-07 10:04:00,107 INFO org.apache.hadoop.hbase.regionserver.Leases: regionserver60020.leaseChecker closed leases
2012-08-07 10:04:00,324 INFO org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@5374a6e2
2012-08-07 10:04:00,324 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown hook
2012-08-07 10:04:00,324 INFO org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs shutdown hook thread.
2012-08-07 10:04:00,325 INFO org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook finished.