2012-03-28 00:37:10,246 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished snapshotting, commencing flushing stores
2012-03-28 00:37:11,172 INFO org.apache.hadoop.hbase.regionserver.Store: Renaming flushed file at hdfs://hadoop1-m1:8020/hbase/gs_us
ers/884e8d17ca610996f0a77b1447040696/.tmp/8120817685620369923 to hdfs://hadoop1-m1:8020/hbase/gs_users/884e8d17ca610996f0a77b1447040
696/properties/2392116451968132287
2012-03-28 00:37:11,203 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://hadoop1-m1:8020/hbase/gs_users/884e8d17ca61099
6f0a77b1447040696/properties/2392116451968132287, entries=56219, sequenceid=1502978979, memsize=14.1m, filesize=943.8k
2012-03-28 00:37:11,210 WARN org.apache.hadoop.hdfs.DFSClient: Error while syncing
java.io.IOException: All datanodes 10.1.104.10:50010 are bad. Aborting...
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2766)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2305)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2477)
2012-03-28 00:37:11,210 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append. Requesting close of hlog
java.io.IOException: Reflection
at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:147)
at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:980)
at org.apache.hadoop.hbase.regionserver.wal.HLog.completeCacheFlush(HLog.java:1158)
at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1007)
at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:898)
at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:850)
at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:394)
at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:202)
at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:222)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor144.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:145)
... 8 more
Caused by: java.io.IOException: All datanodes 10.1.104.10:50010 are bad. Aborting...
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2766)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2305)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2477)
2012-03-28 00:37:11,211 ERROR org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flush failed for region gs_users,952082|zm
7jO0/hIa7D6L9UuZ8lpQ==,1331103806762.884e8d17ca610996f0a77b1447040696.
java.io.IOException: Reflection
at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:147)
at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:980)
at org.apache.hadoop.hbase.regionserver.wal.HLog.completeCacheFlush(HLog.java:1158)
at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1007)
at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:898)
at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:850)
at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:394)
at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:202)
at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:222)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor144.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:145)
... 8 more
Caused by: java.io.IOException: All datanodes 10.1.104.10:50010 are bad. Aborting...
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2766)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2305)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2477)
2012-03-28 00:37:11,233 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
2012-03-28 00:37:11,260 INFO org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush thread woke up with memory above low water.
2012-03-28 00:37:11,261 INFO org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush of region gs_users,5788111|xF/t9gKEK8piWSb7
J7ssOw==,1331189249775.8e4a5b50380fd92602dcb3f94ba7f653. due to global heap pressure
2012-03-28 00:37:11,261 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memstore flush for gs_users,5788111|xF/t9gKEK8pi
WSb7J7ssOw==,1331189249775.8e4a5b50380fd92602dcb3f94ba7f653., current region memstore size 13.7m
2012-03-28 00:37:11,261 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished snapshotting, commencing flushing stores
2012-03-28 00:37:11,285 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server serverName=hadoop1-s08.farm
-ny.gigya.com,60020,1331195007300, load=(requests=3363, regions=1799, usedHeap=5539, maxHeap=7983): IOE in log roller
java.io.IOException: All datanodes 10.1.104.10:50010 are bad. Aborting...
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2766)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2305)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2477)
2012-03-28 00:37:11,285 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: requests=13305, regions=1799, stor
es=1809, storefiles=2762, storefileIndexSize=2797, memstoreSize=2752, compactionQueueSize=619, flushQueueSize=0, usedHeap=5605, maxH
eap=7983, blockCacheSize=24281344, blockCacheFree=2905629952, blockCacheCount=9, blockCacheHitCount=20685, blockCacheMissCount=24310
8337, blockCacheEvictedCount=0, blockCacheHitRatio=0, blockCacheHitCachingRatio=99
2012-03-28 00:37:11,285 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: IOE in log roller
2012-03-28 00:37:11,286 INFO org.apache.hadoop.hbase.regionserver.LogRoller: LogRoller exiting.