1. 2012-03-28 00:37:10,246 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished snapshotting, commencing flushing stores
  2. 2012-03-28 00:37:11,172 INFO org.apache.hadoop.hbase.regionserver.Store: Renaming flushed file at hdfs://hadoop1-m1:8020/hbase/gs_us
  3. ers/884e8d17ca610996f0a77b1447040696/.tmp/8120817685620369923 to hdfs://hadoop1-m1:8020/hbase/gs_users/884e8d17ca610996f0a77b1447040
  4. 696/properties/2392116451968132287
  5. 2012-03-28 00:37:11,203 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://hadoop1-m1:8020/hbase/gs_users/884e8d17ca61099
  6. 6f0a77b1447040696/properties/2392116451968132287, entries=56219, sequenceid=1502978979, memsize=14.1m, filesize=943.8k
  7. 2012-03-28 00:37:11,210 WARN org.apache.hadoop.hdfs.DFSClient: Error while syncing
  8. java.io.IOException: All datanodes 10.1.104.10:50010 are bad. Aborting...
  9. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2766)
  10. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2305)
  11. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2477)
  12. 2012-03-28 00:37:11,210 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append. Requesting close of hlog
  13. java.io.IOException: Reflection
  14. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:147)
  15. at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:980)
  16. at org.apache.hadoop.hbase.regionserver.wal.HLog.completeCacheFlush(HLog.java:1158)
  17. at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1007)
  18. at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:898)
  19. at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:850)
  20. at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:394)
  21. at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:202)
  22. at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:222)
  23. Caused by: java.lang.reflect.InvocationTargetException
  24. at sun.reflect.GeneratedMethodAccessor144.invoke(Unknown Source)
  25. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  26. at java.lang.reflect.Method.invoke(Method.java:597)
  27. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:145)
  28. ... 8 more
  29. Caused by: java.io.IOException: All datanodes 10.1.104.10:50010 are bad. Aborting...
  30. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2766)
  31. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2305)
  32. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2477)
  33. 2012-03-28 00:37:11,211 ERROR org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flush failed for region gs_users,952082|zm
  34. 7jO0/hIa7D6L9UuZ8lpQ==,1331103806762.884e8d17ca610996f0a77b1447040696.
  35. java.io.IOException: Reflection
  36. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:147)
  37. at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:980)
  38. at org.apache.hadoop.hbase.regionserver.wal.HLog.completeCacheFlush(HLog.java:1158)
  39. at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1007)
  40. at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:898)
  41. at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:850)
  42. at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:394)
  43. at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:202)
  44. at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:222)
  45. Caused by: java.lang.reflect.InvocationTargetException
  46. at sun.reflect.GeneratedMethodAccessor144.invoke(Unknown Source)
  47. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  48. at java.lang.reflect.Method.invoke(Method.java:597)
  49. at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:145)
  50. ... 8 more
  51. Caused by: java.io.IOException: All datanodes 10.1.104.10:50010 are bad. Aborting...
  52. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2766)
  53. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2305)
  54. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2477)
  55. 2012-03-28 00:37:11,233 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
  56. 2012-03-28 00:37:11,260 INFO org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush thread woke up with memory above low water.
  57. 2012-03-28 00:37:11,261 INFO org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush of region gs_users,5788111|xF/t9gKEK8piWSb7
  58. J7ssOw==,1331189249775.8e4a5b50380fd92602dcb3f94ba7f653. due to global heap pressure
  59. 2012-03-28 00:37:11,261 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memstore flush for gs_users,5788111|xF/t9gKEK8pi
  60. WSb7J7ssOw==,1331189249775.8e4a5b50380fd92602dcb3f94ba7f653., current region memstore size 13.7m
  61. 2012-03-28 00:37:11,261 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished snapshotting, commencing flushing stores
  62. 2012-03-28 00:37:11,285 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server serverName=hadoop1-s08.farm
  63. -ny.gigya.com,60020,1331195007300, load=(requests=3363, regions=1799, usedHeap=5539, maxHeap=7983): IOE in log roller
  64. java.io.IOException: All datanodes 10.1.104.10:50010 are bad. Aborting...
  65. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2766)
  66. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2305)
  67. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2477)
  68. 2012-03-28 00:37:11,285 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: requests=13305, regions=1799, stor
  69. es=1809, storefiles=2762, storefileIndexSize=2797, memstoreSize=2752, compactionQueueSize=619, flushQueueSize=0, usedHeap=5605, maxH
  70. eap=7983, blockCacheSize=24281344, blockCacheFree=2905629952, blockCacheCount=9, blockCacheHitCount=20685, blockCacheMissCount=24310
  71. 8337, blockCacheEvictedCount=0, blockCacheHitRatio=0, blockCacheHitCachingRatio=99
  72. 2012-03-28 00:37:11,285 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: IOE in log roller
  73. 2012-03-28 00:37:11,286 INFO org.apache.hadoop.hbase.regionserver.LogRoller: LogRoller exiting.