This week only. Pastebin PRO Accounts Christmas Special! Don't miss out!Want more features on Pastebin? Sign Up, it's FREE!
Guest

failed_region_server_log_extract

By: tha-dude on Feb 9th, 2013  |  syntax: None  |  size: 5.35 KB  |  views: 10  |  expires: Never
download  |  raw  |  embed  |  report abuse  |  print
Text below is selected. Please press Ctrl+C to copy to your clipboard. (⌘+C on Mac)
  1. 2013-02-09 17:22:16,884 INFO org.apache.hadoop.hdfs.DFSClient: Successfully connected to /x.xx.xx.xxx:50010 for block 3338988663738511252
  2. 2013-02-09 17:22:16,889 ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open of region=XXXXXXX,'\xD7\x02\xD8\xEE\xDFB\xEA\xAEg\x9F\x83\xB6\x0E\x8B\x92,1358932887309.080cb02f15d15b52c6395a4e3e9227f4.
  3. java.io.FileNotFoundException: File does not exist: /hbase/XXXXXXX/dbbc5aacf7ac6536b12ac50106a05797/t/365203cec6ea4ae3984bb6198e162f18
  4.  
  5. # <similar messages repeated frequently before eventually>
  6.  
  7. 2013-02-09 17:22:16,968 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: XXXXXXX,'\xD7\x02\xD8\xEE\xDFB\xEA\xAEg\x9F\x83\xB6\x0E\x8B\x92,1358932887309.080cb02f15d15b52c6395a4e3e9227f4.
  8. 2013-02-09 17:22:16,983 INFO org.apache.hadoop.hbase.regionserver.HRegion: Setting up tabledescriptor config now ...
  9. 2013-02-09 17:22:16,989 WARN org.apache.hadoop.hdfs.DFSClient: Failed to connect to /x.xx.xx.xxx:50010 for block, add to deadNodes and continue. java.net.SocketException: Too many open files
  10. java.net.SocketException: Too many open files
  11.  
  12. # <similar messages repeated frequently before eventually>
  13.  
  14. 2013-02-09 17:23:04,492 WARN org.mortbay.log: EXCEPTION
  15. java.io.IOException: Too many open files
  16.     at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
  17.     at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
  18.     at org.mortbay.jetty.nio.SelectChannelConnector$1.acceptChannel(SelectChannelConnector.java:75)
  19.     at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:686)
  20.     at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
  21.     at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
  22.     at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
  23.     at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
  24.  
  25. # <similar messages repeated frequently before eventually>
  26. # <"Old" and "New" below are the same IP - the HDFS Namenode>
  27.  
  28. 2013-02-09 17:23:19,256 WARN org.apache.hadoop.ipc.Client: Address change detected. Old: xxx-x.xxxxx.net/xxx.xx.xxx.net:8020 New: xxx-x.xxxxx.net:8020
  29. 2013-02-09 17:23:29,259 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: xxx-x.xxxxx.net:8020. Already tried 9 time(s).
  30. 2013-02-09 17:23:29,259 WARN org.apache.hadoop.hdfs.DFSClient: Failed to renew lease for DFSClient_NONMAPREDUCE_-1025027312_1 for 61 seconds (>= soft-limit =60 seconds.) Closing all files being written ...
  31. java.io.IOException: Failed on local exception: java.net.SocketException: Too many open files; Host Details : local host is: "java.net.UnknownHostException: xxx-x.xxxxx.net: xxx-x.xxxxx.net"; destination host is: "xxx-x.xxxxx.net":8020;
  32. ...
  33. 2013-02-09 17:23:30,123 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not sync. Requesting close of hlog
  34. java.io.IOException: Reflection
  35.  
  36. # ...
  37. # ...
  38.  
  39. Caused by: java.io.IOException: Lease timeout of 0 seconds expired.
  40.  
  41. # ...
  42. # ...
  43.  
  44. 2013-02-09 17:23:30,142 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server xxx-x.xxxxx.net,60020,1360401373855: IOE in log roller
  45. java.io.IOException: cannot get log writer
  46. ...
  47. 2013-02-09 17:23:30,143 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: []
  48. 2013-02-09 17:23:30,153 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: requestsPerSecond=0, numberOfOnlineRegions=298, numberOfStores=595, numberOfStorefiles=524, storefileIndexSizeMB=1, rootIndexSizeKB=1308, totalStaticIndexSizeKB=172853, totalStaticBloomSizeKB=0, memstoreSizeMB=0, mbInMemoryWithoutWAL=0, numberOfPutsWithoutWAL=0, readRequestsCount=0, writeRequestsCount=0, compactionQueueSize=0, flushQueueSize=0, usedHeapMB=93, maxHeapMB=910, blockCacheSizeMB=1.87, blockCacheFreeMB=225.69, blockCacheCount=0, blockCacheHitCount=81647, blockCacheMissCount=1115306, blockCacheEvictedCount=470, blockCacheHitRatio=6%, blockCacheHitCachingRatio=99%, hdfsBlocksLocalityIndex=56, deleteRequestLatencyMean=0, deleteRequestLatencyCount=0, deleteRequestLatencyMedian=0, deleteRequestLatency75th=0, deleteRequestLatency95th=0, deleteRequestLatency99th=0, deleteRequestLatency999th=0, getRequestLatencyMean=0, getRequestLatencyCount=0, getRequestLatencyMedian=0, getRequestLatency75th=0, getRequestLatency95th=0, getRequestLatency99th=0, getRequestLatency999th=0, putRequestLatencyMean=0, putRequestLatencyCount=0, putRequestLatencyMedian=0, putRequestLatency75th=0, putRequestLatency95th=0, putRequestLatency99th=0, putRequestLatency999th=0, fsReadLatencyHistogramMean=3477759.2, fsReadLatencyHistogramCount=334666, fsReadLatencyHistogramMedian=783510.5, fsReadLatencyHistogram75th=1391941.25, fsReadLatencyHistogram95th=2903448.7, fsReadLatencyHistogram99th=25171631.81, fsReadLatencyHistogram999th=362953061.22, fsWriteLatencyHistogramMean=878568.13, fsWriteLatencyHistogramCount=334657, fsWriteLatencyHistogramMedian=146157, fsWriteLatencyHistogram75th=214405.75, fsWriteLatencyHistogram95th=2004682.6, fsWriteLatencyHistogram99th=3239171.75, fsWriteLatencyHistogram999th=4738101.09
  49. 2013-02-09 17:23:30,155 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: IOE in log roller
  50. 2013-02-09 17:23:30,155 INFO org.apache.hadoop.hbase.regionserver.LogRoller: LogRoller exiting.
clone this paste RAW Paste Data