Pastebin launched a little side project called VERYVIRAL.com, check it out ;-) Want more features on Pastebin? Sign Up, it's FREE!
Guest

Untitled

By: a guest on Sep 12th, 2012  |  syntax: None  |  size: 35.76 KB  |  views: 21  |  expires: Never
download  |  raw  |  embed  |  report abuse  |  print
Text below is selected. Please press Ctrl+C to copy to your clipboard. (⌘+C on Mac)
  1. 2012-09-12 10:17:06,989 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Received dynamic protocol exec call with protocolName com.movenetworks.a3.realtime.hbase.coprocessor.CubeCoprocessorProtocol
  2. 2012-09-12 10:17:06,989 DEBUG com.movenetworks.a3.realtime.hbase.coprocessor.CubeCoprocessorImplementation: Calling updateCube [table: 'compound5'; key: 'cdn-dev\x09Level 3\x09arm 7 -1\x09\x80\x00\x019\xBBD\xEC`']
  3. 2012-09-12 10:17:06,989 DEBUG com.movenetworks.a3.realtime.hbase.coprocessor.CubeCoprocessorImplementation: Starting update table [table: 'compound5'; key: 'cdn-dev\x09Level 3\x09arm 7 -1\x09\x80\x00\x019\xBBD\xEC`'; data: [v=0ms, b=0, bc=0, l=0, s=0, d=0, u=0, c=0, hOk=1037, hErr=0, uniq#null]]
  4. 2012-09-12 10:17:06,990 DEBUG com.movenetworks.a3.realtime.hbase.coprocessor.CubeCoprocessorImplementation: Ending update table [table: 'compound5'; key: 'cdn-dev\x09Level 3\x09arm 7 -1\x09\x80\x00\x019\xBBD\xEC`']
  5. 2012-09-12 10:17:06,990 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Received dynamic protocol exec call with protocolName com.movenetworks.a3.realtime.hbase.coprocessor.CubeCoprocessorProtocol
  6. 2012-09-12 10:17:06,990 DEBUG com.movenetworks.a3.realtime.hbase.coprocessor.CubeCoprocessorImplementation: Calling updateCube [table: 'compound0'; key: 'isp-pub-chn\x09charter communications\x09CBD46B77\x09GEONWS\x09\x80\x00\x019\xBBD\xEC`']
  7. 2012-09-12 10:17:06,990 DEBUG com.movenetworks.a3.realtime.hbase.coprocessor.CubeCoprocessorImplementation: Starting update table [table: 'compound0'; key: 'isp-pub-chn\x09charter communications\x09CBD46B77\x09GEONWS\x09\x80\x00\x019\xBBD\xEC`'; data: [v=598000ms, b=1985, bc=1, l=0, s=304, d=156037529, u=275892, c=1, hOk=0, hErr=0, uniq#692]]
  8. 2012-09-12 10:17:06,990 DEBUG com.movenetworks.a3.realtime.hbase.coprocessor.CubeCoprocessorImplementation: Ending update table [table: 'compound0'; key: 'isp-pub-chn\x09charter communications\x09CBD46B77\x09GEONWS\x09\x80\x00\x019\xBBD\xEC`']
  9. 2012-09-12 10:17:06,990 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Received dynamic protocol exec call with protocolName com.movenetworks.a3.realtime.hbase.coprocessor.CubeCoprocessorProtocol
  10. 2012-09-12 10:17:06,990 DEBUG com.movenetworks.a3.realtime.hbase.coprocessor.CubeCoprocessorImplementation: Calling updateCube [table: 'compound0'; key: 'pub-con\x09CBD46B77\x09united states\x09\x80\x00\x019\xBBD\xEC`']
  11. 2012-09-12 10:17:06,990 DEBUG com.movenetworks.a3.realtime.hbase.coprocessor.CubeCoprocessorImplementation: Starting update table [table: 'compound0'; key: 'pub-con\x09CBD46B77\x09united states\x09\x80\x00\x019\xBBD\xEC`'; data: [v=598000ms, b=1985, bc=1, l=0, s=304, d=156037529, u=275892, c=1, hOk=0, hErr=0, uniq#692]]
  12. 2012-09-12 10:17:06,991 DEBUG com.movenetworks.a3.realtime.hbase.coprocessor.CubeCoprocessorImplementation: Ending update table [table: 'compound0'; key: 'pub-con\x09CBD46B77\x09united states\x09\x80\x00\x019\xBBD\xEC`']
  13. 2012-09-12 10:17:06,991 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Received dynamic protocol exec call with protocolName com.movenetworks.a3.realtime.hbase.coprocessor.CubeCoprocessorProtocol
  14. 2012-09-12 10:17:06,991 DEBUG com.movenetworks.a3.realtime.hbase.coprocessor.CubeCoprocessorImplementation: Calling updateCube [table: 'compound0'; key: 'pub-con-reg\x09CBD46B77\x09united states\x09missouri\x09\x80\x00\x019\xBBD\xEC`']
  15. 2012-09-12 10:17:06,991 DEBUG com.movenetworks.a3.realtime.hbase.coprocessor.CubeCoprocessorImplementation: Starting update table [table: 'compound0'; key: 'pub-con-reg\x09CBD46B77\x09united states\x09missouri\x09\x80\x00\x019\xBBD\xEC`'; data: [v=598000ms, b=1985, bc=1, l=0, s=304, d=156037529, u=275892, c=1, hOk=0, hErr=0, uniq#692]]
  16. 2012-09-12 10:17:06,991 DEBUG com.movenetworks.a3.realtime.hbase.coprocessor.CubeCoprocessorImplementation: Ending update table [table: 'compound0'; key: 'pub-con-reg\x09CBD46B77\x09united states\x09missouri\x09\x80\x00\x019\xBBD\xEC`']
  17. 2012-09-12 10:17:06,991 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Received dynamic protocol exec call with protocolName com.movenetworks.a3.realtime.hbase.coprocessor.CubeCoprocessorProtocol
  18. 2012-09-12 10:17:06,991 DEBUG com.movenetworks.a3.realtime.hbase.coprocessor.CubeCoprocessorImplementation: Calling updateCube [table: 'facts'; key: 'Level 3\x09att services inc.\x09CBD46B77\x09\x80\x00\x00\x00\x00\x00\x009RGTI\x09\x80\x00\x00\x00\x00\x0B\xF7\xA1Globo Esporte\x09united states\x09texas\x09richardson\x09arm 7 -1\x092.5.0.154\x09\x80\x00\x019\xBBD\xEC`']
  19. 2012-09-12 10:17:06,991 DEBUG com.movenetworks.a3.realtime.hbase.coprocessor.CubeCoprocessorImplementation: Starting update table [table: 'facts'; key: 'Level 3\x09att services inc.\x09CBD46B77\x09\x80\x00\x00\x00\x00\x00\x009RGTI\x09\x80\x00\x00\x00\x00\x0B\xF7\xA1Globo Esporte\x09united states\x09texas\x09richardson\x09arm 7 -1\x092.5.0.154\x09\x80\x00\x019\xBBD\xEC`'; data: [v=596000ms, b=1998, bc=1, l=0, s=302, d=156205157, u=275231, c=1, hOk=0, hErr=0, uniq#692]]
  20. 2012-09-12 10:17:06,992 DEBUG com.movenetworks.a3.realtime.hbase.coprocessor.CubeCoprocessorImplementation: Ending update table [table: 'facts'; key: 'Level 3\x09att services inc.\x09CBD46B77\x09\x80\x00\x00\x00\x00\x00\x009RGTI\x09\x80\x00\x00\x00\x00\x0B\xF7\xA1Globo Esporte\x09united states\x09texas\x09richardson\x09arm 7 -1\x092.5.0.154\x09\x80\x00\x019\xBBD\xEC`']
  21. 2012-09-12 10:17:06,992 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Received dynamic protocol exec call with protocolName com.movenetworks.a3.realtime.hbase.coprocessor.CubeCoprocessorProtocol
  22. 2012-09-12 10:17:06,992 DEBUG com.movenetworks.a3.realtime.hbase.coprocessor.CubeCoprocessorImplementation: Calling updateCube [table: 'compound1'; key: 'cdn-dev-isp\x09Level 3\x09arm 7 -1\x09att services inc.\x09\x80\x00\x019\xBBD\xEC`']
  23. 2012-09-12 10:17:06,992 DEBUG com.movenetworks.a3.realtime.hbase.coprocessor.CubeCoprocessorImplementation: Starting update table [table: 'compound1'; key: 'cdn-dev-isp\x09Level 3\x09arm 7 -1\x09att services inc.\x09\x80\x00\x019\xBBD\xEC`'; data: [v=0ms, b=0, bc=0, l=0, s=0, d=0, u=0, c=0, hOk=116, hErr=0, uniq#null]]
  24. 2012-09-12 10:17:06,992 DEBUG com.movenetworks.a3.realtime.hbase.coprocessor.CubeCoprocessorImplementation: Ending update table [table: 'compound1'; key: 'cdn-dev-isp\x09Level 3\x09arm 7 -1\x09att services inc.\x09\x80\x00\x019\xBBD\xEC`']
  25. 2012-09-12 10:17:06,992 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Received dynamic protocol exec call with protocolName com.movenetworks.a3.realtime.hbase.coprocessor.CubeCoprocessorProtocol
  26. 2012-09-12 10:17:06,992 DEBUG com.movenetworks.a3.realtime.hbase.coprocessor.CubeCoprocessorImplementation: Calling updateCube [table: 'compound1'; key: 'cdn-pub-aid\x09Level 3\x09CBD46B77\x09\x80\x00\x00\x00\x00\x0B\xF7\xA1\x80\x00\x019\xBBD\xEC`']
  27. 2012-09-12 10:17:06,992 DEBUG com.movenetworks.a3.realtime.hbase.coprocessor.CubeCoprocessorImplementation: Starting update table [table: 'compound1'; key: 'cdn-pub-aid\x09Level 3\x09CBD46B77\x09\x80\x00\x00\x00\x00\x0B\xF7\xA1\x80\x00\x019\xBBD\xEC`'; data: [v=0ms, b=0, bc=0, l=0, s=0, d=0, u=0, c=0, hOk=116, hErr=0, uniq#null]]
  28. 2012-09-12 10:17:06,993 DEBUG com.movenetworks.a3.realtime.hbase.coprocessor.CubeCoprocessorImplementation: Ending update table [table: 'compound1'; key: 'cdn-pub-aid\x09Level 3\x09CBD46B77\x09\x80\x00\x00\x00\x00\x0B\xF7\xA1\x80\x00\x019\xBBD\xEC`']
  29. 2012-09-12 10:17:06,993 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Received dynamic protocol exec call with protocolName com.movenetworks.a3.realtime.hbase.coprocessor.CubeCoprocessorProtocol
  30. 2012-09-12 10:17:06,993 DEBUG com.movenetworks.a3.realtime.hbase.coprocessor.CubeCoprocessorImplementation: Starting cubeRegion [table:'compound0'; range:1347465660000-1347466560000; startRow='pub-chn\x09\x10'; stopRow='pub-chn\x09^?\x00]
  31. 2012-09-12 10:17:25,291 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.6 GB, free=356.85 MB, max=1.95 GB, blocks=23634, accesses=224114313, hits=217918327, hitRatio=97.23%, cachingAccesses=217970409, cachingHits=217758841, cachingHitsRatio=99.90%, evictions=2, evicted=187934, evictedPerRun=93967.0
  32. 2012-09-12 10:18:26,799 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Block cache LRU eviction started; Attempting to free 199.62 MB of total=1.66 GB
  33. 2012-09-12 10:18:26,821 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Block cache LRU eviction completed; freed=199.65 MB, total=1.46 GB, single=327 MB, multi=1.32 GB, memory=161.44 KB
  34. 2012-09-12 10:18:53,481 DEBUG org.apache.hadoop.hbase.util.CompoundBloomFilterWriter: Compacted Bloom chunk #46 from [109306 max keys, 131072 bytes] to [54653 max keys, 65536 bytes]
  35. 2012-09-12 10:18:53,487 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom added to HFile (hdfs://hdpmgr001:8020/hbase/compound3/3f09cec26b1ca74440322a3a78fcf465/.tmp/7eb44ec218b44261a4fb773c29b9419a): org.apache.hadoop.hbase.util.CompoundBloomFilterWriter@40d50ac3
  36. 2012-09-12 10:18:53,497 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for 7eb44ec218b44261a4fb773c29b9419a
  37. 2012-09-12 10:18:53,498 INFO org.apache.hadoop.hbase.regionserver.Store: Renaming compacted file at hdfs://hdpmgr001:8020/hbase/compound3/3f09cec26b1ca74440322a3a78fcf465/.tmp/7eb44ec218b44261a4fb773c29b9419a to hdfs://hdpmgr001:8020/hbase/compound3/3f09cec26b1ca74440322a3a78fcf465/d/7eb44ec218b44261a4fb773c29b9419a
  38. 2012-09-12 10:18:53,508 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for 7eb44ec218b44261a4fb773c29b9419a
  39. 2012-09-12 10:18:53,569 INFO org.apache.hadoop.hbase.regionserver.Store: Completed major compaction of 4 file(s) in d of compound3,isp-con-reg\x09comcast cable communications inc.\x09united states\x09tennessee\x09\x80\x00\x018\xD9 \xEC\xD0,1345737950174.3f09cec26b1ca74440322a3a78fcf465. into 7eb44ec218b44261a4fb773c29b9419a, size=884.9m; total size for store is 884.9m
  40. 2012-09-12 10:18:53,569 INFO org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: completed compaction: regionName=compound3,isp-con-reg\x09comcast cable communications inc.\x09united states\x09tennessee\x09\x80\x00\x018\xD9 \xEC\xD0,1345737950174.3f09cec26b1ca74440322a3a78fcf465., storeName=d, fileCount=4, fileSize=885.4m, priority=3, time=2222727379134555; duration=2mins, 59sec
  41. 2012-09-12 10:18:53,569 DEBUG org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: CompactSplitThread status: compaction_queue=(0:0), split_queue=0
  42. 2012-09-12 10:22:25,291 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.58 GB, free=381.33 MB, max=1.95 GB, blocks=23575, accesses=256513273, hits=250253280, hitRatio=97.55%, cachingAccesses=250308716, cachingHits=250093306, cachingHitsRatio=99.91%, evictions=3, evicted=191835, evictedPerRun=63945.0
  43. 2012-09-12 10:24:16,126 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Block cache LRU eviction started; Attempting to free 199.6 MB of total=1.66 GB
  44. 2012-09-12 10:24:16,145 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Block cache LRU eviction completed; freed=199.68 MB, total=1.46 GB, single=321.75 MB, multi=1.33 GB, memory=161.44 KB
  45. 2012-09-12 10:27:03,285 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Block cache LRU eviction started; Attempting to free 199.6 MB of total=1.66 GB
  46. 2012-09-12 10:27:03,303 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Block cache LRU eviction completed; freed=199.62 MB, total=1.46 GB, single=321.62 MB, multi=1.33 GB, memory=161.44 KB
  47. 2012-09-12 10:27:25,291 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.49 GB, free=466.07 MB, max=1.95 GB, blocks=23027, accesses=275677044, hits=269412075, hitRatio=97.72%, cachingAccesses=269472488, cachingHits=269252102, cachingHitsRatio=99.91%, evictions=5, evicted=197359, evictedPerRun=39471.80078125
  48. 2012-09-12 10:29:50,010 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Block cache LRU eviction started; Attempting to free 199.59 MB of total=1.66 GB
  49. 2012-09-12 10:29:50,028 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Block cache LRU eviction completed; freed=199.62 MB, total=1.46 GB, single=321.56 MB, multi=1.33 GB, memory=161.44 KB
  50. 2012-09-12 10:32:25,291 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.59 GB, free=363.07 MB, max=1.95 GB, blocks=25562, accesses=299900027, hits=293630271, hitRatio=97.90%, cachingAccesses=293695477, cachingHits=293470304, cachingHitsRatio=99.92%, evictions=6, evicted=199611, evictedPerRun=33268.5
  51. 2012-09-12 10:33:36,696 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Block cache LRU eviction started; Attempting to free 199.64 MB of total=1.66 GB
  52. 2012-09-12 10:33:36,715 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Block cache LRU eviction completed; freed=199.7 MB, total=1.46 GB, single=321.5 MB, multi=1.33 GB, memory=161.44 KB
  53. 2012-09-12 10:35:49,213 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of d because one (major) compacted file only and oldestTime 4187029213ms is < ttl=8640000000
  54. 2012-09-12 10:35:49,213 DEBUG org.apache.hadoop.hbase.regionserver.Store: Major compaction triggered on store d; time since last major compaction 89999982ms
  55. 2012-09-12 10:35:49,213 DEBUG org.apache.hadoop.hbase.regionserver.Store: Major compaction triggered on store d; time since last major compaction 89999982ms
  56. 2012-09-12 10:35:49,214 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Large Compaction requested: regionName=compound5,cdn-con-reg-cty\x09Level 3\x09united states\x09california\x09pleasanton\x09\x80\x00\x019\x09\xCEe\x88,1345820270955.dd009cd3fe6a247d754e9dd1e56a983c., storeName=d, fileCount=3, fileSize=1018.4m (993.0m, 23.0m, 2.4m), priority=4, time=2223922909926363; Because: regionserver60020.compactionChecker requests major compaction; use default priority; compaction_queue=(0:0), split_queue=0
  57. 2012-09-12 10:35:49,214 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on d in region compound5,cdn-con-reg-cty\x09Level 3\x09united states\x09california\x09pleasanton\x09\x80\x00\x019\x09\xCEe\x88,1345820270955.dd009cd3fe6a247d754e9dd1e56a983c.
  58. 2012-09-12 10:35:49,214 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of d because one (major) compacted file only and oldestTime 2730829214ms is < ttl=8640000000
  59. 2012-09-12 10:35:49,214 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of d because one (major) compacted file only and oldestTime 3738064214ms is < ttl=8640000000
  60. 2012-09-12 10:35:49,214 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of d because one (major) compacted file only and oldestTime 4779229214ms is < ttl=8640000000
  61. 2012-09-12 10:35:49,214 DEBUG org.apache.hadoop.hbase.regionserver.Store: Major compaction triggered on store d; time since last major compaction 109871383ms
  62. 2012-09-12 10:35:49,214 DEBUG org.apache.hadoop.hbase.regionserver.Store: Major compaction triggered on store d; time since last major compaction 109871383ms
  63. 2012-09-12 10:35:49,214 INFO org.apache.hadoop.hbase.regionserver.Store: Starting compaction of 3 file(s) in d of compound5,cdn-con-reg-cty\x09Level 3\x09united states\x09california\x09pleasanton\x09\x80\x00\x019\x09\xCEe\x88,1345820270955.dd009cd3fe6a247d754e9dd1e56a983c. into tmpdir=hdfs://hdpmgr001:8020/hbase/compound5/dd009cd3fe6a247d754e9dd1e56a983c/.tmp, seqid=2395951678, totalSize=1018.4m
  64. 2012-09-12 10:35:49,214 DEBUG org.apache.hadoop.hbase.regionserver.Store: Compacting hdfs://hdpmgr001:8020/hbase/compound5/dd009cd3fe6a247d754e9dd1e56a983c/d/3175e7053eef4f9788e22dc4e511d15d, keycount=5725553, bloomtype=ROW, size=993.0m
  65. 2012-09-12 10:35:49,214 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Large Compaction requested: regionName=compound0,isp-pub-chn\x09comcast cable communications inc.\x09CBD46B77\x09MBC1\x09\x80\x00\x019[\x0D\x1E0,1346260951439.8b2e6079105c337a485fbafd65492407., storeName=d, fileCount=3, fileSize=724.2m (704.4m, 18.2m, 1.6m), priority=4, time=2223922910469018; Because: regionserver60020.compactionChecker requests major compaction; use default priority; compaction_queue=(1:0), split_queue=0
  66. 2012-09-12 10:35:49,214 DEBUG org.apache.hadoop.hbase.regionserver.Store: Compacting hdfs://hdpmgr001:8020/hbase/compound5/dd009cd3fe6a247d754e9dd1e56a983c/d/5d93db6119444576be2c93218edb1ba7, keycount=132086, bloomtype=ROW, size=23.0m
  67. 2012-09-12 10:35:49,214 DEBUG org.apache.hadoop.hbase.regionserver.Store: Compacting hdfs://hdpmgr001:8020/hbase/compound5/dd009cd3fe6a247d754e9dd1e56a983c/d/1233ba4e86c845efb4c8b98682e2121a, keycount=13514, bloomtype=ROW, size=2.4m
  68. 2012-09-12 10:35:49,270 DEBUG org.apache.hadoop.hbase.io.hfile.HFileWriterV2: Initialized with CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
  69. 2012-09-12 10:35:49,270 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom filter type for hdfs://hdpmgr001:8020/hbase/compound5/dd009cd3fe6a247d754e9dd1e56a983c/.tmp/1215d92794e349a1a0d262f631e4ecbd: ROW, CompoundBloomFilterWriter
  70. 2012-09-12 10:37:25,291 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.6 GB, free=353.93 MB, max=1.95 GB, blocks=25707, accesses=316574786, hits=310233534, hitRatio=97.99%, cachingAccesses=310302031, cachingHits=310073555, cachingHitsRatio=99.92%, evictions=7, evicted=202769, evictedPerRun=28967.0
  71. 2012-09-12 10:39:04,473 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom added to HFile (hdfs://hdpmgr001:8020/hbase/compound5/dd009cd3fe6a247d754e9dd1e56a983c/.tmp/1215d92794e349a1a0d262f631e4ecbd): org.apache.hadoop.hbase.util.CompoundBloomFilterWriter@6fa310cf
  72. 2012-09-12 10:39:04,483 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for 1215d92794e349a1a0d262f631e4ecbd
  73. 2012-09-12 10:39:04,483 INFO org.apache.hadoop.hbase.regionserver.Store: Renaming compacted file at hdfs://hdpmgr001:8020/hbase/compound5/dd009cd3fe6a247d754e9dd1e56a983c/.tmp/1215d92794e349a1a0d262f631e4ecbd to hdfs://hdpmgr001:8020/hbase/compound5/dd009cd3fe6a247d754e9dd1e56a983c/d/1215d92794e349a1a0d262f631e4ecbd
  74. 2012-09-12 10:39:04,492 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for 1215d92794e349a1a0d262f631e4ecbd
  75. 2012-09-12 10:39:04,523 INFO org.apache.hadoop.hbase.regionserver.Store: Completed major compaction of 3 file(s) in d of compound5,cdn-con-reg-cty\x09Level 3\x09united states\x09california\x09pleasanton\x09\x80\x00\x019\x09\xCEe\x88,1345820270955.dd009cd3fe6a247d754e9dd1e56a983c. into 1215d92794e349a1a0d262f631e4ecbd, size=1018.3m; total size for store is 1018.3m
  76. 2012-09-12 10:39:04,523 INFO org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: completed compaction: regionName=compound5,cdn-con-reg-cty\x09Level 3\x09united states\x09california\x09pleasanton\x09\x80\x00\x019\x09\xCEe\x88,1345820270955.dd009cd3fe6a247d754e9dd1e56a983c., storeName=d, fileCount=3, fileSize=1018.4m, priority=4, time=2223922909926363; duration=3mins, 15sec
  77. 2012-09-12 10:39:04,523 DEBUG org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: CompactSplitThread status: compaction_queue=(1:0), split_queue=0
  78. 2012-09-12 10:39:04,523 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on d in region compound0,isp-pub-chn\x09comcast cable communications inc.\x09CBD46B77\x09MBC1\x09\x80\x00\x019[\x0D\x1E0,1346260951439.8b2e6079105c337a485fbafd65492407.
  79. 2012-09-12 10:39:04,523 INFO org.apache.hadoop.hbase.regionserver.Store: Starting compaction of 3 file(s) in d of compound0,isp-pub-chn\x09comcast cable communications inc.\x09CBD46B77\x09MBC1\x09\x80\x00\x019[\x0D\x1E0,1346260951439.8b2e6079105c337a485fbafd65492407. into tmpdir=hdfs://hdpmgr001:8020/hbase/compound0/8b2e6079105c337a485fbafd65492407/.tmp, seqid=2395093497, totalSize=724.2m
  80. 2012-09-12 10:39:04,523 DEBUG org.apache.hadoop.hbase.regionserver.Store: Compacting hdfs://hdpmgr001:8020/hbase/compound0/8b2e6079105c337a485fbafd65492407/d/803c94ccd5354aa3ad609c67c6094756, keycount=4527985, bloomtype=ROW, size=704.4m
  81. 2012-09-12 10:39:04,523 DEBUG org.apache.hadoop.hbase.regionserver.Store: Compacting hdfs://hdpmgr001:8020/hbase/compound0/8b2e6079105c337a485fbafd65492407/d/6e0460b6d27144fe920682db51fed1d8, keycount=116718, bloomtype=ROW, size=18.2m
  82. 2012-09-12 10:39:04,523 DEBUG org.apache.hadoop.hbase.regionserver.Store: Compacting hdfs://hdpmgr001:8020/hbase/compound0/8b2e6079105c337a485fbafd65492407/d/0c3d02551e88439fb27e7100d47c893f, keycount=10440, bloomtype=ROW, size=1.6m
  83. 2012-09-12 10:39:04,580 DEBUG org.apache.hadoop.hbase.io.hfile.HFileWriterV2: Initialized with CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
  84. 2012-09-12 10:39:04,580 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom filter type for hdfs://hdpmgr001:8020/hbase/compound0/8b2e6079105c337a485fbafd65492407/.tmp/08dcb72196ae4a37ba5c5cbdd2cc68a1: ROW, CompoundBloomFilterWriter
  85. 2012-09-12 10:39:07,588 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Block cache LRU eviction started; Attempting to free 199.6 MB of total=1.66 GB
  86. 2012-09-12 10:39:07,609 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Block cache LRU eviction completed; freed=199.62 MB, total=1.46 GB, single=340.12 MB, multi=1.31 GB, memory=161.44 KB
  87. 2012-09-12 10:41:26,742 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom added to HFile (hdfs://hdpmgr001:8020/hbase/compound0/8b2e6079105c337a485fbafd65492407/.tmp/08dcb72196ae4a37ba5c5cbdd2cc68a1): org.apache.hadoop.hbase.util.CompoundBloomFilterWriter@6782aeb0
  88. 2012-09-12 10:41:26,749 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for 08dcb72196ae4a37ba5c5cbdd2cc68a1
  89. 2012-09-12 10:41:26,749 INFO org.apache.hadoop.hbase.regionserver.Store: Renaming compacted file at hdfs://hdpmgr001:8020/hbase/compound0/8b2e6079105c337a485fbafd65492407/.tmp/08dcb72196ae4a37ba5c5cbdd2cc68a1 to hdfs://hdpmgr001:8020/hbase/compound0/8b2e6079105c337a485fbafd65492407/d/08dcb72196ae4a37ba5c5cbdd2cc68a1
  90. 2012-09-12 10:41:26,760 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for 08dcb72196ae4a37ba5c5cbdd2cc68a1
  91. 2012-09-12 10:41:26,792 INFO org.apache.hadoop.hbase.regionserver.Store: Completed major compaction of 3 file(s) in d of compound0,isp-pub-chn\x09comcast cable communications inc.\x09CBD46B77\x09MBC1\x09\x80\x00\x019[\x0D\x1E0,1346260951439.8b2e6079105c337a485fbafd65492407. into 08dcb72196ae4a37ba5c5cbdd2cc68a1, size=724.1m; total size for store is 724.1m
  92. 2012-09-12 10:41:26,793 INFO org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: completed compaction: regionName=compound0,isp-pub-chn\x09comcast cable communications inc.\x09CBD46B77\x09MBC1\x09\x80\x00\x019[\x0D\x1E0,1346260951439.8b2e6079105c337a485fbafd65492407., storeName=d, fileCount=3, fileSize=724.2m, priority=4, time=2223922910469018; duration=2mins, 22sec
  93. 2012-09-12 10:41:26,793 DEBUG org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: CompactSplitThread status: compaction_queue=(0:0), split_queue=0
  94. 2012-09-12 10:42:25,291 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.64 GB, free=316.99 MB, max=1.95 GB, blocks=26292, accesses=327862035, hits=321343919, hitRatio=98.01%, cachingAccesses=321416174, cachingHits=321183740, cachingHitsRatio=99.92%, evictions=8, evicted=206143, evictedPerRun=25767.875
  95. 2012-09-12 10:42:25,957 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Block cache LRU eviction started; Attempting to free 199.6 MB of total=1.66 GB
  96. 2012-09-12 10:42:25,977 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Block cache LRU eviction completed; freed=199.62 MB, total=1.46 GB, single=425.6 MB, multi=1.23 GB, memory=161.44 KB
  97. 2012-09-12 10:42:32,759 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Block cache LRU eviction started; Attempting to free 199.6 MB of total=1.66 GB
  98. 2012-09-12 10:42:32,779 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Block cache LRU eviction completed; freed=199.62 MB, total=1.46 GB, single=569.18 MB, multi=1.08 GB, memory=161.44 KB
  99. 2012-09-12 10:42:39,534 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Block cache LRU eviction started; Attempting to free 199.62 MB of total=1.66 GB
  100. ... about 20 many more like these ---^
  101. 2012-09-12 10:44:52,466 DEBUG com.movenetworks.a3.realtime.hbase.coprocessor.CubeCoprocessorImplementation: Starting cubeRegion [table:'compound0'; range:1347465600000-1347466500000; startRow='pub-chn\x09\x10'; stopRow='pub-chn\x09^?\x00]
  102. 2012-09-12 10:44:52,467 DEBUG com.movenetworks.a3.realtime.hbase.coprocessor.CubeCoprocessorImplementation:     Matched: 863
  103. 2012-09-12 10:44:52,467 DEBUG com.movenetworks.a3.realtime.hbase.coprocessor.CubeCoprocessorImplementation:     Not matched: 0
  104. 2012-09-12 10:44:52,469 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":1785038,"call":"execCoprocessor([B@44b5e600, cubeRegion(com.movenetworks.a3.realtime.hbase.coprocessor.CubeRequest@20f2d3e), rpc version=1, client version=0, methodsFingerPrint=0), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.100.37.125:54143","starttimems":1347466507429,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"execCoprocessor"}
  105. 2012-09-12 10:44:52,470 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call execCoprocessor([B@44b5e600, cubeRegion(com.movenetworks.a3.realtime.hbase.coprocessor.CubeRequest@20f2d3e), rpc version=1, client version=0, methodsFingerPrint=0), rpc version=1, client version=29, methodsFingerPrint=54742778 from 10.100.37.125:54143: output error
  106. 2012-09-12 10:44:52,471 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 8 on 60020 caught: java.nio.channels.ClosedChannelException
  107.         at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(Unknown Source)
  108.         at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
  109.         at org.apache.hadoop.hbase.ipc.HBaseServer.channelIO(HBaseServer.java:1710)
  110.         at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1653)
  111.         at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:924)
  112.         at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:1003)
  113.         at org.apache.hadoop.hbase.ipc.HBaseServer$Call.sendResponseIfReady(HBaseServer.java:409)
  114.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1346)
  115.  
  116. 2012-09-12 10:44:52,471 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Received dynamic protocol exec call with protocolName com.movenetworks.a3.realtime.hbase.coprocessor.CubeCoprocessorProtocol
  117. 2012-09-12 10:44:52,471 DEBUG com.movenetworks.a3.realtime.hbase.coprocessor.CubeCoprocessorImplementation: Calling updateCube [table: 'compound1'; key: 'cdn-pub-aid\x09Level 3\x09CBD46B77\x09\x80\x00\x00\x00\x00\x0B\xE5\xDC\x80\x00\x019\xBBD\xEC`']
  118. 2012-09-12 10:44:52,471 DEBUG com.movenetworks.a3.realtime.hbase.coprocessor.CubeCoprocessorImplementation: Starting update table [table: 'compound1'; key: 'cdn-pub-aid\x09Level 3\x09CBD46B77\x09\x80\x00\x00\x00\x00\x0B\xE5\xDC\x80\x00\x019\xBBD\xEC`'; data: [v=0ms, b=0, bc=0, l=0, s=0, d=0, u=0, c=0, hOk=209, hErr=0, uniq#null]]
  119. 2012-09-12 10:44:52,472 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call getClosestRowBefore([B@37462354, [B@3c859135, [B@65004d9d), rpc version=1, client version=29, methodsFingerPrint=54742778 from 10.100.37.130:44990: output error
  120. 2012-09-12 10:44:52,474 WARN org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 2 on 60020 caught: java.nio.channels.ClosedChannelException
  121.         at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(Unknown Source)
  122.         at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
  123.         at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1653)
  124.         at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:924)
  125.         at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:1003)
  126.         at org.apache.hadoop.hbase.ipc.HBaseServer$Call.sendResponseIfReady(HBaseServer.java:409)
  127.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1346)
  128.  
  129. 2012-09-12 10:44:52,475 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call getClosestRowBefore([B@38db1aab, [B@38b3edb6, [B@2cbb64cf), rpc version=1, client version=29, methodsFingerPrint=54742778 from 10.100.37.130:44991: output error
  130. 2012-09-12 10:44:52,481 WARN org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 1 on 60020 caught: java.nio.channels.ClosedChannelException
  131.         at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(Unknown Source)
  132.         at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
  133.         at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1653)
  134.         at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:924)
  135.         at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:1003)
  136.         at org.apache.hadoop.hbase.ipc.HBaseServer$Call.sendResponseIfReady(HBaseServer.java:409)
  137.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1346)
  138.  
  139. 2012-09-12 10:44:52,482 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call getClosestRowBefore([B@bb71d28, [B@22db331a, [B@64f801d6), rpc version=1, client version=29, methodsFingerPrint=54742778 from 10.100.37.128:44787: output error
  140. 2012-09-12 10:44:52,484 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call getClosestRowBefore([B@1c7153cb, [B@54344303, [B@331ba525), rpc version=1, client version=29, methodsFingerPrint=54742778 from 10.100.37.128:44788: output error
  141. 2012-09-12 10:44:52,484 WARN org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 3 on 60020 caught: java.nio.channels.ClosedChannelException
  142.         at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(Unknown Source)
  143.         at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
  144.         at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1653)
  145.         at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:924)
  146.         at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:1003)
  147.         at org.apache.hadoop.hbase.ipc.HBaseServer$Call.sendResponseIfReady(HBaseServer.java:409)
  148.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1346)
  149.  
  150. 2012-09-12 10:44:52,485 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call getClosestRowBefore([B@155a5d4f, [B@5fa3f87c, [B@60ec5f2), rpc version=1, client version=29, methodsFingerPrint=54742778 from 10.100.37.130:44992: output error
  151. 2012-09-12 10:44:52,485 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call getClosestRowBefore([B@108ab9d, [B@603ad0f2, [B@356bf139), rpc version=1, client version=29, methodsFingerPrint=54742778 from 10.100.37.129:34752: output error
  152. 2012-09-12 10:44:52,485 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call getClosestRowBefore([B@2ebbf60e, [B@3617291a, [B@2a8b85b4), rpc version=1, client version=29, methodsFingerPrint=54742778 from 10.100.37.129:34754: output error
  153. 2012-09-12 10:44:52,485 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call getClosestRowBefore([B@2df7023e, [B@33b44c05, [B@1eaf6c8), rpc version=1, client version=29, methodsFingerPrint=54742778 from 10.100.37.129:34753: output error
  154. 2012-09-12 10:44:52,487 WARN org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 2 on 60020 caught: java.nio.channels.ClosedChannelException
  155.         at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(Unknown Source)
  156.         at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
  157.         at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1653)
  158.         at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:924)
  159.         at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:1003)
  160.         at org.apache.hadoop.hbase.ipc.HBaseServer$Call.sendResponseIfReady(HBaseServer.java:409)
  161.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1346)
  162.  
  163.  
  164. .... more exceptions like these
  165.  
  166.  
  167. 2012-09-12 10:44:52,554 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call openScanner([B@1fbf0f3b, {"timeRange":[0,9223372036854775807],"batch":-1,"startRow":"","stopRow":"","totalColumns":0,"cacheBlocks":true,"families":{"info":[]},"maxVersions":1,"caching":-1}), rpc version=1, client version=29, methodsFingerPrint=54742778 from 10.100.37.120:40725: output error
  168. 2012-09-12 10:44:52,554 WARN org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 2 on 60020 caught: java.nio.channels.ClosedChannelException
  169.         at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(Unknown Source)
  170.         at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
  171.         at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1653)
  172.         at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:924)
  173.         at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:1003)
  174.         at org.apache.hadoop.hbase.ipc.HBaseServer$Call.sendResponseIfReady(HBaseServer.java:409)
  175.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1346)
  176.  
  177. 2012-09-12 10:44:52,555 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call openScanner([B@4befeffe, {"timeRange":[0,9223372036854775807],"batch":-1,"startRow":"","stopRow":"","totalColumns":0,"cacheBlocks":true,"families":{"info":[]},"maxVersions":1,"caching":-1}), rpc version=1, client version=29, methodsFingerPrint=54742778 from 10.100.37.120:40712: output error
  178. 2012-09-12 10:44:52,555 WARN org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 1 on 60020 caught: java.nio.channels.ClosedChannelException
  179.         at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(Unknown Source)
  180.         at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
  181.         at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1653)
  182.         at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:924)
  183.         at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:1003)
  184.         at org.apache.hadoop.hbase.ipc.HBaseServer$Call.sendResponseIfReady(HBaseServer.java:409)
  185.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1346)
  186.  
  187. 2012-09-12 10:44:52,555 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call openScanner([B@4378c6be, {"timeRange":[0,9223372036854775807],"batch":-1,"startRow":"","stopRow":"","totalColumns":0,"cacheBlocks":true,"families":{"info":[]},"maxVersions":1,"caching":-1}), rpc version=1, client version=29, methodsFingerPrint=54742778 from 10.100.37.120:40705: output error
  188. 2012-09-12 10:44:52,555 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call openScanner([B@14a2bfe7, {"timeRange":[0,9223372036854775807],"batch":-1,"startRow":"","stopRow":"","totalColumns":0,"cacheBlocks":true,"families":{"info":[]},"maxVersions":1,"caching":-1}), rpc version=1, client version=29, methodsFingerPrint=54742778 from 10.100.37.120:40709: output error
  189. 2012-09-12 10:44:52,555 WARN org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 8 on 60020 caught: java.nio.channels.ClosedChannelException
  190.         at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(Unknown Source)
  191.         at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
  192.         at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1653)
  193.         at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:924)
  194.         at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:1003)
  195.         at org.apache.hadoop.hbase.ipc.HBaseServer$Call.sendResponseIfReady(HBaseServer.java:409)
  196.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1346)