Want more features on Pastebin? Sign Up, it's FREE!
Guest

HBase Region Server Log

By: ronbuckley on Aug 6th, 2012  |  syntax: None  |  size: 90.46 KB  |  views: 42  |  expires: Never
download  |  raw  |  embed  |  report abuse  |  print
Text below is selected. Please press Ctrl+C to copy to your clipboard. (⌘+C on Mac)
  1.  
  2. 2012-08-06 11:38:28,774 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom filter type for hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/8589b9b3058d46d1a75bb176222fb7a8: ROW, CompoundBloomFilterWriter
  3. 2012-08-06 11:38:29,946 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom added to HFile (hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/8589b9b3058d46d1a75bb176222fb7a8): org.apache.hadoop.hbase.util.CompoundBloomFilterWriter@5c88970e
  4. 2012-08-06 11:38:29,949 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for 8589b9b3058d46d1a75bb176222fb7a8
  5. 2012-08-06 11:38:29,954 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for 8589b9b3058d46d1a75bb176222fb7a8
  6. 2012-08-06 11:38:29,954 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/data/8589b9b3058d46d1a75bb176222fb7a8, entries=509142, sequenceid=1792261769, memsize=155.4m, filesize=5.5m
  7. 2012-08-06 11:38:29,982 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/stats/24b9e95b3fbe41cdb0e26f94ffa91c52, entries=11, sequenceid=1792261769, memsize=2.4k, filesize=740.0
  8. 2012-08-06 11:38:30,522 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~155.4m/162935904, currentsize=680.0/680 for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. in 1806ms, sequenceid=1792261769, compaction requested=false
  9. 2012-08-06 11:38:41,188 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom filter type for hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/8e8d795a6d514eb99b26ed6462879c09: ROW, CompoundBloomFilterWriter
  10. 2012-08-06 11:38:42,310 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom added to HFile (hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/8e8d795a6d514eb99b26ed6462879c09): org.apache.hadoop.hbase.util.CompoundBloomFilterWriter@75f8d143
  11. 2012-08-06 11:38:42,314 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for 8e8d795a6d514eb99b26ed6462879c09
  12. 2012-08-06 11:38:42,334 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for 8e8d795a6d514eb99b26ed6462879c09
  13. 2012-08-06 11:38:42,334 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/data/8e8d795a6d514eb99b26ed6462879c09, entries=445472, sequenceid=1792262048, memsize=135.9m, filesize=4.9m
  14. 2012-08-06 11:38:42,367 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/stats/ee63c14cdafe4f1b82ca76fbf5e22ab8, entries=12, sequenceid=1792262048, memsize=2.6k, filesize=754.0
  15. 2012-08-06 11:38:48,245 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1 Got:
  16. java.io.EOFException: hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/.logs/wchddb37pxdu.prod.oclc.org,9009,1337721421002/wchddb37pxdu.prod.oclc.org%2C9009%2C1337721421002.1344267499281, entryStart=138461295, pos=144248320, end=144248320, edit=10
  17.         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  18.         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
  19.         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
  20.         at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
  21.         at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.addFileInfoToException(SequenceFileLogReader.java:252)
  22.         at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.next(SequenceFileLogReader.java:208)
  23.         at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.readAllEntriesToReplicateOrNextFile(ReplicationSource.java:427)
  24.         at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:306)
  25. Caused by: java.io.EOFException
  26.         at java.io.DataInputStream.readFully(DataInputStream.java:180)
  27.         at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:63)
  28.         at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:101)
  29.         at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2001)
  30.         at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1901)
  31.         at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1947)
  32.         at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.next(SequenceFileLogReader.java:206)
  33.         ... 2 more
  34. 2012-08-06 11:38:48,246 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1 Got EOF while reading, looks like this file is broken? hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/.logs/wchddb37pxdu.prod.oclc.org,9009,1337721421002/wchddb37pxdu.prod.oclc.org%2C9009%2C1337721421002.1344267499281
  35. 2012-08-06 11:38:58,576 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1 Got:
  36. java.io.EOFException: hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/.logs/wchddb37pxdu.prod.oclc.org,9009,1337721421002/wchddb37pxdu.prod.oclc.org%2C9009%2C1337721421002.1344267499281, entryStart=166527116, pos=172119040, end=172119040, edit=12
  37.         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  38.         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
  39.         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
  40.         at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
  41.         at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.addFileInfoToException(SequenceFileLogReader.java:252)
  42.         at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.next(SequenceFileLogReader.java:208)
  43.         at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.readAllEntriesToReplicateOrNextFile(ReplicationSource.java:427)
  44.         at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:306)
  45. Caused by: java.io.EOFException
  46.         at java.io.DataInputStream.readFully(DataInputStream.java:180)
  47.         at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:63)
  48.         at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:101)
  49.         at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2001)
  50.         at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1901)
  51.         at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1947)
  52.         at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.next(SequenceFileLogReader.java:206)
  53.         ... 2 more
  54. 2012-08-06 11:38:58,577 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1 Got EOF while reading, looks like this file is broken? hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/.logs/wchddb37pxdu.prod.oclc.org,9009,1337721421002/wchddb37pxdu.prod.oclc.org%2C9009%2C1337721421002.1344267499281
  55. 2012-08-06 11:39:01,450 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 15 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 291.3m is >= than blocking 256.0m size
  56. 2012-08-06 11:39:01,829 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 21 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 291.3m is >= than blocking 256.0m size
  57. 2012-08-06 11:39:45,314 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 20 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 291.3m is >= than blocking 256.0m size
  58. 2012-08-06 11:40:02,976 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 27 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 291.3m is >= than blocking 256.0m size
  59. 2012-08-06 11:40:03,420 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 14 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 291.3m is >= than blocking 256.0m size
  60. 2012-08-06 11:40:46,397 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 31 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 291.3m is >= than blocking 256.0m size
  61. 2012-08-06 11:41:04,501 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 35 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 291.3m is >= than blocking 256.0m size
  62. 2012-08-06 11:41:05,250 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 36 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 291.3m is >= than blocking 256.0m size
  63. 2012-08-06 11:41:47,484 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 39 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 291.3m is >= than blocking 256.0m size
  64. 2012-08-06 11:42:06,202 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 13 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 291.3m is >= than blocking 256.0m size
  65. 2012-08-06 11:42:06,300 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 4 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 291.3m is >= than blocking 256.0m size
  66. 2012-08-06 11:42:48,489 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 2 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 291.3m is >= than blocking 256.0m size
  67. 2012-08-06 11:42:49,705 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 2 on 9009'
  68. 2012-08-06 11:42:49,705 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 39 on 9009'
  69. 2012-08-06 11:42:49,705 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 4 on 9009'
  70. 2012-08-06 11:42:49,705 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 20 on 9009'
  71. 2012-08-06 11:42:49,705 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 35 on 9009'
  72. 2012-08-06 11:42:49,705 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 27 on 9009'
  73. 2012-08-06 11:42:49,705 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~136.0m/142563440, currentsize=155.4m/162930704 for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. in 248567ms, sequenceid=1792262048, compaction requested=true
  74. 2012-08-06 11:42:49,706 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 15 on 9009'
  75. 2012-08-06 11:42:49,705 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 31 on 9009'
  76. 2012-08-06 11:42:49,706 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 21 on 9009'
  77. 2012-08-06 11:42:49,705 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 14 on 9009'
  78. 2012-08-06 11:42:49,705 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 36 on 9009'
  79. 2012-08-06 11:42:49,705 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 13 on 9009'
  80. 2012-08-06 11:42:49,706 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on data in region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.
  81. 2012-08-06 11:42:49,706 INFO org.apache.hadoop.hbase.regionserver.Store: Starting compaction of 3 file(s) in data of Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. into tmpdir=hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp, seqid=1792262048, totalSize=60.9m
  82. 2012-08-06 11:42:49,707 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"processingtimems":62222,"ts":9223372036854775807,"client":"10.192.98.34:46807","starttimems":1344267707484,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"delete","totalColumns":1,"table":"Works","families":{"data":[{"timestamp":1344267769706,"qualifier":"ocl:xwc::690708517","vlen":0}]},"row":"763297266:und"}
  83. 2012-08-06 11:42:49,708 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"processingtimems":166730,"ts":9223372036854775807,"client":"10.192.98.18:45125","starttimems":1344267602976,"queuetimems":1,"class":"HRegionServer","responsesize":0,"method":"delete","totalColumns":1,"table":"Works","families":{"data":[{"timestamp":1344267769706,"qualifier":"ocl:xwc::690712033","vlen":0}]},"row":"763297266:und"}
  84. 2012-08-06 11:42:49,710 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"processingtimems":248290,"client":"10.192.98.34:46807","timeRange":[0,9223372036854775807],"starttimems":1344267521418,"responsesize":9164166,"class":"HRegionServer","table":"Works","cacheBlocks":true,"families":{"data":["ALL"]},"row":"763297266:und","queuetimems":0,"method":"get","totalColumns":1,"maxVersions":1}
  85. 2012-08-06 11:42:49,743 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom filter type for hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/442e10d687c74c30b23074e5b034a1ec: ROW, CompoundBloomFilterWriter
  86. 2012-08-06 11:42:49,750 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Roll /hbase/.logs/wchddb37pxdu.prod.oclc.org,9009,1337721421002/wchddb37pxdu.prod.oclc.org%2C9009%2C1337721421002.1344267499281, entries=2426, filesize=188830434.  for /hbase/.logs/wchddb37pxdu.prod.oclc.org,9009,1337721421002/wchddb37pxdu.prod.oclc.org%2C9009%2C1337721421002.1344267769705
  87. 2012-08-06 11:42:49,777 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"processingtimems":105275,"ts":9223372036854775807,"client":"10.192.98.51:44258","starttimems":1344267664501,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"delete","totalColumns":1,"table":"Works","families":{"data":[{"timestamp":1344267769706,"qualifier":"ocl:xwc::690712255","vlen":0}]},"row":"763297266:und"}
  88. 2012-08-06 11:42:49,802 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"processingtimems":43502,"ts":9223372036854775807,"client":"10.192.98.18:45125","starttimems":1344267726299,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"delete","totalColumns":1,"table":"Works","families":{"data":[{"timestamp":1344267769790,"qualifier":"ocl:xwc::690712033","vlen":0}]},"row":"763297266:und"}
  89. 2012-08-06 11:42:49,803 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"processingtimems":184489,"ts":9223372036854775807,"client":"10.192.98.34:46807","starttimems":1344267585313,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"delete","totalColumns":1,"table":"Works","families":{"data":[{"timestamp":1344267769801,"qualifier":"ocl:xwc::690708517","vlen":0}]},"row":"763297266:und"}
  90. 2012-08-06 11:42:49,805 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"processingtimems":228355,"ts":9223372036854775807,"client":"10.192.98.51:44258","starttimems":1344267541449,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"delete","totalColumns":1,"table":"Works","families":{"data":[{"timestamp":1344267769803,"qualifier":"ocl:xwc::690712255","vlen":0}]},"row":"763297266:und"}
  91. 2012-08-06 11:42:49,807 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"processingtimems":123410,"ts":9223372036854775807,"client":"10.192.98.34:46807","starttimems":1344267646396,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"delete","totalColumns":1,"table":"Works","families":{"data":[{"timestamp":1344267769804,"qualifier":"ocl:xwc::690708517","vlen":0}]},"row":"763297266:und"}
  92. 2012-08-06 11:42:49,809 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"processingtimems":104558,"ts":9223372036854775807,"client":"10.192.98.18:45125","starttimems":1344267665250,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"delete","totalColumns":1,"table":"Works","families":{"data":[{"timestamp":1344267769808,"qualifier":"ocl:xwc::690712033","vlen":0}]},"row":"763297266:und"}
  93. 2012-08-06 11:42:49,811 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"processingtimems":166390,"ts":9223372036854775807,"client":"10.192.98.51:44258","starttimems":1344267603420,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"delete","totalColumns":1,"table":"Works","families":{"data":[{"timestamp":1344267769808,"qualifier":"ocl:xwc::690712255","vlen":0}]},"row":"763297266:und"}
  94. 2012-08-06 11:42:49,813 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"processingtimems":227983,"ts":9223372036854775807,"client":"10.192.98.18:45125","starttimems":1344267541829,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"delete","totalColumns":1,"table":"Works","families":{"data":[{"timestamp":1344267769810,"qualifier":"ocl:xwc::690712033","vlen":0}]},"row":"763297266:und"}
  95. 2012-08-06 11:42:49,814 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"processingtimems":43611,"ts":9223372036854775807,"client":"10.192.98.51:44258","starttimems":1344267726202,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"delete","totalColumns":1,"table":"Works","families":{"data":[{"timestamp":1344267769813,"qualifier":"ocl:xwc::690712255","vlen":0}]},"row":"763297266:und"}
  96. 2012-08-06 11:42:49,822 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom filter type for hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/745fe30f00774720b742ec68e208d5de: ROW, CompoundBloomFilterWriter
  97. 2012-08-06 11:42:50,985 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom added to HFile (hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/745fe30f00774720b742ec68e208d5de): org.apache.hadoop.hbase.util.CompoundBloomFilterWriter@760576d9
  98. 2012-08-06 11:42:50,990 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for 745fe30f00774720b742ec68e208d5de
  99. 2012-08-06 11:42:50,996 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for 745fe30f00774720b742ec68e208d5de
  100. 2012-08-06 11:42:50,996 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/data/745fe30f00774720b742ec68e208d5de, entries=509106, sequenceid=1792263953, memsize=155.4m, filesize=5.1m
  101. 2012-08-06 11:42:51,020 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/stats/c1f2292852ee4915a8ab852e2bfdc710, entries=8, sequenceid=1792263953, memsize=1.8k, filesize=729.0
  102. 2012-08-06 11:42:52,316 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~155.4m/162931400, currentsize=2.7k/2760 for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. in 2610ms, sequenceid=1792263953, compaction requested=false
  103. 2012-08-06 11:42:55,545 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom added to HFile (hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/442e10d687c74c30b23074e5b034a1ec): org.apache.hadoop.hbase.util.CompoundBloomFilterWriter@79493dc9
  104. 2012-08-06 11:42:55,578 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for 442e10d687c74c30b23074e5b034a1ec
  105. 2012-08-06 11:42:55,578 INFO org.apache.hadoop.hbase.regionserver.Store: Renaming compacted file at hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/442e10d687c74c30b23074e5b034a1ec to hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/data/442e10d687c74c30b23074e5b034a1ec
  106. 2012-08-06 11:42:55,585 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for 442e10d687c74c30b23074e5b034a1ec
  107. 2012-08-06 11:42:56,903 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 3 file(s) in data of Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. into 442e10d687c74c30b23074e5b034a1ec, size=50.5m; total size for store is 935.7m
  108. 2012-08-06 11:42:56,903 INFO org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: completed compaction: regionName=Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6., storeName=data, fileCount=3, fileSize=60.9m, priority=3, time=15936905292664027; duration=7sec
  109. 2012-08-06 11:42:56,903 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on stats in region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.
  110. 2012-08-06 11:42:56,904 INFO org.apache.hadoop.hbase.regionserver.Store: Starting compaction of 3 file(s) in stats of Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. into tmpdir=hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp, seqid=1792262048, totalSize=4.5m
  111. 2012-08-06 11:42:57,603 INFO org.apache.hadoop.hbase.regionserver.Store: Renaming compacted file at hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/ec66a36d18014a9d968426387a250aec to hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/stats/ec66a36d18014a9d968426387a250aec
  112. 2012-08-06 11:42:57,664 INFO org.apache.hadoop.hbase.regionserver.Store: Completed major compaction of 3 file(s) in stats of Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. into ec66a36d18014a9d968426387a250aec, size=4.5m; total size for store is 4.5m
  113. 2012-08-06 11:42:57,664 INFO org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: completed compaction: regionName=Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6., storeName=stats, fileCount=3, fileSize=4.5m, priority=4, time=15936905292745027; duration=0sec
  114. 2012-08-06 11:43:06,864 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom filter type for hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/cd172c73de844f1f91b3ffdf17b52ed2: ROW, CompoundBloomFilterWriter
  115. 2012-08-06 11:43:07,966 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom added to HFile (hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/cd172c73de844f1f91b3ffdf17b52ed2): org.apache.hadoop.hbase.util.CompoundBloomFilterWriter@6be954c2
  116. 2012-08-06 11:43:07,969 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for cd172c73de844f1f91b3ffdf17b52ed2
  117. 2012-08-06 11:43:07,975 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for cd172c73de844f1f91b3ffdf17b52ed2
  118. 2012-08-06 11:43:07,975 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/data/cd172c73de844f1f91b3ffdf17b52ed2, entries=509049, sequenceid=1792264092, memsize=155.3m, filesize=5.1m
  119. 2012-08-06 11:43:08,069 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/stats/63408310be644929980186d4fcce9c65, entries=9, sequenceid=1792264092, memsize=2.0k, filesize=703.0
  120. 2012-08-06 11:43:11,534 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1 Got:
  121. java.io.EOFException: hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/.logs/wchddb37pxdu.prod.oclc.org,9009,1337721421002/wchddb37pxdu.prod.oclc.org%2C9009%2C1337721421002.1344267769705, entryStart=82765345, pos=84585984, end=84585984, edit=8
  122.         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  123.         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
  124.         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
  125.         at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
  126.         at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.addFileInfoToException(SequenceFileLogReader.java:252)
  127.         at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.next(SequenceFileLogReader.java:208)
  128.         at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.readAllEntriesToReplicateOrNextFile(ReplicationSource.java:427)
  129.         at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:306)
  130. Caused by: java.io.EOFException
  131.         at java.io.DataInputStream.readFully(DataInputStream.java:180)
  132.         at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:63)
  133.         at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:101)
  134.         at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2001)
  135.         at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1901)
  136.         at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1947)
  137.         at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.next(SequenceFileLogReader.java:206)
  138.         ... 2 more
  139. 2012-08-06 11:43:11,535 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1 Got EOF while reading, looks like this file is broken? hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/.logs/wchddb37pxdu.prod.oclc.org,9009,1337721421002/wchddb37pxdu.prod.oclc.org%2C9009%2C1337721421002.1344267769705
  140. 2012-08-06 11:43:16,360 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1 Got:
  141. java.io.EOFException: hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/.logs/wchddb37pxdu.prod.oclc.org,9009,1337721421002/wchddb37pxdu.prod.oclc.org%2C9009%2C1337721421002.1344267769705, entryStart=101122117, pos=102617600, end=102617600, edit=0
  142.         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  143.         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
  144.         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
  145.         at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
  146.         at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.addFileInfoToException(SequenceFileLogReader.java:252)
  147.         at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.next(SequenceFileLogReader.java:208)
  148.         at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.readAllEntriesToReplicateOrNextFile(ReplicationSource.java:393)
  149.         at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:306)
  150. Caused by: java.io.EOFException
  151.         at java.io.DataInputStream.readFully(DataInputStream.java:180)
  152.         at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:63)
  153.         at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:101)
  154.         at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2001)
  155.         at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1901)
  156.         at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1947)
  157.         at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.next(SequenceFileLogReader.java:206)
  158.         ... 2 more
  159. 2012-08-06 11:43:23,019 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 6 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  160. 2012-08-06 11:43:24,022 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 19 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  161. 2012-08-06 11:43:35,229 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 11 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  162. 2012-08-06 11:43:35,229 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 13 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  163. 2012-08-06 11:43:35,229 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 14 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  164. 2012-08-06 11:43:35,230 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 41 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  165. 2012-08-06 11:43:35,230 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 5 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  166. 2012-08-06 11:43:35,230 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 21 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  167. 2012-08-06 11:43:35,230 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 46 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  168. 2012-08-06 11:43:35,291 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 40 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  169. 2012-08-06 11:43:45,005 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 33 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  170. 2012-08-06 11:43:45,008 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 36 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  171. 2012-08-06 11:43:45,009 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 4 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  172. 2012-08-06 11:43:45,009 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 25 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  173. 2012-08-06 11:43:45,010 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 37 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  174. 2012-08-06 11:43:45,014 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 32 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  175. 2012-08-06 11:43:45,015 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 48 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  176. 2012-08-06 11:43:45,017 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 7 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  177. 2012-08-06 11:44:08,170 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 9 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  178. 2012-08-06 11:44:08,181 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 22 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  179. 2012-08-06 11:44:08,182 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 12 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  180. 2012-08-06 11:44:08,184 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 44 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  181. 2012-08-06 11:44:08,184 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 27 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  182. 2012-08-06 11:44:08,187 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 35 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  183. 2012-08-06 11:44:08,187 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 23 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  184. 2012-08-06 11:44:08,189 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 38 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  185. 2012-08-06 11:44:11,100 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 30 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  186. 2012-08-06 11:44:18,624 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 20 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  187. 2012-08-06 11:44:18,624 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 8 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  188. 2012-08-06 11:44:18,630 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 18 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  189. 2012-08-06 11:44:18,699 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 45 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  190. 2012-08-06 11:44:18,700 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 39 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  191. 2012-08-06 11:44:18,704 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 26 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  192. 2012-08-06 11:44:18,705 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 15 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  193. 2012-08-06 11:44:18,708 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 2 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  194. 2012-08-06 11:44:24,127 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 17 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  195. 2012-08-06 11:44:25,246 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 49 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  196. 2012-08-06 11:44:33,318 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 1 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  197. 2012-08-06 11:44:33,323 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 10 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  198. 2012-08-06 11:44:33,324 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 28 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  199. 2012-08-06 11:44:33,324 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 16 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  200. 2012-08-06 11:44:33,324 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 0 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  201. 2012-08-06 11:44:33,326 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 34 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  202. 2012-08-06 11:44:33,332 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 47 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  203. 2012-08-06 11:44:33,333 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 3 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  204. 2012-08-06 11:44:36,345 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 24 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  205. 2012-08-06 11:44:36,346 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 31 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  206. 2012-08-06 11:44:36,347 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 43 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  207. 2012-08-06 11:44:36,348 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 42 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size
  208. 2012-08-06 11:51:28,139 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~155.4m/162905016, currentsize=116.5m/122172880 for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. in 501337ms, sequenceid=1792264092, compaction requested=true
  209. 2012-08-06 11:51:28,139 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on data in region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.
  210. 2012-08-06 11:51:28,140 INFO org.apache.hadoop.hbase.regionserver.Store: Starting compaction of 3 file(s) in data of Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. into tmpdir=hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp, seqid=1792264092, totalSize=60.7m
  211. 2012-08-06 11:51:28,140 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 42 on 9009'
  212. 2012-08-06 11:51:28,140 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 43 on 9009'
  213. 2012-08-06 11:51:28,140 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 31 on 9009'
  214. 2012-08-06 11:51:28,140 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 24 on 9009'
  215. 2012-08-06 11:51:28,140 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 49 on 9009'
  216. 2012-08-06 11:51:28,140 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 40 on 9009'
  217. 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 46 on 9009'
  218. 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 21 on 9009'
  219. 2012-08-06 11:51:28,140 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 11 on 9009'
  220. 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 7 on 9009'
  221. 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 32 on 9009'
  222. 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 37 on 9009'
  223. 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 4 on 9009'
  224. 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 19 on 9009'
  225. 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 3 on 9009'
  226. 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 47 on 9009'
  227. 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 10 on 9009'
  228. 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 28 on 9009'
  229. 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 1 on 9009'
  230. 2012-08-06 11:51:28,142 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 6 on 9009'
  231. 2012-08-06 11:51:28,142 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 26 on 9009'
  232. 2012-08-06 11:51:28,142 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 15 on 9009'
  233. 2012-08-06 11:51:28,142 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 2 on 9009'
  234. 2012-08-06 11:51:28,142 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 39 on 9009'
  235. 2012-08-06 11:51:28,142 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 45 on 9009'
  236. 2012-08-06 11:51:28,142 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 18 on 9009'
  237. 2012-08-06 11:51:28,143 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 20 on 9009'
  238. 2012-08-06 11:51:28,143 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":411795,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@2b400c61), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267876347,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  239. 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 13 on 9009'
  240. 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 14 on 9009'
  241. 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 41 on 9009'
  242. 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 5 on 9009'
  243. 2012-08-06 11:51:28,143 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 12 on 9009'
  244. 2012-08-06 11:51:28,144 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":411798,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@28e005dc), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267876345,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  245. 2012-08-06 11:51:28,143 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 44 on 9009'
  246. 2012-08-06 11:51:28,143 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 35 on 9009'
  247. 2012-08-06 11:51:28,143 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 27 on 9009'
  248. 2012-08-06 11:51:28,143 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 23 on 9009'
  249. 2012-08-06 11:51:28,143 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 38 on 9009'
  250. 2012-08-06 11:51:28,145 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":414811,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@78fb735b), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.51:44258","starttimems":1344267873333,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  251. 2012-08-06 11:51:28,145 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":472915,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@ad62131), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267815230,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  252. 2012-08-06 11:51:28,146 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":411799,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@701d1d85), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267876346,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  253. 2012-08-06 11:51:28,143 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 8 on 9009'
  254. 2012-08-06 11:51:28,143 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":463125,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@6a676982), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267825017,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  255. 2012-08-06 11:51:28,142 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":411793,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@4513c778), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267876348,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  256. 2012-08-06 11:51:28,142 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 30 on 9009'
  257. 2012-08-06 11:51:28,141 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"processingtimems":501069,"client":"10.192.98.18:45125","timeRange":[0,9223372036854775807],"starttimems":1344267787070,"responsesize":9163014,"class":"HRegionServer","table":"Works","cacheBlocks":true,"families":{"data":["ALL"]},"row":"763297266:und","queuetimems":1,"method":"get","totalColumns":1,"maxVersions":1}
  258. 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 16 on 9009'
  259. 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 0 on 9009'
  260. 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 34 on 9009'
  261. 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 17 on 9009'
  262. 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 36 on 9009'
  263. 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 33 on 9009'
  264. 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 25 on 9009'
  265. 2012-08-06 11:51:28,147 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":472916,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@75131419), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267815230,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  266. 2012-08-06 11:51:28,147 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":463132,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@378cb15b), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267825014,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  267. 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 48 on 9009'
  268. 2012-08-06 11:51:28,147 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":429442,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@769dd905), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.51:44258","starttimems":1344267858704,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  269. 2012-08-06 11:51:28,146 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"processingtimems":422899,"ts":9223372036854775807,"client":"10.192.98.51:44258","starttimems":1344267865246,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"delete","totalColumns":1,"table":"Works","families":{"data":[{"timestamp":1344268288140,"qualifier":"ocl:xwc::690712263","vlen":0}]},"row":"763297266:und"}
  270. 2012-08-06 11:51:28,145 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":414827,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@6781c633), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.51:44258","starttimems":1344267873317,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  271. 2012-08-06 11:51:28,145 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 9 on 9009'
  272. 2012-08-06 11:51:28,144 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 22 on 9009'
  273. 2012-08-06 11:51:28,144 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":472852,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@ba0c37b), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267815291,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  274. 2012-08-06 11:51:28,149 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":472919,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@166b7164), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267815229,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  275. 2012-08-06 11:51:28,150 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":429526,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@7cb051e6), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.51:44258","starttimems":1344267858623,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  276. 2012-08-06 11:51:28,150 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":463139,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@1bd112fe), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267825010,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  277. 2012-08-06 11:51:28,151 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":429521,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@39c4a668), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.51:44258","starttimems":1344267858630,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  278. 2012-08-06 11:51:28,153 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":439965,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@1441423e), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267848187,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  279. 2012-08-06 11:51:28,153 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call get([B@564d6e0, {"timeRange":[0,9223372036854775807],"totalColumns":1,"cacheBlocks":true,"families":{"stats":["count"]},"maxVersions":1,"row":"763297266:und"}), rpc version=1, client version=29, methodsFingerPrint=54742778 from 10.192.98.60:55529: output error
  280. 2012-08-06 11:51:28,150 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"processingtimems":484127,"ts":9223372036854775807,"client":"10.192.98.51:44258","starttimems":1344267804022,"queuetimems":1,"class":"HRegionServer","responsesize":0,"method":"delete","totalColumns":1,"table":"Works","families":{"data":[{"timestamp":1344268288146,"qualifier":"ocl:xwc::690712263","vlen":0}]},"row":"763297266:und"}
  281. 2012-08-06 11:51:28,161 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":472931,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@3fc30d9f), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267815229,"queuetimems":1,"class":"HRegionServer","responsesize":0,"method":"multi"}
  282. 2012-08-06 11:51:28,161 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":429452,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@5f4d4222), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.51:44258","starttimems":1344267858708,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  283. 2012-08-06 11:51:28,158 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 37 on 9009 caught: java.nio.channels.ClosedChannelException
  284.         at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:133)
  285.         at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
  286.         at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1653)
  287.         at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:924)
  288.         at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:1003)
  289.         at org.apache.hadoop.hbase.ipc.HBaseServer$Call.sendResponseIfReady(HBaseServer.java:409)
  290.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1346)
  291.  
  292. 2012-08-06 11:51:28,162 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":439972,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@4930f70f), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267848189,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  293. 2012-08-06 11:51:28,161 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":414828,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@265428f6), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.51:44258","starttimems":1344267873332,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  294. 2012-08-06 11:51:28,161 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":472930,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@1302f1da), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267815230,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  295. 2012-08-06 11:51:28,163 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":472933,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@5f046b28), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267815229,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  296. 2012-08-06 11:51:28,163 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":439971,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@768a849a), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267848182,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  297. 2012-08-06 11:51:28,164 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":414830,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@68ac091d), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.51:44258","starttimems":1344267873323,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  298. 2012-08-06 11:51:28,165 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom filter type for hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/133021bb2ddd41da87656e7b12d2c46b: ROW, CompoundBloomFilterWriter
  299. 2012-08-06 11:51:28,168 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Roll /hbase/.logs/wchddb37pxdu.prod.oclc.org,9009,1337721421002/wchddb37pxdu.prod.oclc.org%2C9009%2C1337721421002.1344267769705, entries=637, filesize=129608690.  for /hbase/.logs/wchddb37pxdu.prod.oclc.org,9009,1337721421002/wchddb37pxdu.prod.oclc.org%2C9009%2C1337721421002.1344268288139
  300. 2012-08-06 11:51:28,187 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":414848,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@1bac39d), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.51:44258","starttimems":1344267873324,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  301. 2012-08-06 11:51:28,187 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":463164,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@2afec9b7), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267825008,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  302. 2012-08-06 11:51:28,187 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":472943,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@e3bf6dd), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267815229,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  303. 2012-08-06 11:51:28,188 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":429480,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@2946da24), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.51:44258","starttimems":1344267858699,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  304. 2012-08-06 11:51:28,188 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":463170,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@6a9b86a7), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267825009,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  305. 2012-08-06 11:51:28,188 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":429475,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@8a5749f), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.51:44258","starttimems":1344267858704,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  306. 2012-08-06 11:51:28,387 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call get([B@6999eae8, {"timeRange":[0,9223372036854775807],"totalColumns":1,"cacheBlocks":true,"families":{"stats":["count"]},"maxVersions":1,"row":"763297266:und"}), rpc version=1, client version=29, methodsFingerPrint=54742778 from 10.192.98.60:55805: output error
  307. 2012-08-06 11:51:28,392 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 13 on 9009 caught: java.nio.channels.ClosedChannelException
  308.         at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:133)
  309.         at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
  310.         at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1653)
  311.         at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:924)
  312.         at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:1003)
  313.         at org.apache.hadoop.hbase.ipc.HBaseServer$Call.sendResponseIfReady(HBaseServer.java:409)
  314.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1346)
  315.  
  316. 2012-08-06 11:51:28,843 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":485824,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@11dee2bb), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267803018,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  317. 2012-08-06 11:51:28,843 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":415519,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@651410f), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.51:44258","starttimems":1344267873323,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  318. 2012-08-06 11:51:28,844 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"processingtimems":437743,"ts":9223372036854775807,"client":"10.192.98.18:45125","starttimems":1344267851100,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"delete","totalColumns":1,"table":"Works","families":{"data":[{"timestamp":1344268288842,"qualifier":"ocl:xwc::690712037","vlen":0}]},"row":"763297266:und"}
  319. 2012-08-06 11:51:28,848 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":463838,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@518515a8), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267825009,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  320. 2012-08-06 11:51:28,849 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":430149,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@7a6b3cef), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.51:44258","starttimems":1344267858699,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  321. 2012-08-06 11:51:28,850 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":440665,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@13bda00d), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267848184,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  322. 2012-08-06 11:51:28,850 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":463844,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@52c32f37), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267825005,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  323. 2012-08-06 11:51:28,851 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call get([B@101ad9e8, {"timeRange":[0,9223372036854775807],"totalColumns":1,"cacheBlocks":true,"families":{"stats":["count"]},"maxVersions":1,"row":"763297266:und"}), rpc version=1, client version=29, methodsFingerPrint=54742778 from 10.192.98.60:55956: output error
  324. 2012-08-06 11:51:28,855 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 25 on 9009 caught: java.nio.channels.ClosedChannelException
  325.         at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:133)
  326.         at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
  327.         at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1653)
  328.         at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:924)
  329.         at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:1003)
  330.         at org.apache.hadoop.hbase.ipc.HBaseServer$Call.sendResponseIfReady(HBaseServer.java:409)
  331.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1346)
  332.  
  333. 2012-08-06 11:51:28,857 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":463841,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@78ab6d2b), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267825015,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  334. 2012-08-06 11:51:28,926 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom filter type for hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/d3232be635e14fc2ae3f89f5ccefa6da: ROW, CompoundBloomFilterWriter
  335. 2012-08-06 11:51:28,995 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":430370,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@70e6feb6), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.51:44258","starttimems":1344267858624,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  336. 2012-08-06 11:51:28,995 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":440813,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@7a28db5b), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267848181,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  337. 2012-08-06 11:51:28,995 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":415671,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@10597205), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.51:44258","starttimems":1344267873323,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  338. 2012-08-06 11:51:28,999 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":440828,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@4ec324b), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267848170,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  339. 2012-08-06 11:51:29,280 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":425152,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@2bea77e4), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267864127,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  340. 2012-08-06 11:51:29,282 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call get([B@52a05f78, {"timeRange":[0,9223372036854775807],"totalColumns":1,"cacheBlocks":true,"families":{"stats":["count"]},"maxVersions":1,"row":"763297266:und"}), rpc version=1, client version=29, methodsFingerPrint=54742778 from 10.192.98.60:41806: output error
  341. 2012-08-06 11:51:29,285 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 17 on 9009 caught: java.nio.channels.ClosedChannelException
  342.         at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:133)
  343.         at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
  344.         at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1653)
  345.         at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:924)
  346.         at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:1003)
  347.         at org.apache.hadoop.hbase.ipc.HBaseServer$Call.sendResponseIfReady(HBaseServer.java:409)
  348.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1346)
  349.  
  350. 2012-08-06 11:51:29,289 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":415962,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@2beec48b), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.51:44258","starttimems":1344267873326,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  351. 2012-08-06 11:51:29,290 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":441105,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@7f025f5d), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267848184,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  352. 2012-08-06 11:51:29,292 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":441104,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@349c87da), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267848187,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
  353. 2012-08-06 11:51:29,294 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call get([B@4835a20c, {"timeRange":[0,9223372036854775807],"totalColumns":1,"cacheBlocks":true,"families":{"stats":["count"]},"maxVersions":1,"row":"763297266:und"}), rpc version=1, client version=29, methodsFingerPrint=54742778 from 10.192.98.60:41940: output error
  354. 2012-08-06 11:51:29,298 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 36 on 9009 caught: java.nio.channels.ClosedChannelException
  355.         at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:133)
  356.         at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
  357.         at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1653)
  358.         at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:924)
  359.         at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:1003)
  360.         at org.apache.hadoop.hbase.ipc.HBaseServer$Call.sendResponseIfReady(HBaseServer.java:409)
  361.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1346)
  362.  
  363. 2012-08-06 11:51:29,789 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call get([B@54685ac4, {"timeRange":[0,9223372036854775807],"totalColumns":1,"cacheBlocks":true,"families":{"stats":["count"]},"maxVersions":1,"row":"763297266:und"}), rpc version=1, client version=29, methodsFingerPrint=54742778 from 10.192.98.60:42133: output error
  364. 2012-08-06 11:51:29,796 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 48 on 9009 caught: java.nio.channels.ClosedChannelException
  365.         at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:133)
  366.         at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
  367.         at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1653)
  368.         at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:924)
  369.         at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:1003)
  370.         at org.apache.hadoop.hbase.ipc.HBaseServer$Call.sendResponseIfReady(HBaseServer.java:409)
  371.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1346)
  372.  
  373. 2012-08-06 11:51:29,877 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom added to HFile (hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/d3232be635e14fc2ae3f89f5ccefa6da): org.apache.hadoop.hbase.util.CompoundBloomFilterWriter@39be9f73
  374. 2012-08-06 11:51:29,881 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for d3232be635e14fc2ae3f89f5ccefa6da
  375. 2012-08-06 11:51:29,888 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for d3232be635e14fc2ae3f89f5ccefa6da
  376. 2012-08-06 11:51:29,888 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/data/d3232be635e14fc2ae3f89f5ccefa6da, entries=445386, sequenceid=1792264611, memsize=135.9m, filesize=4.5m
  377. 2012-08-06 11:51:29,926 WARN org.apache.hadoop.hdfs.DFSClient: Failed to connect to /10.192.98.60:48359 for file /hbase/Works/1cfc92481a74ec278337871563d189d6/data/1293805bb9e347fca67dde8a03d96245 for block -6262844052051593064:java.io.EOFException
  378.         at java.io.DataInputStream.readShort(DataInputStream.java:298)
  379.         at org.apache.hadoop.hdfs.DFSClient$RemoteBlockReader.newBlockReader(DFSClient.java:1664)
  380.         at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.getBlockReader(DFSClient.java:2383)
  381.         at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.fetchBlockByteRange(DFSClient.java:2272)
  382.         at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2438)
  383.         at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:46)
  384.         at org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1034)
  385.         at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1335)
  386.         at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:266)
  387.         at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209)
  388.         at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:519)
  389.         at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:534)
  390.         at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178)
  391.         at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111)
  392.         at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:83)
  393.         at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1721)
  394.         at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:2865)
  395.         at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1434)
  396.         at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1426)
  397.         at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1402)
  398.         at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:3692)
  399.         at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:3585)
  400.         at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1785)
  401.         at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source)
  402.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  403.         at java.lang.reflect.Method.invoke(Method.java:597)
  404.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
  405.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
  406.  
  407. 2012-08-06 11:51:29,945 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/stats/7497f33631fa4a429133c48399a14421, entries=46, sequenceid=1792264611, memsize=10.1k, filesize=1009.0
  408. 2012-08-06 11:51:29,945 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~135.9m/142545064, currentsize=38.9m/40743208 for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. in 1103ms, sequenceid=1792264611, compaction requested=false
  409. 2012-08-06 11:51:34,318 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom added to HFile (hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/133021bb2ddd41da87656e7b12d2c46b): org.apache.hadoop.hbase.util.CompoundBloomFilterWriter@50ed654a
  410. 2012-08-06 11:51:34,321 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for 133021bb2ddd41da87656e7b12d2c46b
  411. 2012-08-06 11:51:34,321 INFO org.apache.hadoop.hbase.regionserver.Store: Renaming compacted file at hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/133021bb2ddd41da87656e7b12d2c46b to hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/data/133021bb2ddd41da87656e7b12d2c46b
  412. 2012-08-06 11:51:34,327 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for 133021bb2ddd41da87656e7b12d2c46b
  413. 2012-08-06 11:51:35,803 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 3 file(s) in data of Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. into 133021bb2ddd41da87656e7b12d2c46b, size=50.5m; total size for store is 935.1m
  414. 2012-08-06 11:51:35,804 INFO org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: completed compaction: regionName=Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6., storeName=data, fileCount=3, fileSize=60.7m, priority=3, time=15937423726224027; duration=7sec
  415. 2012-08-06 11:51:35,804 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on stats in region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.
  416. 2012-08-06 11:51:35,804 INFO org.apache.hadoop.hbase.regionserver.Store: Starting compaction of 3 file(s) in stats of Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. into tmpdir=hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp, seqid=1792264092, totalSize=4.5m
  417. 2012-08-06 11:51:35,943 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom filter type for hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/0b0bf9015ca544e1801cbe2341c54978: ROW, CompoundBloomFilterWriter
  418. 2012-08-06 11:51:36,548 INFO org.apache.hadoop.hbase.regionserver.Store: Renaming compacted file at hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/c536d9ec3f654bdc8a69bde865e1ea10 to hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/stats/c536d9ec3f654bdc8a69bde865e1ea10
  419. 2012-08-06 11:51:36,607 INFO org.apache.hadoop.hbase.regionserver.Store: Completed major compaction of 3 file(s) in stats of Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. into c536d9ec3f654bdc8a69bde865e1ea10, size=4.5m; total size for store is 4.5m
  420. 2012-08-06 11:51:36,607 INFO org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: completed compaction: regionName=Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6., storeName=stats, fileCount=3, fileSize=4.5m, priority=4, time=15937423726321027; duration=0sec
  421. 2012-08-06 11:51:36,937 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom added to HFile (hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/0b0bf9015ca544e1801cbe2341c54978): org.apache.hadoop.hbase.util.CompoundBloomFilterWriter@7d6297bf
  422. 2012-08-06 11:51:36,941 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for 0b0bf9015ca544e1801cbe2341c54978
  423. 2012-08-06 11:51:36,947 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for 0b0bf9015ca544e1801cbe2341c54978
  424. 2012-08-06 11:51:36,947 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/data/0b0bf9015ca544e1801cbe2341c54978, entries=445430, sequenceid=1792265416, memsize=135.9m, filesize=4.5m
  425. 2012-08-06 11:51:37,006 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/stats/af53e21b8a5b47f9ad0beba0ba2ebc45, entries=146, sequenceid=1792265416, memsize=31.9k, filesize=1.6k
  426. 2012-08-06 11:51:40,308 INFO org.apache.hadoop.hbase.regionserver.HRegion: Fi=false
  427. 2012-08-06 12:19:06,279 INFO org.apache.hadoop.hbase.regionserver.Store: Renaming compacted file at hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/247bb10105924f6e90c42500ab33e2bc to hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/stats/247bb10105924f6e90c42500ab33e2bc
  428. 2012-08-06 12:19:06,333 INFO org.apache.hadoop.hbase.regionserver.Store: Completed major compaction of 3 file(s) in stats of Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. into 247bb10105924f6e90c42500ab33e2bc, size=4.5m; total size for store is 4.5m
  429. 2012-08-06 12:19:06,333 INFO org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: completed compaction: regionName=Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6., storeName=stats, fileCount=3, fileSize=4.5m, priority=4, time=15939074088410027; duration=0sec
  430. 2012-08-06 12:19:13,050 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom filter type for hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/95288f988cea42cfa79b9a716b4b134b: ROW, CompoundBloomFilterWriter
  431. 2012-08-06 12:19:14,147 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom added to HFile (hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/95288f988cea42cfa79b9a716b4b134b): org.apache.hadoop.hbase.util.CompoundBloomFilterWriter@33988e2c
  432. 2012-08-06 12:19:14,151 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for 95288f988cea42cfa79b9a716b4b134b
  433. 2012-08-06 12:19:14,155 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for 95288f988cea42cfa79b9a716b4b134b
  434. 2012-08-06 12:19:14,156 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/data/95288f988cea42cfa79b9a716b4b134b, entries=508741, sequenceid=1792275006, memsize=155.3m, filesize=5.5m
  435. 2012-08-06 12:19:14,183 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/stats/82690dff3ca84bfba7348323bff49df7, entries=18, sequenceid=1792275006, memsize=3.9k, filesize=847.0
  436. 2012-08-06 12:19:14,762 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~155.3m/162810016, currentsize=912.0/912 for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. in 1764ms, sequenceid=1792275006, compaction requested=false
  437. 2012-08-06 12:19:17,141 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1 Got:
  438. java.io.EOFException: hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/.logs/wchddb37pxdu.prod.oclc.org,9009,1337721421002/wchddb37pxdu.prod.oclc.org%2C9009%2C1337721421002.1344269945676, entryStart=36732535, pos=38683136, end=38683136, edit=1
  439.         at sun.reflect.GeneratedConstructorAccessor170.newInstance(Unknown Source)
  440.         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
  441.         at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
  442.         at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.addFileInfoToException(SequenceFileLogReader.java:252)
  443.         at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.next(SequenceFileLogReader.java:208)
  444.         at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.readAllEntriesToReplicateOrNextFile(ReplicationSource.java:427)
  445.         at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:306)
  446. Caused by: java.io.EOFException
  447.         at java.io.DataInputStream.readFully(DataInputStream.java:180)
  448.         at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:63)
  449.         at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:101)
  450.         at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2001)
  451.         at org.apache.hado
  452. -PROD- prodcon@wchddb37pxdu:/prod/hadoop/prod/hbase-0.92.1/logs$
  453. -PROD- prodcon@wchddb37pxdu:/prod/hadoop/prod/hbase-0.92.1/logs$
clone this paste RAW Paste Data