2012-08-06 11:38:28,774 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom filter type for hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/8589b9b3058d46d1a75bb176222fb7a8: ROW, CompoundBloomFilterWriter 2012-08-06 11:38:29,946 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom added to HFile (hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/8589b9b3058d46d1a75bb176222fb7a8): org.apache.hadoop.hbase.util.CompoundBloomFilterWriter@5c88970e 2012-08-06 11:38:29,949 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for 8589b9b3058d46d1a75bb176222fb7a8 2012-08-06 11:38:29,954 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for 8589b9b3058d46d1a75bb176222fb7a8 2012-08-06 11:38:29,954 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/data/8589b9b3058d46d1a75bb176222fb7a8, entries=509142, sequenceid=1792261769, memsize=155.4m, filesize=5.5m 2012-08-06 11:38:29,982 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/stats/24b9e95b3fbe41cdb0e26f94ffa91c52, entries=11, sequenceid=1792261769, memsize=2.4k, filesize=740.0 2012-08-06 11:38:30,522 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~155.4m/162935904, currentsize=680.0/680 for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. in 1806ms, sequenceid=1792261769, compaction requested=false 2012-08-06 11:38:41,188 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom filter type for hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/8e8d795a6d514eb99b26ed6462879c09: ROW, CompoundBloomFilterWriter 2012-08-06 11:38:42,310 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom added to HFile (hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/8e8d795a6d514eb99b26ed6462879c09): org.apache.hadoop.hbase.util.CompoundBloomFilterWriter@75f8d143 2012-08-06 11:38:42,314 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for 8e8d795a6d514eb99b26ed6462879c09 2012-08-06 11:38:42,334 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for 8e8d795a6d514eb99b26ed6462879c09 2012-08-06 11:38:42,334 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/data/8e8d795a6d514eb99b26ed6462879c09, entries=445472, sequenceid=1792262048, memsize=135.9m, filesize=4.9m 2012-08-06 11:38:42,367 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/stats/ee63c14cdafe4f1b82ca76fbf5e22ab8, entries=12, sequenceid=1792262048, memsize=2.6k, filesize=754.0 2012-08-06 11:38:48,245 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1 Got: java.io.EOFException: hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/.logs/wchddb37pxdu.prod.oclc.org,9009,1337721421002/wchddb37pxdu.prod.oclc.org%2C9009%2C1337721421002.1344267499281, entryStart=138461295, pos=144248320, end=144248320, edit=10 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.addFileInfoToException(SequenceFileLogReader.java:252) at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.next(SequenceFileLogReader.java:208) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.readAllEntriesToReplicateOrNextFile(ReplicationSource.java:427) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:306) Caused by: java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:180) at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:63) at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:101) at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2001) at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1901) at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1947) at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.next(SequenceFileLogReader.java:206) ... 2 more 2012-08-06 11:38:48,246 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1 Got EOF while reading, looks like this file is broken? hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/.logs/wchddb37pxdu.prod.oclc.org,9009,1337721421002/wchddb37pxdu.prod.oclc.org%2C9009%2C1337721421002.1344267499281 2012-08-06 11:38:58,576 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1 Got: java.io.EOFException: hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/.logs/wchddb37pxdu.prod.oclc.org,9009,1337721421002/wchddb37pxdu.prod.oclc.org%2C9009%2C1337721421002.1344267499281, entryStart=166527116, pos=172119040, end=172119040, edit=12 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.addFileInfoToException(SequenceFileLogReader.java:252) at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.next(SequenceFileLogReader.java:208) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.readAllEntriesToReplicateOrNextFile(ReplicationSource.java:427) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:306) Caused by: java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:180) at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:63) at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:101) at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2001) at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1901) at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1947) at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.next(SequenceFileLogReader.java:206) ... 2 more 2012-08-06 11:38:58,577 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1 Got EOF while reading, looks like this file is broken? hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/.logs/wchddb37pxdu.prod.oclc.org,9009,1337721421002/wchddb37pxdu.prod.oclc.org%2C9009%2C1337721421002.1344267499281 2012-08-06 11:39:01,450 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 15 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 291.3m is >= than blocking 256.0m size 2012-08-06 11:39:01,829 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 21 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 291.3m is >= than blocking 256.0m size 2012-08-06 11:39:45,314 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 20 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 291.3m is >= than blocking 256.0m size 2012-08-06 11:40:02,976 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 27 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 291.3m is >= than blocking 256.0m size 2012-08-06 11:40:03,420 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 14 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 291.3m is >= than blocking 256.0m size 2012-08-06 11:40:46,397 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 31 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 291.3m is >= than blocking 256.0m size 2012-08-06 11:41:04,501 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 35 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 291.3m is >= than blocking 256.0m size 2012-08-06 11:41:05,250 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 36 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 291.3m is >= than blocking 256.0m size 2012-08-06 11:41:47,484 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 39 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 291.3m is >= than blocking 256.0m size 2012-08-06 11:42:06,202 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 13 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 291.3m is >= than blocking 256.0m size 2012-08-06 11:42:06,300 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 4 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 291.3m is >= than blocking 256.0m size 2012-08-06 11:42:48,489 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 2 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 291.3m is >= than blocking 256.0m size 2012-08-06 11:42:49,705 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 2 on 9009' 2012-08-06 11:42:49,705 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 39 on 9009' 2012-08-06 11:42:49,705 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 4 on 9009' 2012-08-06 11:42:49,705 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 20 on 9009' 2012-08-06 11:42:49,705 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 35 on 9009' 2012-08-06 11:42:49,705 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 27 on 9009' 2012-08-06 11:42:49,705 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~136.0m/142563440, currentsize=155.4m/162930704 for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. in 248567ms, sequenceid=1792262048, compaction requested=true 2012-08-06 11:42:49,706 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 15 on 9009' 2012-08-06 11:42:49,705 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 31 on 9009' 2012-08-06 11:42:49,706 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 21 on 9009' 2012-08-06 11:42:49,705 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 14 on 9009' 2012-08-06 11:42:49,705 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 36 on 9009' 2012-08-06 11:42:49,705 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 13 on 9009' 2012-08-06 11:42:49,706 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on data in region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 2012-08-06 11:42:49,706 INFO org.apache.hadoop.hbase.regionserver.Store: Starting compaction of 3 file(s) in data of Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. into tmpdir=hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp, seqid=1792262048, totalSize=60.9m 2012-08-06 11:42:49,707 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"processingtimems":62222,"ts":9223372036854775807,"client":"10.192.98.34:46807","starttimems":1344267707484,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"delete","totalColumns":1,"table":"Works","families":{"data":[{"timestamp":1344267769706,"qualifier":"ocl:xwc::690708517","vlen":0}]},"row":"763297266:und"} 2012-08-06 11:42:49,708 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"processingtimems":166730,"ts":9223372036854775807,"client":"10.192.98.18:45125","starttimems":1344267602976,"queuetimems":1,"class":"HRegionServer","responsesize":0,"method":"delete","totalColumns":1,"table":"Works","families":{"data":[{"timestamp":1344267769706,"qualifier":"ocl:xwc::690712033","vlen":0}]},"row":"763297266:und"} 2012-08-06 11:42:49,710 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"processingtimems":248290,"client":"10.192.98.34:46807","timeRange":[0,9223372036854775807],"starttimems":1344267521418,"responsesize":9164166,"class":"HRegionServer","table":"Works","cacheBlocks":true,"families":{"data":["ALL"]},"row":"763297266:und","queuetimems":0,"method":"get","totalColumns":1,"maxVersions":1} 2012-08-06 11:42:49,743 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom filter type for hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/442e10d687c74c30b23074e5b034a1ec: ROW, CompoundBloomFilterWriter 2012-08-06 11:42:49,750 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Roll /hbase/.logs/wchddb37pxdu.prod.oclc.org,9009,1337721421002/wchddb37pxdu.prod.oclc.org%2C9009%2C1337721421002.1344267499281, entries=2426, filesize=188830434. for /hbase/.logs/wchddb37pxdu.prod.oclc.org,9009,1337721421002/wchddb37pxdu.prod.oclc.org%2C9009%2C1337721421002.1344267769705 2012-08-06 11:42:49,777 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"processingtimems":105275,"ts":9223372036854775807,"client":"10.192.98.51:44258","starttimems":1344267664501,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"delete","totalColumns":1,"table":"Works","families":{"data":[{"timestamp":1344267769706,"qualifier":"ocl:xwc::690712255","vlen":0}]},"row":"763297266:und"} 2012-08-06 11:42:49,802 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"processingtimems":43502,"ts":9223372036854775807,"client":"10.192.98.18:45125","starttimems":1344267726299,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"delete","totalColumns":1,"table":"Works","families":{"data":[{"timestamp":1344267769790,"qualifier":"ocl:xwc::690712033","vlen":0}]},"row":"763297266:und"} 2012-08-06 11:42:49,803 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"processingtimems":184489,"ts":9223372036854775807,"client":"10.192.98.34:46807","starttimems":1344267585313,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"delete","totalColumns":1,"table":"Works","families":{"data":[{"timestamp":1344267769801,"qualifier":"ocl:xwc::690708517","vlen":0}]},"row":"763297266:und"} 2012-08-06 11:42:49,805 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"processingtimems":228355,"ts":9223372036854775807,"client":"10.192.98.51:44258","starttimems":1344267541449,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"delete","totalColumns":1,"table":"Works","families":{"data":[{"timestamp":1344267769803,"qualifier":"ocl:xwc::690712255","vlen":0}]},"row":"763297266:und"} 2012-08-06 11:42:49,807 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"processingtimems":123410,"ts":9223372036854775807,"client":"10.192.98.34:46807","starttimems":1344267646396,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"delete","totalColumns":1,"table":"Works","families":{"data":[{"timestamp":1344267769804,"qualifier":"ocl:xwc::690708517","vlen":0}]},"row":"763297266:und"} 2012-08-06 11:42:49,809 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"processingtimems":104558,"ts":9223372036854775807,"client":"10.192.98.18:45125","starttimems":1344267665250,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"delete","totalColumns":1,"table":"Works","families":{"data":[{"timestamp":1344267769808,"qualifier":"ocl:xwc::690712033","vlen":0}]},"row":"763297266:und"} 2012-08-06 11:42:49,811 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"processingtimems":166390,"ts":9223372036854775807,"client":"10.192.98.51:44258","starttimems":1344267603420,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"delete","totalColumns":1,"table":"Works","families":{"data":[{"timestamp":1344267769808,"qualifier":"ocl:xwc::690712255","vlen":0}]},"row":"763297266:und"} 2012-08-06 11:42:49,813 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"processingtimems":227983,"ts":9223372036854775807,"client":"10.192.98.18:45125","starttimems":1344267541829,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"delete","totalColumns":1,"table":"Works","families":{"data":[{"timestamp":1344267769810,"qualifier":"ocl:xwc::690712033","vlen":0}]},"row":"763297266:und"} 2012-08-06 11:42:49,814 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"processingtimems":43611,"ts":9223372036854775807,"client":"10.192.98.51:44258","starttimems":1344267726202,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"delete","totalColumns":1,"table":"Works","families":{"data":[{"timestamp":1344267769813,"qualifier":"ocl:xwc::690712255","vlen":0}]},"row":"763297266:und"} 2012-08-06 11:42:49,822 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom filter type for hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/745fe30f00774720b742ec68e208d5de: ROW, CompoundBloomFilterWriter 2012-08-06 11:42:50,985 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom added to HFile (hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/745fe30f00774720b742ec68e208d5de): org.apache.hadoop.hbase.util.CompoundBloomFilterWriter@760576d9 2012-08-06 11:42:50,990 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for 745fe30f00774720b742ec68e208d5de 2012-08-06 11:42:50,996 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for 745fe30f00774720b742ec68e208d5de 2012-08-06 11:42:50,996 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/data/745fe30f00774720b742ec68e208d5de, entries=509106, sequenceid=1792263953, memsize=155.4m, filesize=5.1m 2012-08-06 11:42:51,020 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/stats/c1f2292852ee4915a8ab852e2bfdc710, entries=8, sequenceid=1792263953, memsize=1.8k, filesize=729.0 2012-08-06 11:42:52,316 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~155.4m/162931400, currentsize=2.7k/2760 for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. in 2610ms, sequenceid=1792263953, compaction requested=false 2012-08-06 11:42:55,545 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom added to HFile (hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/442e10d687c74c30b23074e5b034a1ec): org.apache.hadoop.hbase.util.CompoundBloomFilterWriter@79493dc9 2012-08-06 11:42:55,578 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for 442e10d687c74c30b23074e5b034a1ec 2012-08-06 11:42:55,578 INFO org.apache.hadoop.hbase.regionserver.Store: Renaming compacted file at hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/442e10d687c74c30b23074e5b034a1ec to hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/data/442e10d687c74c30b23074e5b034a1ec 2012-08-06 11:42:55,585 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for 442e10d687c74c30b23074e5b034a1ec 2012-08-06 11:42:56,903 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 3 file(s) in data of Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. into 442e10d687c74c30b23074e5b034a1ec, size=50.5m; total size for store is 935.7m 2012-08-06 11:42:56,903 INFO org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: completed compaction: regionName=Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6., storeName=data, fileCount=3, fileSize=60.9m, priority=3, time=15936905292664027; duration=7sec 2012-08-06 11:42:56,903 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on stats in region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 2012-08-06 11:42:56,904 INFO org.apache.hadoop.hbase.regionserver.Store: Starting compaction of 3 file(s) in stats of Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. into tmpdir=hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp, seqid=1792262048, totalSize=4.5m 2012-08-06 11:42:57,603 INFO org.apache.hadoop.hbase.regionserver.Store: Renaming compacted file at hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/ec66a36d18014a9d968426387a250aec to hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/stats/ec66a36d18014a9d968426387a250aec 2012-08-06 11:42:57,664 INFO org.apache.hadoop.hbase.regionserver.Store: Completed major compaction of 3 file(s) in stats of Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. into ec66a36d18014a9d968426387a250aec, size=4.5m; total size for store is 4.5m 2012-08-06 11:42:57,664 INFO org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: completed compaction: regionName=Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6., storeName=stats, fileCount=3, fileSize=4.5m, priority=4, time=15936905292745027; duration=0sec 2012-08-06 11:43:06,864 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom filter type for hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/cd172c73de844f1f91b3ffdf17b52ed2: ROW, CompoundBloomFilterWriter 2012-08-06 11:43:07,966 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom added to HFile (hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/cd172c73de844f1f91b3ffdf17b52ed2): org.apache.hadoop.hbase.util.CompoundBloomFilterWriter@6be954c2 2012-08-06 11:43:07,969 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for cd172c73de844f1f91b3ffdf17b52ed2 2012-08-06 11:43:07,975 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for cd172c73de844f1f91b3ffdf17b52ed2 2012-08-06 11:43:07,975 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/data/cd172c73de844f1f91b3ffdf17b52ed2, entries=509049, sequenceid=1792264092, memsize=155.3m, filesize=5.1m 2012-08-06 11:43:08,069 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/stats/63408310be644929980186d4fcce9c65, entries=9, sequenceid=1792264092, memsize=2.0k, filesize=703.0 2012-08-06 11:43:11,534 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1 Got: java.io.EOFException: hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/.logs/wchddb37pxdu.prod.oclc.org,9009,1337721421002/wchddb37pxdu.prod.oclc.org%2C9009%2C1337721421002.1344267769705, entryStart=82765345, pos=84585984, end=84585984, edit=8 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.addFileInfoToException(SequenceFileLogReader.java:252) at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.next(SequenceFileLogReader.java:208) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.readAllEntriesToReplicateOrNextFile(ReplicationSource.java:427) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:306) Caused by: java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:180) at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:63) at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:101) at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2001) at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1901) at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1947) at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.next(SequenceFileLogReader.java:206) ... 2 more 2012-08-06 11:43:11,535 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1 Got EOF while reading, looks like this file is broken? hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/.logs/wchddb37pxdu.prod.oclc.org,9009,1337721421002/wchddb37pxdu.prod.oclc.org%2C9009%2C1337721421002.1344267769705 2012-08-06 11:43:16,360 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1 Got: java.io.EOFException: hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/.logs/wchddb37pxdu.prod.oclc.org,9009,1337721421002/wchddb37pxdu.prod.oclc.org%2C9009%2C1337721421002.1344267769705, entryStart=101122117, pos=102617600, end=102617600, edit=0 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.addFileInfoToException(SequenceFileLogReader.java:252) at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.next(SequenceFileLogReader.java:208) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.readAllEntriesToReplicateOrNextFile(ReplicationSource.java:393) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:306) Caused by: java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:180) at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:63) at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:101) at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2001) at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1901) at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1947) at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.next(SequenceFileLogReader.java:206) ... 2 more 2012-08-06 11:43:23,019 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 6 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:43:24,022 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 19 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:43:35,229 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 11 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:43:35,229 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 13 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:43:35,229 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 14 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:43:35,230 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 41 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:43:35,230 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 5 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:43:35,230 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 21 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:43:35,230 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 46 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:43:35,291 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 40 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:43:45,005 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 33 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:43:45,008 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 36 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:43:45,009 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 4 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:43:45,009 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 25 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:43:45,010 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 37 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:43:45,014 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 32 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:43:45,015 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 48 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:43:45,017 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 7 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:44:08,170 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 9 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:44:08,181 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 22 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:44:08,182 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 12 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:44:08,184 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 44 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:44:08,184 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 27 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:44:08,187 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 35 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:44:08,187 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 23 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:44:08,189 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 38 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:44:11,100 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 30 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:44:18,624 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 20 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:44:18,624 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 8 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:44:18,630 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 18 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:44:18,699 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 45 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:44:18,700 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 39 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:44:18,704 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 26 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:44:18,705 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 15 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:44:18,708 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 2 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:44:24,127 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 17 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:44:25,246 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 49 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:44:33,318 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 1 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:44:33,323 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 10 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:44:33,324 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 28 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:44:33,324 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 16 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:44:33,324 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 0 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:44:33,326 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 34 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:44:33,332 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 47 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:44:33,333 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 3 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:44:36,345 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 24 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:44:36,346 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 31 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:44:36,347 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 43 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:44:36,348 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 42 on 9009' on region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6.: memstore size 271.9m is >= than blocking 256.0m size 2012-08-06 11:51:28,139 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~155.4m/162905016, currentsize=116.5m/122172880 for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. in 501337ms, sequenceid=1792264092, compaction requested=true 2012-08-06 11:51:28,139 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on data in region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 2012-08-06 11:51:28,140 INFO org.apache.hadoop.hbase.regionserver.Store: Starting compaction of 3 file(s) in data of Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. into tmpdir=hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp, seqid=1792264092, totalSize=60.7m 2012-08-06 11:51:28,140 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 42 on 9009' 2012-08-06 11:51:28,140 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 43 on 9009' 2012-08-06 11:51:28,140 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 31 on 9009' 2012-08-06 11:51:28,140 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 24 on 9009' 2012-08-06 11:51:28,140 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 49 on 9009' 2012-08-06 11:51:28,140 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 40 on 9009' 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 46 on 9009' 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 21 on 9009' 2012-08-06 11:51:28,140 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 11 on 9009' 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 7 on 9009' 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 32 on 9009' 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 37 on 9009' 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 4 on 9009' 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 19 on 9009' 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 3 on 9009' 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 47 on 9009' 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 10 on 9009' 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 28 on 9009' 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 1 on 9009' 2012-08-06 11:51:28,142 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 6 on 9009' 2012-08-06 11:51:28,142 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 26 on 9009' 2012-08-06 11:51:28,142 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 15 on 9009' 2012-08-06 11:51:28,142 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 2 on 9009' 2012-08-06 11:51:28,142 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 39 on 9009' 2012-08-06 11:51:28,142 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 45 on 9009' 2012-08-06 11:51:28,142 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 18 on 9009' 2012-08-06 11:51:28,143 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 20 on 9009' 2012-08-06 11:51:28,143 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":411795,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@2b400c61), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267876347,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 13 on 9009' 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 14 on 9009' 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 41 on 9009' 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 5 on 9009' 2012-08-06 11:51:28,143 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 12 on 9009' 2012-08-06 11:51:28,144 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":411798,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@28e005dc), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267876345,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,143 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 44 on 9009' 2012-08-06 11:51:28,143 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 35 on 9009' 2012-08-06 11:51:28,143 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 27 on 9009' 2012-08-06 11:51:28,143 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 23 on 9009' 2012-08-06 11:51:28,143 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 38 on 9009' 2012-08-06 11:51:28,145 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":414811,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@78fb735b), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.51:44258","starttimems":1344267873333,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,145 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":472915,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@ad62131), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267815230,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,146 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":411799,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@701d1d85), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267876346,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,143 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 8 on 9009' 2012-08-06 11:51:28,143 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":463125,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@6a676982), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267825017,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,142 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":411793,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@4513c778), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267876348,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,142 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 30 on 9009' 2012-08-06 11:51:28,141 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"processingtimems":501069,"client":"10.192.98.18:45125","timeRange":[0,9223372036854775807],"starttimems":1344267787070,"responsesize":9163014,"class":"HRegionServer","table":"Works","cacheBlocks":true,"families":{"data":["ALL"]},"row":"763297266:und","queuetimems":1,"method":"get","totalColumns":1,"maxVersions":1} 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 16 on 9009' 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 0 on 9009' 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 34 on 9009' 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 17 on 9009' 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 36 on 9009' 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 33 on 9009' 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 25 on 9009' 2012-08-06 11:51:28,147 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":472916,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@75131419), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267815230,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,147 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":463132,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@378cb15b), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267825014,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,141 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 48 on 9009' 2012-08-06 11:51:28,147 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":429442,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@769dd905), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.51:44258","starttimems":1344267858704,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,146 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"processingtimems":422899,"ts":9223372036854775807,"client":"10.192.98.51:44258","starttimems":1344267865246,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"delete","totalColumns":1,"table":"Works","families":{"data":[{"timestamp":1344268288140,"qualifier":"ocl:xwc::690712263","vlen":0}]},"row":"763297266:und"} 2012-08-06 11:51:28,145 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":414827,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@6781c633), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.51:44258","starttimems":1344267873317,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,145 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 9 on 9009' 2012-08-06 11:51:28,144 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 'IPC Server handler 22 on 9009' 2012-08-06 11:51:28,144 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":472852,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@ba0c37b), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267815291,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,149 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":472919,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@166b7164), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267815229,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,150 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":429526,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@7cb051e6), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.51:44258","starttimems":1344267858623,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,150 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":463139,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@1bd112fe), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267825010,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,151 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":429521,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@39c4a668), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.51:44258","starttimems":1344267858630,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,153 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":439965,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@1441423e), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267848187,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,153 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call get([B@564d6e0, {"timeRange":[0,9223372036854775807],"totalColumns":1,"cacheBlocks":true,"families":{"stats":["count"]},"maxVersions":1,"row":"763297266:und"}), rpc version=1, client version=29, methodsFingerPrint=54742778 from 10.192.98.60:55529: output error 2012-08-06 11:51:28,150 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"processingtimems":484127,"ts":9223372036854775807,"client":"10.192.98.51:44258","starttimems":1344267804022,"queuetimems":1,"class":"HRegionServer","responsesize":0,"method":"delete","totalColumns":1,"table":"Works","families":{"data":[{"timestamp":1344268288146,"qualifier":"ocl:xwc::690712263","vlen":0}]},"row":"763297266:und"} 2012-08-06 11:51:28,161 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":472931,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@3fc30d9f), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267815229,"queuetimems":1,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,161 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":429452,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@5f4d4222), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.51:44258","starttimems":1344267858708,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,158 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 37 on 9009 caught: java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:133) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324) at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1653) at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:924) at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:1003) at org.apache.hadoop.hbase.ipc.HBaseServer$Call.sendResponseIfReady(HBaseServer.java:409) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1346) 2012-08-06 11:51:28,162 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":439972,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@4930f70f), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267848189,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,161 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":414828,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@265428f6), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.51:44258","starttimems":1344267873332,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,161 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":472930,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@1302f1da), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267815230,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,163 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":472933,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@5f046b28), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267815229,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,163 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":439971,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@768a849a), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267848182,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,164 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":414830,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@68ac091d), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.51:44258","starttimems":1344267873323,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,165 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom filter type for hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/133021bb2ddd41da87656e7b12d2c46b: ROW, CompoundBloomFilterWriter 2012-08-06 11:51:28,168 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Roll /hbase/.logs/wchddb37pxdu.prod.oclc.org,9009,1337721421002/wchddb37pxdu.prod.oclc.org%2C9009%2C1337721421002.1344267769705, entries=637, filesize=129608690. for /hbase/.logs/wchddb37pxdu.prod.oclc.org,9009,1337721421002/wchddb37pxdu.prod.oclc.org%2C9009%2C1337721421002.1344268288139 2012-08-06 11:51:28,187 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":414848,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@1bac39d), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.51:44258","starttimems":1344267873324,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,187 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":463164,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@2afec9b7), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267825008,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,187 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":472943,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@e3bf6dd), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267815229,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,188 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":429480,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@2946da24), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.51:44258","starttimems":1344267858699,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,188 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":463170,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@6a9b86a7), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267825009,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,188 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":429475,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@8a5749f), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.51:44258","starttimems":1344267858704,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,387 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call get([B@6999eae8, {"timeRange":[0,9223372036854775807],"totalColumns":1,"cacheBlocks":true,"families":{"stats":["count"]},"maxVersions":1,"row":"763297266:und"}), rpc version=1, client version=29, methodsFingerPrint=54742778 from 10.192.98.60:55805: output error 2012-08-06 11:51:28,392 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 13 on 9009 caught: java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:133) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324) at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1653) at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:924) at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:1003) at org.apache.hadoop.hbase.ipc.HBaseServer$Call.sendResponseIfReady(HBaseServer.java:409) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1346) 2012-08-06 11:51:28,843 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":485824,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@11dee2bb), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267803018,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,843 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":415519,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@651410f), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.51:44258","starttimems":1344267873323,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,844 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"processingtimems":437743,"ts":9223372036854775807,"client":"10.192.98.18:45125","starttimems":1344267851100,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"delete","totalColumns":1,"table":"Works","families":{"data":[{"timestamp":1344268288842,"qualifier":"ocl:xwc::690712037","vlen":0}]},"row":"763297266:und"} 2012-08-06 11:51:28,848 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":463838,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@518515a8), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267825009,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,849 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":430149,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@7a6b3cef), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.51:44258","starttimems":1344267858699,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,850 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":440665,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@13bda00d), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267848184,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,850 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":463844,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@52c32f37), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267825005,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,851 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call get([B@101ad9e8, {"timeRange":[0,9223372036854775807],"totalColumns":1,"cacheBlocks":true,"families":{"stats":["count"]},"maxVersions":1,"row":"763297266:und"}), rpc version=1, client version=29, methodsFingerPrint=54742778 from 10.192.98.60:55956: output error 2012-08-06 11:51:28,855 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 25 on 9009 caught: java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:133) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324) at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1653) at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:924) at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:1003) at org.apache.hadoop.hbase.ipc.HBaseServer$Call.sendResponseIfReady(HBaseServer.java:409) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1346) 2012-08-06 11:51:28,857 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":463841,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@78ab6d2b), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267825015,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,926 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom filter type for hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/d3232be635e14fc2ae3f89f5ccefa6da: ROW, CompoundBloomFilterWriter 2012-08-06 11:51:28,995 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":430370,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@70e6feb6), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.51:44258","starttimems":1344267858624,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,995 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":440813,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@7a28db5b), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267848181,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,995 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":415671,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@10597205), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.51:44258","starttimems":1344267873323,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:28,999 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":440828,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@4ec324b), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267848170,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:29,280 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":425152,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@2bea77e4), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267864127,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:29,282 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call get([B@52a05f78, {"timeRange":[0,9223372036854775807],"totalColumns":1,"cacheBlocks":true,"families":{"stats":["count"]},"maxVersions":1,"row":"763297266:und"}), rpc version=1, client version=29, methodsFingerPrint=54742778 from 10.192.98.60:41806: output error 2012-08-06 11:51:29,285 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 17 on 9009 caught: java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:133) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324) at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1653) at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:924) at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:1003) at org.apache.hadoop.hbase.ipc.HBaseServer$Call.sendResponseIfReady(HBaseServer.java:409) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1346) 2012-08-06 11:51:29,289 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":415962,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@2beec48b), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.51:44258","starttimems":1344267873326,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:29,290 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":441105,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@7f025f5d), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267848184,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:29,292 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":441104,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@349c87da), rpc version=1, client version=29, methodsFingerPrint=54742778","client":"10.192.98.34:46807","starttimems":1344267848187,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} 2012-08-06 11:51:29,294 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call get([B@4835a20c, {"timeRange":[0,9223372036854775807],"totalColumns":1,"cacheBlocks":true,"families":{"stats":["count"]},"maxVersions":1,"row":"763297266:und"}), rpc version=1, client version=29, methodsFingerPrint=54742778 from 10.192.98.60:41940: output error 2012-08-06 11:51:29,298 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 36 on 9009 caught: java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:133) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324) at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1653) at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:924) at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:1003) at org.apache.hadoop.hbase.ipc.HBaseServer$Call.sendResponseIfReady(HBaseServer.java:409) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1346) 2012-08-06 11:51:29,789 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call get([B@54685ac4, {"timeRange":[0,9223372036854775807],"totalColumns":1,"cacheBlocks":true,"families":{"stats":["count"]},"maxVersions":1,"row":"763297266:und"}), rpc version=1, client version=29, methodsFingerPrint=54742778 from 10.192.98.60:42133: output error 2012-08-06 11:51:29,796 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 48 on 9009 caught: java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:133) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324) at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1653) at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:924) at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:1003) at org.apache.hadoop.hbase.ipc.HBaseServer$Call.sendResponseIfReady(HBaseServer.java:409) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1346) 2012-08-06 11:51:29,877 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom added to HFile (hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/d3232be635e14fc2ae3f89f5ccefa6da): org.apache.hadoop.hbase.util.CompoundBloomFilterWriter@39be9f73 2012-08-06 11:51:29,881 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for d3232be635e14fc2ae3f89f5ccefa6da 2012-08-06 11:51:29,888 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for d3232be635e14fc2ae3f89f5ccefa6da 2012-08-06 11:51:29,888 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/data/d3232be635e14fc2ae3f89f5ccefa6da, entries=445386, sequenceid=1792264611, memsize=135.9m, filesize=4.5m 2012-08-06 11:51:29,926 WARN org.apache.hadoop.hdfs.DFSClient: Failed to connect to /10.192.98.60:48359 for file /hbase/Works/1cfc92481a74ec278337871563d189d6/data/1293805bb9e347fca67dde8a03d96245 for block -6262844052051593064:java.io.EOFException at java.io.DataInputStream.readShort(DataInputStream.java:298) at org.apache.hadoop.hdfs.DFSClient$RemoteBlockReader.newBlockReader(DFSClient.java:1664) at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.getBlockReader(DFSClient.java:2383) at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.fetchBlockByteRange(DFSClient.java:2272) at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2438) at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:46) at org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1034) at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1335) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:266) at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:209) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:519) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:534) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:178) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:111) at org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:83) at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1721) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:2865) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1434) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1426) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1402) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:3692) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:3585) at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1785) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326) 2012-08-06 11:51:29,945 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/stats/7497f33631fa4a429133c48399a14421, entries=46, sequenceid=1792264611, memsize=10.1k, filesize=1009.0 2012-08-06 11:51:29,945 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~135.9m/142545064, currentsize=38.9m/40743208 for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. in 1103ms, sequenceid=1792264611, compaction requested=false 2012-08-06 11:51:34,318 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom added to HFile (hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/133021bb2ddd41da87656e7b12d2c46b): org.apache.hadoop.hbase.util.CompoundBloomFilterWriter@50ed654a 2012-08-06 11:51:34,321 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for 133021bb2ddd41da87656e7b12d2c46b 2012-08-06 11:51:34,321 INFO org.apache.hadoop.hbase.regionserver.Store: Renaming compacted file at hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/133021bb2ddd41da87656e7b12d2c46b to hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/data/133021bb2ddd41da87656e7b12d2c46b 2012-08-06 11:51:34,327 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for 133021bb2ddd41da87656e7b12d2c46b 2012-08-06 11:51:35,803 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 3 file(s) in data of Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. into 133021bb2ddd41da87656e7b12d2c46b, size=50.5m; total size for store is 935.1m 2012-08-06 11:51:35,804 INFO org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: completed compaction: regionName=Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6., storeName=data, fileCount=3, fileSize=60.7m, priority=3, time=15937423726224027; duration=7sec 2012-08-06 11:51:35,804 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on stats in region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. 2012-08-06 11:51:35,804 INFO org.apache.hadoop.hbase.regionserver.Store: Starting compaction of 3 file(s) in stats of Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. into tmpdir=hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp, seqid=1792264092, totalSize=4.5m 2012-08-06 11:51:35,943 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom filter type for hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/0b0bf9015ca544e1801cbe2341c54978: ROW, CompoundBloomFilterWriter 2012-08-06 11:51:36,548 INFO org.apache.hadoop.hbase.regionserver.Store: Renaming compacted file at hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/c536d9ec3f654bdc8a69bde865e1ea10 to hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/stats/c536d9ec3f654bdc8a69bde865e1ea10 2012-08-06 11:51:36,607 INFO org.apache.hadoop.hbase.regionserver.Store: Completed major compaction of 3 file(s) in stats of Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. into c536d9ec3f654bdc8a69bde865e1ea10, size=4.5m; total size for store is 4.5m 2012-08-06 11:51:36,607 INFO org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: completed compaction: regionName=Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6., storeName=stats, fileCount=3, fileSize=4.5m, priority=4, time=15937423726321027; duration=0sec 2012-08-06 11:51:36,937 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom added to HFile (hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/0b0bf9015ca544e1801cbe2341c54978): org.apache.hadoop.hbase.util.CompoundBloomFilterWriter@7d6297bf 2012-08-06 11:51:36,941 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for 0b0bf9015ca544e1801cbe2341c54978 2012-08-06 11:51:36,947 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for 0b0bf9015ca544e1801cbe2341c54978 2012-08-06 11:51:36,947 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/data/0b0bf9015ca544e1801cbe2341c54978, entries=445430, sequenceid=1792265416, memsize=135.9m, filesize=4.5m 2012-08-06 11:51:37,006 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/stats/af53e21b8a5b47f9ad0beba0ba2ebc45, entries=146, sequenceid=1792265416, memsize=31.9k, filesize=1.6k 2012-08-06 11:51:40,308 INFO org.apache.hadoop.hbase.regionserver.HRegion: Fi=false 2012-08-06 12:19:06,279 INFO org.apache.hadoop.hbase.regionserver.Store: Renaming compacted file at hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/247bb10105924f6e90c42500ab33e2bc to hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/stats/247bb10105924f6e90c42500ab33e2bc 2012-08-06 12:19:06,333 INFO org.apache.hadoop.hbase.regionserver.Store: Completed major compaction of 3 file(s) in stats of Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. into 247bb10105924f6e90c42500ab33e2bc, size=4.5m; total size for store is 4.5m 2012-08-06 12:19:06,333 INFO org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: completed compaction: regionName=Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6., storeName=stats, fileCount=3, fileSize=4.5m, priority=4, time=15939074088410027; duration=0sec 2012-08-06 12:19:13,050 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom filter type for hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/95288f988cea42cfa79b9a716b4b134b: ROW, CompoundBloomFilterWriter 2012-08-06 12:19:14,147 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Bloom added to HFile (hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/.tmp/95288f988cea42cfa79b9a716b4b134b): org.apache.hadoop.hbase.util.CompoundBloomFilterWriter@33988e2c 2012-08-06 12:19:14,151 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for 95288f988cea42cfa79b9a716b4b134b 2012-08-06 12:19:14,155 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROW CompoundBloomFilter metadata for 95288f988cea42cfa79b9a716b4b134b 2012-08-06 12:19:14,156 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/data/95288f988cea42cfa79b9a716b4b134b, entries=508741, sequenceid=1792275006, memsize=155.3m, filesize=5.5m 2012-08-06 12:19:14,183 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/Works/1cfc92481a74ec278337871563d189d6/stats/82690dff3ca84bfba7348323bff49df7, entries=18, sequenceid=1792275006, memsize=3.9k, filesize=847.0 2012-08-06 12:19:14,762 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~155.3m/162810016, currentsize=912.0/912 for region Works,73658752:eng,1337915462169.1cfc92481a74ec278337871563d189d6. in 1764ms, sequenceid=1792275006, compaction requested=false 2012-08-06 12:19:17,141 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1 Got: java.io.EOFException: hdfs://wchddb01pxdu.prod.oclc.org:9008/hbase/.logs/wchddb37pxdu.prod.oclc.org,9009,1337721421002/wchddb37pxdu.prod.oclc.org%2C9009%2C1337721421002.1344269945676, entryStart=36732535, pos=38683136, end=38683136, edit=1 at sun.reflect.GeneratedConstructorAccessor170.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.addFileInfoToException(SequenceFileLogReader.java:252) at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.next(SequenceFileLogReader.java:208) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.readAllEntriesToReplicateOrNextFile(ReplicationSource.java:427) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:306) Caused by: java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:180) at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:63) at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:101) at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2001) at org.apache.hado -PROD- prodcon@wchddb37pxdu:/prod/hadoop/prod/hbase-0.92.1/logs$ -PROD- prodcon@wchddb37pxdu:/prod/hadoop/prod/hbase-0.92.1/logs$