API tools faq
paste
Advertisement
V_R

Large 100M table index failure

V_R
Oct 2nd, 2013
88
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 12.08 KB | None | 0 0
raw download clone embed print report
  1. 013-10-02 21:10:20,565 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":12979,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@7b7a3cfa), rpc version=1, client version=29, methodsFingerPrint=-1368823753","client":"10.202.50.48:46216","starttimems":1380748207583,"queuetimems":1,"class":"HRegionServer","responsesize":0,"method":"multi"}
  2. 2013-10-02 21:10:20,723 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":12802,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@3c93845e), rpc version=1, client version=29, methodsFingerPrint=-1368823753","client":"10.202.50.48:46216","starttimems":1380748207920,"queuetimems":6,"class":"HRegionServer","responsesize":0,"method":"multi"}
  3. 2013-10-02 21:10:20,724 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":12753,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@168bb90), rpc version=1, client version=29, methodsFingerPrint=-1368823753","client":"10.202.50.48:46216","starttimems":1380748207970,"queuetimems":22,"class":"HRegionServer","responsesize":0,"method":"multi"}
  4. 2013-10-02 21:10:21,819 INFO org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom and NO DeleteFamily was added to HFile (hdfs://sjc1-eng-perf-g1-name01.carrieriq.com:9000/hbase/DB_IDX_LARGE/7ab578edd1c25f123fe94ef60229a424/.tmp/c050c8ed1399427cad1a7b1eedfcf609)
  5. 2013-10-02 21:10:21,819 INFO org.apache.hadoop.hbase.regionserver.Store: Flushed , sequenceid=405586893, memsize=257.4 M, into tmp file hdfs://sjc1-eng-perf-g1-name01.carrieriq.com:9000/hbase/DB_IDX_LARGE/7ab578edd1c25f123fe94ef60229a424/.tmp/c050c8ed1399427cad1a7b1eedfcf609
  6. 2013-10-02 21:10:22,010 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://sjc1-eng-perf-g1-name01.carrieriq.com:9000/hbase/DB_IDX_LARGE/7ab578edd1c25f123fe94ef60229a424/_0/c050c8ed1399427cad1a7b1eedfcf609, entries=1369760, sequenceid=405586893, filesize=13.9 M
  7. 2013-10-02 21:10:22,017 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~241.9 M/253693472, currentsize=134.3 M/140807800 for region DB_IDX_LARGE,,1380747793249.7ab578edd1c25f123fe94ef60229a424. in 19869ms, sequenceid=405586893, compaction requested=false
  8. 2013-10-02 21:10:22,019 WARN org.apache.hadoop.hdfs.DomainSocketFactory: The short-circuit local reads feature is disabled because libhadoop cannot be loaded.
  9. 2013-10-02 21:10:22,047 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Roll /hbase/.logs/sjc1-eng-perf-g1-grid01.carrieriq.com,60020,1380652581548/sjc1-eng-perf-g1-grid01.carrieriq.com%2C60020%2C1380652581548.1380748201137, entries=198, filesize=91381786. for /hbase/.logs/sjc1-eng-perf-g1-grid01.carrieriq.com,60020,1380652581548/sjc1-eng-perf-g1-grid01.carrieriq.com%2C60020%2C1380652581548.1380748222017
  10. 2013-10-02 21:10:25,048 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter type for hdfs://sjc1-eng-perf-g1-name01.carrieriq.com:9000/hbase/DB_IDX_LARGE/7ab578edd1c25f123fe94ef60229a424/.tmp/2e58ef9fef8a443eafa1d1a9e08cff86: CompoundBloomFilterWriter
  11. 2013-10-02 21:10:27,262 INFO org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom and NO DeleteFamily was added to HFile (hdfs://sjc1-eng-perf-g1-name01.carrieriq.com:9000/hbase/DB_IDX_LARGE/7ab578edd1c25f123fe94ef60229a424/.tmp/2e58ef9fef8a443eafa1d1a9e08cff86)
  12. 2013-10-02 21:10:27,262 INFO org.apache.hadoop.hbase.regionserver.Store: Flushed , sequenceid=405587151, memsize=172.7 M, into tmp file hdfs://sjc1-eng-perf-g1-name01.carrieriq.com:9000/hbase/DB_IDX_LARGE/7ab578edd1c25f123fe94ef60229a424/.tmp/2e58ef9fef8a443eafa1d1a9e08cff86
  13. 2013-10-02 21:10:27,439 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://sjc1-eng-perf-g1-name01.carrieriq.com:9000/hbase/DB_IDX_LARGE/7ab578edd1c25f123fe94ef60229a424/_0/2e58ef9fef8a443eafa1d1a9e08cff86, entries=918725, sequenceid=405587151, filesize=9.6 M
  14. 2013-10-02 21:10:27,440 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~192.4 M/201758856, currentsize=49.4 M/51817896 for region DB_IDX_LARGE,,1380747793249.7ab578edd1c25f123fe94ef60229a424. in 2648ms, sequenceid=405587151, compaction requested=true
  15. 2013-10-02 21:10:27,443 WARN org.apache.hadoop.hdfs.DomainSocketFactory: The short-circuit local reads feature is disabled because libhadoop cannot be loaded.
  16. 2013-10-02 21:10:27,484 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Roll /hbase/.logs/sjc1-eng-perf-g1-grid01.carrieriq.com,60020,1380652581548/sjc1-eng-perf-g1-grid01.carrieriq.com%2C60020%2C1380652581548.1380748222017, entries=154, filesize=70110446. for /hbase/.logs/sjc1-eng-perf-g1-grid01.carrieriq.com,60020,1380652581548/sjc1-eng-perf-g1-grid01.carrieriq.com%2C60020%2C1380652581548.1380748227440
  17. 2013-10-02 21:10:30,087 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call multi(org.apache.hadoop.hbase.client.MultiAction@3fdc8e9d), rpc version=1, client version=29, methodsFingerPrint=-1368823753 from 10.202.50.48:46216: output error
  18. 2013-10-02 21:10:30,088 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 20 on 60020 caught: java.io.IOException: Broken pipe
  19. at sun.nio.ch.FileDispatcher.write0(Native Method)
  20. at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
  21. at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69)
  22. at sun.nio.ch.IOUtil.write(IOUtil.java:40)
  23. at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:336)
  24. at org.apache.hadoop.hbase.ipc.HBaseServer.channelIO(HBaseServer.java:1830)
  25. at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1773)
  26. at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:967)
  27. at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:1053)
  28. at org.apache.hadoop.hbase.ipc.HBaseServer$Call.sendResponseIfReady(HBaseServer.java:453)
  29. at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1446)
  30.  
  31. 2013-10-02 21:10:30,287 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server listener on 60020: readAndProcess threw exception java.io.IOException: Connection reset by peer. Count of bytes read: 0
  32. java.io.IOException: Connection reset by peer
  33. at sun.nio.ch.FileDispatcher.read0(Native Method)
  34. at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
  35. at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
  36. at sun.nio.ch.IOUtil.read(IOUtil.java:171)
  37. at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:245)
  38. at org.apache.hadoop.hbase.ipc.HBaseServer.channelIO(HBaseServer.java:1830)
  39. at org.apache.hadoop.hbase.ipc.HBaseServer.channelRead(HBaseServer.java:1796)
  40. at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:1220)
  41. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:748)
  42. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:539)
  43. at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
  44. at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  45. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  46. at java.lang.Thread.run(Thread.java:662)
  47. 2013-10-02 21:10:49,330 INFO org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Waited 90594ms on a compaction to clean up 'too many store files'; waited long enough... proceeding with flush of DB_IDX_LARGE,\xC2\x0B\x02\x00CSApple.com\x00Login\x00\x00\x00\x01E\xBA"\xBD\x10,1380747793249.e564b26ba2d7e2996ee0145069da182a.
  48. 2013-10-02 21:10:50,048 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter type for hdfs://sjc1-eng-perf-g1-name01.carrieriq.com:9000/hbase/DB_IDX_LARGE/e564b26ba2d7e2996ee0145069da182a/.tmp/7ab98ce4496043dba8bb7090528a4a25: CompoundBloomFilterWriter
  49. 2013-10-02 21:10:57,759 INFO org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom and NO DeleteFamily was added to HFile (hdfs://sjc1-eng-perf-g1-name01.carrieriq.com:9000/hbase/DB_IDX_LARGE/e564b26ba2d7e2996ee0145069da182a/.tmp/7ab98ce4496043dba8bb7090528a4a25)
  50. 2013-10-02 21:10:57,759 INFO org.apache.hadoop.hbase.regionserver.Store: Flushed , sequenceid=405587301, memsize=867.4 M, into tmp file hdfs://sjc1-eng-perf-g1-name01.carrieriq.com:9000/hbase/DB_IDX_LARGE/e564b26ba2d7e2996ee0145069da182a/.tmp/7ab98ce4496043dba8bb7090528a4a25
  51. 2013-10-02 21:10:57,772 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://sjc1-eng-perf-g1-name01.carrieriq.com:9000/hbase/DB_IDX_LARGE/e564b26ba2d7e2996ee0145069da182a/_0/7ab98ce4496043dba8bb7090528a4a25, entries=4614741, sequenceid=405587301, filesize=44.8 M
  52. 2013-10-02 21:10:57,773 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~870.3 M/912567368, currentsize=0/0 for region DB_IDX_LARGE,\xC2\x0B\x02\x00CSApple.com\x00Login\x00\x00\x00\x01E\xBA"\xBD\x10,1380747793249.e564b26ba2d7e2996ee0145069da182a. in 8443ms, sequenceid=405587301, compaction requested=false
  53. 2013-10-02 21:11:03,177 INFO org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom and NO DeleteFamily was added to HFile (hdfs://sjc1-eng-perf-g1-name01.carrieriq.com:9000/hbase/DB_IDX_LARGE/e564b26ba2d7e2996ee0145069da182a/.tmp/576f013182bb4c05a31189d926bab0b2)
  54. 2013-10-02 21:11:03,184 INFO org.apache.hadoop.hbase.regionserver.Store: Renaming compacted file at hdfs://sjc1-eng-perf-g1-name01.carrieriq.com:9000/hbase/DB_IDX_LARGE/e564b26ba2d7e2996ee0145069da182a/.tmp/576f013182bb4c05a31189d926bab0b2 to hdfs://sjc1-eng-perf-g1-name01.carrieriq.com:9000/hbase/DB_IDX_LARGE/e564b26ba2d7e2996ee0145069da182a/_0/576f013182bb4c05a31189d926bab0b2
  55. 2013-10-02 21:11:03,275 INFO org.apache.hadoop.hbase.regionserver.Store: Completed major compaction of 8 file(s) in _0 of DB_IDX_LARGE,\xC2\x0B\x02\x00CSApple.com\x00Login\x00\x00\x00\x01E\xBA"\xBD\x10,1380747793249.e564b26ba2d7e2996ee0145069da182a. into 576f013182bb4c05a31189d926bab0b2, size=224.0 M; total size for store is 311.7 M
  56. 2013-10-02 21:11:03,275 INFO org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: completed compaction: regionName=DB_IDX_LARGE,\xC2\x0B\x02\x00CSApple.com\x00Login\x00\x00\x00\x01E\xBA"\xBD\x10,1380747793249.e564b26ba2d7e2996ee0145069da182a., storeName=_0, fileCount=8, fileSize=229.6 M, priority=-1, time=11128113639171386; duration=1mins, 15sec
  57. 2013-10-02 21:11:03,276 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on _0 in region DB_IDX_LARGE,,1380747793249.7ab578edd1c25f123fe94ef60229a424.
  58. 2013-10-02 21:11:03,276 INFO org.apache.hadoop.hbase.regionserver.Store: Starting compaction of 3 file(s) in _0 of DB_IDX_LARGE,,1380747793249.7ab578edd1c25f123fe94ef60229a424. into tmpdir=hdfs://sjc1-eng-perf-g1-name01.carrieriq.com:9000/hbase/DB_IDX_LARGE/7ab578edd1c25f123fe94ef60229a424/.tmp, seqid=405587151, totalSize=80.7 M
  59. 2013-10-02 21:11:03,294 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter type for hdfs://sjc1-eng-perf-g1-name01.carrieriq.com:9000/hbase/DB_IDX_LARGE/7ab578edd1c25f123fe94ef60229a424/.tmp/5158a674fe2a4034ab7cadf1e9c7a3ef: CompoundBloomFilterWriter
  60. 2013-10-02 21:11:19,319 INFO org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom and NO DeleteFamily was added to HFile (hdfs://sjc1-eng-perf-g1-name01.carrieriq.com:9000/hbase/DB_IDX_LARGE/7ab578edd1c25f123fe94ef60229a424/.tmp/5158a674fe2a4034ab7cadf1e9c7a3ef)
  61. 2013-10-02 21:11:19,324 INFO org.apache.hadoop.hbase.regionserver.Store: Renaming compacted file at hdfs://sjc1-eng-perf-g1-name01.carrieriq.com:9000/hbase/DB_IDX_LARGE/7ab578edd1c25f123fe94ef60229a424/.tmp/5158a674fe2a4034ab7cadf1e9c7a3ef to hdfs://sjc1-eng-perf-g1-name01.carrieriq.com:9000/hbase/DB_IDX_LARGE/7ab578edd1c25f123fe94ef60229a424/_0/5158a674fe2a4034ab7cadf1e9c7a3ef
  62. 2013-10-02 21:11:19,355 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 3 file(s) in _0 of DB_IDX_LARGE,,1380747793249.7ab578edd1c25f123fe94ef60229a424. into 5158a674fe2a4034ab7cadf1e9c7a3ef, size=79.1 M; total size for store is 306.4 M
  63. 2013-10-02 21:11:19,355 INFO org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: completed compaction: regionName=DB_IDX_LARGE,,1380747793249.7ab578edd1c25f123fe94ef60229a424., storeName=_0, fileCount=3, fileSize=80.7 M, priority=3, time=11128275126799386; duration=16sec
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!