INFO [ScheduledTasks:1] 2012-06-01 02:44:28,910 GCInspector.java (line 122) GC for ParNew: 337 ms for 1 collections, 3576915528 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:44:31,150 GCInspector.java (line 122) GC for ParNew: 489 ms for 1 collections, 3626474888 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:44:33,629 GCInspector.java (line 122) GC for ParNew: 493 ms for 1 collections, 3699560416 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:44:36,630 GCInspector.java (line 122) GC for ParNew: 491 ms for 1 collections, 3885584352 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:44:37,771 GCInspector.java (line 122) GC for ParNew: 484 ms for 1 collections, 3840039632 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:44:39,216 GCInspector.java (line 122) GC for ParNew: 491 ms for 1 collections, 3915147440 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:44:41,217 GCInspector.java (line 122) GC for ParNew: 500 ms for 1 collections, 4136255504 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:44:42,297 GCInspector.java (line 122) GC for ParNew: 487 ms for 1 collections, 4059457576 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:44:44,317 GCInspector.java (line 122) GC for ParNew: 489 ms for 1 collections, 4277378984 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:44:45,427 GCInspector.java (line 122) GC for ParNew: 490 ms for 1 collections, 4202695376 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:44:47,428 GCInspector.java (line 122) GC for ParNew: 481 ms for 1 collections, 4508279232 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:44:48,582 GCInspector.java (line 122) GC for ParNew: 483 ms for 1 collections, 4456140896 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:44:49,583 GCInspector.java (line 122) GC for ParNew: 471 ms for 1 collections, 4308056960 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:44:50,744 GCInspector.java (line 122) GC for ParNew: 472 ms for 1 collections, 4154909376 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:44:51,968 GCInspector.java (line 122) GC for ParNew: 481 ms for 1 collections, 3962325816 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:44:53,205 GCInspector.java (line 122) GC for ParNew: 487 ms for 1 collections, 3752205336 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:44:54,464 GCInspector.java (line 122) GC for ParNew: 475 ms for 1 collections, 3526183272 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:44:55,755 GCInspector.java (line 122) GC for ParNew: 477 ms for 1 collections, 3334173920 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:44:57,029 GCInspector.java (line 122) GC for ParNew: 478 ms for 1 collections, 3302774152 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:44:58,047 GCInspector.java (line 122) GC for ParNew: 448 ms for 1 collections, 3297815368 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:44:59,048 GCInspector.java (line 122) GC for ParNew: 436 ms for 1 collections, 3374819440 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:45:00,070 GCInspector.java (line 122) GC for ParNew: 450 ms for 1 collections, 3442053288 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:45:01,121 GCInspector.java (line 122) GC for ParNew: 457 ms for 1 collections, 3514939504 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:45:02,158 GCInspector.java (line 122) GC for ParNew: 444 ms for 1 collections, 3587436048 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:45:03,295 GCInspector.java (line 122) GC for ParNew: 495 ms for 1 collections, 3655651224 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:45:04,569 GCInspector.java (line 122) GC for ParNew: 454 ms for 1 collections, 3726744336 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:45:05,843 GCInspector.java (line 122) GC for ParNew: 443 ms for 1 collections, 3797343632 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:45:07,316 GCInspector.java (line 122) GC for ParNew: 495 ms for 1 collections, 3872328256 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:45:08,612 GCInspector.java (line 122) GC for ParNew: 516 ms for 1 collections, 3942107216 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:45:09,885 GCInspector.java (line 122) GC for ParNew: 498 ms for 1 collections, 4014688120 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:45:11,283 GCInspector.java (line 122) GC for ParNew: 505 ms for 1 collections, 4087386328 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:45:12,671 GCInspector.java (line 122) GC for ParNew: 507 ms for 1 collections, 4157370888 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:45:14,026 GCInspector.java (line 122) GC for ParNew: 506 ms for 1 collections, 4228114440 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:45:15,320 GCInspector.java (line 122) GC for ParNew: 472 ms for 1 collections, 4300857192 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:45:16,616 GCInspector.java (line 122) GC for ParNew: 483 ms for 1 collections, 4372262264 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:45:18,003 GCInspector.java (line 122) GC for ParNew: 470 ms for 1 collections, 4444151952 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:45:19,400 GCInspector.java (line 122) GC for ParNew: 482 ms for 1 collections, 4514967664 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:45:20,770 GCInspector.java (line 122) GC for ParNew: 491 ms for 1 collections, 4588120240 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:45:22,095 GCInspector.java (line 122) GC for ParNew: 480 ms for 1 collections, 4658890232 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:45:23,364 GCInspector.java (line 122) GC for ParNew: 480 ms for 1 collections, 4731783776 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:45:24,644 GCInspector.java (line 122) GC for ParNew: 471 ms for 1 collections, 4803944408 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:45:25,973 GCInspector.java (line 122) GC for ParNew: 476 ms for 1 collections, 4876166496 used; max is 5326766080 INFO [GossipTasks:1] 2012-06-01 02:47:05,543 Gossiper.java (line 818) InetAddress /10.0.1.1 is now dead. INFO [ScheduledTasks:1] 2012-06-01 02:47:05,544 GCInspector.java (line 122) GC for ParNew: 62173 ms for 1 collections, 4914675328 used; max is 5326766080 INFO [GossipTasks:1] 2012-06-01 02:47:05,544 Gossiper.java (line 818) InetAddress /10.0.1.8 is now dead. INFO [ScheduledTasks:1] 2012-06-01 02:47:05,545 StatusLogger.java (line 50) Pool Name Active Pending Blocked INFO [GossipTasks:1] 2012-06-01 02:47:05,545 Gossiper.java (line 818) InetAddress /10.0.1.3 is now dead. INFO [ScheduledTasks:1] 2012-06-01 02:47:05,546 StatusLogger.java (line 65) ReadStage 5 5 0 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,608 StatusLogger.java (line 65) RequestResponseStage 0 1 0 INFO [GossipStage:1] 2012-06-01 02:47:05,608 Gossiper.java (line 804) InetAddress /10.0.1.8 is now UP INFO [ScheduledTasks:1] 2012-06-01 02:47:05,608 StatusLogger.java (line 65) ReadRepairStage 0 0 0 INFO [GossipStage:1] 2012-06-01 02:47:05,609 Gossiper.java (line 804) InetAddress /10.0.1.3 is now UP INFO [ScheduledTasks:1] 2012-06-01 02:47:05,609 StatusLogger.java (line 65) MutationStage 0 0 0 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,610 StatusLogger.java (line 65) ReplicateOnWriteStage 0 0 0 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,610 StatusLogger.java (line 65) GossipStage 1 19 0 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,610 StatusLogger.java (line 65) AntiEntropyStage 0 0 0 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,611 StatusLogger.java (line 65) MigrationStage 0 0 0 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,611 StatusLogger.java (line 65) StreamStage 0 0 0 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,611 StatusLogger.java (line 65) MemtablePostFlusher 0 0 0 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,611 StatusLogger.java (line 65) FlushWriter 0 0 0 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,611 StatusLogger.java (line 65) MiscStage 0 0 0 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,612 StatusLogger.java (line 65) InternalResponseStage 0 0 0 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,612 StatusLogger.java (line 65) AntiEntropySessions 0 0 0 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,612 StatusLogger.java (line 65) HintedHandoff 0 0 0 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,612 StatusLogger.java (line 70) CompactionManager 0 0 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,613 StatusLogger.java (line 82) MessagingService n/a 0,0 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,613 StatusLogger.java (line 86) ColumnFamily Memtable ops,data Row cache size/cap Key cache size/cap INFO [ScheduledTasks:1] 2012-06-01 02:47:05,613 StatusLogger.java (line 89) system.NodeIdInfo 3,131 0/0 1/1 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,613 StatusLogger.java (line 89) system.IndexInfo 0,0 0/0 0/1 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,614 StatusLogger.java (line 89) system.LocationInfo 0,0 0/0 1/2 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,614 StatusLogger.java (line 89) system.Versions 3,103 0/0 0/2 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,614 StatusLogger.java (line 89) system.Migrations 0,0 0/0 1/1 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,614 StatusLogger.java (line 89) system.HintsColumnFamily 0,0 0/0 0/1 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,614 StatusLogger.java (line 89) system.Schema 0,0 0/0 1/1 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,615 StatusLogger.java (line 89) ElasticInbox.IndexLabels 234488,113671251 375210/1000000 200000/200000 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,615 StatusLogger.java (line 89) ElasticInbox.MessageMetadata 28761,77170997 0/0 200000/200000 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,615 StatusLogger.java (line 89) ElasticInbox.Counters 488621,227783079 21881/300000 200000/200000 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,615 StatusLogger.java (line 89) ElasticInbox.Accounts 21752,14826519 29216/100000 107939/200000 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,615 GCInspector.java (line 122) GC for ConcurrentMarkSweep: 2273 ms for 2 collections, 4941337800 used; max is 5326766080 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,616 StatusLogger.java (line 50) Pool Name Active Pending Blocked INFO [ScheduledTasks:1] 2012-06-01 02:47:05,625 StatusLogger.java (line 65) ReadStage 5 5 0 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,626 StatusLogger.java (line 65) RequestResponseStage 1 6 0 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,637 StatusLogger.java (line 65) ReadRepairStage 0 0 0 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,637 StatusLogger.java (line 65) MutationStage 0 0 0 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,637 StatusLogger.java (line 65) ReplicateOnWriteStage 0 0 0 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,638 StatusLogger.java (line 65) GossipStage 0 0 0 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,638 StatusLogger.java (line 65) AntiEntropyStage 0 0 0 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,638 StatusLogger.java (line 65) MigrationStage 0 0 0 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,639 StatusLogger.java (line 65) StreamStage 0 0 0 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,639 StatusLogger.java (line 65) MemtablePostFlusher 0 0 0 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,639 StatusLogger.java (line 65) FlushWriter 0 0 0 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,639 StatusLogger.java (line 65) MiscStage 0 0 0 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,640 StatusLogger.java (line 65) InternalResponseStage 0 0 0 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,640 StatusLogger.java (line 65) AntiEntropySessions 0 0 0 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,640 StatusLogger.java (line 65) HintedHandoff 0 0 0 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,640 StatusLogger.java (line 70) CompactionManager 0 0 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,641 StatusLogger.java (line 82) MessagingService n/a 0,0 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,641 StatusLogger.java (line 86) ColumnFamily Memtable ops,data Row cache size/cap Key cache size/cap INFO [ScheduledTasks:1] 2012-06-01 02:47:05,641 StatusLogger.java (line 89) system.NodeIdInfo 3,131 0/0 1/1 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,641 StatusLogger.java (line 89) system.IndexInfo 0,0 0/0 0/1 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,641 StatusLogger.java (line 89) system.LocationInfo 0,0 0/0 1/2 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,642 StatusLogger.java (line 89) system.Versions 3,103 0/0 0/2 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,642 StatusLogger.java (line 89) system.Migrations 0,0 0/0 1/1 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,642 StatusLogger.java (line 89) system.HintsColumnFamily 0,0 0/0 0/1 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,642 StatusLogger.java (line 89) system.Schema 0,0 0/0 1/1 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,644 StatusLogger.java (line 89) ElasticInbox.IndexLabels 234488,113671251 375210/1000000 200000/200000 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,644 StatusLogger.java (line 89) ElasticInbox.MessageMetadata 28761,77170997 0/0 200000/200000 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,644 StatusLogger.java (line 89) ElasticInbox.Counters 488621,227783079 21881/300000 200000/200000 INFO [ScheduledTasks:1] 2012-06-01 02:47:05,644 StatusLogger.java (line 89) ElasticInbox.Accounts 21752,14826519 29216/100000 107939/200000 WARN [ScheduledTasks:1] 2012-06-01 02:47:05,644 GCInspector.java (line 139) Heap is 0.9276431001077486 full. You may need to reduce memtable and/or cache sizes. Cassandra is now reducing cache sizes to free up memory. Adjust reduce_cache_sizes_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically