Cluster Information:
3 node 1.0.5
16GB RAM (8GB assigned to heap)
default cassandra.yaml, except for:
flush_largest_memtables_at: 0.5
Keyspace: MapData007:
Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
Durable Writes: true
Options: [replication_factor:2]
Column Families:
ColumnFamily: HOL (Super)
Key Validation Class: org.apache.cassandra.db.marshal.Int32Type
Default column value validator: org.apache.cassandra.db.marshal.BytesType
Columns sorted by: org.apache.cassandra.db.marshal.Int32Type/org.apache.cassandra.db.marshal.UTF8Type
Row cache size / save period in seconds / keys to save : 0.0/0/all
Row Cache Provider: org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider
Key cache size / save period in seconds: 200000.0/14400
GC grace seconds: 864000
Compaction min/max thresholds: 4/32
Read repair chance: 1.0
Replicate on write: true
Built indexes: []
Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy
Compression Options:
chunk_length_kb: 65536
sstable_compression: org.apache.cassandra.io.compress.SnappyCompressor
ColumnFamily: HOS (Super)
Key Validation Class: org.apache.cassandra.db.marshal.Int32Type
Default column value validator: org.apache.cassandra.db.marshal.BytesType
Columns sorted by: org.apache.cassandra.db.marshal.Int32Type/org.apache.cassandra.db.marshal.UTF8Type
Row cache size / save period in seconds / keys to save : 0.0/0/all
Row Cache Provider: org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider
Key cache size / save period in seconds: 200000.0/14400
GC grace seconds: 864000
Compaction min/max thresholds: 4/32
Read repair chance: 1.0
Replicate on write: true
Built indexes: []
Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy
Compression Options:
chunk_length_kb: 65536
sstable_compression: org.apache.cassandra.io.compress.SnappyCompressor
system.log
INFO [CompactionExecutor:15] 2011-12-06 15:56:07,412 CompactionTask.java (line 213) Compacted to [/data/MapData007/HOS-hb-4339-Data.db,]. 23,179,577 to 27,408,495 (~118% of original) bytes for 765,937 keys at 0.662849MB/s. Time: 39,434ms.
INFO [CompactionExecutor:13] 2011-12-06 15:56:10,790 CompactionTask.java (line 213) Compacted to [/data/MapData007/HOS-hb-4338-Data.db,]. 26,163,431 to 29,211,198 (~111% of original) bytes for 754,833 keys at 0.650112MB/s. Time: 42,851ms.
INFO [CompactionExecutor:13] 2011-12-06 15:56:10,791 CompactionTask.java (line 112) Compacting [SSTableReader(path='/data/MapData007/HOS-hb-4337-Data.db'), SSTableReader(path='/data/MapData007/HOS-hb-4336-Data.db'), SSTableReader(path='/data/MapData007/HOS-hb-4339-Data.db'), SSTableReader(path='/data/MapData007/HOS-hb-4338-Data.db')]
INFO [pool-1-thread-1] 2011-12-06 15:56:11,796 Memtable.java (line 179) CFS(Keyspace='OpsCenter', ColumnFamily='rollups300') liveRatio is 16.36052836052836 (just-counted was 15.875793930839803). calculation took 3ms for 107 columns
INFO [CompactionExecutor:10] 2011-12-06 15:56:13,769 CompactionTask.java (line 213) Compacted to [/data/MapData007/HOS-hb-4340-Data.db,]. 29,306,274 to 29,526,448 (~100% of original) bytes for 580,823 keys at 0.614884MB/s. Time: 45,795ms.
INFO [CompactionExecutor:13] 2011-12-06 15:57:00,138 CompactionTask.java (line 213) Compacted to [/data/MapData007/HOS-hb-4344-Data.db,]. 68,228,761 to 59,685,286 (~87% of original) bytes for 788,409 keys at 1.153471MB/s. Time: 49,347ms.
INFO [CompactionExecutor:9] 2011-12-06 15:57:39,020 CompactionTask.java (line 213) Compacted to [/data/MapData007/HOS-hb-4341-Data.db,]. 207,139,798 to 184,659,335 (~89% of original) bytes for 882,525 keys at 1.343461MB/s. Time: 131,083ms.
INFO [ScheduledTasks:1] 2011-12-06 15:58:24,056 GCInspector.java (line 123) GC for ParNew: 216 ms for 1 collections, 6553195592 used; max is 8485076992
WARN [ScheduledTasks:1] 2011-12-06 16:01:08,676 GCInspector.java (line 140) Heap is 0.9893797413877373 full. You may need to reduce memtable and/or cache sizes. Cassandra is now reducing cache sizes to free up memory. Adjust reduce_cache_sizes_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
WARN [ScheduledTasks:1] 2011-12-06 16:01:08,677 AutoSavingCache.java (line 187) Reducing NodeIdInfo KeyCache capacity from 1 to 0 to reduce memory pressure
WARN [ScheduledTasks:1] 2011-12-06 16:01:08,678 AutoSavingCache.java (line 187) Reducing IndexInfo KeyCache capacity from 1 to 0 to reduce memory pressure
WARN [ScheduledTasks:1] 2011-12-06 16:01:08,678 AutoSavingCache.java (line 187) Reducing LocationInfo KeyCache capacity from 2 to 1 to reduce memory pressure
ERROR [ReadStage:43] 2011-12-06 16:01:08,679 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[ReadStage:43,5,main]
java.lang.OutOfMemoryError: Java heap space
at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:75)
at org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:62)
at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:44)
at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:39)
at org.apache.cassandra.io.util.CompressedSegmentedFile.getSegment(CompressedSegmentedFile.java:63)
at org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:798)
at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:48)
at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:78)
at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:214)
at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:62)
at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1275)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1161)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1128)
at org.apache.cassandra.db.Table.getRow(Table.java:375)
at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:53)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)