I did some more analysis and I think the compaction for this CF stopped after
we did a update column family to increase the key cache. Other CF compactions
were going on without any issues.
I did another update column family to the same CF with same values as before
and the compaction started again.
It it interesting to note that this happened on only one of the node in a 8
node cluster. Also it is the same node where the command was issued using the
cli.
Is it normal to issue update column family to increase the key cache when
cassandra is actively serving traffic in a production environment?
Are there any known issues/interactions with compaction and updating column
family in 1.0?
Thanks
Ramesh
Ramesh Natarajan <rames...@gmail.com> wrote:
>Looks like compaction for this column family stopped after some time.
>The last message for this column family in the system.log is
>
>
>INFO [MigrationStage:1] 2011-10-25 16:57:00,385 Migration.java (line
>119) Applying migration 43f106c0-ff54-11e0-0000-68877f281daf Update
>column family to
>org.apache.cassandra.config.CFMetaData@86c50e4[cfId=1000,ksName=MSA,cfName=uid,cfType=Standard,comparator=org.apache.cassandra.db.marshal.BytesType,subcolumncomparator=<null>,comment=,rowCacheSize=0.0,keyCacheSize=1.3E7,readRepairChance=1.0,replicateOnWrite=true,gcGraceSeconds=3600,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=4,maxCompactionThreshold=32,rowCacheSavePeriodInSeconds=0,keyCacheSavePeriodInSeconds=14400,rowCacheKeysToSave=2147483647,rowCacheProvider=org.apache.cassandra.cache.SerializingCacheProvider@7f32ad0d,mergeShardsChance=0.1,keyAlias=java.nio.HeapByteBuffer[pos=0
>lim=3 cap=3],column_metadata={},compactionStrategyClass=class
>org.apache.cassandra.db.compaction.LeveledCompactionStrategy,compactionStrategyOptions={sstable_size_in_mb=10},compressionOptions={}]
> INFO [MigrationStage:1] 2011-10-25 16:57:00,389
>ColumnFamilyStore.java (line 664) Enqueuing flush of
>Memtable-Migrations@1386279942(8806/11007 serialized/live bytes, 1
>ops)
> INFO [FlushWriter:307] 2011-10-25 16:57:00,389 Memtable.java (line
>237) Writing Memtable-Migrations@1386279942(8806/11007 serialized/live
>bytes, 1 ops)
> INFO [MigrationStage:1] 2011-10-25 16:57:00,389
>ColumnFamilyStore.java (line 664) Enqueuing flush of
>Memtable-Schema@1156898891(4336/5420 serialized/live bytes, 3 ops)
> INFO [FlushWriter:307] 2011-10-25 16:57:00,402 Memtable.java (line
>273) Completed flushing
>/var/lib/cassandra/data/system/Migrations-h-48-Data.db (8870 bytes)
> INFO [FlushWriter:307] 2011-10-25 16:57:00,402 Memtable.java (line
>237) Writing Memtable-Schema@1156898891(4336/5420 serialized/live
>bytes, 3 ops)
> INFO [FlushWriter:307] 2011-10-25 16:57:00,413 Memtable.java (line
>273) Completed flushing
>/var/lib/cassandra/data/system/Schema-h-48-Data.db (4486 bytes)
> INFO [CompactionExecutor:23] 2011-10-25 16:57:00,929
>CompactionTask.java (line 119) Compacting
>[SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9016-Data.db'),
>SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9046-Data.db'),
>SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9042-Data.db'),
>SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9039-Data.db'),
>SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9043-Data.db'),
>SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9045-Data.db'),
>SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9044-Data.db'),
>SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9015-Data.db'),
>SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9040-Data.db'),
>SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9054-Data.db'),
>SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9047-Data.db'),
>SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9041-Data.db')]
> INFO [MigrationStage:1] 2011-10-25 16:57:06,693 Migration.java (line
>119) Applying migration 47b1b840-ff54-11e0-0000-68877f281daf Update
>column family to
>org.apache.cassandra.config.CFMetaData@23de1953[cfId=1000,ksName=MSA,cfName=uid,cfType=Standard,comparator=org.apache.cassandra.db.marshal.BytesType,subcolumncomparator=<null>,comment=,rowCacheSize=0.0,keyCacheSize=1.5E7,readRepairChance=1.0,replicateOnWrite=true,gcGraceSeconds=3600,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=4,maxCompactionThreshold=32,rowCacheSavePeriodInSeconds=0,keyCacheSavePeriodInSeconds=14400,rowCacheKeysToSave=2147483647,rowCacheProvider=org.apache.cassandra.cache.SerializingCacheProvider@4a50aa8a,mergeShardsChance=0.1,keyAlias=java.nio.HeapByteBuffer[pos=0
>lim=3 cap=3],column_metadata={},compactionStrategyClass=class
>org.apache.cassandra.db.compaction.LeveledCompactionStrategy,compactionStrategyOptions={sstable_size_in_mb=10},compressionOptions={}]
> INFO [MigrationStage:1] 2011-10-25 16:57:06,694
>ColumnFamilyStore.java (line 664) Enqueuing flush of
>Memtable-Migrations@1978429475(8806/11007 serialized/live bytes, 1
>ops)
> INFO [FlushWriter:307] 2011-10-25 16:57:06,694 Memtable.java (line
>237) Writing Memtable-Migrations@1978429475(8806/11007 serialized/live
>bytes, 1 ops)
>
>
>
>The schema for this CF is
>
>create column family uid
> with column_type = 'Standard'
> and comparator = 'BytesType'
> and default_validation_class = 'BytesType'
> and key_validation_class = 'BytesType'
> and rows_cached = 0.0
> and row_cache_save_period = 0
> and row_cache_keys_to_save = 2147483647
> and keys_cached = 1.5E7
> and key_cache_save_period = 14400
> and read_repair_chance = 1.0
> and gc_grace = 3600
> and min_compaction_threshold = 4
> and max_compaction_threshold = 32
> and replicate_on_write = true
> and row_cache_provider = 'SerializingCacheProvider'
> and compaction_strategy =
>'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'
> and compaction_strategy_options = {'sstable_size_in_mb' : '10'};
>
>
>
>Any idea why compaction stopped automatically for this CF?
>
>thanks
>Ramesh
>
>
>
>
>
>On Wed, Oct 26, 2011 at 5:23 AM, Sylvain Lebresne <sylv...@datastax.com> wrote:
>> The estimate for the number of keys is computed by summing the key
>> estimate for each sstable of the CF. For each sstable, the estimate
>> should be fairly good. However, that's when we sum all the sstable estimates
>> that we can loose potentially a lot of precision if there is a lot of rows
>> that
>> have parts in different sstables. But that in turn would suggest a problem
>> with compaction lacking badly behind, especially with leveled compaction.
>>
>> --
>> Sylvain
>>
>> On Wed, Oct 26, 2011 at 3:58 AM, Terry Cumaranatunge <cumar...@gmail.com>
>> wrote:
>>> I have a cluster of 8 nodes all running 1.0. The stats shown on the 1st node
>>> on one of the CFs for the number of keys is much larger than expected. The
>>> first node shows the key count estimate to be 9.2M whereas the rest report
>>> ~650K on each node. The 650K is in the correct neighborhood of the number of
>>> keys that have been inserted. The counts are comparable for all other CFs
>>> across the cluster. I'm using Level compaction, but no compression.
>>>
>>> The 'nodetool ring' shows that the load is equal across all nodes. What
>>> could cause this large disparity in the number of keys? Is this just a stats
>>> issue or does this suggest a functional problem?
>>>
>>> 1st node:
>>> Column Family: uid
>>> SSTable count: 395
>>> Space used (live): 13752624444
>>> Space used (total): 5482088532
>>> Number of Keys (estimate): 9215104
>>> Memtable Columns Count: 514952
>>> Memtable Data Size: 295213448
>>> Memtable Switch Count: 290
>>> Read Count: 193102511
>>> Read Latency: 0.146 ms.
>>> Write Count: 176934874
>>> Write Latency: 0.018 ms.
>>> Pending Tasks: 0
>>> Key cache capacity: 8302131
>>> Key cache size: 8302131
>>> Key cache hit rate: 0.8644664668071792
>>> Row cache: disabled
>>> Compacted row minimum size: 87
>>> Compacted row maximum size: 7007506
>>> Compacted row mean size: 8944
>>> 2nd node:
>>> Column Family: uid
>>> SSTable count: 402
>>> Space used (live): 13723958304
>>> Space used (total): 4044833220
>>> Number of Keys (estimate): 652928
>>> Memtable Columns Count: 170290
>>> Memtable Data Size: 102378904
>>> Memtable Switch Count: 272
>>> Read Count: 192463595
>>> Read Latency: 0.289 ms.
>>> Write Count: 176527238
>>> Write Latency: 0.014 ms.
>>> Pending Tasks: 0
>>> Key cache capacity: 8783058
>>> Key cache size: 8783058
>>> Key cache hit rate: 0.7865727464740025
>>> Row cache: disabled
>>> Compacted row minimum size: 87
>>> Compacted row maximum size: 7007506
>>> Compacted row mean size: 12151
>>> 3rd node:
>>> Column Family: uid
>>> SSTable count: 401
>>> Space used (live): 13204714872
>>> Space used (total): 4030024144
>>> Number of Keys (estimate): 675968
>>> Memtable Columns Count: 42881
>>> Memtable Data Size: 30992298
>>> Memtable Switch Count: 304
>>> Read Count: 190769879
>>> Read Latency: 0.224 ms.
>>> Write Count: 175381826
>>> Write Latency: 0.014 ms.
>>> Pending Tasks: 0
>>> Key cache capacity: 8920108
>>> Key cache size: 8920108
>>> Key cache hit rate: 0.8053563128870577
>>> Row cache: disabled
>>> Compacted row minimum size: 87
>>> Compacted row maximum size: 4866323
>>> Compacted row mean size: 12074
>>>
>>
