On Wed, Oct 26, 2011 at 3:50 PM, Ramesh Natarajan <rames...@gmail.com> wrote:
> I did some more analysis and I think the compaction for this CF stopped after
> we did a update column family to increase the key cache. Other CF compactions
> were going on without any issues.
> I did another update column family to the same CF with same values as before
> and the compaction started again.
> It it interesting to note that this happened on only one of the node in a 8
> node cluster. Also it is the same node where the command was issued using the
> cli.
>
> Is it normal to issue update column family to increase the key cache when
> cassandra is actively serving traffic in a production environment?
>
> Are there any known issues/interactions with compaction and updating column
> family in 1.0?
If the update happens to change and set to 0 the
min_compaction_threshold or max_compaction_threshold, then that would
disable compaction. Now of course it should not do that unless you've
asked it to, and if it did, it should disable compaction cluster-wide,
otherwise that would be a bug. But that's the only way I see for an
update column family to stop compaction.
--
Sylvain
>
> Thanks
> Ramesh
>
> Ramesh Natarajan <rames...@gmail.com> wrote:
>
>>Looks like compaction for this column family stopped after some time.
>>The last message for this column family in the system.log is
>>
>>
>>INFO [MigrationStage:1] 2011-10-25 16:57:00,385 Migration.java (line
>>119) Applying migration 43f106c0-ff54-11e0-0000-68877f281daf Update
>>column family to
>>org.apache.cassandra.config.CFMetaData@86c50e4[cfId=1000,ksName=MSA,cfName=uid,cfType=Standard,comparator=org.apache.cassandra.db.marshal.BytesType,subcolumncomparator=<null>,comment=,rowCacheSize=0.0,keyCacheSize=1.3E7,readRepairChance=1.0,replicateOnWrite=true,gcGraceSeconds=3600,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=4,maxCompactionThreshold=32,rowCacheSavePeriodInSeconds=0,keyCacheSavePeriodInSeconds=14400,rowCacheKeysToSave=2147483647,rowCacheProvider=org.apache.cassandra.cache.SerializingCacheProvider@7f32ad0d,mergeShardsChance=0.1,keyAlias=java.nio.HeapByteBuffer[pos=0
>>lim=3 cap=3],column_metadata={},compactionStrategyClass=class
>>org.apache.cassandra.db.compaction.LeveledCompactionStrategy,compactionStrategyOptions={sstable_size_in_mb=10},compressionOptions={}]
>> INFO [MigrationStage:1] 2011-10-25 16:57:00,389
>>ColumnFamilyStore.java (line 664) Enqueuing flush of
>>Memtable-Migrations@1386279942(8806/11007 serialized/live bytes, 1
>>ops)
>> INFO [FlushWriter:307] 2011-10-25 16:57:00,389 Memtable.java (line
>>237) Writing Memtable-Migrations@1386279942(8806/11007 serialized/live
>>bytes, 1 ops)
>> INFO [MigrationStage:1] 2011-10-25 16:57:00,389
>>ColumnFamilyStore.java (line 664) Enqueuing flush of
>>Memtable-Schema@1156898891(4336/5420 serialized/live bytes, 3 ops)
>> INFO [FlushWriter:307] 2011-10-25 16:57:00,402 Memtable.java (line
>>273) Completed flushing
>>/var/lib/cassandra/data/system/Migrations-h-48-Data.db (8870 bytes)
>> INFO [FlushWriter:307] 2011-10-25 16:57:00,402 Memtable.java (line
>>237) Writing Memtable-Schema@1156898891(4336/5420 serialized/live
>>bytes, 3 ops)
>> INFO [FlushWriter:307] 2011-10-25 16:57:00,413 Memtable.java (line
>>273) Completed flushing
>>/var/lib/cassandra/data/system/Schema-h-48-Data.db (4486 bytes)
>> INFO [CompactionExecutor:23] 2011-10-25 16:57:00,929
>>CompactionTask.java (line 119) Compacting
>>[SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9016-Data.db'),
>>SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9046-Data.db'),
>>SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9042-Data.db'),
>>SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9039-Data.db'),
>>SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9043-Data.db'),
>>SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9045-Data.db'),
>>SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9044-Data.db'),
>>SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9015-Data.db'),
>>SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9040-Data.db'),
>>SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9054-Data.db'),
>>SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9047-Data.db'),
>>SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9041-Data.db')]
>> INFO [MigrationStage:1] 2011-10-25 16:57:06,693 Migration.java (line
>>119) Applying migration 47b1b840-ff54-11e0-0000-68877f281daf Update
>>column family to
>>org.apache.cassandra.config.CFMetaData@23de1953[cfId=1000,ksName=MSA,cfName=uid,cfType=Standard,comparator=org.apache.cassandra.db.marshal.BytesType,subcolumncomparator=<null>,comment=,rowCacheSize=0.0,keyCacheSize=1.5E7,readRepairChance=1.0,replicateOnWrite=true,gcGraceSeconds=3600,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=4,maxCompactionThreshold=32,rowCacheSavePeriodInSeconds=0,keyCacheSavePeriodInSeconds=14400,rowCacheKeysToSave=2147483647,rowCacheProvider=org.apache.cassandra.cache.SerializingCacheProvider@4a50aa8a,mergeShardsChance=0.1,keyAlias=java.nio.HeapByteBuffer[pos=0
>>lim=3 cap=3],column_metadata={},compactionStrategyClass=class
>>org.apache.cassandra.db.compaction.LeveledCompactionStrategy,compactionStrategyOptions={sstable_size_in_mb=10},compressionOptions={}]
>> INFO [MigrationStage:1] 2011-10-25 16:57:06,694
>>ColumnFamilyStore.java (line 664) Enqueuing flush of
>>Memtable-Migrations@1978429475(8806/11007 serialized/live bytes, 1
>>ops)
>> INFO [FlushWriter:307] 2011-10-25 16:57:06,694 Memtable.java (line
>>237) Writing Memtable-Migrations@1978429475(8806/11007 serialized/live
>>bytes, 1 ops)
>>
>>
>>
>>The schema for this CF is
>>
>>create column family uid
>> with column_type = 'Standard'
>> and comparator = 'BytesType'
>> and default_validation_class = 'BytesType'
>> and key_validation_class = 'BytesType'
>> and rows_cached = 0.0
>> and row_cache_save_period = 0
>> and row_cache_keys_to_save = 2147483647
>> and keys_cached = 1.5E7
>> and key_cache_save_period = 14400
>> and read_repair_chance = 1.0
>> and gc_grace = 3600
>> and min_compaction_threshold = 4
>> and max_compaction_threshold = 32
>> and replicate_on_write = true
>> and row_cache_provider = 'SerializingCacheProvider'
>> and compaction_strategy =
>>'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'
>> and compaction_strategy_options = {'sstable_size_in_mb' : '10'};
>>
>>
>>
>>Any idea why compaction stopped automatically for this CF?
>>
>>thanks
>>Ramesh
>>
>>
>>
>>
>>
>>On Wed, Oct 26, 2011 at 5:23 AM, Sylvain Lebresne <sylv...@datastax.com>
>>wrote:
>>> The estimate for the number of keys is computed by summing the key
>>> estimate for each sstable of the CF. For each sstable, the estimate
>>> should be fairly good. However, that's when we sum all the sstable estimates
>>> that we can loose potentially a lot of precision if there is a lot of rows
>>> that
>>> have parts in different sstables. But that in turn would suggest a problem
>>> with compaction lacking badly behind, especially with leveled compaction.
>>>
>>> --
>>> Sylvain
>>>
>>> On Wed, Oct 26, 2011 at 3:58 AM, Terry Cumaranatunge <cumar...@gmail.com>
>>> wrote:
>>>> I have a cluster of 8 nodes all running 1.0. The stats shown on the 1st
>>>> node
>>>> on one of the CFs for the number of keys is much larger than expected. The
>>>> first node shows the key count estimate to be 9.2M whereas the rest report
>>>> ~650K on each node. The 650K is in the correct neighborhood of the number
>>>> of
>>>> keys that have been inserted. The counts are comparable for all other CFs
>>>> across the cluster. I'm using Level compaction, but no compression.
>>>>
>>>> The 'nodetool ring' shows that the load is equal across all nodes. What
>>>> could cause this large disparity in the number of keys? Is this just a
>>>> stats
>>>> issue or does this suggest a functional problem?
>>>>
>>>> 1st node:
>>>> Column Family: uid
>>>> SSTable count: 395
>>>> Space used (live): 13752624444
>>>> Space used (total): 5482088532
>>>> Number of Keys (estimate): 9215104
>>>> Memtable Columns Count: 514952
>>>> Memtable Data Size: 295213448
>>>> Memtable Switch Count: 290
>>>> Read Count: 193102511
>>>> Read Latency: 0.146 ms.
>>>> Write Count: 176934874
>>>> Write Latency: 0.018 ms.
>>>> Pending Tasks: 0
>>>> Key cache capacity: 8302131
>>>> Key cache size: 8302131
>>>> Key cache hit rate: 0.8644664668071792
>>>> Row cache: disabled
>>>> Compacted row minimum size: 87
>>>> Compacted row maximum size: 7007506
>>>> Compacted row mean size: 8944
>>>> 2nd node:
>>>> Column Family: uid
>>>> SSTable count: 402
>>>> Space used (live): 13723958304
>>>> Space used (total): 4044833220
>>>> Number of Keys (estimate): 652928
>>>> Memtable Columns Count: 170290
>>>> Memtable Data Size: 102378904
>>>> Memtable Switch Count: 272
>>>> Read Count: 192463595
>>>> Read Latency: 0.289 ms.
>>>> Write Count: 176527238
>>>> Write Latency: 0.014 ms.
>>>> Pending Tasks: 0
>>>> Key cache capacity: 8783058
>>>> Key cache size: 8783058
>>>> Key cache hit rate: 0.7865727464740025
>>>> Row cache: disabled
>>>> Compacted row minimum size: 87
>>>> Compacted row maximum size: 7007506
>>>> Compacted row mean size: 12151
>>>> 3rd node:
>>>> Column Family: uid
>>>> SSTable count: 401
>>>> Space used (live): 13204714872
>>>> Space used (total): 4030024144
>>>> Number of Keys (estimate): 675968
>>>> Memtable Columns Count: 42881
>>>> Memtable Data Size: 30992298
>>>> Memtable Switch Count: 304
>>>> Read Count: 190769879
>>>> Read Latency: 0.224 ms.
>>>> Write Count: 175381826
>>>> Write Latency: 0.014 ms.
>>>> Pending Tasks: 0
>>>> Key cache capacity: 8920108
>>>> Key cache size: 8920108
>>>> Key cache hit rate: 0.8053563128870577
>>>> Row cache: disabled
>>>> Compacted row minimum size: 87
>>>> Compacted row maximum size: 4866323
>>>> Compacted row mean size: 12074
>>>>
>>>
>
