Looks like compaction for this column family stopped after some time.
The last message for this column family in the system.log is
INFO [MigrationStage:1] 2011-10-25 16:57:00,385 Migration.java (line
119) Applying migration 43f106c0-ff54-11e0-0000-68877f281daf Update
column family to
org.apache.cassandra.config.CFMetaData@86c50e4[cfId=1000,ksName=MSA,cfName=uid,cfType=Standard,comparator=org.apache.cassandra.db.marshal.BytesType,subcolumncomparator=<null>,comment=,rowCacheSize=0.0,keyCacheSize=1.3E7,readRepairChance=1.0,replicateOnWrite=true,gcGraceSeconds=3600,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=4,maxCompactionThreshold=32,rowCacheSavePeriodInSeconds=0,keyCacheSavePeriodInSeconds=14400,rowCacheKeysToSave=2147483647,rowCacheProvider=org.apache.cassandra.cache.SerializingCacheProvider@7f32ad0d,mergeShardsChance=0.1,keyAlias=java.nio.HeapByteBuffer[pos=0
lim=3 cap=3],column_metadata={},compactionStrategyClass=class
org.apache.cassandra.db.compaction.LeveledCompactionStrategy,compactionStrategyOptions={sstable_size_in_mb=10},compressionOptions={}]
INFO [MigrationStage:1] 2011-10-25 16:57:00,389
ColumnFamilyStore.java (line 664) Enqueuing flush of
Memtable-Migrations@1386279942(8806/11007 serialized/live bytes, 1
ops)
INFO [FlushWriter:307] 2011-10-25 16:57:00,389 Memtable.java (line
237) Writing Memtable-Migrations@1386279942(8806/11007 serialized/live
bytes, 1 ops)
INFO [MigrationStage:1] 2011-10-25 16:57:00,389
ColumnFamilyStore.java (line 664) Enqueuing flush of
Memtable-Schema@1156898891(4336/5420 serialized/live bytes, 3 ops)
INFO [FlushWriter:307] 2011-10-25 16:57:00,402 Memtable.java (line
273) Completed flushing
/var/lib/cassandra/data/system/Migrations-h-48-Data.db (8870 bytes)
INFO [FlushWriter:307] 2011-10-25 16:57:00,402 Memtable.java (line
237) Writing Memtable-Schema@1156898891(4336/5420 serialized/live
bytes, 3 ops)
INFO [FlushWriter:307] 2011-10-25 16:57:00,413 Memtable.java (line
273) Completed flushing
/var/lib/cassandra/data/system/Schema-h-48-Data.db (4486 bytes)
INFO [CompactionExecutor:23] 2011-10-25 16:57:00,929
CompactionTask.java (line 119) Compacting
[SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9016-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9046-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9042-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9039-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9043-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9045-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9044-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9015-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9040-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9054-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9047-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9041-Data.db')]
INFO [MigrationStage:1] 2011-10-25 16:57:06,693 Migration.java (line
119) Applying migration 47b1b840-ff54-11e0-0000-68877f281daf Update
column family to
org.apache.cassandra.config.CFMetaData@23de1953[cfId=1000,ksName=MSA,cfName=uid,cfType=Standard,comparator=org.apache.cassandra.db.marshal.BytesType,subcolumncomparator=<null>,comment=,rowCacheSize=0.0,keyCacheSize=1.5E7,readRepairChance=1.0,replicateOnWrite=true,gcGraceSeconds=3600,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=4,maxCompactionThreshold=32,rowCacheSavePeriodInSeconds=0,keyCacheSavePeriodInSeconds=14400,rowCacheKeysToSave=2147483647,rowCacheProvider=org.apache.cassandra.cache.SerializingCacheProvider@4a50aa8a,mergeShardsChance=0.1,keyAlias=java.nio.HeapByteBuffer[pos=0
lim=3 cap=3],column_metadata={},compactionStrategyClass=class
org.apache.cassandra.db.compaction.LeveledCompactionStrategy,compactionStrategyOptions={sstable_size_in_mb=10},compressionOptions={}]
INFO [MigrationStage:1] 2011-10-25 16:57:06,694
ColumnFamilyStore.java (line 664) Enqueuing flush of
Memtable-Migrations@1978429475(8806/11007 serialized/live bytes, 1
ops)
INFO [FlushWriter:307] 2011-10-25 16:57:06,694 Memtable.java (line
237) Writing Memtable-Migrations@1978429475(8806/11007 serialized/live
bytes, 1 ops)
The schema for this CF is
create column family uid
with column_type = 'Standard'
and comparator = 'BytesType'
and default_validation_class = 'BytesType'
and key_validation_class = 'BytesType'
and rows_cached = 0.0
and row_cache_save_period = 0
and row_cache_keys_to_save = 2147483647
and keys_cached = 1.5E7
and key_cache_save_period = 14400
and read_repair_chance = 1.0
and gc_grace = 3600
and min_compaction_threshold = 4
and max_compaction_threshold = 32
and replicate_on_write = true
and row_cache_provider = 'SerializingCacheProvider'
and compaction_strategy =
'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'
and compaction_strategy_options = {'sstable_size_in_mb' : '10'};
Any idea why compaction stopped automatically for this CF?
thanks
Ramesh
On Wed, Oct 26, 2011 at 5:23 AM, Sylvain Lebresne <sylv...@datastax.com> wrote:
> The estimate for the number of keys is computed by summing the key
> estimate for each sstable of the CF. For each sstable, the estimate
> should be fairly good. However, that's when we sum all the sstable estimates
> that we can loose potentially a lot of precision if there is a lot of rows
> that
> have parts in different sstables. But that in turn would suggest a problem
> with compaction lacking badly behind, especially with leveled compaction.
>
> --
> Sylvain
>
> On Wed, Oct 26, 2011 at 3:58 AM, Terry Cumaranatunge <cumar...@gmail.com>
> wrote:
>> I have a cluster of 8 nodes all running 1.0. The stats shown on the 1st node
>> on one of the CFs for the number of keys is much larger than expected. The
>> first node shows the key count estimate to be 9.2M whereas the rest report
>> ~650K on each node. The 650K is in the correct neighborhood of the number of
>> keys that have been inserted. The counts are comparable for all other CFs
>> across the cluster. I'm using Level compaction, but no compression.
>>
>> The 'nodetool ring' shows that the load is equal across all nodes. What
>> could cause this large disparity in the number of keys? Is this just a stats
>> issue or does this suggest a functional problem?
>>
>> 1st node:
>> Column Family: uid
>> SSTable count: 395
>> Space used (live): 13752624444
>> Space used (total): 5482088532
>> Number of Keys (estimate): 9215104
>> Memtable Columns Count: 514952
>> Memtable Data Size: 295213448
>> Memtable Switch Count: 290
>> Read Count: 193102511
>> Read Latency: 0.146 ms.
>> Write Count: 176934874
>> Write Latency: 0.018 ms.
>> Pending Tasks: 0
>> Key cache capacity: 8302131
>> Key cache size: 8302131
>> Key cache hit rate: 0.8644664668071792
>> Row cache: disabled
>> Compacted row minimum size: 87
>> Compacted row maximum size: 7007506
>> Compacted row mean size: 8944
>> 2nd node:
>> Column Family: uid
>> SSTable count: 402
>> Space used (live): 13723958304
>> Space used (total): 4044833220
>> Number of Keys (estimate): 652928
>> Memtable Columns Count: 170290
>> Memtable Data Size: 102378904
>> Memtable Switch Count: 272
>> Read Count: 192463595
>> Read Latency: 0.289 ms.
>> Write Count: 176527238
>> Write Latency: 0.014 ms.
>> Pending Tasks: 0
>> Key cache capacity: 8783058
>> Key cache size: 8783058
>> Key cache hit rate: 0.7865727464740025
>> Row cache: disabled
>> Compacted row minimum size: 87
>> Compacted row maximum size: 7007506
>> Compacted row mean size: 12151
>> 3rd node:
>> Column Family: uid
>> SSTable count: 401
>> Space used (live): 13204714872
>> Space used (total): 4030024144
>> Number of Keys (estimate): 675968
>> Memtable Columns Count: 42881
>> Memtable Data Size: 30992298
>> Memtable Switch Count: 304
>> Read Count: 190769879
>> Read Latency: 0.224 ms.
>> Write Count: 175381826
>> Write Latency: 0.014 ms.
>> Pending Tasks: 0
>> Key cache capacity: 8920108
>> Key cache size: 8920108
>> Key cache hit rate: 0.8053563128870577
>> Row cache: disabled
>> Compacted row minimum size: 87
>> Compacted row maximum size: 4866323
>> Compacted row mean size: 12074
>>
>
