Sylvain - We have similar problem but the discrepancy is not that big.
Do we have to do major compaction to fix it? We did not do 'nodetool
compact', just did repair regularly, which triggers minor compaction.
Thanks,
Daning
On 10/26/2011 03:23 AM, Sylvain Lebresne wrote:
The estimate for the number of keys is computed by summing the key
estimate for each sstable of the CF. For each sstable, the estimate
should be fairly good. However, that's when we sum all the sstable estimates
that we can loose potentially a lot of precision if there is a lot of rows that
have parts in different sstables. But that in turn would suggest a problem
with compaction lacking badly behind, especially with leveled compaction.
--
Sylvain
On Wed, Oct 26, 2011 at 3:58 AM, Terry Cumaranatunge<cumar...@gmail.com> wrote:
I have a cluster of 8 nodes all running 1.0. The stats shown on the 1st node
on one of the CFs for the number of keys is much larger than expected. The
first node shows the key count estimate to be 9.2M whereas the rest report
~650K on each node. The 650K is in the correct neighborhood of the number of
keys that have been inserted. The counts are comparable for all other CFs
across the cluster. I'm using Level compaction, but no compression.
The 'nodetool ring' shows that the load is equal across all nodes. What
could cause this large disparity in the number of keys? Is this just a stats
issue or does this suggest a functional problem?
1st node:
Column Family: uid
SSTable count: 395
Space used (live): 13752624444
Space used (total): 5482088532
Number of Keys (estimate): 9215104
Memtable Columns Count: 514952
Memtable Data Size: 295213448
Memtable Switch Count: 290
Read Count: 193102511
Read Latency: 0.146 ms.
Write Count: 176934874
Write Latency: 0.018 ms.
Pending Tasks: 0
Key cache capacity: 8302131
Key cache size: 8302131
Key cache hit rate: 0.8644664668071792
Row cache: disabled
Compacted row minimum size: 87
Compacted row maximum size: 7007506
Compacted row mean size: 8944
2nd node:
Column Family: uid
SSTable count: 402
Space used (live): 13723958304
Space used (total): 4044833220
Number of Keys (estimate): 652928
Memtable Columns Count: 170290
Memtable Data Size: 102378904
Memtable Switch Count: 272
Read Count: 192463595
Read Latency: 0.289 ms.
Write Count: 176527238
Write Latency: 0.014 ms.
Pending Tasks: 0
Key cache capacity: 8783058
Key cache size: 8783058
Key cache hit rate: 0.7865727464740025
Row cache: disabled
Compacted row minimum size: 87
Compacted row maximum size: 7007506
Compacted row mean size: 12151
3rd node:
Column Family: uid
SSTable count: 401
Space used (live): 13204714872
Space used (total): 4030024144
Number of Keys (estimate): 675968
Memtable Columns Count: 42881
Memtable Data Size: 30992298
Memtable Switch Count: 304
Read Count: 190769879
Read Latency: 0.224 ms.
Write Count: 175381826
Write Latency: 0.014 ms.
Pending Tasks: 0
Key cache capacity: 8920108
Key cache size: 8920108
Key cache hit rate: 0.8053563128870577
Row cache: disabled
Compacted row minimum size: 87
Compacted row maximum size: 4866323
Compacted row mean size: 12074
