I have a cluster of 8 nodes all running 1.0. The stats shown on the 1st node
on one of the CFs for the number of keys is much larger than expected. The
first node shows the key count estimate to be 9.2M whereas the rest report
~650K on each node. The 650K is in the correct neighborhood of the number of
keys that have been inserted. The counts are comparable for all other CFs
across the cluster. I'm using Level compaction, but no compression.
The 'nodetool ring' shows that the load is equal across all nodes. What
could cause this large disparity in the number of keys? Is this just a stats
issue or does this suggest a functional problem?
1st node:
Column Family: uid
SSTable count: 395
Space used (live): 13752624444
Space used (total): 5482088532
Number of Keys (estimate): 9215104
Memtable Columns Count: 514952
Memtable Data Size: 295213448
Memtable Switch Count: 290
Read Count: 193102511
Read Latency: 0.146 ms.
Write Count: 176934874
Write Latency: 0.018 ms.
Pending Tasks: 0
Key cache capacity: 8302131
Key cache size: 8302131
Key cache hit rate: 0.8644664668071792
Row cache: disabled
Compacted row minimum size: 87
Compacted row maximum size: 7007506
Compacted row mean size: 8944
2nd node:
Column Family: uid
SSTable count: 402
Space used (live): 13723958304
Space used (total): 4044833220
Number of Keys (estimate): 652928
Memtable Columns Count: 170290
Memtable Data Size: 102378904
Memtable Switch Count: 272
Read Count: 192463595
Read Latency: 0.289 ms.
Write Count: 176527238
Write Latency: 0.014 ms.
Pending Tasks: 0
Key cache capacity: 8783058
Key cache size: 8783058
Key cache hit rate: 0.7865727464740025
Row cache: disabled
Compacted row minimum size: 87
Compacted row maximum size: 7007506
Compacted row mean size: 12151
3rd node:
Column Family: uid
SSTable count: 401
Space used (live): 13204714872
Space used (total): 4030024144
Number of Keys (estimate): 675968
Memtable Columns Count: 42881
Memtable Data Size: 30992298
Memtable Switch Count: 304
Read Count: 190769879
Read Latency: 0.224 ms.
Write Count: 175381826
Write Latency: 0.014 ms.
Pending Tasks: 0
Key cache capacity: 8920108
Key cache size: 8920108
Key cache hit rate: 0.8053563128870577
Row cache: disabled
Compacted row minimum size: 87
Compacted row maximum size: 4866323
Compacted row mean size: 12074
