Hi All, We're taking advantage of the lull in traffic to go through a production cluster upgrade from 1.2.x (latest) to 2.0.x (latest). We have three nodes with a replication factor of three. I've noticed that the 'space used' has almost doubled as a result of running 'nodetool upgradesstables'.
Anyone have any ideas? Is that to be expected? For comparison, on a node (pre-upgrade): > nodetool cfstats dp.s_evt > Keyspace: dp > Read Count: 190570567 > Read Latency: 2.6280611004164145 ms. > Write Count: 46213651 > Write Latency: 0.08166790944519835 ms. > Pending Tasks: 0 > Column Family: s_evt > SSTable count: 8 > Space used (live): 36269415929 > Space used (total): 36274282945 > SSTable Compression Ratio: 0.23450301499990572 > Number of Keys (estimate): 3213696 > Memtable Columns Count: 2934 > Memtable Data Size: 9561951 > Memtable Switch Count: 1974 > Read Count: 190570567 > Read Latency: 2.628 ms. > Write Count: 46213651 > Write Latency: 0.082 ms. > Pending Tasks: 0 > Bloom Filter False Positives: 1162636 > Bloom Filter False Ratio: 0.73869 > Bloom Filter Space Used: 4492256 > Compacted row minimum size: 373 > Compacted row maximum size: 1996099046 > Compacted row mean size: 63595 > Average live cells per slice (last five minutes): 11.0 > Average tombstones per slice (last five minutes): 0.0 And after upgrading and running 'upgradesstables' (different node): > nodetool cfstats dp.s_evt > Keyspace: dp > Read Count: 1461617 > Read Latency: 4.9734411921864625 ms. > Write Count: 359250 > Write Latency: 0.11328054279749478 ms. > Pending Tasks: 0 > Table: s_evt > SSTable count: 6 > Space used (live), bytes: 71266932602 > Space used (total), bytes: 71266932602 > Off heap memory used (total), bytes: 44853104 > SSTable Compression Ratio: 0.2387480210082192 > Number of keys (estimate): 3307776 > Memtable cell count: 603223 > Memtable data size, bytes: 121913569 > Memtable switch count: 9 > Local read count: 1461617 > Local read latency: 7.248 ms > Local write count: 359250 > Local write latency: 0.110 ms > Pending tasks: 0 > Bloom filter false positives: 2501 > Bloom filter false ratio: 0.01118 > Bloom filter space used, bytes: 4135248 > Bloom filter off heap memory used, bytes: 4135200 > Index summary off heap memory used, bytes: 723576 > Compression metadata off heap memory used, bytes: 39994328 > Compacted partition minimum bytes: 536 > Compacted partition maximum bytes: 2874382626 > Compacted partition mean bytes: 108773 > Average live cells per slice (last five minutes): 11.0 > Average tombstones per slice (last five minutes): 17.0 Column familiy definition: > create column family s_evt with column_type = 'Super' and comparator = > 'TimeUUIDType' and subcomparator = 'UTF8Type'; Also curious why the 'Average tombstones per slice' value has gone from 0 to 17. Note sure if its relevant but way back when we used to write values to that (super) column family with a TTL, but for a long time now its been append only (with no TTL)... Thanks, Dan
