Thanks for the response Jeff. It wasn't snapshots but after running
upgradesstables on all nodes I started a repair and it seems like the file
sizes are reducing:
INFO [CompactionExecutor:1626] 2017-12-30 19:42:36,065 CompactionTask.java
(line 299) Compacted 2 sstables to
[/appdata/lib/cassandra/data/dp/s_evt/dp-s_evt-jb-302,]. 9,663,696,752
bytes to 4,834,895,601 (~50% of original) in 3,899,888ms =
1.182320MB/s. 90,533
total partitions merged to 45,278. Partition merge counts were {1:23,
2:45255, }
Dan Washusen
On Sun, Dec 31, 2017 at 1:51 AM, Jeff Jirsa <jji...@gmail.com> wrote:
> 1.2 to 2.0 was a long time ago for many of us, but I don’t recall anything
> that should have doubled size other than perhaps temporarily during the
> sstable rewrite or snapshots (which may? Be automatic on upgrade).
>
> The bloom filters, sstable count, compression ratio in cfstats all look
> similar, only the size is double, so that sorta hints st maybe a snapshot
>
> You have few sstables, looks like STCS, so it’d be possible that if the
> upgrade is still running, maybe one sstable of the old version still
> (temporarily) exists on disk causing it to be double counted.
>
>
>
> --
> Jeff Jirsa
>
>
> On Dec 29, 2017, at 4:33 PM, Dan Washusen <d...@reactive.org> wrote:
>
> Hi All,
> We're taking advantage of the lull in traffic to go through a production
> cluster upgrade from 1.2.x (latest) to 2.0.x (latest). We have three nodes
> with a replication factor of three. I've noticed that the 'space used' has
> almost doubled as a result of running 'nodetool upgradesstables'.
>
> Anyone have any ideas? Is that to be expected?
>
> For comparison, on a node (pre-upgrade):
>
>> nodetool cfstats dp.s_evt
>> Keyspace: dp
>> Read Count: 190570567
>> Read Latency: 2.6280611004164145 ms.
>> Write Count: 46213651
>> Write Latency: 0.08166790944519835 ms.
>> Pending Tasks: 0
>> Column Family: s_evt
>> SSTable count: 8
>> Space used (live): 36269415929
>> Space used (total): 36274282945
>> SSTable Compression Ratio: 0.23450301499990572
>> Number of Keys (estimate): 3213696
>> Memtable Columns Count: 2934
>> Memtable Data Size: 9561951
>> Memtable Switch Count: 1974
>> Read Count: 190570567
>> Read Latency: 2.628 ms.
>> Write Count: 46213651
>> Write Latency: 0.082 ms.
>> Pending Tasks: 0
>> Bloom Filter False Positives: 1162636
>> Bloom Filter False Ratio: 0.73869
>> Bloom Filter Space Used: 4492256
>> Compacted row minimum size: 373
>> Compacted row maximum size: 1996099046
>> Compacted row mean size: 63595
>> Average live cells per slice (last five minutes): 11.0
>> Average tombstones per slice (last five minutes): 0.0
>
>
> And after upgrading and running 'upgradesstables' (different node):
>
>> nodetool cfstats dp.s_evt
>> Keyspace: dp
>> Read Count: 1461617
>> Read Latency: 4.9734411921864625 ms.
>> Write Count: 359250
>> Write Latency: 0.11328054279749478 ms.
>> Pending Tasks: 0
>> Table: s_evt
>> SSTable count: 6
>> Space used (live), bytes: 71266932602
>> Space used (total), bytes: 71266932602
>> Off heap memory used (total), bytes: 44853104
>> SSTable Compression Ratio: 0.2387480210082192
>> Number of keys (estimate): 3307776
>> Memtable cell count: 603223
>> Memtable data size, bytes: 121913569
>> Memtable switch count: 9
>> Local read count: 1461617
>> Local read latency: 7.248 ms
>> Local write count: 359250
>> Local write latency: 0.110 ms
>> Pending tasks: 0
>> Bloom filter false positives: 2501
>> Bloom filter false ratio: 0.01118
>> Bloom filter space used, bytes: 4135248
>> Bloom filter off heap memory used, bytes: 4135200
>> Index summary off heap memory used, bytes: 723576
>> Compression metadata off heap memory used, bytes: 39994328
>> Compacted partition minimum bytes: 536
>> Compacted partition maximum bytes: 2874382626
>> Compacted partition mean bytes: 108773
>> Average live cells per slice (last five minutes): 11.0
>> Average tombstones per slice (last five minutes): 17.0
>
>
> Column familiy definition:
>
>> create column family s_evt with column_type = 'Super' and comparator =
>> 'TimeUUIDType' and subcomparator = 'UTF8Type';
>
>
>
> Also curious why the 'Average tombstones per slice' value has gone from 0
> to 17. Note sure if its relevant but way back when we used to write values
> to that (super) column family with a TTL, but for a long time now its been
> append only (with no TTL)...
>
> Thanks,
> Dan
>
>
