Nope, just ran out of disk space... So on 1.2.x I had 70GB used on a 200GB
disk and everything was great, with 2.0.x I'm now at 99% used and getting
exception while compacting about insufficient disk space. FML...
Dan Washusen
On Sun, Dec 31, 2017 at 6:47 AM, Dan Washusen <d...@reactive.org> wrote:
> Thanks for the response Jeff. It wasn't snapshots but after running
> upgradesstables on all nodes I started a repair and it seems like the file
> sizes are reducing:
>
> INFO [CompactionExecutor:1626] 2017-12-30 19:42:36,065
> CompactionTask.java (line 299) Compacted 2 sstables to
> [/appdata/lib/cassandra/data/dp/s_evt/dp-s_evt-jb-302,]. 9,663,696,752
> bytes to 4,834,895,601 (~50% of original) in 3,899,888ms = 1.182320MB/s.
> 90,533
> total partitions merged to 45,278. Partition merge counts were {1:23,
> 2:45255, }
>
> Dan Washusen
>
>
> On Sun, Dec 31, 2017 at 1:51 AM, Jeff Jirsa <jji...@gmail.com> wrote:
>
>> 1.2 to 2.0 was a long time ago for many of us, but I don’t recall
>> anything that should have doubled size other than perhaps temporarily
>> during the sstable rewrite or snapshots (which may? Be automatic on
>> upgrade).
>>
>> The bloom filters, sstable count, compression ratio in cfstats all look
>> similar, only the size is double, so that sorta hints st maybe a snapshot
>>
>> You have few sstables, looks like STCS, so it’d be possible that if the
>> upgrade is still running, maybe one sstable of the old version still
>> (temporarily) exists on disk causing it to be double counted.
>>
>>
>>
>> --
>> Jeff Jirsa
>>
>>
>> On Dec 29, 2017, at 4:33 PM, Dan Washusen <d...@reactive.org> wrote:
>>
>> Hi All,
>> We're taking advantage of the lull in traffic to go through a production
>> cluster upgrade from 1.2.x (latest) to 2.0.x (latest). We have three nodes
>> with a replication factor of three. I've noticed that the 'space used' has
>> almost doubled as a result of running 'nodetool upgradesstables'.
>>
>> Anyone have any ideas? Is that to be expected?
>>
>> For comparison, on a node (pre-upgrade):
>>
>>> nodetool cfstats dp.s_evt
>>> Keyspace: dp
>>> Read Count: 190570567
>>> Read Latency: 2.6280611004164145 ms.
>>> Write Count: 46213651
>>> Write Latency: 0.08166790944519835 ms.
>>> Pending Tasks: 0
>>> Column Family: s_evt
>>> SSTable count: 8
>>> Space used (live): 36269415929
>>> Space used (total): 36274282945
>>> SSTable Compression Ratio: 0.23450301499990572
>>> Number of Keys (estimate): 3213696
>>> Memtable Columns Count: 2934
>>> Memtable Data Size: 9561951
>>> Memtable Switch Count: 1974
>>> Read Count: 190570567
>>> Read Latency: 2.628 ms.
>>> Write Count: 46213651
>>> Write Latency: 0.082 ms.
>>> Pending Tasks: 0
>>> Bloom Filter False Positives: 1162636
>>> Bloom Filter False Ratio: 0.73869
>>> Bloom Filter Space Used: 4492256
>>> Compacted row minimum size: 373
>>> Compacted row maximum size: 1996099046
>>> Compacted row mean size: 63595
>>> Average live cells per slice (last five minutes): 11.0
>>> Average tombstones per slice (last five minutes): 0.0
>>
>>
>> And after upgrading and running 'upgradesstables' (different node):
>>
>>> nodetool cfstats dp.s_evt
>>> Keyspace: dp
>>> Read Count: 1461617
>>> Read Latency: 4.9734411921864625 ms.
>>> Write Count: 359250
>>> Write Latency: 0.11328054279749478 ms.
>>> Pending Tasks: 0
>>> Table: s_evt
>>> SSTable count: 6
>>> Space used (live), bytes: 71266932602
>>> Space used (total), bytes: 71266932602
>>> Off heap memory used (total), bytes: 44853104
>>> SSTable Compression Ratio: 0.2387480210082192
>>> Number of keys (estimate): 3307776
>>> Memtable cell count: 603223
>>> Memtable data size, bytes: 121913569
>>> Memtable switch count: 9
>>> Local read count: 1461617
>>> Local read latency: 7.248 ms
>>> Local write count: 359250
>>> Local write latency: 0.110 ms
>>> Pending tasks: 0
>>> Bloom filter false positives: 2501
>>> Bloom filter false ratio: 0.01118
>>> Bloom filter space used, bytes: 4135248
>>> Bloom filter off heap memory used, bytes: 4135200
>>> Index summary off heap memory used, bytes: 723576
>>> Compression metadata off heap memory used, bytes:
>>> 39994328
>>> Compacted partition minimum bytes: 536
>>> Compacted partition maximum bytes: 2874382626
>>> Compacted partition mean bytes: 108773
>>> Average live cells per slice (last five minutes): 11.0
>>> Average tombstones per slice (last five minutes): 17.0
>>
>>
>> Column familiy definition:
>>
>>> create column family s_evt with column_type = 'Super' and comparator =
>>> 'TimeUUIDType' and subcomparator = 'UTF8Type';
>>
>>
>>
>> Also curious why the 'Average tombstones per slice' value has gone from 0
>> to 17. Note sure if its relevant but way back when we used to write values
>> to that (super) column family with a TTL, but for a long time now its been
>> append only (with no TTL)...
>>
>> Thanks,
>> Dan
>>
>>
>
