Amazing how I missed the -Dcassandra.disable_stcs_in_l0=true option - I have LeveledManifest source opened the whole day;-)
On Tue, Nov 18, 2014 at 9:15 PM, Andrei Ivanov <aiva...@iponweb.net> wrote: > Thanks a lot for your support, Marcus - that is useful beyond all > recognition!;-) And I will try #6621 right away. > > Sincerely, Andrei. > > On Tue, Nov 18, 2014 at 8:50 PM, Marcus Eriksson <krum...@gmail.com> wrote: >> you should stick to as small nodes as possible yes :) >> >> There are a few relevant tickets related to bootstrap and LCS: >> https://issues.apache.org/jira/browse/CASSANDRA-6621 - startup with >> -Dcassandra.disable_stcs_in_l0=true to not do STCS in L0 >> https://issues.apache.org/jira/browse/CASSANDRA-7460 - (3.0) send source >> sstable level when bootstrapping >> >> On Tue, Nov 18, 2014 at 3:33 PM, Andrei Ivanov <aiva...@iponweb.net> wrote: >>> >>> OK, got it. >>> >>> Actually, my problem is not that we constantly having many files at >>> L0. Normally, quite a few of them - that is, nodes are managing to >>> compact incoming writes in a timely manner. >>> >>> But it looks like when we join a new node, it receives tons of files >>> from existing nodes (and they end up at L0, right?) and that seems to >>> be where our problems start. In practice, in what I call the "old" >>> cluster, compaction became a problem at ~2TB nodes. (You, know, we are >>> trying to save something on HW - we are running on EC2 with EBS >>> volumes) >>> >>> Do I get it right that, we better stick to cmaller nodes? >>> >>> >>> >>> On Tue, Nov 18, 2014 at 5:20 PM, Marcus Eriksson <krum...@gmail.com> >>> wrote: >>> > No, they will get compacted into smaller sstables in L1+ eventually >>> > (once >>> > you have less than 32 sstables in L0, an ordinary L0 -> L1 compaction >>> > will >>> > happen) >>> > >>> > But, if you consistently get many files in L0 it means that compaction >>> > is >>> > not keeping up with your inserts and you should probably expand your >>> > cluster >>> > (or consider going back to SizeTieredCompactionStrategy for the tables >>> > that >>> > take that many writes) >>> > >>> > /Marcus >>> > >>> > On Tue, Nov 18, 2014 at 2:49 PM, Andrei Ivanov <aiva...@iponweb.net> >>> > wrote: >>> >> >>> >> Marcus, thanks a lot! It explains a lot those huge tables are indeed at >>> >> L0. >>> >> >>> >> It seems that they start to appear as a result of some "massive" >>> >> operations (join, repair, rebuild). What's their fate in the future? >>> >> Will they continue to propagate like this through levels? Is there >>> >> anything that can be done to avoid/solve/prevent this? >>> >> >>> >> My fears here are around a feeling that those big tables (like in my >>> >> "old" cluster) will be hardly compactable in the future... >>> >> >>> >> Sincerely, Andrei. >>> >> >>> >> On Tue, Nov 18, 2014 at 4:27 PM, Marcus Eriksson <krum...@gmail.com> >>> >> wrote: >>> >> > I suspect they are getting size tiered in L0 - if you have too many >>> >> > sstables >>> >> > in L0, we will do size tiered compaction on sstables in L0 to improve >>> >> > performance >>> >> > >>> >> > Use tools/bin/sstablemetadata to get the level for those sstables, if >>> >> > they >>> >> > are in L0, that is probably the reason. >>> >> > >>> >> > /Marcus >>> >> > >>> >> > On Tue, Nov 18, 2014 at 2:06 PM, Andrei Ivanov <aiva...@iponweb.net> >>> >> > wrote: >>> >> >> >>> >> >> Dear all, >>> >> >> >>> >> >> I have the following problem: >>> >> >> - C* 2.0.11 >>> >> >> - LCS with default 160MB >>> >> >> - Compacted partition maximum bytes: 785939 (for cf/table xxx.xxx) >>> >> >> - Compacted partition mean bytes: 6750 (for cf/table xxx.xxx) >>> >> >> >>> >> >> I would expect the sstables to be of +- maximum 160MB. Despite this >>> >> >> I >>> >> >> see files like: >>> >> >> 192M Nov 18 13:00 xxx-xxx-jb-15580-Data.db >>> >> >> or >>> >> >> 631M Nov 18 13:03 xxx-xxx-jb-15583-Data.db >>> >> >> >>> >> >> Am I missing something? What could be the reason? (Actually this is >>> >> >> a >>> >> >> "fresh" cluster - on an "old" one I'm seeing 500GB sstables). I'm >>> >> >> getting really desperate in my attempt to understand what's going >>> >> >> on. >>> >> >> >>> >> >> Thanks in advance Andrei. >>> >> > >>> >> > >>> > >>> > >> >>
