I'm curious what it would take for folks to be OK with merging this into 4.1? How much additional time would you want to feel comfortable?
I should probably have been a little more vigorous in my +1 of Mick's PR. For a little background - I worked on several hundred clusters while at TLP, mostly dealing with stability and performance issues. A lot of them stemmed partially or wholly from the GC settings we ship in the project. Par New with CMS and small new gen results in a lot of premature promotion leading to high pause times into the hundreds of ms which pushes p99 latency through the roof. I'm a big +1 in favor of G1 because it's not just better for most people but it's better for _every_ new Cassandra user. The first experience that people have with the project is important, and our current GC settings are quite bad - so bad they lead to problems with stability in production. The G1 settings are mostly hands off, result in shorter pause times and are a big improvement over the status quo. Most folks don't do GC tuning, they use what we supply, and what we currently supply leads to a poor initial experience with the database. I think we owe the community our best effort even if it means pushing the release back little bit. Just for some additional context, we're (Netflix) running 25K nodes on G1 across a variety of hardware in AWS with wildly varying workloads, and I haven't seen G1 be the root cause of a problem even once. The settings that Mick is proposing are almost identical to what we use (we use half of heap up to 30GB). I'd really appreciate it if we took a second to consider the community effect of another release that ships settings that cause significant pain for our users. Jon On 2022/11/10 21:49:36 Mick Semb Wever wrote: > > > > In case of GC, reasonably extensive performance testing should be the > > expectations. Potentially revisiting some of the G1 params for the 4.1 > > reality - quite a lot has changed since those optional defaults where > > picked. > > > > > I've put our battle-tested g1 opts (from consultants at TLP and DataStax) > in the patch for CASSANDRA-18027 > > In reality it is really not much of a change, g1 does make it simple. > Picking the correct ParallelGCThreads and ConcGCThreads and the floor to > the new heap (XX:NewSize) is still required, though we could do a much > better job of dynamic defaults to them. > > Alex Dejanovski's blog is a starting point: > https://thelastpickle.com/blog/2020/06/29/cassandra_4-0_garbage_collectors_performance_benchmarks.html > where this gc opt set was used (though it doesn't prove why those options > are chosen) > > The bar for objection to sneaking these into 4.1 was intended to be low, > and I stand by those that raise concerns. >
