All right. I’ll clarify then. -0 on switching the default to G1 *this late* just before RC1. -1 on switching the default offheap_objects *for 4.1 RC1*, but all for it in principle, for 4.2, after we run some more test and resolve the concerns raised by Jeff.
Let’s please try to avoid this kind of super late defaults switch going forward? — AY > On 16 Nov 2022, at 03:27, Derek Chen-Becker <de...@chen-becker.org> wrote: > > For the record, I'm +100 on G1. Take it with whatever sized grain of > salt you think appropriate for a relative newcomer to the list, but > I've spent my last 7-8 years dealing with the intersection of > high-throughput, low latency systems and their interaction with GC and > in my personal experience G1 outperforms CMS in all cases and with > significantly less work (zero work, in many cases). The only things > I've seen perform better *with a similar heap footprint* are GenShen > (currently experimental) and Rust (beyond the scope of this topic). > > Derek > > On Tue, Nov 15, 2022 at 4:51 PM Jon Haddad <rustyrazorbl...@apache.org> wrote: >> >> I'm curious what it would take for folks to be OK with merging this into >> 4.1? How much additional time would you want to feel comfortable? >> >> I should probably have been a little more vigorous in my +1 of Mick's PR. >> For a little background - I worked on several hundred clusters while at TLP, >> mostly dealing with stability and performance issues. A lot of them stemmed >> partially or wholly from the GC settings we ship in the project. Par New >> with CMS and small new gen results in a lot of premature promotion leading >> to high pause times into the hundreds of ms which pushes p99 latency through >> the roof. >> >> I'm a big +1 in favor of G1 because it's not just better for most people but >> it's better for _every_ new Cassandra user. The first experience that >> people have with the project is important, and our current GC settings are >> quite bad - so bad they lead to problems with stability in production. The >> G1 settings are mostly hands off, result in shorter pause times and are a >> big improvement over the status quo. >> >> Most folks don't do GC tuning, they use what we supply, and what we >> currently supply leads to a poor initial experience with the database. I >> think we owe the community our best effort even if it means pushing the >> release back little bit. >> >> Just for some additional context, we're (Netflix) running 25K nodes on G1 >> across a variety of hardware in AWS with wildly varying workloads, and I >> haven't seen G1 be the root cause of a problem even once. The settings that >> Mick is proposing are almost identical to what we use (we use half of heap >> up to 30GB). >> >> I'd really appreciate it if we took a second to consider the community >> effect of another release that ships settings that cause significant pain >> for our users. >> >> Jon >> >> On 2022/11/10 21:49:36 Mick Semb Wever wrote: >>>> >>>> In case of GC, reasonably extensive performance testing should be the >>>> expectations. Potentially revisiting some of the G1 params for the 4.1 >>>> reality - quite a lot has changed since those optional defaults where >>>> picked. >>>> >>> >>> >>> I've put our battle-tested g1 opts (from consultants at TLP and DataStax) >>> in the patch for CASSANDRA-18027 >>> >>> In reality it is really not much of a change, g1 does make it simple. >>> Picking the correct ParallelGCThreads and ConcGCThreads and the floor to >>> the new heap (XX:NewSize) is still required, though we could do a much >>> better job of dynamic defaults to them. >>> >>> Alex Dejanovski's blog is a starting point: >>> https://thelastpickle.com/blog/2020/06/29/cassandra_4-0_garbage_collectors_performance_benchmarks.html >>> where this gc opt set was used (though it doesn't prove why those options >>> are chosen) >>> >>> The bar for objection to sneaking these into 4.1 was intended to be low, >>> and I stand by those that raise concerns. >>> > > > > -- > +---------------------------------------------------------------+ > | Derek Chen-Becker | > | GPG Key available at https://keybase.io/dchenbecker and | > | https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org | > | Fngrprnt: EB8A 6480 F0A3 C8EB C1E7 7F42 AFC5 AFEE 96E4 6ACC | > +---------------------------------------------------------------+
