I'm fine with not including G1 in 4.1, but would we consider inclusion for 4.1.X down the road once validation has been done?
Derek On Wed, Nov 16, 2022 at 4:39 PM David Capwell <dcapw...@apple.com> wrote: > > Getting poked in Slack to be more explicit in this thread… > > Switching to G1 on trunk, +1 > Switching to G1 on 4.1, -1. 4.1 is about to be released and this isn’t a bug > fix but a perf improvement ticket and as such should go through validation > that the perf improvements are seen, there is not enough time left for that > added performance work burden so strongly feel it should be pushed to 4.2/5.0 > where it has plenty of time to be validated against. The ticket even asks to > avoid validating the claims; saying 'Hoping we can skip due diligence on this > ticket because the data is "in the past” already”'. Others have attempted > both shenandoah and ZGC and found mixed results, so nothing leads me to > believe that won’t be true here either. > > > On Nov 16, 2022, at 9:15 AM, J. D. Jordan <jeremiah.jor...@gmail.com> wrote: > > > > Heap - > > +1 for G1 in trunk > > +0 for G1 in 4.1 - I think it’s worthwhile and fairly well tested but I > > understand pushback against changing this so late in the game. > > > > Memtable - > > -1 for off heap in 4.1. I think this needs more testing and isn’t something > > to change at the last minute. > > +1 for running performance/fuzz tests against the alternate memtable > > choices in trunk and switching if they don’t show regressions. > > > >> On Nov 16, 2022, at 10:48 AM, Josh McKenzie <jmcken...@apache.org> wrote: > >> > >> > >> To clarify: -0 here on G1 as default for 4.1 as well; I'd like us to > >> prioritize digging into G1's behavior on small heaps vs. CMS w/our default > >> tuning sooner rather than later. With that info I'd likely be a strong +1 > >> on the shift. > >> > >> -1 on switching to offheap_objects for 4.1 RC; again, think this is just a > >> small step away from being a +1 w/some more rigor around seeing the > >> current state of the technology's intersections. > >> > >> On Wed, Nov 16, 2022, at 7:47 AM, Aleksey Yeshchenko wrote: > >>> All right. I’ll clarify then. > >>> > >>> -0 on switching the default to G1 *this late* just before RC1. > >>> -1 on switching the default offheap_objects *for 4.1 RC1*, but all for it > >>> in principle, for 4.2, after we run some more test and resolve the > >>> concerns raised by Jeff. > >>> > >>> Let’s please try to avoid this kind of super late defaults switch going > >>> forward? > >>> > >>> — > >>> AY > >>> > >>> > On 16 Nov 2022, at 03:27, Derek Chen-Becker <de...@chen-becker.org> > >>> > wrote: > >>> > > >>> > For the record, I'm +100 on G1. Take it with whatever sized grain of > >>> > salt you think appropriate for a relative newcomer to the list, but > >>> > I've spent my last 7-8 years dealing with the intersection of > >>> > high-throughput, low latency systems and their interaction with GC and > >>> > in my personal experience G1 outperforms CMS in all cases and with > >>> > significantly less work (zero work, in many cases). The only things > >>> > I've seen perform better *with a similar heap footprint* are GenShen > >>> > (currently experimental) and Rust (beyond the scope of this topic). > >>> > > >>> > Derek > >>> > > >>> > On Tue, Nov 15, 2022 at 4:51 PM Jon Haddad <rustyrazorbl...@apache.org> > >>> > wrote: > >>> >> > >>> >> I'm curious what it would take for folks to be OK with merging this > >>> >> into 4.1? How much additional time would you want to feel comfortable? > >>> >> > >>> >> I should probably have been a little more vigorous in my +1 of Mick's > >>> >> PR. For a little background - I worked on several hundred clusters > >>> >> while at TLP, mostly dealing with stability and performance issues. A > >>> >> lot of them stemmed partially or wholly from the GC settings we ship > >>> >> in the project. Par New with CMS and small new gen results in a lot of > >>> >> premature promotion leading to high pause times into the hundreds of > >>> >> ms which pushes p99 latency through the roof. > >>> >> > >>> >> I'm a big +1 in favor of G1 because it's not just better for most > >>> >> people but it's better for _every_ new Cassandra user. The first > >>> >> experience that people have with the project is important, and our > >>> >> current GC settings are quite bad - so bad they lead to problems with > >>> >> stability in production. The G1 settings are mostly hands off, result > >>> >> in shorter pause times and are a big improvement over the status quo. > >>> >> > >>> >> Most folks don't do GC tuning, they use what we supply, and what we > >>> >> currently supply leads to a poor initial experience with the database. > >>> >> I think we owe the community our best effort even if it means pushing > >>> >> the release back little bit. > >>> >> > >>> >> Just for some additional context, we're (Netflix) running 25K nodes on > >>> >> G1 across a variety of hardware in AWS with wildly varying workloads, > >>> >> and I haven't seen G1 be the root cause of a problem even once. The > >>> >> settings that Mick is proposing are almost identical to what we use > >>> >> (we use half of heap up to 30GB). > >>> >> > >>> >> I'd really appreciate it if we took a second to consider the community > >>> >> effect of another release that ships settings that cause significant > >>> >> pain for our users. > >>> >> > >>> >> Jon > >>> >> > >>> >> On 2022/11/10 21:49:36 Mick Semb Wever wrote: > >>> >>>> > >>> >>>> In case of GC, reasonably extensive performance testing should be the > >>> >>>> expectations. Potentially revisiting some of the G1 params for the > >>> >>>> 4.1 > >>> >>>> reality - quite a lot has changed since those optional defaults where > >>> >>>> picked. > >>> >>>> > >>> >>> > >>> >>> > >>> >>> I've put our battle-tested g1 opts (from consultants at TLP and > >>> >>> DataStax) > >>> >>> in the patch for CASSANDRA-18027 > >>> >>> > >>> >>> In reality it is really not much of a change, g1 does make it simple. > >>> >>> Picking the correct ParallelGCThreads and ConcGCThreads and the floor > >>> >>> to > >>> >>> the new heap (XX:NewSize) is still required, though we could do a much > >>> >>> better job of dynamic defaults to them. > >>> >>> > >>> >>> Alex Dejanovski's blog is a starting point: > >>> >>> https://thelastpickle.com/blog/2020/06/29/cassandra_4-0_garbage_collectors_performance_benchmarks.html > >>> >>> where this gc opt set was used (though it doesn't prove why those > >>> >>> options > >>> >>> are chosen) > >>> >>> > >>> >>> The bar for objection to sneaking these into 4.1 was intended to be > >>> >>> low, > >>> >>> and I stand by those that raise concerns. > >>> >>> > >>> > > >>> > > >>> > > >>> > -- > >>> > +---------------------------------------------------------------+ > >>> > | Derek Chen-Becker | > >>> > | GPG Key available at https://keybase.io/dchenbecker and | > >>> > | https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org | > >>> > | Fngrprnt: EB8A 6480 F0A3 C8EB C1E7 7F42 AFC5 AFEE 96E4 6ACC | > >>> > +---------------------------------------------------------------+ > >>> > >>> > >> > -- +---------------------------------------------------------------+ | Derek Chen-Becker | | GPG Key available at https://keybase.io/dchenbecker and | | https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org | | Fngrprnt: EB8A 6480 F0A3 C8EB C1E7 7F42 AFC5 AFEE 96E4 6ACC | +---------------------------------------------------------------+
