To clarify: -0 here on G1 as default for 4.1 as well; I'd like us to prioritize digging into G1's behavior on small heaps vs. CMS w/our default tuning sooner rather than later. With that info I'd likely be a strong +1 on the shift.
-1 on switching to offheap_objects for 4.1 RC; again, think this is just a small step away from being a +1 w/some more rigor around seeing the current state of the technology's intersections.
All right. I’ll clarify then.
-0 on switching the default to G1 *this late* just before RC1.
-1 on switching the default offheap_objects *for 4.1 RC1*, but all for it in principle, for 4.2, after we run some more test and resolve the concerns raised by Jeff.
Let’s please try to avoid this kind of super late defaults switch going forward?
—
AY
>
> For the record, I'm +100 on G1. Take it with whatever sized grain of
> salt you think appropriate for a relative newcomer to the list, but
> I've spent my last 7-8 years dealing with the intersection of
> high-throughput, low latency systems and their interaction with GC and
> in my personal experience G1 outperforms CMS in all cases and with
> significantly less work (zero work, in many cases). The only things
> I've seen perform better *with a similar heap footprint* are GenShen
> (currently experimental) and Rust (beyond the scope of this topic).
>
> Derek
>
>>
>> I'm curious what it would take for folks to be OK with merging this into 4.1? How much additional time would you want to feel comfortable?
>>
>> I should probably have been a little more vigorous in my +1 of Mick's PR. For a little background - I worked on several hundred clusters while at TLP, mostly dealing with stability and performance issues. A lot of them stemmed partially or wholly from the GC settings we ship in the project. Par New with CMS and small new gen results in a lot of premature promotion leading to high pause times into the hundreds of ms which pushes p99 latency through the roof.
>>
>> I'm a big +1 in favor of G1 because it's not just better for most people but it's better for _every_ new Cassandra user. The first experience that people have with the project is important, and our current GC settings are quite bad - so bad they lead to problems with stability in production. The G1 settings are mostly hands off, result in shorter pause times and are a big improvement over the status quo.
>>
>> Most folks don't do GC tuning, they use what we supply, and what we currently supply leads to a poor initial experience with the database. I think we owe the community our best effort even if it means pushing the release back little bit.
>>
>> Just for some additional context, we're (Netflix) running 25K nodes on G1 across a variety of hardware in AWS with wildly varying workloads, and I haven't seen G1 be the root cause of a problem even once. The settings that Mick is proposing are almost identical to what we use (we use half of heap up to 30GB).
>>
>> I'd really appreciate it if we took a second to consider the community effect of another release that ships settings that cause significant pain for our users.
>>
>> Jon
>>
>> On 2022/11/10 21:49:36 Mick Semb Wever wrote:
>>>>
>>>> In case of GC, reasonably extensive performance testing should be the
>>>> expectations. Potentially revisiting some of the G1 params for the 4.1
>>>> reality - quite a lot has changed since those optional defaults where
>>>> picked.
>>>>
>>>
>>>
>>> I've put our battle-tested g1 opts (from consultants at TLP and DataStax)
>>> in the patch for CASSANDRA-18027
>>>
>>> In reality it is really not much of a change, g1 does make it simple.
>>> Picking the correct ParallelGCThreads and ConcGCThreads and the floor to
>>> the new heap (XX:NewSize) is still required, though we could do a much
>>> better job of dynamic defaults to them.
>>>
>>> Alex Dejanovski's blog is a starting point:
>>> where this gc opt set was used (though it doesn't prove why those options
>>> are chosen)
>>>
>>> The bar for objection to sneaking these into 4.1 was intended to be low,
>>> and I stand by those that raise concerns.
>>>
>
>
>
> --
> +---------------------------------------------------------------+
> | Derek Chen-Becker |
> | Fngrprnt: EB8A 6480 F0A3 C8EB C1E7 7F42 AFC5 AFEE 96E4 6ACC |
> +---------------------------------------------------------------+
