Thanks for bringing this up to discussion. > I really suffered from the use of Netty recyclers. It makes code much > really harder to maintain. I also made a benchmark and the recycler > allocation is 10~20x slower than a normal heap allocation. > https://gist.github.com/BewareMyPower/3dcc59183c92c76e9c985cced16d049d
Awesome! JMH is a great way to do microbenchmarks and also contains great integration to profilers such as async-profiler and Java Flight Recorder. It would be great to add more microbenchmarks to Pulsar's microbench module [1]. btw. Regarding the particular RecyclerBenchmark, when using JMH, preventing dead code elimination is needed for accurate benchmarks. The JIT compiler aggressively optimizes code it determines doesn't affect the program state. In benchmarks, this can lead to misleading results as operations you're trying to measure might be completely eliminated. The way to avoid this is to return a computed value from the test method or use JMH's org.openjdk.jmh.infra.Blackhole to consume a computed value. In many cases it doesn't change the results, but it's just a detail about correctness of a JMH benchmark. In RecyclerBenchmark, it would be useful to pass all field values to the JMH blackhole to make the comparison fair. > I know it's hard to convince existing code that already uses the Netty > recycler. But I hope for new code, if you want to use a recycler, > please show the benefit rather than a simple "I think it reduces the > GC overhead (though I'm not sure if it's true, but it should be > true)". As a contrast, I can also say modern GC is much stronger than > you might think, especially for short-lived objects. Yes, that's a valid point. For example ZGC Generational GC [2,3] is a significant improvement compared to available GC implementations 10 years ago. What has bothered me in the past is looking at "GC Pauses" in Grafana and assuming that all information is "pauses". It also turned out that the Grafana dashboard has an incorrect label. It's "GC time", but it's not necessarily a "pause" at all (there's a separate metric for a stop-the-world pause, this metric isn't even in the dashboard). This is explained in PR https://github.com/apache/pulsar/pull/18891 where I fixed this for the Grafana charts that are included in apache/pulsar. Using Netty Recycler, will also have a CPU overhead and we don't have a way to measure that. That's why just looking at the "GC time" metric isn't the best approach to even comparing using Netty Recycler or not. Since Netty Recycler can be disabled with system properties, it should be possible to test the system level impact of the Netty Recycler in Pulsar in an OMB test (or any other performance test suite) without making changes to Pulsar. Doing that would give us real data of the impact if we would just stop using the Netty Recycler in Pulsar and remove it. It would also be great to measure if disabling Netty Recycler with a system property makes the JMH benchmark achieve about a similar level of performance. As mentioned before, it would be useful to improve the benchmark to ensure that dead code elimination doesn't impact the results. This would raise the confidence of the results. -Lari 1 - running Pulsar JMH benchmarks - https://github.com/apache/pulsar/tree/master/microbench 2 - https://inside.java/2023/11/28/gen-zgc-explainer/ 3 - https://netflixtechblog.com/bending-pause-times-to-your-will-with-generational-zgc-256629c9386b On Sat, 22 Feb 2025 at 11:02, Yunze Xu <x...@apache.org> wrote: > > Bump this thread again. > > I really suffered from the use of Netty recyclers. It makes code much > really harder to maintain. I also made a benchmark and the recycler > allocation is 10~20x slower than a normal heap allocation. > https://gist.github.com/BewareMyPower/3dcc59183c92c76e9c985cced16d049d > > I know it's hard to convince existing code that already uses the Netty > recycler. But I hope for new code, if you want to use a recycler, > please show the benefit rather than a simple "I think it reduces the > GC overhead (though I'm not sure if it's true, but it should be > true)". As a contrast, I can also say modern GC is much stronger than > you might think, especially for short-lived objects. > > I believe the recycler was used everywhere just because the original > authors thought it would be good, without any real benchmark. I can > hardly see such tricks in other Java projects (e.g. Kafka). If you > know, feel free to share it. > > Thanks, > Yunze > > > > On Fri, Jun 28, 2024 at 11:10 AM Yunze Xu <x...@apache.org> wrote: > > > > Hi all, > > > > I'm doubting the value of the widely used Netty Recycler in Pulsar. > > When I checked the recent commits today, I found even a pair of > > Boolean and Integer is wrapped as a recyclable object. See > > TopicExistsInfo in https://github.com/apache/pulsar/pull/22838. It's > > really a mess, especially compared with a record implementation like: > > > > ```java > > public record TopicExistsInfo(boolean exists, int partitions) {} > > ``` > > > > There was a similar doubt in an issue from early days in 2016: > > https://github.com/netty/netty/issues/5904. We can also see > > https://github.com/elastic/elasticsearch/pull/22452 from that issue > > that ES disables the Netty recycler by default. Yeah, Netty even > > provides a way to disable the recycler. > > > > I don't look into the implementation at the moment so I asked ChatGPT for > > now: > > > > ---- > > Here are some cases when it is not recommended to use Netty Recycler: > > 1.Short-lived Objects: If objects in your application have very short > > lifecycles, meaning they are created and destroyed frequently and > > rapidly, using Recycler may add extra overhead as object pooling and > > reuse may not provide significant performance improvements. > > 2. High Thread Safety Requirements: If your application demands high > > thread safety for objects, and objects are passed between different > > threads frequently, using an object pool may introduce potential > > thread safety issues as object states are shared across threads. > > 3. Limited Memory Constraints: In some cases, object pools may consume > > additional memory, especially when a large number of objects need to > > be instantiated. If memory usage is a critical consideration, using an > > object pool may increase memory consumption. > > 4. Low Object Creation Cost: If the cost of creating objects is low, > > meaning object initialization overhead is minimal, and object reuse > > has little impact on performance, then using an object pool may not be > > worthwhile as the benefits of reuse may be offset by the management > > overhead of the object pool. > > ---- > > > > At very least, it makes sense to me that short-lived objects and costs > > with low object creation. i.e. some simple tuple structures can be > > just implemented as a record. JVM GC is evolving and the recycling for > > such objects should not be high. > > > > Thanks, > > Yunze
