My earlier comment may have come across as a bit stand-offish. If so, my
apologies for that. For what it's worth, Gianluca, if you wrote an Assembly
compiler I'd say that puts you well ahead of me in terms of experience and
qualification! 😂
To try and demonstrate what I meant by linking to the "build your LLM a
laboratory
<https://brianlovin.com/writing/give-your-agent-a-laboratory-jH5ryjC>"
blog, I started work on a branch
<https://github.com/apache/groovy/compare/GROOVY_4_0_X...jonnybot0:groovy:INDY-PERF-EXPLORATION?expand=1>
that would create some discrete JMH benchmarks that could potentially be
the small samples that Jochen was asking for. I've got just enough
experience with JMH to be a wise fool here, so I'm not even asking for a
review until I can prove these benchmarks are useful. Indeed, only the very
first part of the (LLM-generated) plan
<https://github.com/apache/groovy/compare/GROOVY_4_0_X...jonnybot0:groovy:INDY-PERF-EXPLORATION?expand=1#diff-f515504702ff1f9d88b86f6befd12ef0fdd93c6c265751b9b7e0bacb0c051261>
is implemented.
My plan is to use them to help me (a performance engineering and language
development neophyte) evaluate James Fredley's PR
<https://github.com/apache/groovy/pull/2374>, which looks like a concrete
step towards addressing some of the indy performance concerns based on real
world data. If my little foray into benchmarks here is a distraction from
that, please ignore it. James's PR deserves more attention. That said, if
these benchmarks are useful to the conversation, or you just want to play
with them, I'd recommend tweaking the JMH configuration so that it only
runs a few iterations on your first pass. JMH is rigorous by default, so
it'll happily spend an hour or more running benchmarks if you just run
./gradlew performance:jmh. That's fine for CI, but you may want to do more
of a "quick sniff" test
// Add this to subprojects/performance/build.gradle and tweak using
the available options according to taste
//
https://github.com/melix/jmh-gradle-plugin?tab=readme-ov-file#configuration-options
// Then run the benchmark you're interested in with the benchInclude
property, like -PbenchInclude=ColdCall
jmh {
iterations = 1
warmupIterations = 1
fork = 1
}
I've attached an example from running all the benchmarks on my machine with
the above limitation. It took about 8 minutes with Java 17.
My little experience with benchmarks is that Richard Feynman's warning is
very apt: "The first principle is that you must not fool yourself and you
are the easiest person to fool." Let the reader beware! 🙂
I'd be very grateful for feedback from anyone on this thread as to whether
they think these benchmarks might be a useful step in advancing the state
of the art in Groovy performance.
Best,
Jonny
On Mon, Jan 19, 2026 at 3:47 PM MG <[email protected]> wrote:
>
> 1. Evidently if we had a compact source sample, progressing towards
> fixing this would most likely be easy, but in the absence of that I was
> just trying to give the feedback I can.
> 2. I might misinterpret what you are trying to say, but primitive
> optimizations playing a role would suprise me, since what we do a lot in
> our code is (deeply) nested GStrings with a lot of embedded objects
> representing database table and column references, created using
> reflection, many of them short lived.
> 3. The code that does these operations is in the 2 low level modules
> that fixed the performance problems when we switched them to
> @CompileStatic.
>
> Am 19.01.2026 um 07:22 schrieb Jochen Theodorou:
>
> I was talking about a specific case and without primitive optimizations.
>
> On 1/18/26 22:54, MG wrote:
>
> @And for the long run the performance is actually comparable:
>
> 1. I want to reiterate that this is /not /what we observered in any of
> the real world cases we used as benchmarks.
> 2. While performance typically improved slightly with more iterations,
> this never led to a convergence towards the same runtime behavior.
> 3. In addition for many real life use cases it would not matter/help
> even if performance became identical between INDY & non-INDY after,
> say 1000 or 10000 calls, because this is not a threshold ever
> reached in practice, since we e.g. auto restart our application each
> night.
> 4. Imho therefore performance would need to be fixed in a way that
> works from the first call, not only at any point in the future...
>
> Cheers,
> mg
>
> Am 18.01.2026 um 06:23 schrieb Jochen Theodorou:
>
>
>
> On 1/17/26 21:36, Дилян Палаузов wrote:
>
> Hello,
>
> I do not think that this is going to make progress, until somebody shares
> a minimal example, which has performance differences between INDY and
> classic bytecode generation. That is: .groovy files, build environment and
> instructions how to run the files to notice significant performance
> difference.
>
>
>
> That in itself is actually not so difficult. See GROOVY-11842 for example.
> While the issue itself is about something being in the callstack, that
> should not be there, I there also talk about how Groovy 3 with and without
> primitive optimizations performs here in the first few iterations. And for
> the long run the performance is actually comparable.
>
> File:
>
> int foo(){1}
>
> int a
> long t1,t2
> O_MAX = 1000
> long[] ts = new long[O_MAX]
> for (int o=0; o<O_MAX; o++) {
> t1 = System.nanoTime()
> for (int i=0; i<10_000; i++) {
> a=foo()
> }
> t2 = System.nanoTime()
> ts[o] = t2-t1
> }
>
> def f = new File("out.txt")
> for (int i =0; i<O_MAX; i++) {
> f << i +": " + ts[i] + "\n"
> }
> println "done"
>
>
> Environment: just java command line with GroovyMain or groovy command.
>
> But what does it actually say? And is this really relevant in a big
> application? Those are difficult to answer.
>
> And that is your whole point I assume. it needs simple enough to actually
> get what you test, but complex enough to be relevant for the application.
> This one here surely is not
>
> bye Jochen
>
>
>
>
>
>
>
Benchmark
(n) Mode Cnt Score Error Units
o.a.g.bench.GeneratedHashCodeBench.generated_hashcode_on_instance_with_null_properties
N/A thrpt 86928.560 ops/ms
o.a.g.bench.dispatch.CacheInvalidationBench.groovy_bimorphic
N/A thrpt 42732.767 ops/ms
o.a.g.bench.dispatch.CacheInvalidationBench.groovy_changingTypes
N/A thrpt 983.041 ops/ms
o.a.g.bench.dispatch.CacheInvalidationBench.groovy_megamorphic10
N/A thrpt 855.380 ops/ms
o.a.g.bench.dispatch.CacheInvalidationBench.groovy_megamorphic8
N/A thrpt 929.317 ops/ms
o.a.g.bench.dispatch.CacheInvalidationBench.groovy_monomorphic
N/A thrpt 1231743.511 ops/ms
o.a.g.bench.dispatch.CacheInvalidationBench.groovy_polymorphic3
N/A thrpt 38771.149 ops/ms
o.a.g.bench.dispatch.CacheInvalidationBench.groovy_randomOrder
N/A thrpt 1554.455 ops/ms
o.a.g.bench.dispatch.CacheInvalidationBench.java_bimorphic
N/A thrpt 352001.046 ops/ms
o.a.g.bench.dispatch.CacheInvalidationBench.java_changingTypes
N/A thrpt 256880.865 ops/ms
o.a.g.bench.dispatch.CacheInvalidationBench.java_megamorphic10
N/A thrpt 257158.605 ops/ms
o.a.g.bench.dispatch.CacheInvalidationBench.java_megamorphic8
N/A thrpt 254637.514 ops/ms
o.a.g.bench.dispatch.CacheInvalidationBench.java_monomorphic
N/A thrpt 410903.996 ops/ms
o.a.g.bench.dispatch.CacheInvalidationBench.java_polymorphic3
N/A thrpt 258136.760 ops/ms
o.a.g.bench.dispatch.CacheInvalidationBench.java_randomOrder
N/A thrpt 255642.803 ops/ms
o.a.g.bench.dispatch.CallsiteBench.dispatch_1_monomorphic_groovy
N/A thrpt 22990.075 ops/ms
o.a.g.bench.dispatch.CallsiteBench.dispatch_1_monomorphic_java
N/A thrpt 8191.934 ops/ms
o.a.g.bench.dispatch.CallsiteBench.dispatch_3_polymorphic_groovy
N/A thrpt 616.566 ops/ms
o.a.g.bench.dispatch.CallsiteBench.dispatch_3_polymorphic_java
N/A thrpt 6038.183 ops/ms
o.a.g.bench.dispatch.CallsiteBench.dispatch_8_megamorphic_groovy
N/A thrpt 20.015 ops/ms
o.a.g.bench.dispatch.CallsiteBench.dispatch_8_megamorphic_java
N/A thrpt 5701.109 ops/ms
o.a.g.bench.indy.ThresholdSensitivityBench.batch_collectionIteration
N/A thrpt 8607.541 ops/ms
o.a.g.bench.indy.ThresholdSensitivityBench.batch_repeatMethodCall
N/A thrpt 40820811.430 ops/ms
o.a.g.bench.indy.ThresholdSensitivityBench.java_batchIteration
N/A thrpt 89386.403 ops/ms
o.a.g.bench.indy.ThresholdSensitivityBench.java_webRequest
N/A thrpt 219863.221 ops/ms
o.a.g.bench.indy.ThresholdSensitivityBench.mixed_cachedAndNew
N/A thrpt 3879.692 ops/ms
o.a.g.bench.indy.ThresholdSensitivityBench.polymorphic_interfaceDispatch
N/A thrpt 301320.858 ops/ms
o.a.g.bench.indy.ThresholdSensitivityBench.polymorphic_randomAccess
N/A thrpt 405535.052 ops/ms
o.a.g.bench.indy.ThresholdSensitivityBench.property_dynamic
N/A thrpt 17222.653 ops/ms
o.a.g.bench.indy.ThresholdSensitivityBench.property_getter
N/A thrpt 1510286.557 ops/ms
o.a.g.bench.indy.ThresholdSensitivityBench.property_setter
N/A thrpt 505027.918 ops/ms
o.a.g.bench.indy.ThresholdSensitivityBench.webRequest_fewCalls
N/A thrpt 324877.132 ops/ms
o.a.g.bench.indy.ThresholdSensitivityBench.webRequest_singleCall
N/A thrpt 445182.327 ops/ms
o.a.g.bench.indy.ThresholdSensitivityBench.webRequest_typicalController
N/A thrpt 7414.351 ops/ms
o.a.g.bench.orm.PropertyAccessBench.collection_collectProperty
N/A thrpt 60.359 ops/ms
o.a.g.bench.orm.PropertyAccessBench.collection_findAllByProperty
N/A thrpt 259.141 ops/ms
o.a.g.bench.orm.PropertyAccessBench.collection_findByProperty
N/A thrpt 469.485 ops/ms
o.a.g.bench.orm.PropertyAccessBench.collection_iterateMultipleProps
N/A thrpt 873875.400 ops/ms
o.a.g.bench.orm.PropertyAccessBench.collection_iterateSingleProp
N/A thrpt 1544939.369 ops/ms
o.a.g.bench.orm.PropertyAccessBench.collection_spreadOperator
N/A thrpt 462.749 ops/ms
o.a.g.bench.orm.PropertyAccessBench.hierarchy_computedPath
N/A thrpt 2306.121 ops/ms
o.a.g.bench.orm.PropertyAccessBench.hierarchy_traverseUp
N/A thrpt 889.574 ops/ms
o.a.g.bench.orm.PropertyAccessBench.java_collectionCollect
N/A thrpt 642.408 ops/ms
o.a.g.bench.orm.PropertyAccessBench.java_collectionIterate
N/A thrpt 404719.193 ops/ms
o.a.g.bench.orm.PropertyAccessBench.java_multipleGetters
N/A thrpt 308695.103 ops/ms
o.a.g.bench.orm.PropertyAccessBench.java_nestedAccess
N/A thrpt 492495.068 ops/ms
o.a.g.bench.orm.PropertyAccessBench.java_singleGetter
N/A thrpt 500966.574 ops/ms
o.a.g.bench.orm.PropertyAccessBench.nested_computedProperty
N/A thrpt 5593.703 ops/ms
o.a.g.bench.orm.PropertyAccessBench.nested_deepAccess
N/A thrpt 278867.881 ops/ms
o.a.g.bench.orm.PropertyAccessBench.nested_multipleAccess
N/A thrpt 959847.741 ops/ms
o.a.g.bench.orm.PropertyAccessBench.nested_oneLevel
N/A thrpt 1478662.364 ops/ms
o.a.g.bench.orm.PropertyAccessBench.simple_computedProperty
N/A thrpt 8409.522 ops/ms
o.a.g.bench.orm.PropertyAccessBench.simple_computedWithLogic
N/A thrpt 8296.906 ops/ms
o.a.g.bench.orm.PropertyAccessBench.simple_multipleGetters
N/A thrpt 956653.598 ops/ms
o.a.g.bench.orm.PropertyAccessBench.simple_singleGetter
N/A thrpt 1828572.525 ops/ms
o.a.g.bench.orm.PropertyAccessBench.view_generateReport
N/A thrpt 1598.767 ops/ms
o.a.g.bench.orm.PropertyAccessBench.view_renderList
N/A thrpt 51.698 ops/ms
o.a.g.bench.orm.PropertyAccessBench.view_renderOrderDetails
N/A thrpt 4189.320 ops/ms
o.a.g.bench.orm.PropertyAccessBench.view_renderSingle
N/A thrpt 2992.087 ops/ms
o.a.g.plugin.GroovyRunnerRegistryBench.linkedListIterator
N/A thrpt 310184.275 ops/ms
o.a.g.plugin.GroovyRunnerRegistryBench.registryIterator
N/A thrpt 143684.995 ops/ms
o.a.g.bench.AckermannBench.groovy
5 avgt 0.291 ms/op
o.a.g.bench.AckermannBench.groovy
6 avgt 1.289 ms/op
o.a.g.bench.AckermannBench.groovy
7 avgt 5.551 ms/op
o.a.g.bench.AckermannBench.java
5 avgt 0.058 ms/op
o.a.g.bench.AckermannBench.java
6 avgt 0.264 ms/op
o.a.g.bench.AckermannBench.java
7 avgt 1.171 ms/op
o.a.g.bench.AckermannBench.java
8 avgt 4.708 ms/op
o.a.g.bench.AryBench.groovy
10 avgt 0.383 ms/op
o.a.g.bench.AryBench.groovy
100 avgt 3.877 ms/op
o.a.g.bench.AryBench.groovy
1000 avgt 37.669 ms/op
o.a.g.bench.AryBench.groovy
1000000 avgt 35113.156 ms/op
o.a.g.bench.AryBench.java
10 avgt 0.004 ms/op
o.a.g.bench.AryBench.java
100 avgt 0.010 ms/op
o.a.g.bench.AryBench.java
1000 avgt 0.063 ms/op
o.a.g.bench.AryBench.java
1000000 avgt 115.060 ms/op
o.a.g.bench.FiboBench.groovy
30 avgt 9.706 ms/op
o.a.g.bench.FiboBench.groovy
31 avgt 16.747 ms/op
o.a.g.bench.FiboBench.groovy
32 avgt 25.479 ms/op
o.a.g.bench.FiboBench.groovy
33 avgt 44.255 ms/op
o.a.g.bench.FiboBench.groovy
34 avgt 65.906 ms/op
o.a.g.bench.FiboBench.java
30 avgt 2.288 ms/op
o.a.g.bench.FiboBench.java
31 avgt 3.694 ms/op
o.a.g.bench.FiboBench.java
32 avgt 6.469 ms/op
o.a.g.bench.FiboBench.java
33 avgt 10.321 ms/op
o.a.g.bench.FiboBench.java
34 avgt 15.866 ms/op
o.a.g.bench.indy.ColdCallBench.cold_01_createObjectOnly
N/A avgt 0.004 us/op
o.a.g.bench.indy.ColdCallBench.cold_02_createAndCallOne
N/A avgt 0.014 us/op
o.a.g.bench.indy.ColdCallBench.cold_03_createAndCallMultiple
N/A avgt 0.014 us/op
o.a.g.bench.indy.ColdCallBench.cold_04_propertyAccess
N/A avgt 0.132 us/op
o.a.g.bench.indy.ColdCallBench.cold_05_factoryCreateAndCall
N/A avgt 0.005 us/op
o.a.g.bench.indy.ColdCallBench.cold_06_factoryCreateAndCallMultiple
N/A avgt 0.040 us/op
o.a.g.bench.indy.ColdCallBench.collection_01_spreadOperator
N/A avgt 11.989 us/op
o.a.g.bench.indy.ColdCallBench.collection_02_chainedOperations
N/A avgt 11.825 us/op
o.a.g.bench.indy.ColdCallBench.gstring_01_interpolation
N/A avgt 0.186 us/op
o.a.g.bench.indy.ColdCallBench.java_01_createAndCall
N/A avgt 0.002 us/op
o.a.g.bench.indy.ColdCallBench.java_02_createAndCallMultiple
N/A avgt 0.003 us/op
o.a.g.bench.indy.ColdCallBench.static_01_methodCall
N/A avgt 0.002 us/op
o.a.g.bench.indy.ColdCallBench.warm_01_singleMethod
N/A avgt ≈ 10⁻³ us/op
o.a.g.bench.indy.ColdCallBench.warm_02_multipleMethods
N/A avgt 0.001 us/op
o.a.g.bench.indy.ColdCallBench.warm_03_propertyAccess
N/A avgt 0.107 us/op
o.a.g.bench.indy.WarmupBehaviorBench.java_baseline
N/A avgt 2.031 ns/op
o.a.g.bench.indy.WarmupBehaviorBench.warmup_00_cold
N/A avgt 12.163 ns/op
o.a.g.bench.indy.WarmupBehaviorBench.warmup_01_after10
N/A avgt 0.439 ns/op
o.a.g.bench.indy.WarmupBehaviorBench.warmup_02_after100
N/A avgt 0.437 ns/op
o.a.g.bench.indy.WarmupBehaviorBench.warmup_03_after1000
N/A avgt 0.436 ns/op
o.a.g.bench.indy.WarmupBehaviorBench.warmup_04_after5000
N/A avgt 0.435 ns/op
o.a.g.bench.indy.WarmupBehaviorBench.warmup_05_after10000
N/A avgt 0.432 ns/op
o.a.g.bench.indy.WarmupBehaviorBench.warmup_06_after15000
N/A avgt 0.488 ns/op
o.a.g.bench.indy.WarmupBehaviorBench.warmup_07_after50000
N/A avgt 0.438 ns/op
o.a.g.bench.indy.WarmupBehaviorBench.warmup_08_fullyWarmed
N/A avgt 0.440 ns/op
