This seems like a great improvement to cut build time. The results are very encouraging. We will have to see if the parallel tests increase the test flakiness.
Can we also test it with Spark? Spark and Flink are the heavy modules. But we can roll out the changes in phases. On Tue, Nov 18, 2025 at 7:53 AM Maximilian Michels <[email protected]> wrote: > Hi Eduard, > > That's a good point. I just ran the following command which completed > in 3 min 27 sec: > > ./gradlew clean :iceberg-core:check -DtestParallelism=8 > > Omitting the testParallelism flag, the same build target takes 13 min 29 > sec. > > We see a similar speedup as for the Flink tests. It does look like we > would benefit from this feature in other modules. In the PR, I enabled > the parallel CI test execution for Flink, we could also enable it for > iceberg-core, to allow for faster feedback on the PRs. Of course, the > general caveats with parallel test execution apply. We will have to > check which other modules can safely run tests in parallel without any > flakiness. > > Cheers, > Max > > On Tue, Nov 18, 2025 at 11:59 AM Eduard Tudenhöfner > <[email protected]> wrote: > > > > Thanks Max for looking into this. Have we tried increasing the > parallelism for other modules than Flink and did those pass successfully? > Given that we're running more and more tests with more format versions, it > would be quite helpful to bring CI time down if possible. > > > > > > On Tue, Nov 18, 2025 at 12:29 AM Rodrigo Meneses <[email protected]> > wrote: > >> > >> +1 (non-binding) > >> > >> as reducing test execution time will improve developer productivity! > >> > >> > >> On Mon, Nov 17, 2025 at 5:12 AM Maximilian Michels <[email protected]> > wrote: > >>> > >>> Hi, > >>> > >>> What do you think about the following change? > >>> https://github.com/apache/iceberg/pull/13675 > >>> > >>> Background: > >>> > >>> We build modules in parallel, but test execution is serial in Iceberg. > >>> I found that increasing the test parallelism for a module greatly > >>> reduces the test execution time. For example, on my machine (MacBook > >>> Pro M1 Max) it takes ~32 minutes to execute the Flink tests via: > >>> > >>> ./gradlew :iceberg-flink:iceberg-flink-2.1:check > >>> > >>> But if I increase `maxParallelForks` in the test section of the Flink > >>> build file to 8, it will only take 9 minutes, which is a speedup of > >>> 3.5x. > >>> > >>> Instead of doing this manually, we can do it via the property added in > above PR: > >>> > >>> ./gradlew :iceberg-flink:iceberg-flink-2.1:check > -DtestParallelism=8 > >>> > >>> I also added an 'auto' config where we take half of the available > CPUs, e.g.: > >>> > >>> ./gradlew :iceberg-flink:iceberg-flink-2.1:check > -DtestParallelism=auto > >>> > >>> It is important to note that any modules which already set > >>> `maxParallelForks` will not be affected by this change. So we can > >>> still enforce non-parallel test execution. Also, the change is opt-in > >>> and doesn't change the default build configuration. > >>> > >>> Any objections against merging this functionality? > >>> > >>> Cheers, > >>> Max >
