Hi Eduard,
That's a good point. I just ran the following command which completed
in 3 min 27 sec:
./gradlew clean :iceberg-core:check -DtestParallelism=8
Omitting the testParallelism flag, the same build target takes 13 min 29 sec.
We see a similar speedup as for the Flink tests. It does look like we
would benefit from this feature in other modules. In the PR, I enabled
the parallel CI test execution for Flink, we could also enable it for
iceberg-core, to allow for faster feedback on the PRs. Of course, the
general caveats with parallel test execution apply. We will have to
check which other modules can safely run tests in parallel without any
flakiness.
Cheers,
Max
On Tue, Nov 18, 2025 at 11:59 AM Eduard Tudenhöfner
<[email protected]> wrote:
>
> Thanks Max for looking into this. Have we tried increasing the parallelism
> for other modules than Flink and did those pass successfully? Given that
> we're running more and more tests with more format versions, it would be
> quite helpful to bring CI time down if possible.
>
>
> On Tue, Nov 18, 2025 at 12:29 AM Rodrigo Meneses <[email protected]> wrote:
>>
>> +1 (non-binding)
>>
>> as reducing test execution time will improve developer productivity!
>>
>>
>> On Mon, Nov 17, 2025 at 5:12 AM Maximilian Michels <[email protected]> wrote:
>>>
>>> Hi,
>>>
>>> What do you think about the following change?
>>> https://github.com/apache/iceberg/pull/13675
>>>
>>> Background:
>>>
>>> We build modules in parallel, but test execution is serial in Iceberg.
>>> I found that increasing the test parallelism for a module greatly
>>> reduces the test execution time. For example, on my machine (MacBook
>>> Pro M1 Max) it takes ~32 minutes to execute the Flink tests via:
>>>
>>> ./gradlew :iceberg-flink:iceberg-flink-2.1:check
>>>
>>> But if I increase `maxParallelForks` in the test section of the Flink
>>> build file to 8, it will only take 9 minutes, which is a speedup of
>>> 3.5x.
>>>
>>> Instead of doing this manually, we can do it via the property added in
>>> above PR:
>>>
>>> ./gradlew :iceberg-flink:iceberg-flink-2.1:check -DtestParallelism=8
>>>
>>> I also added an 'auto' config where we take half of the available CPUs,
>>> e.g.:
>>>
>>> ./gradlew :iceberg-flink:iceberg-flink-2.1:check -DtestParallelism=auto
>>>
>>> It is important to note that any modules which already set
>>> `maxParallelForks` will not be affected by this change. So we can
>>> still enforce non-parallel test execution. Also, the change is opt-in
>>> and doesn't change the default build configuration.
>>>
>>> Any objections against merging this functionality?
>>>
>>> Cheers,
>>> Max
