nevzheng commented on PR #13553:
URL: https://github.com/apache/gravitino/pull/13553#issuecomment-5882301426
**Follow-up on this PR's approach — closing the loop on further concurrency
work.**
After this PR shipped (shared container + tuning, 26m25s → 7m04s, 3.7x), we
evaluated two further architectural changes on top of it to push past the
existing per-fork concurrency cap: **thread-scoped parallelism**
(ScopedReference — per-thread isolation of the 4 core DB singletons instead of
per-process) and **classloader-scoped parallelism** (per-lane classloaders,
zero production-code changes). Both were spiked, measured on CI-matched
hardware, and **rejected**:
- **ScopedReference**: broke real concurrency-serialization tests ("waits
for uncommitted delete," row-level lock serialization) at the lowest
concurrency level tested, and ran 5x *slower* (35m54s) than this PR's own tuned
baseline — worse on both correctness and speed.
- **Classloader isolation**: `NoClassDefFoundError` inside Testcontainers'
resource-cleanup hook, crossing the per-lane classloader boundary —
nondeterministic, sometimes failing the build outright and sometimes silently
skipping cleanup while still reporting green.
Neither is a tuning problem fixable with more iteration — both are
structural mismatches between sharing test infrastructure below the process
level and libraries (JDBC connection semantics, Testcontainers' Docker client)
not designed for that. Recommend **not pursuing either further at this time**.
This PR's container-sharing + tuning approach stands as the shipped result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
