voonhous opened a new pull request, #19928: URL: https://github.com/apache/hudi/pull/19928
### Describe the issue this Pull Request addresses Part of #19524, stacked on the shared-session and state-scoping PRs (successors of #19921 and #19923). Supersedes the proof of concept #19925. The Java CI wall clock is set by the Scala SQL shards, and scalatest has no fork count, so the lever is running suites concurrently in one JVM. This PR turns that on for the three SQL-only shards and measures it. ### Summary and Changelog - Pom properties `hudi.scalatest.parallel` (default false) and `hudi.scalatest.threads` (default 0) mapped to the scalatest plugin's `parallel` and `threadCount`: suites run concurrently, tests within a suite stay sequential. Set to true and 2 on dml-1, dml-2 and other-1, next to the shared-session flag they carry. - Two fixes the local two-thread loop exposed: the per-test cleanup owns a table only by the suite's name prefix or as a session temp view (derived names such as `<generated>_pt` were dropped by other suites), and two helpers that sorted log file paths as strings now sort by the log file comparator (the write token precedes the instant in the name, so stage ids compared as text). The second is latent on master. Measurement, over three runs of each of the three shards: the scalatest "Run completed" wall clock against the same job serial. dml-1 and other-1 are the two jobs setting the run's critical runtime of about 41 minutes under #19892, so this is where the wall clock should move. A result inside the serial range is no gain. | shard | serial under #19892 (its measured runs) | sample 1 | sample 2 | sample 3 | |---|---|---|---|---| | dml-1 (dml.others) | about 30 min of tests | pending | pending | pending | | dml-2 (dml.insert, dml.schema) | about 23 | pending | pending | pending | | other-1 (procedure, ddl) | about 32 | pending | pending | pending | | run's longest job | 41 to 44 | pending | pending | pending | <details> <summary>Evidence from the proof of concept (#19925)</summary> On master's layout the single dml job (dml-1 plus dml-2 together) at two threads: 27:58, 35:27 and 35:32 ([run 34697891947](https://github.com/apache/hudi/actions/runs/34697891947), attempts 1 to 3) against 41:22 on master's fastest runner and 52 to 56 on its typical one, the same 326 test names each time, both threads busy 96 to 98% of the time. Locally, twenty iterations of twelve interaction-heavy suites at two threads were green, and the whole dml tree at two threads passed in 22 minutes. </details> ### Impact CI configuration and test infrastructure only. ### Risk Level low. Off by default; on three shards whose suites were prepared for it in the two PRs below and looped locally at two threads. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
