voonhous opened a new pull request, #19932: URL: https://github.com/apache/hudi/pull/19932
### Describe the issue this Pull Request addresses Part of #19524, stacked on #19928 (phase 4 of the Scala parallel work). With dml-1, dml-2 and other-1 running suites two at a time, the run's longest job became scala-other-2 at 35 to 37 minutes: the only Scala shard still serial, because its packages mix 33 suites on `HoodieSparkSqlTestBase` (26.5 of its 28.4 minutes of tests) with 10 suites that build their own SparkContext (1.9 minutes). ### Summary and Changelog - New ScalaTest tag annotation `SharedSessionSuite` (`@TagAnnotation`, `@Inherited`) on `HoodieSparkSqlTestBase`, so every suite on that base carries the tag without being listed, including the procedure base's subclasses. - scala-other-2 runs only the tagged suites of its packages (`-DtagsToInclude`), in shared-session mode two at a time, the same flags as the other three SQL shards. - The untagged suites of the same packages (`-DtagsToExclude`) run as an extra step of test-spark-java17-ftb, the shortest Java 17 job. A new suite lands on this serial side unless it extends the base. Job count unchanged at fourteen. Measurement, three samples: scala-other-2's scalatest wall clock against its serial 28:24, the ftb job against its 21 to 28, and the run's longest job against 35 to 37. | job | before | sample 1 | sample 2 | sample 3 | |---|---|---|---|---| | scala-other-2 tests | 28:24 serial | pending | pending | pending | | ftb job total | 21 to 28 | pending | pending | pending | | run's longest job | 35 to 37, scala-other-2 | pending | pending | pending | <details> <summary>Why a tag rather than package filters or a suite list</summary> The two families live in the same packages, and the plugin's `wildcardSuites` cannot exclude. An explicit list of the ten own-context suites would silently route a new own-context suite into the shared-session job, where it would break. The inherited tag makes the shared-session side opt-in by base class, verified locally against the project's ScalaTest: a Java annotation with `@TagAnnotation` and `@Inherited` on a base class is seen by `-n` and `-l` through two levels of subclassing. </details> ### Impact CI configuration and test infrastructure only. ### Risk Level low. Off by default like the rest of the stack; the ten own-context suites keep a private SparkContext. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
