voonhous opened a new pull request, #19923: URL: https://github.com/apache/hudi/pull/19923
### Describe the issue this Pull Request addresses Part of #19524, stacked on #19921 (its two commits are included here until it merges). #19921 made the Scala SQL suites able to share one SparkContext behind a property; this PR removes the cross-suite state that would break once those suites run concurrently in a shard. Still serial: no scheduling change, no timing claim. ### Summary and Changelog - Literal table names (`blob_*`, `vec_*`, two more) and literal database names (`hudi_database`, `test_database`, `testdb`) in the SQL suites now come from `generateTableName`, so no two suites can name the same catalog object. Assertions that embedded a literal build from the same value. - `HoodieInMemoryHashIndex` (test-only, "ONLY USE FOR LOCAL TESTING") keeps one map per table instead of one for the JVM, and can clear a single table. The test base clears globally only in per-suite mode; in shared mode it clears a table's entry when the table is dropped. - New `ExclusiveSuite` mixin: a fair read/write lock where every test holds the read side and an exclusive suite holds the write side for its whole run. Applied to the four suites that mutate JVM-wide state (shared Hadoop conf, persisted RDDs, metrics registry). No effect until suites run in parallel. - `TestInsertTable4` scopes its stage listener to the jobs the test started (a local property, since Hudi overwrites the job group on its own jobs) and removes it in a `finally`. - `TestRepairsProcedure` no longer closes the JVM-cached `FileSystem`. <details> <summary>Why per-table keying in the index rather than serializing the INMEMORY suites</summary> Ten SQL suites across all three shards use the INMEMORY index, and the base cleared the JVM-wide map after every test. Under concurrency one suite's clear, or a same-key write, misroutes another suite's upserts into wrong results, not a crash. Keying the map by base path removes the interaction at the source; the stale-location guard in `tagLocation` (commit-time validity check) is unchanged. Serializing ten suites would have cost most of the gain the later phase is after. </details> <details> <summary>Verification</summary> | run | mode | result | |---|---|---| | the 12 touched suites: TestBlobDataType, TestCreateTable, TestInsertTable4, TestRepairsProcedure, TestSpark3DDL, TestClusteringBinaryCopyStrategy, TestSparkCatalogSync, TestCDCForSparkSQL, TestSparkSqlWithTimestampKeyGenerator, TestMergeIntoLogOnlyTable, TestPartialUpdateForMergeInto, TestHoodieCommandMetrics (Spark 3.5 / Scala 2.12 / JDK 11, maven-free scalatest run) | shared | 138 tests, 0 failures, one base session; covers both ExclusiveSuite suites, the four INMEMORY suites on the per-table index, and the scoped listener | | test-compile of the diff, Spark 4.2 / Scala 2.13 / JDK 17, fresh reactor | - | green | | scalastyle on hudi-spark, checkstyle on hudi-client-common | - | clean | Default mode is covered by CI: the other-tests shard runs the ddl, procedure, feature and command suites with the property off, and the dml shard runs its packages in shared mode. </details> ### Impact Test infrastructure plus one test-only index class. No user-facing change. ### Risk Level low. With the property off, the only behaviour change is the per-table keying of a test-only index, which the existing INMEMORY suites exercise on every CI run. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
