voonhous opened a new pull request, #20201: URL: https://github.com/apache/hudi/pull/20201
### Describe the issue this Pull Request addresses **Stacked on #20029.** The first two commits belong to that PR and drop out on rebase once it merges; only the last two are for review here. #20029 restores nested predicate pushdown for Hudi scans. On Spark 4.1+, a predicate on a variant that `PushVariantIntoScan` rewrote is then pushed as a filter on the projection struct's ordinal-named member, e.g. ``GreaterThan(v.`0`,5)``. Two gaps follow: - No test pins what the reader does with such a filter. - Under schema-on-read, the filter rebuild runs before the variant guard and throws ``cannot find filter col name: v.`0` ``, hiding the guard's message that names the conf to turn off. ### Summary and Changelog - `ParquetSchemaEvolutionUtils.rebuildFilterFromParquet` collapses a filter on a name the query schema does not hold to `AlwaysTrue`, as it already does for a column the file lacks. The read then reaches `validateNoShreddedVariants` and fails with the guard message, as on master today. - Tests pin the `PushedFilters` of `variant_get` predicates per `pushVariantIntoScan` arm (top-level on COW and MOR, nested `s.inner`) beside row counts both arms share, plus a schema-on-read leg that expects the guard. <details> <summary>Why the pushed filter cannot change results</summary> No file carries ``v.`0` `` as a parquet leaf: a file holds `v.metadata`, `v.value` and, when shredded, `v.typed_value...`. Spark's `ParquetFilters` is built from the footer schema, finds no column under that name and creates no predicate, so no row group is skipped and no other column is filtered. File slices with log files only keep record-key filters. Spark evaluates the predicate above the scan in every case. </details> <details> <summary>Schema-on-read error before and after</summary> | | `where variant_get(v, '$.a', 'int') > 0` under schema-on-read | |---|---| | master | guard: `Column 'v' is a variant requested in Spark's full-variant projection shape ...` | | #20029 alone | ``IllegalArgumentException: cannot find filter col name: v.`0` from querySchema ...`` | | this PR | guard, as on master | </details> ### Impact No change for reads that work today. A schema-on-read read with a `variant_get` predicate keeps failing with the guard message once #20029 is in. Real support is #18285. ### Risk Level low. Dropping a pushed filter cannot change results: Spark applies it again above the scan. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
