voonhous opened a new pull request, #20184: URL: https://github.com/apache/hudi/pull/20184
### Describe the issue this Pull Request addresses branch-1.2.x does not have #20136 (issue #20135). On Spark 4.1, where `spark.sql.variant.pushVariantIntoScan` is on by default, a variant column added by `ALTER TABLE ... ADD COLUMNS` without schema-on-read cannot be read from the files written before the DDL. The pushed-down read fails with `INVALID_VARIANT_SHREDDING_SCHEMA` on COW and MOR, over base files and log blocks alike. Hudi's readers turn nested schema pruning off, so `HoodieParquetReadSupport.trimParquetSchema` kept the missing top-level column in the parquet read schema as a plain group, and Spark built its variant converter over it. Reads with the rule off are correct. ### Summary and Changelog Port of master `827644e8699b`. - `HoodieParquetReadSupport`, `TestHoodieParquetReadSupport`: applied as on master. A top-level field the file does not have is dropped from the read schema, the same way missing nested fields already are. - `TestVariantPushVariantIntoScan`: master's end-to-end test runs over shredded files, which this branch cannot write. This PR adds an unshredded version on the existing table type x rule x record type sweep. It reads over the old files only, after an insert that carries `v2`, after an update of one row per file group (the COW merge handle; on MOR beside a log block written before the DDL) and, on MOR, after compaction. <details> <summary>Local runs, Spark 4.1.1</summary> | State | Result | |:--|:--| | tests without the prod change | new test fails on the 4 rule-on legs (`INVALID_VARIANT_SHREDDING_SCHEMA`); unit test 2 pass, 1 fail | | port | unit test 3/3; `TestVariantPushVariantIntoScan` 4 tests, 24 sweep legs, all pass | </details> ### Impact Row-based parquet reads of a file that lacks a requested top-level column now drop it from the read schema and return null for it. On this branch only the Spark 4.1+ rule-on reads of a DDL-added variant failed before; add-column reads of plain columns return the same rows. ### Risk Level low ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
