voonhous commented on issue #18285: URL: https://github.com/apache/hudi/issues/18285#issuecomment-5869224297
Schema evolution beside a variant column with `spark.sql.variant.pushVariantIntoScan` on (Spark 4.1+, master): where it stands and what is left, following the `addMissingFields` thread (https://github.com/apache/hudi/pull/19783#discussion_r3888224040). Covered on master: - [x] Column add beside a top-level variant, pushdown on and off, COW and MOR, followed by compaction and clustering (`TestVariantShreddingMixedLayouts` section G; `TestVariantDataType` "schema evolves"). - [x] Implicit type widening of a sibling under a nested variant projection, COW (#19783: the `addMissingFields` arm and "Implicit widening of a sibling keeps the nested variant projection"). - [x] Schema-on-read plus the projection struct fails fast, including the inline compaction, clustering and CDC routes (`validateNoShreddedVariants`, pinned in "Schema-on-read reads of shredded variant files fail fast"). Open: - [ ] Implicit sibling widening on MOR with log files (native parquet and avro blocks), pushdown on. The #19783 leg is COW only, and the MOR merge is where its bug lived. - [ ] The same widening with a top-level variant beside the widened column. The #19783 leg uses a nested-only table. - [ ] Explicit `ALTER TABLE ... ALTER COLUMN n TYPE bigint` on a sibling without schema-on-read (`AlterHoodieTableChangeColumnCommand`, then the same implicit read path), COW and MOR. - [ ] Add a second VARIANT column by DDL without schema-on-read, then `variant_get` on it across old and new files, pushdown on, COW and MOR. - [ ] Rename under schema-on-read with pushdown on: pin the loud failure. Today it dies in Spark codegen rather than through the guard. Real support is this ticket, and the DDL itself is blocked by #19766 under default configs. Not needed: - Drop and reposition of the variant column are schema-on-read only and belong to this ticket's own list above. - Type promotion of the variant column itself: no format defines one, Spark's `UpCastRule` refuses it, and Hudi's compatibility checker refuses it. See the scope comment above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
