voonhous opened a new issue, #20032: URL: https://github.com/apache/hudi/issues/20032
### Task Description **What needs to be done:** Record the Spark 4.0 position on `spark.sql.variant.pushVariantIntoScan` so the review question "what happens on Spark 4.0 with a VARIANT column and `variant_get` on COW and MOR" has a fixed answer, and decide whether the manual opt-in on 4.0 needs a guard. 1. Spark 4.0.x ships the flag **off**. It is `.internal()` with `createWithDefault(false)` in [4.0.2](https://github.com/apache/spark/blob/v4.0.2/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L4757-L4764). Spark 4.1 flipped it to `true` in [SPARK-54454](https://github.com/apache/spark/commit/d50e9b7fee5d) ([4.1.1 source](https://github.com/apache/spark/blob/v4.1.1/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L5539-L5546)). The `.version("4.0.0")` tag only records when the key was added. 2. On stock 4.0 the rewrite never fires. The scan requests native `VariantType` and Hudi reads COW and MOR base files and log blocks unprojected. Pinned by [TestVariantDataType](https://github.com/apache/hudi/blob/7430134280ae5684daee64ece7660a77577bf65f/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/schema/TestVariantDataType.scala#L139) and by the [4.1+ gate](https://github.com/apache/hudi/blob/7430134280ae5684daee64ece7660a77577bf65f/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/schema/TestVariantShreddingMixedLayouts.scala#L69) on every projection test in TestVariantShreddingMixedLayouts. No fallthrough into schema-change logic on the default path. 3. Residual: the rule matches the same relation on [4.0](https://github.com/apache/spark/blob/v4.0.2/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PushVariantIntoScan.scala#L276-L280) and [4.1](https://github.com/apache/spark/blob/v4.1.1/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PushVariantIntoScan.scala#L279-L283) (any `HadoopFsRelation` whose format is a `ParquetFileFormat`), and Hudi's file format [extends ParquetFileFormat](https://github.com/apache/hudi/blob/7430134280ae5684daee64ece7660a77577bf65f/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedFileFormat.scala#L98). Setting the flag to `true` by hand on 4.0 therefore delivers the projection struct to Hudi, where nothing recognises it. Code-read only, not executed; details below. **Why this task is needed:** So the question does not get re-asked on every variant PR, and so the manual-override gap is a recorded decision rather than an unknown. **Proposal:** no 4.0 projection handling (#18935 is removing the 4.0 variant shims). If anything, a 4.0-only guard that rejects the projection struct with a clear message. <details> <summary>Why a manual opt-in on 4.0 is unhandled (Hudi at 7430134280ae)</summary> - `SparkAdapter.isVariantProjectionStruct` [defaults to false](https://github.com/apache/hudi/blob/7430134280ae5684daee64ece7660a77577bf65f/hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/hudi/SparkAdapter.scala#L514); only [Spark4_1Adapter overrides it](https://github.com/apache/hudi/blob/7430134280ae5684daee64ece7660a77577bf65f/hudi-spark-datasource/hudi-spark4.1.x/src/main/scala/org/apache/spark/sql/adapter/Spark4_1Adapter.scala#L233). [Spark4_0Adapter](https://github.com/apache/hudi/blob/7430134280ae5684daee64ece7660a77577bf65f/hudi-spark-datasource/hudi-spark4.0.x/src/main/scala/org/apache/spark/sql/adapter/Spark4_0Adapter.scala) has no variant override at all. - `HoodieSparkSchemaConverters.isSparkVariantProjectionStruct` [short-circuits on gteqSpark4_1](https://github.com/apache/hudi/blob/7430134280ae5684daee64ece7660a77577bf65f/hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/avro/HoodieSparkSchemaConverters.scala#L532), so the projected struct is not mapped back to a HoodieSchema variant. - `supportBatch` only [forces row reads](https://github.com/apache/hudi/blob/7430134280ae5684daee64ece7660a77577bf65f/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedFileFormat.scala#L181-L189) when the adapter reports a projection struct or on 4.1+, so on 4.0 the vectorized reader stays on. - `BaseSpark4Adapter.isDataTypeEqualForPhysicalSchema` returns `None` for the (projection struct, VariantType) pair on 4.0, so the request falls into the generic type-comparison path. What that path does with it has not been run. Related: #18674 restored the 4.0 unprojected read after the 4.1 alignment; #19768 is the open 4.1 payload-merger case. </details> ### Task Type Other ### Related Issues **Parent feature issue:** n/a **Related issues:** #18935, #19768, #18674 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
