voonhous opened a new issue, #20037:
URL: https://github.com/apache/hudi/issues/20037
### Bug Description
**What happened:**
On release-1.2.1 with Spark 4.1, whose default is
`spark.sql.variant.pushVariantIntoScan=true`, a MOR read that merges a log file
mishandles variant columns:
- **Top-level `v variant`:** a row whose variant was nulled through the log
disappears from `where v is null`. The projector turns the null into a struct
of nulls instead of a NULL struct. Both log block formats.
- **Nested `s struct<inner: variant>`:** `variant_get(s.inner, ...)` or
`cast(s.inner as string)` fails the task. On x86 it is `SparkOutOfMemoryError:
Unable to acquire 863570212 bytes`; on aarch64 a JVM SIGBUS in
`StubRoutines::forward_copy_longs`. The merged row still holds a raw VariantVal
at `s.inner` while the plan reads it as the projected struct.
COW is correct, and every read with the conf off is correct. Master fixed
both in #19783 (`c59987a024cd`, recursive overlay and projector with the null
guard); release-1.2.1 carries #18674 but not #19783.
**What you expected:**
The same rows on both conf arms, as on master.
**Steps to reproduce:**
1. Spark 4.1.1, release-1.2.1 head `ddd37c05521a`, a MOR table `(id int, s
struct<inner: variant>, ts long)`.
2. `insert` five rows, then `update ... set s = named_struct('inner',
parse_json(...)) where id >= 3`.
3. `select id, cast(s.inner as string) from t`.
Evidence: draft PR #20034 carries the test; its CI run
([job](https://github.com/apache/hudi/actions/runs/35850779561/job/107147676422))
fails 4 of 16 legs, exactly the MOR conf-on ones, and passes the other 12.
Same matrix locally.
**Fix:** hand-port #19783 (a cherry-pick conflicts in 10 of 13 files: the
mixed-layouts suite and Spark4_2Adapter are absent, the reader context sits on
the #18674 buffer-hook design). Code scope: overlay and needs-projection
recursion in `SparkFileFormatInternalRowReaderContext`, a
`containsVariantProjection` helper on SparkAdapter, the recursive projector
with the null guard, the SparkSchemaTransformUtils arm, plus the #20034 suite
as the regression test. No prerequisite PRs; the shredded-read fixes do not
apply since 1.2.1 has no shredding writer config.
### Environment
**Hudi version:** release-1.2.1 (`ddd37c05521a`, 1.2.1-rc2)
**Query engine:** Spark 4.1.1, JDK 17
**Relevant configs:** `spark.sql.variant.pushVariantIntoScan=true` (Spark
4.1 default); MOR table; any record type
### Logs and Stack Trace
<details>
<summary>CI leg verdicts (spark4.1 Scala DML job on #20034)</summary>
```
LEG FAIL toplevel:mor:true:AVRO: TestFailedException: Expected Array([4]),
but got Array()
LEG FAIL toplevel:mor:true:SPARK: TestFailedException: Expected Array([4]),
but got Array()
LEG FAIL nested:mor:true:AVRO: SparkException: ... SparkOutOfMemoryError:
[UNABLE_TO_ACQUIRE_MEMORY] Unable to acquire 863570212 bytes of memory, got
281579768
LEG FAIL nested:mor:true:SPARK: SparkException: ... (same)
Tests: succeeded 260, failed 2, canceled 2, ignored 1
```
Local aarch64 run: `SIGBUS (0xa) BUS_ADRALN` in `v
~StubRoutines::forward_copy_longs`, executor task thread.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
