voonhous opened a new issue, #20040:
URL: https://github.com/apache/hudi/issues/20040
**Describe the problem you faced**
On Spark 4.1+ with `spark.sql.variant.pushVariantIntoScan=true` (the
default), a query that extracts several paths from a variant column fails with
`java.lang.NegativeArraySizeException` once a row lacks enough of those paths.
Reproduced on master with eight `try_variant_get(v, '$.x', 'bigint')` paths
over a row that holds only `$.a`:
- MOR table with a log file in the latest slice: any such query.
- Bootstrapped table (METADATA_ONLY), COW or MOR: any such query that also
needs a meta column, and every query on MOR.
Snapshot reads of a COW table without bootstrap are unaffected. Fewer
extracted paths (or rows that hold most of them) pass, which is why the
existing pushVariantIntoScan legs never hit it.
**Cause**
PushVariantIntoScan rewrites the variant column into a projection struct in
the scan schema. `SparkFileFormatInternalRowReaderContext` reads base-file rows
in that shape and rewrites log rows into it (#19783), but two row writers are
still built from the engine `HoodieSchema`, whose variant field converts to
`VariantType`:
1. the output converter that projects the reader's required schema down to
the requested one (`FileGroupReaderSchemaHandler.getOutputConverter` via
`BaseSparkInternalRecordContext.projectRecord`), present whenever merge columns
or `_hoodie_commit_time` widen the required schema;
2. the bootstrap skeleton/data join
(`BaseSparkInternalRowReaderContext.getBootstrapProjection`).
A `VariantType`-typed writer copies the struct through
`UnsafeRow.getVariant`, which reads the struct's null bitset as the variant
value length. That is byte-identical while the low bitset word stays below the
struct's byte size, and throws once enough pushed fields are null (seven null
fields of eight: `NegativeArraySizeException: -186`).
**To Reproduce**
```sql
create table t (id int, v variant, ts long) using hudi tblproperties
(primaryKey = 'id', preCombineField = 'ts', type = 'mor');
insert into t select id, parse_json(concat('{"a":', id, '}')), 1000 from
range(0, 10);
update t set v = parse_json(concat('{"a":', 100 + id, '}')), ts = 1001 where
id >= 5;
select try_variant_get(v, '$.a', 'bigint'), try_variant_get(v, '$.b',
'bigint'), try_variant_get(v, '$.c', 'bigint'),
try_variant_get(v, '$.d', 'bigint'), try_variant_get(v, '$.e',
'bigint'), try_variant_get(v, '$.f', 'bigint'),
try_variant_get(v, '$.g', 'bigint'), try_variant_get(v, '$.h',
'bigint') from t where id = 2;
```
**Expected behavior**
`2, null, null, null, null, null, null, null`.
**Environment Description**
* Hudi version : master (7430134280ae)
* Spark version : 4.1.1 (pushVariantIntoScan on by default)
* Storage : local
**Stacktrace**
<details><summary>MOR read</summary>
```
java.lang.NegativeArraySizeException: -186
at
org.apache.spark.unsafe.types.VariantVal.readFromUnsafeRow(VariantVal.java:79)
...
at
org.apache.spark.sql.HoodieInternalRowUtils$.$anonfun$genUnsafeStructWriter$3(HoodieInternalRowUtils.scala:210)
at
org.apache.spark.sql.HoodieInternalRowUtils$.$anonfun$modifiableRowWriter$2(HoodieInternalRowUtils.scala:239)
at
org.apache.spark.sql.HoodieInternalRowUtils$.$anonfun$genUnsafeRowWriter$2(HoodieInternalRowUtils.scala:165)
at
org.apache.hudi.BaseSparkInternalRecordContext.lambda$projectRecord$0(BaseSparkInternalRecordContext.java:221)
at
org.apache.hudi.common.table.read.BufferedRecord.project(BufferedRecord.java:90)
at
org.apache.hudi.common.table.read.HoodieFileGroupReader.next(HoodieFileGroupReader.java:351)
```
</details>
<details><summary>Bootstrap read</summary>
```
java.lang.NegativeArraySizeException: -186
at
org.apache.spark.unsafe.types.VariantVal.readFromUnsafeRow(VariantVal.java:79)
...
at
org.apache.hudi.BaseSparkInternalRowReaderContext.lambda$getBootstrapProjection$2(BaseSparkInternalRowReaderContext.java:96)
at
org.apache.hudi.SparkFileFormatInternalRowReaderContext$$anon$2.doHasNext(SparkFileFormatInternalRowReaderContext.scala:405)
at
org.apache.hudi.common.util.collection.CachingIterator.hasNext(CachingIterator.java:30)
at
org.apache.hudi.common.table.read.HoodieFileGroupReader.hasNext(HoodieFileGroupReader.java:339)
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
