This is an automated email from the ASF dual-hosted git repository.
voonhous pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new b549b1ca0de7 docs: pushVariantIntoScan workaround for 1.2.x (#20159)
b549b1ca0de7 is described below
commit b549b1ca0de7c293d7d491b7e7ed50baa4ba84bc
Author: voonhous <[email protected]>
AuthorDate: Thu Oct 1 14:08:44 2026 +0800
docs: pushVariantIntoScan workaround for 1.2.x (#20159)
* docs: pushVariantIntoScan workaround for 1.2.x
The 1.2.1 known issue on VARIANT reads under Spark 4.1 also applies to
1.2.0 and is wider than stated: two more failures only happen with
the PushVariantIntoScan rule on, one of them on Copy-on-Write tables.
List all four, say where each is fixed, and show how to turn the rule
off when a read hits one of them.
Correct the Spark 4.0 sentence: with the rule turned on there, every
read of a table with a VARIANT column fails.
Add the same caution to the VARIANT section of the 1.2.0 and 1.2.1
SQL DDL pages and to the 1.2.0 release notes, and describe the
setting on the current SQL DDL page.
* docs: flag the silent null-VARIANT case in sql_ddl
The versioned sql_ddl caution only told users to react to a failing
read. The MOR null-VARIANT case raises no error, so repeat the release
notes guidance to keep the rule off for those reads.
---
website/docs/sql_ddl.md | 9 +++++++-
website/releases/release-1.2.md | 29 ++++++++++++++++++++-----
website/versioned_docs/version-1.2.0/sql_ddl.md | 15 ++++++++++++-
website/versioned_docs/version-1.2.1/sql_ddl.md | 15 ++++++++++++-
4 files changed, 60 insertions(+), 8 deletions(-)
diff --git a/website/docs/sql_ddl.md b/website/docs/sql_ddl.md
index 150a8197ffc0..205a81387dfa 100644
--- a/website/docs/sql_ddl.md
+++ b/website/docs/sql_ddl.md
@@ -2,7 +2,7 @@
title: SQL DDL
summary: "In this page, we discuss using SQL DDL commands with Hudi"
toc: true
-last_modified_at: 2026-08-07T20:54:32+05:30
+last_modified_at: 2026-09-30T18:50:00+08:00
---
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
@@ -333,6 +333,13 @@ decodable in application code if needed. What you do
**not** get on Spark 3.x:
| Flink < 2.1 | Throws `UnsupportedOperationException` on VARIANT columns. |
| Flink ≥ 2.1 | Surfaces VARIANT as `ROW<metadata BYTES, value BYTES>`. Flink
can read the underlying struct but cannot decode it as a variant value. |
+Spark's `PushVariantIntoScan` rule is controlled by
`spark.sql.variant.pushVariantIntoScan` (on by
+default from Spark 4.1, off by default in Spark 4.0). Hudi reads the rewritten
scan on Spark 4.1+.
+If a read of a VARIANT column fails with the rule on, set it to `false`: Spark
then reads the
+whole VARIANT and evaluates `variant_get` and casts after the scan, with the
same results. On
+Spark 4.0 keep it off: Hudi rejects a read of a table with a VARIANT column
while it is on. For
+Hudi 1.2.0 and 1.2.1 on Spark 4.1, see the [1.2 release
notes](/releases/release-1.2#known-issues).
+
VARIANT columns are not supported on Lance-backed tables.
Only unshredded VARIANT is exposed at the user-facing level: every Spark/Hudi
schema conversion
diff --git a/website/releases/release-1.2.md b/website/releases/release-1.2.md
index 5f854617182e..1bdadc3c35e5 100644
--- a/website/releases/release-1.2.md
+++ b/website/releases/release-1.2.md
@@ -2,7 +2,7 @@
title: "Release 1.2"
layout: releases
toc: true
-last_modified_at: 2026-09-24T18:00:00-08:00
+last_modified_at: 2026-09-30T18:50:00+08:00
---
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
@@ -50,15 +50,30 @@ The Trino-Hudi connector now lives in the Hudi repo
(RFC-105, [#18837](https://g
## Known Issues
-On Spark 4.1, `spark.sql.variant.pushVariantIntoScan` is on by default. With
it on, snapshot reads of Merge-on-Read tables with `VARIANT` columns can go
wrong when log files are merged:
+These issues affect 1.2.0 and 1.2.1. Spark's `PushVariantIntoScan` rule is
controlled by `spark.sql.variant.pushVariantIntoScan`, which is on by default
in Spark 4.1 and off by default in Spark 4.0.
+
+On Spark 4.1 with the rule on, reads of `VARIANT` columns can go wrong. On
Merge-on-Read tables, when a snapshot read merges log files:
* A top-level `VARIANT` value set to null by an update is read back as a
struct of nulls instead of `NULL`, so a filter such as `v IS NULL` misses the
row.
* A `VARIANT` nested inside a struct can crash the executor JVM.
+* A query that extracts several paths from one `VARIANT` can fail with
`NegativeArraySizeException` when rows lack some of those paths.
-Copy-on-Write tables are not affected. Spark 4.0 is affected only if
`spark.sql.variant.pushVariantIntoScan` is turned on, since it is off by
default there. A fix is tracked in
[#20039](https://github.com/apache/hudi/pull/20039).
+On Copy-on-Write and Merge-on-Read tables:
-:::tip
-Until the fix is released, set `spark.sql.variant.pushVariantIntoScan=false`
for Spark 4.1 reads of Merge-on-Read tables with `VARIANT` columns.
+* A `VARIANT` column added with `ALTER TABLE ... ADD COLUMNS` cannot be read
from files written before it was added. The read fails with
`INVALID_VARIANT_SHREDDING_SCHEMA`.
+
+Reads with the rule off are not affected. The first two are fixed on the 1.2.x
branch ([#20039](https://github.com/apache/hudi/pull/20039)); no release
carries that fix yet. The other two are tracked in
[#20040](https://github.com/apache/hudi/issues/20040) and
[#20135](https://github.com/apache/hudi/issues/20135).
+
+On Spark 4.0 the rule is not supported. With it turned on, every read of a
table that has a `VARIANT` column fails with `HoodieSchemaException: Illegal
initial character: 0`, so leave it at its default there.
+
+:::tip Workaround
+If a read hits one of these, turn the rule off for the session:
+
+```sql
+SET spark.sql.variant.pushVariantIntoScan=false;
+```
+
+or start Spark with `--conf spark.sql.variant.pushVariantIntoScan=false`.
Spark then reads the whole `VARIANT` and evaluates `variant_get` and casts
after the scan. The setting is per Spark session and also applies to the
non-Hudi Parquet tables read in it. The first issue raises no error, so keep
the rule off for Spark 4.1 reads of Merge-on-Read tables whose `VARIANT`
columns can be null.
:::
## Raw Release Notes
@@ -146,6 +161,10 @@ Engine support: native `VARIANT` keyword on Spark 4.0+; on
Spark 3.x, VARIANT su
**Limitations as of 1.2.0**: Shredding is not supported in this release. Hudi
1.2.0 stores the entire VARIANT as the binary value payload plus its
key-dictionary metadata; frequently accessed fields are not projected into
native typed columns. As a result, predicates and projections over fields
inside a VARIANT are not pushed down at the column level. A query that filters
on a nested field (for example `WHERE tool_calls[0].name = 'search'`) reads and
decodes the binary payload rather tha [...]
+:::caution
+On Spark 4.1, some reads of `VARIANT` columns fail or miss rows on 1.2.0 and
1.2.1. Setting `spark.sql.variant.pushVariantIntoScan=false` avoids them. See
[Known Issues](#known-issues).
+:::
+
##### BLOB
Images, PDFs, audio, video, and model artifacts are increasingly first-class
inputs and outputs of data pipelines, but storing them in a separate object
store and joining by path is awkward, error-prone, and breaks transactional
guarantees. Putting raw bytes into a `BINARY` column works for small payloads
but bloats every scan that doesn't need the bytes.
diff --git a/website/versioned_docs/version-1.2.0/sql_ddl.md
b/website/versioned_docs/version-1.2.0/sql_ddl.md
index 150a8197ffc0..2e31fc6dcf0f 100644
--- a/website/versioned_docs/version-1.2.0/sql_ddl.md
+++ b/website/versioned_docs/version-1.2.0/sql_ddl.md
@@ -2,7 +2,7 @@
title: SQL DDL
summary: "In this page, we discuss using SQL DDL commands with Hudi"
toc: true
-last_modified_at: 2026-08-07T20:54:32+05:30
+last_modified_at: 2026-09-30T18:50:00+08:00
---
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
@@ -333,6 +333,19 @@ decodable in application code if needed. What you do
**not** get on Spark 3.x:
| Flink < 2.1 | Throws `UnsupportedOperationException` on VARIANT columns. |
| Flink ≥ 2.1 | Surfaces VARIANT as `ROW<metadata BYTES, value BYTES>`. Flink
can read the underlying struct but cannot decode it as a variant value. |
+:::caution Spark 4.1 and `spark.sql.variant.pushVariantIntoScan`
+In Hudi 1.2.0 and 1.2.1, reads of VARIANT columns can return wrong rows or
fail while Spark's
+`PushVariantIntoScan` rule is on, which is the Spark 4.1 default: on
Merge-on-Read tables when
+log files are merged, and on any table for a VARIANT column added with
+`ALTER TABLE ... ADD COLUMNS`. If a read hits one of these, set
+`spark.sql.variant.pushVariantIntoScan=false` for the session. One case raises
no error: on
+Merge-on-Read, a VARIANT set to null by an update is read back as a struct of
nulls, so
+`v IS NULL` misses the row. Keep the rule off for Spark 4.1 reads of
Merge-on-Read tables whose
+VARIANT columns can be null. On Spark 4.0 keep the setting at
+its default (`false`): with it on, every read of a table with a VARIANT column
fails. See the
+[1.2 release notes](/releases/release-1.2#known-issues).
+:::
+
VARIANT columns are not supported on Lance-backed tables.
Only unshredded VARIANT is exposed at the user-facing level: every Spark/Hudi
schema conversion
diff --git a/website/versioned_docs/version-1.2.1/sql_ddl.md
b/website/versioned_docs/version-1.2.1/sql_ddl.md
index 150a8197ffc0..2e31fc6dcf0f 100644
--- a/website/versioned_docs/version-1.2.1/sql_ddl.md
+++ b/website/versioned_docs/version-1.2.1/sql_ddl.md
@@ -2,7 +2,7 @@
title: SQL DDL
summary: "In this page, we discuss using SQL DDL commands with Hudi"
toc: true
-last_modified_at: 2026-08-07T20:54:32+05:30
+last_modified_at: 2026-09-30T18:50:00+08:00
---
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
@@ -333,6 +333,19 @@ decodable in application code if needed. What you do
**not** get on Spark 3.x:
| Flink < 2.1 | Throws `UnsupportedOperationException` on VARIANT columns. |
| Flink ≥ 2.1 | Surfaces VARIANT as `ROW<metadata BYTES, value BYTES>`. Flink
can read the underlying struct but cannot decode it as a variant value. |
+:::caution Spark 4.1 and `spark.sql.variant.pushVariantIntoScan`
+In Hudi 1.2.0 and 1.2.1, reads of VARIANT columns can return wrong rows or
fail while Spark's
+`PushVariantIntoScan` rule is on, which is the Spark 4.1 default: on
Merge-on-Read tables when
+log files are merged, and on any table for a VARIANT column added with
+`ALTER TABLE ... ADD COLUMNS`. If a read hits one of these, set
+`spark.sql.variant.pushVariantIntoScan=false` for the session. One case raises
no error: on
+Merge-on-Read, a VARIANT set to null by an update is read back as a struct of
nulls, so
+`v IS NULL` misses the row. Keep the rule off for Spark 4.1 reads of
Merge-on-Read tables whose
+VARIANT columns can be null. On Spark 4.0 keep the setting at
+its default (`false`): with it on, every read of a table with a VARIANT column
fails. See the
+[1.2 release notes](/releases/release-1.2#known-issues).
+:::
+
VARIANT columns are not supported on Lance-backed tables.
Only unshredded VARIANT is exposed at the user-facing level: every Spark/Hudi
schema conversion
