Thanks for the proposal. The strongest motivation is that a table owner
cannot currently require name-safe structured values across writes,
comparisons, and other expressions. I agree that INSERT BY NAME alone does
not address that requirement, and preserving existing STRUCT behavior is
important.


Serge Rielau <[email protected]> 于2026年11月18日周三 08:18写道:

> Hi all,
>
> I'd like to propose a new SQL data type, OBJECT, for *structured*
> objects. It is a structured sibling of STRUCT in which attribute names
> are part of the type, so a value is matched by name, not by position.
>
> *The problem*
>
> Today STRUCT matches fields by position. With ANSI mode on, inserting 
> named_struct('salary',
> 150000, 'is_contractor', true) into a column of type STRUCT<name STRING,
> title STRING> succeeds and stores name = '150000' and title = 'true'.
> Comparison behaves the same way: named_struct('a', 1, 'b', 2) =
> named_struct('b', 1, 'a', 2) returns true. INSERT ... BY NAME helps, but
> it is per statement, the table owner cannot require it, and comparison,
> CAST, and set operations remain positional.
>
> *The proposal*
>
> OBJECT keeps STRUCT's shape (ordered, typed attributes, field access,
> nesting). Assignment and comparison require names to match at each
> position, with no implicit reordering. Reordering needs an explicit cast
> that remaps by name:
>
> CAST(object('firstname', 'Ada', 'lastname', 'Lovelace')
>      AS OBJECT<lastname STRING, firstname STRING>);
>
> It would come with OBJECT_APPEND, OBJECT_DELETE, and OBJECT_PICK, and
> explicit casts to and from STRUCT. It is gated by
> spark.sql.objectType.enabled, default false. STRUCT is unchanged, and no
> new on-disk format is introduced.
>
> *Not in scope*
>
> changing STRUCT, Snowflake-style open semi-structured objects (VARIANT
> and MAP already cover those), CREATE TYPE UDTs, or a new storage format.
>
> *Estimate*
>
> about 3 months. The proposal lists mid-term and final checkpoints in the
> SPIP.
>
> The full SPIP (motivation, design, risks, rejected alternatives) is in the
> JIRA ticket: SPARK-61931
> <https://issues.apache.org/jira/browse/SPARK-61931>. I would welcome
> feedback on the problem statement, the naming, and whether the rejected
> alternatives are the right ones to consider.
>
> Thanks, Serge Rielau
>

Reply via email to