This makes much more sense to refer to the fields by name. The default match by position behavior of STRUCT can be surprising.
On 2026/11/19 18:04:25 Xiao Li wrote: > Thanks for the proposal. The strongest motivation is that a table owner > cannot currently require name-safe structured values across writes, > comparisons, and other expressions. I agree that INSERT BY NAME alone does > not address that requirement, and preserving existing STRUCT behavior is > important. > > > Serge Rielau <[email protected]> 于2026年11月18日周三 08:18写道: > > > Hi all, > > > > I'd like to propose a new SQL data type, OBJECT, for *structured* > > objects. It is a structured sibling of STRUCT in which attribute names > > are part of the type, so a value is matched by name, not by position. > > > > *The problem* > > > > Today STRUCT matches fields by position. With ANSI mode on, inserting > > named_struct('salary', > > 150000, 'is_contractor', true) into a column of type STRUCT<name STRING, > > title STRING> succeeds and stores name = '150000' and title = 'true'. > > Comparison behaves the same way: named_struct('a', 1, 'b', 2) = > > named_struct('b', 1, 'a', 2) returns true. INSERT ... BY NAME helps, but > > it is per statement, the table owner cannot require it, and comparison, > > CAST, and set operations remain positional. > > > > *The proposal* > > > > OBJECT keeps STRUCT's shape (ordered, typed attributes, field access, > > nesting). Assignment and comparison require names to match at each > > position, with no implicit reordering. Reordering needs an explicit cast > > that remaps by name: > > > > CAST(object('firstname', 'Ada', 'lastname', 'Lovelace') > > AS OBJECT<lastname STRING, firstname STRING>); > > > > It would come with OBJECT_APPEND, OBJECT_DELETE, and OBJECT_PICK, and > > explicit casts to and from STRUCT. It is gated by > > spark.sql.objectType.enabled, default false. STRUCT is unchanged, and no > > new on-disk format is introduced. > > > > *Not in scope* > > > > changing STRUCT, Snowflake-style open semi-structured objects (VARIANT > > and MAP already cover those), CREATE TYPE UDTs, or a new storage format. > > > > *Estimate* > > > > about 3 months. The proposal lists mid-term and final checkpoints in the > > SPIP. > > > > The full SPIP (motivation, design, risks, rejected alternatives) is in the > > JIRA ticket: SPARK-61931 > > <https://issues.apache.org/jira/browse/SPARK-61931>. I would welcome > > feedback on the problem statement, the naming, and whether the rejected > > alternatives are the right ones to consider. > > > > Thanks, Serge Rielau > > > --------------------------------------------------------------------- To unsubscribe e-mail: [email protected]
