I would like to review this conversation. The feature is very useful is on its own, but the derived column concept could be used for several ongoing projects: - Non-materialized columns: https://lists.apache.org/thread.html/4h53r1nvvgxqymfvx7vcyhqgh1fk8180 - Iceberg secondary indexes: https://lists.apache.org/thread/gtpn4ydovmoxb1bffz5wcnsso20n379c
Was there any more progress in this regard? Is someone driving this feature, or shall I jump in? Thanks, Peter Szehon Ho <[email protected]> ezt írta (időpont: 2026. aug. 11., K, 0:41): > HI Prasant, sure thanks! I added some brief context to the doc from our > previous research: > > 1. Put the official SQL spec definition > 2. Define 'Generated Column' > 3. Define 'Virtual Column' > 4. Clarify what we gain/lose with deterministic expressions. > > I didn't modify the proposal yet, just wanted to make sure we are on the > same page on the definition of derived column as we go into the proposal. > Hope it looks ok with you. Maybe we can also chat on slack channel if it's > more convenient, up to you folks. > > Hi Stefan, it's a good point. Some database like SQLServer seem to also > support a similar concept (computed column), but there's a lot that the > column cannot do if it's non-deterministic. As in the points I put in the > doc, these are thing like VIRTUAL column, partition column, index key, > check constraint, even Update. So while its not so useful in the > traditional database sense, it may make sense to let the engine have an > option. > > Aside from determinism, another question is portability, ie have engines > (at least different instance of same engine) understand each other's column > expression. Sounds like we want to align with UDF/View spec, and upcoming > Check Constraint and Default Value spec, as mentioned in Andrei's email? > > Thanks > Szehon > > On Mon, Aug 10, 2026 at 5:23 AM Prashant Sharma <[email protected]> > wrote: > >> I am fairly new to Iceberg. Thanks for the encouragement. Here is a first >> draft (still WIP) >> https://docs.google.com/document/d/1u93kyIrTc8bt9VSTB4O0GM9S7zuXekqixJR6Id2FZC4/edit?usp=sharing. >> Please take a look ! >> >> https://github.com/apache/iceberg/issues/15923 >> >> @Andrei Tserakhau <[email protected]> >> Your summary has helped a lot in catching up with parallel work and also >> the history - Thanks a lot ! >> >> @[email protected] <[email protected]> and @[email protected] >> <[email protected]> Happy to collaborate - was trying to add you as >> editor - let me know your google account enabled id. >> >> In the proposal document above, I have also tried to address two problem >> which are common to multiple v4 features >> >> 1. Expressions reuse >> 2. Backward compatibility: V4-features need the ability to know data was >> modified by an engine unaware of the feature and how does the engine tell >> that? For example: derived columns feature on optimizer rewrite could give >> incorrect results. >> >> On Fri, Aug 7, 2026 at 9:09 PM Stefan Grafberger via dev < >> [email protected]> wrote: >> >>> Hi all, >>> >>> Thanks a lot for bringing up this topic! >>> >>> I think one important topic to consider here is multi-modal workloads. >>> Both Lance and other databases like BigQuery use derived columns also >>> for embedding generation, where materializing the output of >>> expressions like embedding functions provides huge cost and compute >>> savings. >>> >>> - Lance Embedding Generation: https://docs.lancedb.com/embedding >>> - BigQuery Autonomous Embedding Generation: >>> >>> https://docs.cloud.google.com/bigquery/docs/autonomous-embedding-generation >>> >>> To support these multi-modal AI patterns in Iceberg, it would be >>> beneficial not to broadly prohibit all non-deterministic expressions >>> in the spec. >>> >>> Instead, it might be worth distinguishing between: >>> 1. Purely deterministic expressions (e.g., LOWER(col)), >>> 2. Purely non-deterministic expressions (e.g., RAND(), UUID()), and >>> 3. Non-deterministic but reusable expressions (e.g., AI.EMBED() in >>> BigQuery), where the value generation is AI based, so there's some >>> non-determinism involved, but once generated, the materialized result >>> acts as a reusable derived property. >>> >>> By making this distinction at the spec level, we could allow reusable >>> non-deterministic expressions without sacrificing the predictability >>> needed for query optimization. >>> >>> Curious to hear what others think about supporting such AI use cases. >>> >>> Best, >>> Stefan >>> >>> On Thu, Aug 6, 2026 at 8:09 PM Szehon Ho <[email protected]> >>> wrote: >>> > >>> > Hi, nice, I have also been looking at implementing this from the Spark >>> DSV2 side, would love to collaborate on this as well. >>> > >>> > Yes it seems like we can go further with the expression spec. I found >>> the determinism problem to be deeper than it seemed at first glance, a lot >>> of popular generation expressions in Delta, Spark are in fact dependent on >>> engine configs like timezone, which makes data skipping a bit tricky, we've >>> only recently started thinking about this issue. >>> > >>> > Thanks, >>> > Szehon >>> > >>> > On Thu, Aug 6, 2026 at 9:07 AM Daniel Weeks <[email protected]> wrote: >>> >> >>> >> What Andrei captured is a great summary of related proposals and >>> ongoing work. >>> >> >>> >> While there's no formal proposal, discussions have occurred regarding >>> using expressions and the default value expressions concept to expose >>> generated columns. >>> >> >>> >> We'd love for someone to create a formal proposal to expand upon >>> these existing proposals/features as it's a pretty natural extension from >>> what we have in progress already. >>> >> >>> >> I'd be happy to work with you on this if you would like to pick this >>> up, >>> >> -Dan >>> >> >>> >> On Thu, Aug 6, 2026 at 8:38 AM Andrei Tserakhau via dev < >>> [email protected]> wrote: >>> >>> >>> >>> Thanks for bringing this to the list, Prashant. >>> >>> >>> >>> On your question: no, there's no active work on a format-level >>> >>> derived/computed column concept in Iceberg, and as far as I can tell >>> >>> from the archives none has been proposed before -- so you're not >>> >>> duplicating anything. Some adjacent prior art that may be useful. >>> >>> >>> >>> Delta has had generated columns since OSS Delta 1.0.0 (May 2021), >>> with >>> >>> your exact motivation: a column materialized from an expression, used >>> >>> as a partition column, lets the engine infer a filter on it from a >>> >>> filter on the source and skip files. Two costs showed up in the five >>> >>> years since. First, the expression is stored as a Spark SQL string >>> >>> (delta.generationExpression), so anything that isn't Spark has to >>> parse >>> >>> Spark SQL to know what the column means -- which is roughly where >>> >>> presto.derived-columns.spec.json would put Iceberg, one dialect over. >>> >>> Second, Delta gates it behind a writer feature, so a writer that >>> >>> doesn't understand the invariant is refused rather than allowed to >>> >>> silently break it. Your RFC is candid that stale values produce >>> >>> incorrect results once rewrite is enabled, and table properties are >>> >>> defined as hints, so an unaware writer is within its rights to ignore >>> >>> them. Worth deciding early which of those two you want. >>> >>> >>> >>> On both counts there's an existing pattern in Iceberg you'd probably >>> >>> rather join than parallel. The expressions spec merged in June >>> >>> (https://github.com/apache/iceberg/pull/16652) -- field-ID >>> references >>> >>> that survive renames, dialect-specific behavior pushed into UDF >>> >>> references. Two features are already building on it: Huaxin's >>> >>> constraint support ( >>> https://lists.apache.org/thread/mv0gjm1o9v6wzx6dxkdw6yp5b9119w2d), >>> >>> which stores CHECK expressions as Expression objects rather than SQL >>> >>> strings, and Dan's default value expressions in V4 >>> >>> (https://lists.apache.org/thread/w0xqrm0dpnsgvw0dyvy4r34y0dtzmn7f, >>> PR >>> >>> 16777). The constraint proposal also tracks validity per snapshot >>> >>> rather than on the constraint, which might be a lighter answer to >>> your >>> >>> staleness problem than a format-version bump -- an unaware writer >>> would >>> >>> degrade you to "rewrite disabled" rather than "wrong answer." >>> >>> >>> >>> Two scope notes. Your item 3 (stats and a bloom filter with no stored >>> >>> data) overlaps the secondary index work fairly directly -- Péter has >>> a >>> >>> draft spec (https://github.com/apache/iceberg/pull/16961) and >>> there's >>> >>> an active sync series. And it's worth stating explicitly that this >>> >>> covers deterministic expressions only: Delta put deterministic >>> >>> expression-derived columns and non-deterministic value generation >>> >>> (identity/sequence) under one GENERATED ALWAYS AS umbrella, and >>> >>> separating them afterwards was painful, since pushdown rewrite needs >>> >>> reproducibility and value generation needs the opposite. Your RFC >>> >>> already excludes non-deterministic functions, so this is just making >>> >>> that explicit in the spec. >>> >>> >>> >>> Last thing: hidden-partition transforms already give metadata-level >>> >>> skipping for monotonic cases with no extra stored column. lower(col) >>> >>> genuinely isn't covered, so the motivation is fair -- but saying >>> where >>> >>> transforms already suffice and where a stored column is actually >>> needed >>> >>> would make the proposal harder to argue with. >>> >>> >>> >>> Best, >>> >>> Andrei >>> >>> >>> >>> On Thu, Aug 6, 2026 at 12:28 PM Prashant Sharma < >>> [email protected]> wrote: >>> >>>> >>> >>>> Hello All, >>> >>>> >>> >>>> Recently, I have been involved in bringing derived columns(aka >>> generated columns) for the Iceberg connector in Presto Link: >>> https://github.com/prestodb/rfcs/pull/61. We currently use table >>> properties to store the extra metadata for derived columns. However, we >>> cannot achieve the following: >>> >>>> >>> >>>> 1. Cross engine compatibility: How do we know if the derived >>> columns metadata is in sync i.e. if other engines unaware of derived >>> columns write to the table/alter the columns etc... >>> >>>> 2. UDF portability and versioning: which is already addressed by >>> (Iceberg UDF spec) >>> >>>> 3. VIRTUAL columns: We store all the column metadata/metrics/stats >>> including a bloom filter but not the actual data. A poor man's index. >>> >>>> >>> >>>> We are considering a Iceberg spec level support to address all of >>> the above. Just wondering, is there work already happening in this >>> direction elsewhere? >>> >>>> >>> >>>> If not, I have started an issue: >>> https://github.com/apache/iceberg/issues/15923 , will be adding more >>> details soon. >>> >>>> >>> >>>> Thanks, >>> >>>> Prashant >>> >>
