Hi Peter, Sorry for delay, Let me put out what I have got and then you can contribute
On Thu, 15 Oct 2026 at 4:36 PM, Péter Váry <[email protected]> wrote: > I would like to review this conversation. > The feature is very useful is on its own, but the derived column concept > could be used for several ongoing projects: > - Non-materialized columns: > https://lists.apache.org/thread.html/4h53r1nvvgxqymfvx7vcyhqgh1fk8180 > - Iceberg secondary indexes: > https://lists.apache.org/thread/gtpn4ydovmoxb1bffz5wcnsso20n379c > > Was there any more progress in this regard? > Is someone driving this feature, or shall I jump in? > > Thanks, > Peter > > Szehon Ho <[email protected]> ezt írta (időpont: 2026. aug. 11., K, > 0:41): > >> HI Prasant, sure thanks! I added some brief context to the doc from our >> previous research: >> >> 1. Put the official SQL spec definition >> 2. Define 'Generated Column' >> 3. Define 'Virtual Column' >> 4. Clarify what we gain/lose with deterministic expressions. >> >> I didn't modify the proposal yet, just wanted to make sure we are on the >> same page on the definition of derived column as we go into the proposal. >> Hope it looks ok with you. Maybe we can also chat on slack channel if it's >> more convenient, up to you folks. >> >> Hi Stefan, it's a good point. Some database like SQLServer seem to also >> support a similar concept (computed column), but there's a lot that the >> column cannot do if it's non-deterministic. As in the points I put in the >> doc, these are thing like VIRTUAL column, partition column, index key, >> check constraint, even Update. So while its not so useful in the >> traditional database sense, it may make sense to let the engine have an >> option. >> >> Aside from determinism, another question is portability, ie have engines >> (at least different instance of same engine) understand each other's column >> expression. Sounds like we want to align with UDF/View spec, and upcoming >> Check Constraint and Default Value spec, as mentioned in Andrei's email? >> >> Thanks >> Szehon >> >> On Mon, Aug 10, 2026 at 5:23 AM Prashant Sharma <[email protected]> >> wrote: >> >>> I am fairly new to Iceberg. Thanks for the encouragement. Here is a >>> first draft (still WIP) >>> https://docs.google.com/document/d/1u93kyIrTc8bt9VSTB4O0GM9S7zuXekqixJR6Id2FZC4/edit?usp=sharing. >>> Please take a look ! >>> >>> https://github.com/apache/iceberg/issues/15923 >>> >>> @Andrei Tserakhau <[email protected]> >>> Your summary has helped a lot in catching up with parallel work and also >>> the history - Thanks a lot ! >>> >>> @[email protected] <[email protected]> and >>> @[email protected] <[email protected]> Happy to collaborate - was >>> trying to add you as editor - let me know your google account enabled id. >>> >>> In the proposal document above, I have also tried to address two problem >>> which are common to multiple v4 features >>> >>> 1. Expressions reuse >>> 2. Backward compatibility: V4-features need the ability to know data was >>> modified by an engine unaware of the feature and how does the engine tell >>> that? For example: derived columns feature on optimizer rewrite could give >>> incorrect results. >>> >>> On Fri, Aug 7, 2026 at 9:09 PM Stefan Grafberger via dev < >>> [email protected]> wrote: >>> >>>> Hi all, >>>> >>>> Thanks a lot for bringing up this topic! >>>> >>>> I think one important topic to consider here is multi-modal workloads. >>>> Both Lance and other databases like BigQuery use derived columns also >>>> for embedding generation, where materializing the output of >>>> expressions like embedding functions provides huge cost and compute >>>> savings. >>>> >>>> - Lance Embedding Generation: https://docs.lancedb.com/embedding >>>> - BigQuery Autonomous Embedding Generation: >>>> >>>> https://docs.cloud.google.com/bigquery/docs/autonomous-embedding-generation >>>> >>>> To support these multi-modal AI patterns in Iceberg, it would be >>>> beneficial not to broadly prohibit all non-deterministic expressions >>>> in the spec. >>>> >>>> Instead, it might be worth distinguishing between: >>>> 1. Purely deterministic expressions (e.g., LOWER(col)), >>>> 2. Purely non-deterministic expressions (e.g., RAND(), UUID()), and >>>> 3. Non-deterministic but reusable expressions (e.g., AI.EMBED() in >>>> BigQuery), where the value generation is AI based, so there's some >>>> non-determinism involved, but once generated, the materialized result >>>> acts as a reusable derived property. >>>> >>>> By making this distinction at the spec level, we could allow reusable >>>> non-deterministic expressions without sacrificing the predictability >>>> needed for query optimization. >>>> >>>> Curious to hear what others think about supporting such AI use cases. >>>> >>>> Best, >>>> Stefan >>>> >>>> On Thu, Aug 6, 2026 at 8:09 PM Szehon Ho <[email protected]> >>>> wrote: >>>> > >>>> > Hi, nice, I have also been looking at implementing this from the >>>> Spark DSV2 side, would love to collaborate on this as well. >>>> > >>>> > Yes it seems like we can go further with the expression spec. I >>>> found the determinism problem to be deeper than it seemed at first glance, >>>> a lot of popular generation expressions in Delta, Spark are in fact >>>> dependent on engine configs like timezone, which makes data skipping a bit >>>> tricky, we've only recently started thinking about this issue. >>>> > >>>> > Thanks, >>>> > Szehon >>>> > >>>> > On Thu, Aug 6, 2026 at 9:07 AM Daniel Weeks <[email protected]> >>>> wrote: >>>> >> >>>> >> What Andrei captured is a great summary of related proposals and >>>> ongoing work. >>>> >> >>>> >> While there's no formal proposal, discussions have occurred >>>> regarding using expressions and the default value expressions concept to >>>> expose generated columns. >>>> >> >>>> >> We'd love for someone to create a formal proposal to expand upon >>>> these existing proposals/features as it's a pretty natural extension from >>>> what we have in progress already. >>>> >> >>>> >> I'd be happy to work with you on this if you would like to pick this >>>> up, >>>> >> -Dan >>>> >> >>>> >> On Thu, Aug 6, 2026 at 8:38 AM Andrei Tserakhau via dev < >>>> [email protected]> wrote: >>>> >>> >>>> >>> Thanks for bringing this to the list, Prashant. >>>> >>> >>>> >>> On your question: no, there's no active work on a format-level >>>> >>> derived/computed column concept in Iceberg, and as far as I can tell >>>> >>> from the archives none has been proposed before -- so you're not >>>> >>> duplicating anything. Some adjacent prior art that may be useful. >>>> >>> >>>> >>> Delta has had generated columns since OSS Delta 1.0.0 (May 2021), >>>> with >>>> >>> your exact motivation: a column materialized from an expression, >>>> used >>>> >>> as a partition column, lets the engine infer a filter on it from a >>>> >>> filter on the source and skip files. Two costs showed up in the five >>>> >>> years since. First, the expression is stored as a Spark SQL string >>>> >>> (delta.generationExpression), so anything that isn't Spark has to >>>> parse >>>> >>> Spark SQL to know what the column means -- which is roughly where >>>> >>> presto.derived-columns.spec.json would put Iceberg, one dialect >>>> over. >>>> >>> Second, Delta gates it behind a writer feature, so a writer that >>>> >>> doesn't understand the invariant is refused rather than allowed to >>>> >>> silently break it. Your RFC is candid that stale values produce >>>> >>> incorrect results once rewrite is enabled, and table properties are >>>> >>> defined as hints, so an unaware writer is within its rights to >>>> ignore >>>> >>> them. Worth deciding early which of those two you want. >>>> >>> >>>> >>> On both counts there's an existing pattern in Iceberg you'd probably >>>> >>> rather join than parallel. The expressions spec merged in June >>>> >>> (https://github.com/apache/iceberg/pull/16652) -- field-ID >>>> references >>>> >>> that survive renames, dialect-specific behavior pushed into UDF >>>> >>> references. Two features are already building on it: Huaxin's >>>> >>> constraint support ( >>>> https://lists.apache.org/thread/mv0gjm1o9v6wzx6dxkdw6yp5b9119w2d), >>>> >>> which stores CHECK expressions as Expression objects rather than SQL >>>> >>> strings, and Dan's default value expressions in V4 >>>> >>> (https://lists.apache.org/thread/w0xqrm0dpnsgvw0dyvy4r34y0dtzmn7f, >>>> PR >>>> >>> 16777). The constraint proposal also tracks validity per snapshot >>>> >>> rather than on the constraint, which might be a lighter answer to >>>> your >>>> >>> staleness problem than a format-version bump -- an unaware writer >>>> would >>>> >>> degrade you to "rewrite disabled" rather than "wrong answer." >>>> >>> >>>> >>> Two scope notes. Your item 3 (stats and a bloom filter with no >>>> stored >>>> >>> data) overlaps the secondary index work fairly directly -- Péter >>>> has a >>>> >>> draft spec (https://github.com/apache/iceberg/pull/16961) and >>>> there's >>>> >>> an active sync series. And it's worth stating explicitly that this >>>> >>> covers deterministic expressions only: Delta put deterministic >>>> >>> expression-derived columns and non-deterministic value generation >>>> >>> (identity/sequence) under one GENERATED ALWAYS AS umbrella, and >>>> >>> separating them afterwards was painful, since pushdown rewrite needs >>>> >>> reproducibility and value generation needs the opposite. Your RFC >>>> >>> already excludes non-deterministic functions, so this is just making >>>> >>> that explicit in the spec. >>>> >>> >>>> >>> Last thing: hidden-partition transforms already give metadata-level >>>> >>> skipping for monotonic cases with no extra stored column. lower(col) >>>> >>> genuinely isn't covered, so the motivation is fair -- but saying >>>> where >>>> >>> transforms already suffice and where a stored column is actually >>>> needed >>>> >>> would make the proposal harder to argue with. >>>> >>> >>>> >>> Best, >>>> >>> Andrei >>>> >>> >>>> >>> On Thu, Aug 6, 2026 at 12:28 PM Prashant Sharma < >>>> [email protected]> wrote: >>>> >>>> >>>> >>>> Hello All, >>>> >>>> >>>> >>>> Recently, I have been involved in bringing derived columns(aka >>>> generated columns) for the Iceberg connector in Presto Link: >>>> https://github.com/prestodb/rfcs/pull/61. We currently use table >>>> properties to store the extra metadata for derived columns. However, we >>>> cannot achieve the following: >>>> >>>> >>>> >>>> 1. Cross engine compatibility: How do we know if the derived >>>> columns metadata is in sync i.e. if other engines unaware of derived >>>> columns write to the table/alter the columns etc... >>>> >>>> 2. UDF portability and versioning: which is already addressed by >>>> (Iceberg UDF spec) >>>> >>>> 3. VIRTUAL columns: We store all the column metadata/metrics/stats >>>> including a bloom filter but not the actual data. A poor man's index. >>>> >>>> >>>> >>>> We are considering a Iceberg spec level support to address all of >>>> the above. Just wondering, is there work already happening in this >>>> direction elsewhere? >>>> >>>> >>>> >>>> If not, I have started an issue: >>>> https://github.com/apache/iceberg/issues/15923 , will be adding more >>>> details soon. >>>> >>>> >>>> >>>> Thanks, >>>> >>>> Prashant >>>> >>>
