I don't think that we are trying to solve the problem of generated columns right now. I think that should be part of the schema so it is separate from this work.
You might be suggesting that we use generated columns to store stats instead of the current proposal. I don't think that is a good idea because it adds extra fields to the schema with different behavior that breaks assumptions for schemas. For instance, stats-only fields are not fields in the schema and should not be displayed as part of the table schema. Generated columns are part of table data, which is why they belong in the schema. If we add stats-only fields to the schema, then anyone consuming the schema needs to know what to filter out before passing it along. Similarly, schema fields have specific types that change only when a user modifies them. Stats fields are derived and can change when schema changes affect them, so we would not be able to produce all field types without binding the schema to a subset of itself. This would also complicate composition. Right now, stats-only fields are not nested within the schema, they are independent. If we were allowing a field to be a stat-only field then we have to come up with rules around where they can be used (not in map keys?) and then check those rules. How would you handle a struct with only one field that is a stats-only field that is not actually present in the schema? The proposal is much simpler because we don't need to worry about many of these concerns. Ryan On Thu, Oct 22, 2026 at 10:15 PM Péter Váry <[email protected]> wrote: > Hi Ryan, > > Thanks for putting this together! > > I had the same impression when I took a look at the derived columns > proposal, and we updated that. > > Now, a schema would look like this: > > schema { > 1: id: required int > 2: name: optional string > 3: hilbert: optional long (generated: stats-only: hilbert(1, 2)) > } > > Or in JSON: > { > [...] > "fields": [ > { > "id": 3, > "name": "id", > "required": false, > "type": "long", > "generated": > { > "strategy": "stats-only", // on-write, on-read(?) > "expression": [...] > } > }, > [...] > ] > } > > This would help removing the extra rules around the id generation, stat > generation and we likely need the generated columns too. > > Schema already contains expressions, elements that are not stored in the > files, and stats-only fields are optional so just return `null` when > queried. > > WDYT? > > Thanks, > Peter > > > On Fri, Oct 23, 2026, 00:56 Ryan Blue <[email protected]> wrote: > >> Hi everyone, >> >> We've been working on the spec changes for non-materialized columns in >> v4, which was previously discussed here: >> https://lists.apache.org/thread/4h53r1nvvgxqymfvx7vcyhqgh1fk8180 >> >> I think the spec is about ready. Please take a look here: >> https://github.com/apache/iceberg/pull/18405 >> >> I ended up using "stats-only fields" since I think that's a more >> straightforward name than "non-materialized columns". The changes are what >> was discussed in the last thread. >> >> I also considered whether it would be a good idea to unify this with >> materialized columns that are generated by an expression, like default >> value expressions or generated columns. I decided not to do that for the >> same reasons we chose not to separate expressions from check constraints. >> That would require adding an expression and then adding a column that >> depends on it. That indirection isn't needed and I think generated columns >> should have an expression in the definition. >> >> Please take a look at the PR and comment. I'd like to start a vote soon >> if there is consensus. Thanks! >> >> Ryan >> >
