Hey Sergei, Thanks for bringing this up and for making the effort to bring this forward. I'm the person who added fixed[16] encoded UUIDs and timestamp-nanos to Avro a while ago. Keep in mind that we started with int64 nanos in Avro, so we might need to add wider types.
Both nanoseconds and local time zones were added to Avro beyond the workaround in Iceberg, but we never updated the notes in the Iceberg spec. So I agree we need to update the spec to reflect these changes, including informing the reader with the historical context of the adjust-to-utc. I think it makes sense to drop the adjust-to-utc property when writing in V4 in favor of local types, but we still need to support it when reading older versions. For data files, versions can be mixed since after upgrading the table version, the data files are not rewritten. First, I would propose a spec change, which is a bit more involved: https://iceberg.apache.org/contribute/#what-is-an-improvement-proposal. Once there is consensus around the spec change, the implementation can follow. Hope this helps, and looking forward to the proposed changes to the specification. Kind regards, Fokko On 2026/08/31 14:59:41 Sergei Nikolaev wrote: > Hi all, > > I'd like to discuss how Avro timestamps should be represented in V4. > Personally, I'm in favor of following the Avro 1.12.0 spec and dropping the > Iceberg-specific conventions, but I'd love to hear other ideas on this. > > The goals are: > 1. Make it easier to write Avro-encoded data into Iceberg tables with > ingestion tools like the Kafka Iceberg Sink and the Flink Iceberg Sink > 2. Allow standard Avro files to be added to a table with no modifications > needed > 3. Generally avoid diverging from the Avro spec when there's no real need > to :) > > > Currently, [the Appendix A - Avro](https://iceberg.apache.org/spec/#avro) > mandates the following mappings: > > |Type|Avro type|Notes| > |--- |--- |--- | > |**`timestamp`** | `{ "type": "long", "logicalType": > "timestamp-micros", "adjust-to-utc": false }` | Stores microseconds from > 1970-01-01 00:00:00.000000. [1] | > |**`timestamptz`** | `{ "type": "long", "logicalType": > "timestamp-micros", "adjust-to-utc": true }` | Stores microseconds from > 1970-01-01 00:00:00.000000 UTC. [1] | > |**`timestamp_ns`** | `{ "type": "long", "logicalType": "timestamp-nanos" > , "adjust-to-utc": false }` | Stores nanoseconds from 1970-01-01 > 00:00:00.000000000. [1], [2] | > |**`timestamptz_ns`** | `{ "type": "long", "logicalType": "timestamp-nanos" > , "adjust-to-utc": true }` | Stores nanoseconds from 1970-01-01 > 00:00:00.000000000 UTC. [1], [2] | > > Notes: > 1. Avro type annotation `adjust-to-utc` is an Iceberg convention; default > value is `false` if not present. > 2. Avro logical type `timestamp-nanos` is an Iceberg convention; the Avro > specification does not define this type. > > > Avro 1.10.0 (released 2020-07-01) added `local-timestamp-millis` and > `local-timestamp-micros`, and 1.12.0 (2024-08-05) added `timestamp-nanos` > and `local-timestamp-nanos` on top of that. One option would be to simply > add support for `local-timestamp-(micros|nanos)` for timestamps without a > zone, on top of the existing mappings. > The catch is that timestamp logical types have been UTC-adjusted by > definition since they were introduced in Avro 1.8.0 (2016-01-29), whereas > the Iceberg convention treats `timestamp-(micros|nanos)` without > `adjust-to-utc` as local. In practice this is probably only a spec-level > inconsistency, since the core library has always [set `adjust-to-utc` > explicitly, going back at least to 0.7.0]( > https://github.com/apache/iceberg/blob/apache-iceberg-0.7.0-incubating/core/src/main/java/org/apache/iceberg/avro/TypeToSchema.java#L53-L54). > The only place timestamp types can actually appear in v1-v3 manifests is in > partition data, when the partition spec applies an `identity` or `void` > transform to a timestamp field - there are no static timestamp fields > defined anywhere else in the spec. > > > Since Avro has had everything Iceberg needs for over two years now, rather > than growing the spec further, I'd rather follow upstream and drop > `adjust-to-utc` entirely in V4: > > |Type|Avro type|Notes| > |--- |--- |--- | > |**`timestamp`** | `{ "type": "long", "logicalType": > "local-timestamp-micros" }` | Stores microseconds from 1970-01-01 > 00:00:00.000000. | > |**`timestamptz`** | `{ "type": "long", "logicalType": > "timestamp-micros" }` | Stores microseconds from 1970-01-01 > 00:00:00.000000 UTC. | > |**`timestamp_ns`** | `{ "type": "long", "logicalType": > "local-timestamp-nanos" }` | Stores nanoseconds from 1970-01-01 > 00:00:00.000000000. | > |**`timestamptz_ns`** | `{ "type": "long", "logicalType": "timestamp-nanos" > }` | Stores nanoseconds from 1970-01-01 00:00:00.000000000 UTC. | > > No notes needed. > > > According to [the ongoing work](https://github.com/apache/iceberg/pull/16025), > manifests will get a per-entry `format_version` `int` field (0: PRE-V4, 4: > V4). Readers could use this field to interpret `timestamp-*`: > - `format_version=0`: read `adjust-to-utc`, defaulting to `false` > - `format_version>=4`: always read as UTC-adjusted, ignore `adjust-to-utc` > > This would resolve the conflict and remove any ambiguity. It would also let > users add external Avro files to a table without rewriting them - a file > gets `format_version=0` if its embedded writer schema is Iceberg v1-3 spec > compliant, or `format_version=4` if it's a plain Avro schema. > > Does this seem like a valid use of the `format_version` field? > > The work to support `local-timestamp-*` and to read `timestamp-*` as > UTC-adjusted is already underway in [PR 17196]( > https://github.com/apache/iceberg/pull/17196), and I'm starting this > discussion to help move it forward. There are several other related issues > and PRs linked from that PR's description, so this isn't a new topic - > users have wanted "new" Avro logical types integrated into Iceberg for a > while now. > > I'm looking forward to your thoughts! > > Best regards, > Sergei Nikolaev >
