Re: [ANNOUNCE] Apache Iceberg release 1.8.0

Hi, I agree with Fokko's sum-up and call for action. Regards JB On Wed, Feb 19, 2025 at 11:34 AM Fokko Driesprong wrote: > > Thanks for the historical context here. I wasn't aware of the equivalent > between -1 and null. > > It looks like we're all aligned on not retroactively changing the spe

Re: [ANNOUNCE] Apache Iceberg release 1.8.0

Thanks for the historical context here. I wasn't aware of the equivalent between -1 and null. It looks like we're all aligned on not retroactively changing the spec. Let's wrap this up with some actions: - To revert the changes, I've raised the PRs (1.8.x

Re: [ANNOUNCE] Apache Iceberg release 1.8.0

I agree that we shouldn't retroactively change the spec to align with the implementation. We're trying to strike a balance here and I think the notes are an effective way to convey that while omitting or producing null with v1/v2 is technically to spec, it's going to be incompatible with most impl

Re: [ANNOUNCE] Apache Iceberg release 1.8.0

`For compatibility with existing libraries, we should maintain that `-1` is equivalent to no snapshot and it should be written for v1/v2.` The only issue I have with this is are we saying that for v1 and v2 we are changing the spec to say that current-snapshot-id is required? Or are we adding an i

Re: [ANNOUNCE] Apache Iceberg release 1.8.0

I would agree the best path forward is to note the current behavior for v1/v2 since that's well established and address the behavior in v3. For compatibility with existing libraries, we should maintain that `-1` is equivalent to no snapshot and it should be written for v1/v2. With V3 we should su

Re: [ANNOUNCE] Apache Iceberg release 1.8.0

+1 to reverting PT 11560 in main and 1.8.1. That avoids unnecessary incompatibility with older readers. I also agree that we should update the spec to say what Russell suggests: > that -1 has meant "no current snapshot" in the past and is equivalent to missing/null. That's a correct description o

Re: [ANNOUNCE] Apache Iceberg release 1.8.0

I had a couple of small fixes that would be great to get into 1.8.1: https://github.com/apache/iceberg/pull/12305 https://github.com/apache/iceberg/pull/12224 I added those to the GitHub 1.8.1 milestone in case this is possible. Thanks, Bryan > On Feb 18, 2025, at 8:53 AM, Robert Stupp wrote: >

Re: [ANNOUNCE] Apache Iceberg release 1.8.0

On 18.02.25 17:10, Fokko Driesprong wrote: Reality is that Iceberg did write '-1' into current-snapshot-id (and other "non-exist" marker values for schema/spec/sort) instead of omitting the field. Yes, but this is wrong. The spec dictates

Re: [ANNOUNCE] Apache Iceberg release 1.8.0

> > Reality is that Iceberg did write '-1' into current-snapshot-id (and other > "non-exist" marker values for schema/spec/sort) instead of omitting the > field. Yes, but this is wrong. The spec dictates under current-snapshot-id: long ID

Re: [ANNOUNCE] Apache Iceberg release 1.8.0

The only thing I think I agree with is defining that -1 has meant "no current snapshot" in the past and is equivalent to a missing/nuil (if we have to specify that) . I don't think there is any reason to change the behavior of writing null / missing unless that's really a point of confusion for fo

Re: [ANNOUNCE] Apache Iceberg release 1.8.0

Correcting myself: schema/spec/sort seem to be always present - please ignore that part in my previous email. The valid values for those fields however should be defined. On 18.02.25 14:29, Robert Stupp wrote: Reality is that Iceberg did write '-1' into current-snapshot-id (and other "non-ex

Re: [ANNOUNCE] Apache Iceberg release 1.8.0

Reality is that Iceberg did write '-1' into current-snapshot-id (and other "non-exist" marker values for schema/spec/sort) instead of omitting the field. Side note: the table-spec says that these fields are optional, but nothing about whether it is nullable. The spec should at least be amend

Re: [ANNOUNCE] Apache Iceberg release 1.8.0

It sounds like the argument here is that we should change the Spec for V1, V2, and V3 to mark current-snapshot-id as required. Then we should change all other implementations to follow this new standard. I'm not sure that is a good solution going forwards but I'm not sure of how we can support cata

Re: [ANNOUNCE] Apache Iceberg release 1.8.0

Hey Robert, The thing is, that -1 cannot "go away". Yes, I agree, but that's also the case for null, as the field is optional in the spec . Therefore we support both in PyIceberg

Re: [ANNOUNCE] Apache Iceberg release 1.8.0

Hi Fokko, sure, in general "absent" or "null" would be cleaner. But now we have two representations for the same case - I suspect most went with the "reference behavior". The thing is, that -1 cannot "go away". I'd prefer to keep the previous behavior - otherwise implementations may fall ba

Re: [ANNOUNCE] Apache Iceberg release 1.8.0

Hey Robert, Thanks for raising this. snapshot-ID -1 isn't per-se invalid, because the valid values are not > defined in the spec. For me, this is invalid, since there is no snapshot with -1 in the snapshots property. In the tests with the PR, you can see that there are no snapshots

Re: [ANNOUNCE] Apache Iceberg release 1.8.0

Feels like https://github.com/apache/iceberg/pull/11560 introduced a behavior change. snapshot-ID -1 isn't per-se invalid, because the valid values are not defined in the spec. Previous Iceberg-Java versions always produced -1 if there's no current snapshot - 1.8 produces `null` in that case

Re: [ANNOUNCE] Apache Iceberg release 1.8.0

I think we wanted to perform a check against the endpoint support like other view operations since they aren't expected to be implemented. PR here is what I believe we want. -Dan On Thu, Feb 13, 2025 at 7:37 AM Eduard Tudenhöfner wrote: > This is

Re: [ANNOUNCE] Apache Iceberg release 1.8.0

This is most likely caused by the additional checks that were added in https://github.com/apache/iceberg/pull/11756. On Thu, Feb 13, 2025 at 4:16 PM Gabor Kaszab wrote: > Hi, > > I think it's because of this patch: > https://github.com/apache/iceberg/commit/a2b8008da7bc26e03248a3560d1cc7e849

Re: [ANNOUNCE] Apache Iceberg release 1.8.0

Hi, I think it's because of this patch: https://github.com/apache/iceberg/commit/a2b8008da7bc26e03248a3560d1cc7e8499d Before that replaceTransaction() didn't fail if the server didn't support views, with this change it does. Anyway, I see this was merged into 1.7 too. Didn't you see the same i

Re: [ANNOUNCE] Apache Iceberg release 1.8.0

Hi, There is a breaking change in 1.8.0 that prevents Iceberg 1.8 clients to talk to pre-1.8 Iceberg REST services. The errors manifest as something like: java.lang.UnsupportedOperationException: Server does not support endpoint: HEAD /v1/{prefix}/namespaces/{namespace}/views/{view} at

[ANNOUNCE] Apache Iceberg release 1.8.0

I'm pleased to announce the release of Apache Iceberg 1.8.0! Apache Iceberg is an open table format for huge analytic datasets. Iceberg delivers high query performance for tables with tens of petabytes of data, along with atomic commits, concurrent writes, and SQL-compatible table evolution. This