Re: [DISCUSS] Proposal for Iceberg 1.9.2 Release to Fix Critical REST Client Issue

Sorry I didn't get to reply here, I think the fix Ajantha is contributing is extremely important but probably out of scope for a patch release. Because it will take a bit of manual intervention to fix after jumping to the next version I think we should save this for 1.10.0 which also should come ou

Re: [DISCUSS] V4 - Parquet as Metadata File Format

t; Let me know the results of your POC and happy to collaborate on this work. > > > - Ajantha > > On Fri, Jun 13, 2025 at 3:16 AM Russell Spitzer > wrote: > >> All we have to do is add the parquet module as a test dependency, working >> on a poc now. I don't t

Re: [DISCUSS] V4 - Parquet as Metadata File Format

se metadata structure can change? > > - Ajantha > > On Thu, May 29, 2025 at 11:37 PM Russell Spitzer < > russell.spit...@gmail.com> wrote: > >> Hi Y'all >> >> As discussed in the last community sync, we are beginning to gather up >> folks who are int

Re: [DISCUSS] June board report

Looking good! On Wed, Jun 4, 2025 at 4:21 PM Ryan Blue wrote: > Hi everyone, > > Here’s my draft of our board report for June. I went through the old syncs > for highlights, but please reply if you want me to add any more! > > Ryan > Description: > > Apache Iceberg is a table format for huge ana

Re: [DISCUSS] v4 - One file commits

I’m also super excited about this idea On Thu, May 29, 2025 at 3:37 PM Amogh Jahagirdar <2am...@gmail.com> wrote: > Thanks for kicking this thread off Ryan, I'm interested in helping out > here! I've been working on a proposal in this area and it would be great to > collaborate with different fol

[DISCUSS] API: Rename RowDelta deleteFile to removeRows

Ryan pointed out to me that whenI added the "deleteFile" method I was not following the convention already being used within the RowDelta operation and instead had copied the OverwriteFiles API. To fix this I think it would be great to change the API to "removeRows" to match the other APIs in the c

Re: [Discuss] Make identity(String sourceName, String targetName) Public

gt; public scope > * identity transform builder is the only one where targetName builder is > not public > * handle the partition column rename use case > > So it seems reasonable to me. > > > On Wed, May 21, 2025 at 2:49 PM Russell Spitzer > wrote: > >> Hi Y

[DISCUSS] V4 - Parquet as Metadata File Format

Hi Y'all As discussed in the last community sync, we are beginning to gather up folks who are interested in various efforts for Iceberg V4. To that end, I'd like to use this thread as a gathering point for folks interested in the metadata file format shift to Parquet. I wrote a quick abstract to d

Re: [DISCUSS] Enabling more Meetups

: > > > On 2025/05/23 19:24:28 Russell Spitzer wrote: > > Hey Y'all > > > > Basically I would like to get the PMC out of the meetup approval > business > ... > > Please let me know what you think, > > (Board hat) > > A critical role of a P

[ANNOUNCE] Apache Iceberg release 1.9.1

I'm pleased to announce the release of Apache Iceberg 1.9.1! Apache Iceberg is an open table format for huge analytic datasets. Iceberg delivers high query performance for tables with tens of petabytes of data, along with atomic commits, concurrent writes, and SQL-compatible table evolution. This

Re: [DISCUSS] Enabling more Meetups

t; Le mar. 27 mai 2025 à 17:52, Ryan Blue a écrit : >> >> >> >> JB, can you give us a bit more context about why you're recommending >> those pages? Do they have policies that already do what is being suggested? >> Do they impose limits that mean we could not

Re: [RESULT][VOTE] Release Apache Iceberg 1.9.1 RC1

I forgot Peter! I'm so sorry! That should have been +1: 4 (binding), 5 (non-binding). {Russell, Steven Wu, Fokko, Peter} | {JB, Karuppayya, Kevin, Huaxin, Aihua} +0: 0 (binding), 0 (non-binding) -1: 0 (binding), 0 (non-binding) On Tue, May 27, 2025 at 4:29 PM Russell Spitzer

[RESULT][VOTE] Release Apache Iceberg 1.9.1 RC1

>> >> On Tue, May 27, 2025 at 9:06 AM Russell Spitzer < >> russell.spit...@gmail.com> wrote: >> >>> For all those who haven't seen this before, GPG key signing is a very >>> "early hacker" sort of thing. The idea is the only way to

Re: [VOTE] Release Apache Iceberg 1.9.1 RC1

ignature, checksum, license and ran some tests. >> >> On Tue, May 27, 2025 at 9:06 AM Russell Spitzer < >> russell.spit...@gmail.com> wrote: >> >>> For all those who haven't seen this before, GPG key signing is a very >>> "early hacker" so

Re: [DISCUSS] Apache Iceberg 1.10.0 release

Thanks Steven! I know we are going to have a busy June but I think pretty much everything (except Geo Types) is close to being ready in the reference Spark implementation. On Tue, May 27, 2025 at 1:08 PM Steven Wu wrote: > > As discussed in the community sync, we are planning for the next 1.10.0

Re: Wide tables in V4

than > writing to a separate file, I am not sure how much worse it is though. > > On Tue, May 27, 2025 at 9:40 AM Russell Spitzer > wrote: > >> I think that "after the fact" modification is one of the requirements >> here, IE: Updating a single column without

Re: Wide tables in V4

at tooling actually supports this though. Could be >> interesting to see what the history of this is. >> https://lists.apache.org/thread/rcv1cxndp113shjybfcldh6nq1t3lcq3, >> https://lists.apache.org/thread/k5nv310yp315fttcz213l8o0vmnd7vyw >> >> On Tue, May 27, 2025 at 8

Re: [VOTE] Release Apache Iceberg 1.9.1 RC1

in 'apache-iceberg-1.9.1.tar.gz' >> gpg: Signature made Wed May 21 15:19:17 2025 PDT >> gpg:using RSA key xxx >> gpg: Good signature from "Russell Spitzer (CODE SIGNING KEY) >> " [unknown] >> gpg: WARNING: This key is not certified

Re: Wide tables in V4

I have to agree that while there can be some fixes in Parquet, we fundamentally need a way to split a "row group" or something like that between separate files. If that's something we can do in the parquet project that would be great but it feels like we need to start exploring more drastic options

[DISCUSS] Enabling more Meetups

Hey Y'all I know there has been a lot of confusion over who is allowed to host a meetup and what it can be called. Technically, I believe the PMC is supposed to be involved in anything involving the Apache Iceberg trademark but I'd like to pre-approve a bunch of meetups if we have consensus. Basi

Re: [VOTE] Release Apache Iceberg 1.9.1 RC1

;> anymore) >>> https://github.com/apache/iceberg/issues/13122 >>> >>> If it is induced by 1.9.0, we should probably fix it and include it in >>> RC? >>> No functional impact. >>> >>> - Ajantha >>> >>> On Thu, May

[DISCUSS] Overview of the Table Specification Lifecycle and how to Make Changes

Hi Y'all, As the community grows I wanted to get some of our tribal knowledge written down and get some consensus on what I think isn't really written down anywhere. Please take a look at let me know if you have any questions or comments. I'm imagining this would go either parallel to Contributing

Re: [VOTE] Release Apache Iceberg 1.9.1 RC0

de in the > following release. I feel it's a regression which didn't catch in 1.9.0 and > make sense to fix in 1.9.1. > > Thanks Russell. Busy with oncall to miss today's sync. I guess we agree to > move forward to make this fix? > > > > On Wed, May 21, 202

[VOTE] Release Apache Iceberg 1.9.1 RC1

Hi Y'all, I propose that we release the following RC as the official Apache Iceberg 1.9.1 release. The commit ID is f40208ae6fb2f33e578c2637d3dea1db18739f31 * This corresponds to the tag: apache-iceberg-1.9.1-rc1 * https://github.com/apache/iceberg/commits/apache-iceberg-1.9.1-rc1 * https://githu

Re: [VOTE] Release Apache Iceberg 1.9.1 RC0

probably need > to work on a better/complete fix. > > I'm not sure reverting this change would make sense either. I'm more > in favor of continuing the 1.9.1 vote. > > Regards > JB > > On Mon, May 19, 2025 at 6:25 PM Russell Spitzer > wrote: > > > &

[Discuss] Make identity(String sourceName, String targetName) Public

Hi Y'all We've been considering making a change to the Identity Partition Transform builder. Unlikely all of the other builders, Identity doesn't allow you to make an Identity Transform with a name different from the column you are transforming. We want to be able to construct in memory a TableMe

Re: [VOTE] Adopt the v3 spec changes

+1 (binding) On Tue, May 20, 2025 at 11:02 AM Steven Wu wrote: > +1 (binding) > > On Tue, May 20, 2025 at 5:25 AM Manu Zhang > wrote: > >> +1 (non-binding). Thanks Ryan for driving this and everyone contributing >> to the new features. >> >> Regards, >> Manu >> >> Péter Váry 于2025年5月20日 周二20:14

Re: [VOTE] Release Apache Iceberg 1.9.1 RC0

t; >>>>> It runs tests against several catalogs, including HMS, Glue, >>>>> JDBC (PostgreSQL), REST (Polaris, Unity, S3 Tables, Tabular), Nessie, and >>>>> Snowflake. >>>>> >>>>> >>>>> >>>>> BR, >>>>> >>&g

Re: [VOTE] Release Apache Iceberg 1.9.1 RC0

+1 (Binding) All the normal things and also checked manually ---> scala> import org.apache.iceberg.IcebergBuild import org.apache.iceberg.IcebergBuild scala> IcebergBuild.version res0: String = 1.9.1 On Fri, May 16, 2025 at 4:32 PM Russell Spitzer wrote: > Hi Y'all, >

[VOTE] Release Apache Iceberg 1.9.1 RC0

Hi Y'all, I propose that we release the following RC as the official Apache Iceberg 1.9.1 release. The commit ID is 5541cf84b9e139d8dd22db44db7f592c3a2d * This corresponds to the tag: apache-iceberg-1.9.1-rc0 * https://github.com/apache/iceberg/commits/apache-iceberg-1.9.1-rc0 * https://githu

Re: [Discuss] Iceberg 1.9.1 Release

Steven explained the Flink issue to me, Flink 2.0 isn't in 1.9.0 so not an issue. On Fri, May 16, 2025 at 12:20 PM Russell Spitzer wrote: > Ok so far of the lists proposed above I only picked 2 fixes that apply > cleanly and (we double checked) > actually apply to 1.9.0. Som

Re: [Discuss] Iceberg 1.9.1 Release

On Tue, May 13, 2025, 06:42 Jean-Baptiste Onofré wrote: >> >>> Hi >>> >>> I did a fix/improvement on Avro. I will propose to do new Avro releases. >>> Maybe worth to include in Iceberg 1.9.1 if the timing is ok. >>> >>> Regards

Re: Spark 4.0/Iceberg Integration Merged – Spark 3.5 Merges Can Resume

Thanks for getting this in! On Wed, May 14, 2025 at 5:39 PM huaxin gao wrote: > Dear all, > > Thank you so much for your patience and support! > > The Spark 4.0/Iceberg integration PR has now been merged. You can go ahead > and resume normal merging on Spark 3.5. > > Really appreciate everyone’s

Re: [VOTE] Clarify writer requirements in the spec to prevent orphan DVs

+1 (Binding) On Wed, May 14, 2025 at 10:52 AM Anton Okolnychyi wrote: > Hi all, > > I propose the following update to the spec to clarify that writers must > remove any deletion vector that applies to a data file when that data file > is removed from the metadata, as previously brought up by Ste

Re: [Discuss] Iceberg 1.9.1 Release

t; >> * Core: Disallow creation of invalid PartitionSpec #12887 >> <https://github.com/apache/iceberg/pull/12887> >> * GCP: Support multiple storage credential prefixes #12881 >> <https://github.com/apache/iceberg/pull/12881> >> * Build, Core

Re: [Discuss] Iceberg 1.9.1 Release

e Onofré wrote: > Hi Russ > > Yes, agree. Your PR is good and already merged. > > I don't have anything blocker for 1.9.1 (still working on source-ids, > but definitely not for 1.9.1). > > Thanks ! > Regards > JB > > On Fri, May 2, 2025 at 11:47 PM Russel

Re: [DISCUSS] [REST SPEC] Add first-row-id in the data files for Row Lineage

Makes sense to me, perhaps we should also add in a test that checks that the Datafile api object and the REst spec are always in sync? On Mon, May 12, 2025 at 10:52 AM Amogh Jahagirdar <2am...@gmail.com> wrote: > Thanks Prashant, I definitely agree the first_row_id will need to be added > to the

Re: [VOTE] Merge details about GZip metadata files to the spec.

+1 (binding) On Mon, May 12, 2025 at 5:32 AM Eduard Tudenhöfner wrote: > +1 (binding) > > On Mon, May 12, 2025 at 3:45 AM Gang Wu wrote: > >> +1 (non-binding) >> >> On Mon, May 12, 2025 at 3:27 AM Kevin Liu wrote: >> >>> +1 (non-binding) >>> >>> Thanks for starting a vote. >>> >>> There's extr

Re: [VOTE] Add encryption key updates to REST spec

+1 (binding) On Thu, May 8, 2025 at 10:37 AM Amogh Jahagirdar <2am...@gmail.com> wrote: > +1 (binding) > > On Thu, May 8, 2025 at 9:29 AM Eduard Tudenhöfner < > etudenhoef...@apache.org> wrote: > >> +1 (binding) >> >> On Thu, May 8, 2025 at 5:23 PM Ryan Blue wrote: >> >>> Hi everyone, >>> >>> I’

Re: [VOTE] Minor clarification for Geo Spec

+1 (bind) On Wed, May 7, 2025 at 7:32 AM Eduard Tudenhöfner wrote: > +1 (binding) > > On Wed, May 7, 2025 at 4:14 AM Gang Wu wrote: > >> The clarification is simple and clear from the writer's perspective. >> >> CMIW, the implication is that reader should drop bbox with any NaN value >> regardl

[Discuss] Iceberg 1.9.1 Release

Hey y'all! Thanks to @suilis we have learned that IcebergBuild.version() is returning unspecified for Iceberg 1.9.0. I have a PR up to fix this and I think this is a clear reason to do a 1.9.1 as soon a

Re: [DISCUSS] Finalizing the v3 spec

Sounds good to me, I think we can move ahead with this, for all intents and purposes I think we are past any breaking changes for Spec V3 and should consider it "stable" for implementation purposes. I want to work on some official descriptions of our spec versioning / library process to better expl

Re: [VOTE] Add encryption keys to table metadata

+1 On Wed, Apr 30, 2025 at 11:36 AM Szehon Ho wrote: > +1 > > Thanks > Szehon > > On Wed, Apr 30, 2025 at 4:10 AM Eduard Tudenhöfner < > etudenhoef...@apache.org> wrote: > >> +1 (binding) >> >> On Tue, Apr 29, 2025 at 9:29 PM Ryan Blue wrote: >> >>> Hi everyone, >>> >>> I’d like to propose merg

Re: [DISCUSS] Finalizing the v3 spec

We should probably come to a resolution on the compressed metadata.json name as well, although that's mostly retroactive. V3 would be the place where we could officially change the naming convention. I'm also interested in getting a release with the full implementation of V3 as it currently stands

Re: [VOTE] Release Apache Iceberg 1.9.0 RC2

+1 Binding Did the normal tests Rat Check and such Signature Checks out - Eduard's -- I do keep having an issue with some of the S3 Signing tests which require me to keep adjusting my local config inorder for the tests to past. (Requested ports already bound) So that would be nice to clean up if p

Re: [VOTE] Small spec change for default values

+1 (Binding) On Mon, Apr 21, 2025 at 4:21 PM Ryan Blue wrote: > Hi everyone, > > I’d like to vote on the spec changes in PR 12841 > . This is a small change > that makes handling default values for structs much easier. Initially, we > allowed both a

Re: Iceberg Interval Types Proposal

I think this is a pretty good idea for us to adopt in terms of compatibility with other systems and I really appreciate that Naren made sure to use a broad enough definition to support all available engines. I'm really interested to know how other folks feel about this proposal and I hope we can re

Re: [VOTE] Spec Update: Variant Field Lower/Upper Bounds

+1 (binding) On Fri, Apr 18, 2025 at 12:45 PM Ron Ortloff wrote: > +1 (non-binding) > > On Fri, Apr 18, 2025 at 9:47 AM Ryan Blue wrote: > >> +1 (binding) >> >> On Thu, Apr 17, 2025 at 8:27 PM Aihua Xu wrote: >> >>> Hi all, >>> >>> I'd like to initiate a vote to include a spec update for suppo

Re: [VOTE] Update row lineage spec ID assignment

+1 On Thu, Apr 17, 2025 at 12:30 PM Ryan Blue wrote: > Adding my own +1. > > On Thu, Apr 17, 2025 at 10:19 AM Daniel Weeks wrote: > >> +1 (binding) >> >> I think this update really helps ensure row ids will be present and >> reliable for upgraded tables. Thanks Ryan! >> >> On Wed, Apr 16, 2025

Re: [VOTE] Simplify multi-argument field-id(s) encoding

+1 (Bind) On Thu, Apr 17, 2025 at 8:14 AM Jean-Baptiste Onofré wrote: > +1 (non binding) (as said in the PR :)) > > Thanks ! > > Regards > JB > > On Thu, Apr 17, 2025 at 3:00 PM Fokko Driesprong wrote: > > > > Hi everyone, > > > > A while ago, I sent out a DISCUSS around simplifying the encodin

Re: [DISCUSS] Introducing Iceberg Features ?

I'm not a big fan of this, I am currently a strong supporter of the V3 is V3 approach. This is one of the reasons we decided to make row-lineage mandatory, we want to avoid encouraging engines from selectively adopting requirements. On Tue, Apr 15, 2025 at 1:42 PM Fokko Driesprong wrote: > Hey J

Re: [VOTE] Release Apache Iceberg 1.9.0 RC0

> JB > > On Tue, Apr 15, 2025 at 6:39 PM Russell Spitzer > wrote: > > > > We had a ticket about improving kafka connect distribution, > https://github.com/apache/iceberg/issues/12507 because the current docs > require you to build your own kafka-connect zip

Re: [VOTE] Release Apache Iceberg 1.9.0 RC0

We had a ticket about improving kafka connect distribution, https://github.com/apache/iceberg/issues/12507 because the current docs require you to build your own kafka-connect zip. On Mon, Apr 14, 2025 at 8:15 AM Jean-Baptiste Onofré wrote: > Another point that I saw: the kafka-connect runtime d

Re: [VOTE] Row lineage required for v3

+1 On Mon, Mar 31, 2025 at 2:22 PM Amogh Jahagirdar <2am...@gmail.com> wrote: > +1 (binding) > > Thanks Dan! > > On Mon, Mar 31, 2025 at 1:20 PM Ryan Blue wrote: > >> +1 >> >> On Mon, Mar 31, 2025 at 12:01 PM Anton Okolnychyi >> wrote: >> >>> +1 (binding) >>> >>> - Anton >>> >>> пн, 31 бер. 202

Re: [Discuss] Apache Iceberg 1.9.0 release

github.com/apache/iceberg/pull/12120? >>>>> >>>>> On Tue, Mar 18, 2025 at 3:18 AM Steve Loughran >>>>> wrote: >>>>> >>>>>> >>>>>> Can I get this reviewed and merged; gives all hadoop filesystems with &g

Re: [DISCUSS] Multi-arg transforms

1. This makes sense to me, it was the only one requested in the past. It should allow "IN" as well. 2.What's the suggestion here? To allow both source-id and source-ids in V3 but error out if the two don't match? Trying to determine how the validation would look in both cases On Tue, Mar 25, 2025

Re: [VOTE] Minor simplifications for Geo Spec

+1 (bind On Fri, Mar 21, 2025 at 11:53 AM Steve Zhang wrote: > +1 (non-binding) > > Thanks, > Steve Zhang > > > > On Mar 18, 2025, at 6:29 PM, Gang Wu wrote: > > +1 (non-binding) > > >

Re: [DISCUSS] Row lineage required for v3

I think I'm in favor of this but I would like some way of knowing whether or not a snapshot was produced while preserving row_ids or not. Just so we can make it clear on read what the row-lineage behavior of the writer was without knowing what system wrote the data. On Thu, Mar 20, 2025 at 10:43 A

Re: [VOTE] Release Apache Iceberg 1.7.2 RC3

+1 (binding) verified sigs, checksums, rat. Ran Build/test On Tue, Mar 18, 2025 at 10:45 PM Amogh Jahagirdar <2am...@gmail.com> wrote: > +1 (binding) > verified signatures/checksums/RAT checks. Ran build/test against JDK17 > > Thanks, > Amogh Jahagirdar > > On Tue, Mar 18, 2025 at 8:57 PM Kevin L

Re: [DISCUSS] Inconsistency between java implementation and spec about partition-spec and schema in v1 table

Our general policy in the past has been to accept potentially invalid metadata as long as we can reason about it or until we get to a point where we have to throw an error. Read liberally and write strictly. I wouldn’t want to change the spec to match an incorrect writer but there is nothing wron

Re: [Discuss] Apache Iceberg 1.9.0 release

Can you please rehash the plan? I thought we just did a release last month and were aiming for a three month schedule. I may have missed something On Mon, Mar 17, 2025 at 6:00 AM Ajantha Bhat wrote: > Hey everyone, > > Following the plan from the 1.8.0 release, I'll be handling the 1.9.0 > relea

Re: [DISCUSS] Rename iceberg repo to iceberg-java ?

I generally dislike renames unless we are going to get some tangible benefit out of it. Consistency is never something I really care that much about in repository names since no one is really using those on a daily basis and they aren't really used programmatically. If we did do this rename would

Re: [VOTE] Improve OpenAPI documentation around how NamespaceNotEmptyException is treated

+1 On Fri, Mar 14, 2025 at 10:17 AM Jean-Baptiste Onofré wrote: > +1 (non binding) > > Regards > JB > > On Fri, Mar 14, 2025 at 3:10 PM Eduard Tudenhöfner > wrote: > > > > Hey everyone, > > > > I'd like to hold a quick VOTE on #12518 that improves the documentation > around how NamespaceNotEmpt

Re: [ANNOUNCE] Apache Iceberg Pre-Summit Community Meetup in SF

I'm flying in early enough to join as well! On Mon, Mar 10, 2025 at 4:38 PM Fokko Driesprong wrote: > Nice, thanks for organizing this Sung, and thanks to Bloomberg for > sponsoring. I've just signed up! > > Kind regards, > Fokko > > Op ma 10 mrt 2025 om 16:22 schreef Jean-Baptiste Onofré : > >>

Re: [VOTE] Release Apache Iceberg 1.8.1 RC1

+1 Checked Sigs and Checksum Ran Rat Ran full build/test On Tue, Feb 25, 2025 at 11:30 AM Driesprong, Fokko wrote: > +1 (binding) > >- Checked signatures and checksum >- Checked licenses >- Spotchecked NOTICE/LICENSE > > Kind regards, > Fokko > > Op di 25 feb 2025 om 16:56 schree

Re: [VOTE] Java implementation notes around current-snapshot-id

+1 On Mon, Feb 24, 2025 at 12:55 PM Fokko Driesprong wrote: > Hi everyone, > > Recently, there was confusion > about > valid values for the current-snapshot-id, which led to implementation > notes

Re: [VOTE] Allow Row-Lineage with Equality Deletes

;>>>> Thanks Russell! >>>>>> >>>>>> On Thu, Feb 20, 2025 at 1:57 AM Fokko Driesprong >>>>>> wrote: >>>>>> >>>>>>> +1 >>>>>>> >>>>>>> Thanks Russell! >>>

[VOTE] Allow Row-Lineage with Equality Deletes

The PR: https://github.com/apache/iceberg/pull/12230 is basically ready now. So let's do a last vote to make sure everyone is aware of the upcoming change. Before: Equality deletes are not allowed to be used when row-lineage is enabled After: Equality deletes are allowed to be used when row-line

Re: [ANNOUNCE] Apache Iceberg release 1.8.0

ior in the "Implementation Notes" section. >> >> > How about reverting #11560 for 1.8.1, and then reinstating this for >> 2.0.0? >> >> I think we need to fix this at a format version boundary, not a library >> version boundary. I'd be up for reinsta

Re: [ANNOUNCE] Apache Iceberg release 1.8.0

nges for the fields > * Define the "schema/spec/sort not present" values (the fields are > optional for v1 but required for v2+v3). > * OR Define that "schema/spec/sort must be absent" if there is no current > schema/spec/sort. > > WDYT? > > On 17.02.25 21:07,

Re: [Discuss] Print un-pretty metadata JSON files without whitespace

+0 - I would be surprised if post compression sizes were that different but minifying json is a pretty standard practice for over the wire transfers On Mon, Feb 17, 2025 at 1:51 PM Steve Zhang wrote: > +1. Configure table property `write.metadata.compression-codec` to gzip is > usually suggested

Re: [ANNOUNCE] Apache Iceberg release 1.8.0

It sounds like the argument here is that we should change the Spec for V1, V2, and V3 to mark current-snapshot-id as required. Then we should change all other implementations to follow this new standard. I'm not sure that is a good solution going forwards but I'm not sure of how we can support cata

Re: Remove deprecated table properties

+1 to remove in 1.9 On Mon, Feb 17, 2025 at 4:20 AM Fokko Driesprong wrote: > Hi everyone, > > While reviewing the LocationProvider equivalent of PyIceberg, I noticed > some old code in the Java codebase that I felt could be cleaned up. You > can find the PR over here

Re: [DISCUSS] Consolidate docs under Concepts and Project/Terms

term, specs), "Concepts" is > probably not accurate. Maybe "Spec" is more appropriate. > > On Thu, Feb 13, 2025 at 9:41 AM Russell Spitzer > wrote: > >> I think we should do an even bigger change. IMHO, Project should have >> information about inter

Re: [DISCUSS] Consolidate docs under Concepts and Project/Terms

I think we should do an even bigger change. IMHO, Project should have information about interacting with the project so Community Contributing Implementation Status Multi-engine Support How to Release ASF Then have Concepts include all the technical details * Spec Terms On Thu, Feb 13, 2025 at 1

Re: [VOTE] Add overwriteRequested to RegisterTableRequest in REST spec

+1 On Wed, Feb 12, 2025 at 5:30 PM Steve Zhang wrote: > Hi Iceberg Community, > > I'm working on supporting the registration of iceberg metadata for an > existing table in the catalog. As part of this work, I'm proposing to add > an optional boolean field in RegisterTableRequest. > > I'd lik

Re: [Discussion] Spec change for Row Lineage - Allow Equality Deletes

t said, I'm not convinced that this worth the complexity and effort. > Especially since between maintenance job runs the lineage info is still > invalid. > > > On Wed, Feb 12, 2025, 19:06 Russell Spitzer > wrote: > >> I'm not sure I follow how one could figur

Re: [Discussion] Spec change for Row Lineage - Allow Equality Deletes

gt;>> >>>>> On Tue, Feb 11, 2025 at 7:39 PM Gang Wu wrote: >>>>> >>>>>> Hi Russell, >>>>>> >>>>>> Thanks for supporting equality deletes to row lineage! >>>>>> >>>>>> > accept that "

[Discussion] Spec change for Row Lineage - Allow Equality Deletes

Hi Y'all, As we have been working on the row lineage implementation I've been reached out to by a few folks in the community who are interested in changing our defined behavior around equality deletes. Currently when Row Lineage is enabled, the spec says to disable equality deletes for the table

Re: [VOTE] Add RemoveSchemas update type to REST spec

+1 On Tue, Feb 11, 2025 at 9:15 AM Fokko Driesprong wrote: > +1 > > Op di 11 feb 2025 om 13:52 schreef Jean-Baptiste Onofré : > >> +1 (non binding) >> >> Regards >> JB >> >> On Tue, Feb 11, 2025 at 3:38 AM Gabor Kaszab >> wrote: >> > >> > Hi Iceberg Community, >> > >> > I'm working on removing

Re: Table metadata swap not work for REST Catalog (#12134)

I still would like a "register table" force" option On Mon, Feb 10, 2025 at 5:06 PM Steve Zhang wrote: > Thank you Dan for your detailed reply. Based on your explanation, do you > think it would be worthwhile to support non-linear or complete metadata > replacements in the REST implementation? I

Re: [VOTE] Simplify multi-arg table metadata

+1 On Mon, Feb 10, 2025 at 2:50 AM Eduard Tudenhöfner wrote: > +1 > > On Mon, Feb 10, 2025 at 7:40 AM Péter Váry > wrote: > >> +1 >> >> On Mon, Feb 10, 2025, 03:44 Manu Zhang wrote: >> >>> +1 (non-binding) >>> >>> On Mon, Feb 10, 2025 at 10:25 AM roryqi wrote: >>> +1 xianjin 于

Re: [VOTE] Add Geometry and Geography types for V3

+1 On Fri, Feb 7, 2025 at 12:57 AM Denny Lee wrote: > +1 (non-binding) - super exciting! > > On Thu, Feb 6, 2025 at 3:52 PM rdb...@gmail.com wrote: > >> +1 >> >> Awesome to see this ready to go! >> >> On Thu, Feb 6, 2025 at 12:01 PM Szehon Ho >> wrote: >> >>> Hi everyone >>> >>> We would like

Re: Welcome Huaxin Gao as a committer!

Congratulations! On Thu, Feb 6, 2025 at 11:35 AM Péter Váry wrote: > Congratulations! > > Matt Topol ezt írta (időpont: 2025. febr. 6., > Cs, 10:40): > >> Congrats! Welcome! >> >> On Thu, Feb 6, 2025, 10:19 AM Raúl Cumplido wrote: >> >>> Congrats Huaxin! >>> >>> El jue, 6 feb 2025 a las 10:16,

Re: FileRewrite API refactor

te anyway, and focusing on data file rewriting would allow us to > remove some generics from the API. > > WDYT? > > Russell Spitzer ezt írta (időpont: 2025. jan. > 21., K, 17:11): > >> To bump this back up, I think this is a pretty important change to the >> core l

Re: [VOTE] Update partition stats spec for V3

+1 On Sat, Feb 1, 2025 at 3:01 AM Anton Okolnychyi wrote: > Hi all, > > I propose the following updates to our partition stats spec in V3: > > - Modify `position_delete_record_count` to include a sum of position > deletes across position delete files and DVs > - Keep `position_delete_file_count`

Re: guideline for interface change

In the API module we tend to be very careful and follow a deprecation schedule. So we would deprecate the old method and make a new one with the different return type. This would then be removed in the next big release. On Sat, Feb 1, 2025 at 8:03 AM Péter Váry wrote: > Can we deprecate the old

Re: [DISCUSS] Clarify delete counts handling in partition stats

Sounds reasonable, I think the intent was that N/A is different then 0 but that only makes sense for V1. For V2/V3 0 makes sense On Sat, Feb 1, 2025 at 3:15 AM Anton Okolnychyi wrote: > Hi all, > > I propose to clarify our delete counts handling in partition stats. We > have the following metric

Re: [VOTE] Add Variant type to Iceberg Spec

). >>> >>> >>>> 3. There is very little in our change set here that specifically >>>> references the Parquet spec except for our reference link to it. >>> >>> >>> This cuts both ways? What is the rush to get this into V3 if it can >>

Re: [VOTE] Release Apache Iceberg 1.7.2 rc0

+1 (binding) Checked licensing and sha and gpg sig On Mon, Jan 27, 2025 at 3:02 PM Fokko Driesprong wrote: > +1 (binding) > > Ran signature/checksum/license: > > *➜ **Desktop* gpg --verify apache-iceberg-1.7.2.tar.gz.asc > > gpg: assuming signed data in 'apache-iceberg-1.7.2.tar.gz' > > gpg: S

Re: [DISCUSS/VOTE] Add in ChangeLog Reserved Field IDs to Spec and Decrement Row Lineage Reserved IDs

Thanks everyone, I'll be merging that fix ASAP On Mon, Jan 27, 2025 at 6:01 AM Fokko Driesprong wrote: > +1 > > Op ma 27 jan 2025 om 10:54 schreef Honah J. : > >> +1, thanks for driving this! >> >> Best Regards, >> Honah >> >> On Sun, Jan 26, 2025 at 3:20 PM Steven Wu wrote: >> >>> +1 >>> >>> O

[DISCUSS/VOTE] Add in ChangeLog Reserved Field IDs to Spec and Decrement Row Lineage Reserved IDs

We added reserved fields into the Apache Iceberg repo to use with ChangeLog views but these were never added to the spec. When Row Lineage was added, those IDs inadvertently collided with the ID's already set. In this PR I add in the ChangeLog

Re: [VOTE] REST API changes for freshness-aware table loading

+1 'd on the PR earlier, but for the record +1 here as well :) On Fri, Jan 24, 2025 at 1:57 PM rdb...@gmail.com wrote: > +1 > > Thanks, Gabor! > > On Fri, Jan 24, 2025 at 9:25 AM Christian Thiel < > christian.t.b...@gmail.com> wrote: > >> +1 (non binding). Thanks Gabor! >> >> Daniel Weeks schri

Re: [DISCUSS, VOTE] OpenAPI Metadata Update for EnableRowLineage

025 at 10:55 AM Fokko Driesprong >>> wrote: >>> >>>> +1 >>>> >>>> Thanks Russell >>>> >>>> Op do 23 jan 2025 om 18:47 schreef Aihua Xu : >>>> >>>>> + (non binding). >>>>> &

Re: Very strange (AI generated) issues

>> * I guess whoever has the tool realised their mistake and either stopped >>> it or removed some confusion >>> * I have my own suspicions (which I am exploring) - but I asked the user >>> to provide information about what tooling they were using (and the user was

Re: [DISCUSS, VOTE] OpenAPI Metadata Update for EnableRowLineage

disable/enable (more forward > thinking as this is the first case quite like this). > > -Dan > > > > On Wed, Jan 22, 2025 at 3:55 PM Amogh Jahagirdar <2am...@gmail.com> wrote: > >> +1 Thanks Russell >> >> On Wed, Jan 22, 2025 at 4:50 PM rdb...@gmail.com

[DISCUSS, VOTE] OpenAPI Metadata Update for EnableRowLineage

Hey Y'all Yet another Row Lineage Spec update. This adds a MetadataUpdate EnableRowLineage to the REST Spec. We briefly talked today about an alternative EnableFeature(Feature Name) API instead but in the absence of other features it doesn't seem like that's really a requirement now. I agreed tha

Iceberg Community Meeting Notes - Jan 15 2025

Hey Y'all! Here are the notes and recording for the Jan 15th meeting! Video - https://www.youtube.com/watch?v=9ZLWQSZvLIw Notes - https://docs.google.com/document/d/1YuGhUdukLP5gGiqCbk0A5_Wifqe2CZWgOd3TbhY3UQg/edit?tab=t.0 Notes : --- - Highlights - Java - Hive

Re: Very strange (AI generated) issues

This is pretty disturbing and I hope that any users out there see that using automated tools to submit issues is just adding noise to the project which makes it very hard for real issues to be addressed. On Wed, Jan 22, 2025 at 6:58 AM Jarek Potiuk wrote: > - Iceberg dev to not flood them :) (i

Re: [VOTE] Document Snapshot Summary Optional Fields as Subsection of Appendix F in Spec

+1 On Tue, Jan 21, 2025 at 2:36 PM rdb...@gmail.com wrote: > +1 > > On Tue, Jan 21, 2025 at 12:20 PM Honah J. wrote: > >> Hi everyone, >> >> In the last VOTE >> thread >> on documenting snapshot summary optional fields, we decid

  1   2   3   4   >