Re: [DISCUSS] Spec update to cover compressed JSON metadata files

I'll plan on starting a vote for the proposed PR tomorrow, unless there are any objections. I look forward to follow-ups on ways we can improve compression here. Thanks, Micah On Tue, Apr 29, 2025 at 10:38 AM Micah Kornfield wrote: > > I wanted to clarify, as others have pointed out, that the

Re: [VOTE] Add encryption keys to table metadata

+1 (non-binding) On Wed, Apr 30, 2025 at 8:06 PM Aihua Xu wrote: > +1 (non-binding). > > On Wed, Apr 30, 2025 at 3:49 PM Daniel Weeks wrote: > >> +1 (binding) >> >> On Wed, Apr 30, 2025 at 1:38 PM Amogh Jahagirdar <2am...@gmail.com> >> wrote: >> >>> +1 (binding) >>> >>> On Wed, Apr 30, 2025 at

Re: [VOTE] Add encryption keys to table metadata

+1 (non-binding). On Wed, Apr 30, 2025 at 3:49 PM Daniel Weeks wrote: > +1 (binding) > > On Wed, Apr 30, 2025 at 1:38 PM Amogh Jahagirdar <2am...@gmail.com> wrote: > >> +1 (binding) >> >> On Wed, Apr 30, 2025 at 1:29 PM Anurag Mantripragada >> wrote: >> >>> +1 (non-binding) >>> >>> ~ Anurag Man

Re: [DISCUSS] Table Identifiers in Iceberg View Spec

> I think that's the lesser evil compared to Iceberg specifying how engines should resolve identifiers I think this is also similar to the previous point. It is the other way around. Right now the spec dictates how to resolve (through employing a view-specific `default-catalog` field). The proposa

Re: [DISCUSS] Table Identifiers in Iceberg View Spec

> I thought "default-catalog" could be set via the USE command. Benny, I think this is a misconception or miscommunication. The USE command has no impact on the `default-catalog` field. In fact, the proposal's direction is exactly to establish that USE command should influence how tables are resol

Re: [VOTE] Add encryption keys to table metadata

+1 (binding) On Wed, Apr 30, 2025 at 1:38 PM Amogh Jahagirdar <2am...@gmail.com> wrote: > +1 (binding) > > On Wed, Apr 30, 2025 at 1:29 PM Anurag Mantripragada > wrote: > >> +1 (non-binding) >> >> ~ Anurag Mantripragada >> >> >> >> >> >> >> >> On Apr 30, 2025, at 11:44 AM, Ryan Blue wrote: >> >

Re: Discuss proposal - IRC APIs for Multi-Statement Multi-Table Transactions

Hi everyone, I enjoyed the discussion/feedback during Iceberg community sync. I am going through feedback and updating my proposal - adding the "Catalog GlobalSequenceNumber" option, and describing how it would look, detailing the use cases better in the document, and rest of feedback. Please also

Re: [DISCUSS] Table Identifiers in Iceberg View Spec

> there is no SQL construct today to explicitly set default-catalog I thought "default-catalog" could be set via the USE command. I generally agree with Dan about requiring consistent catalog names. I think that's the lesser evil compared to Iceberg specifying how engines should resolve identifi

Re: [VOTE] Add encryption keys to table metadata

+1 (binding) On Wed, Apr 30, 2025 at 1:29 PM Anurag Mantripragada wrote: > +1 (non-binding) > > ~ Anurag Mantripragada > > > > > > > > On Apr 30, 2025, at 11:44 AM, Ryan Blue wrote: > > +1 (binding) > > On Wed, Apr 30, 2025 at 10:34 AM Huang-Hsiang Cheng > wrote: > >> +1 (non-binding) >> >> Hu

Re: [VOTE] Add encryption keys to table metadata

+1 (non-binding) ~ Anurag Mantripragada > On Apr 30, 2025, at 11:44 AM, Ryan Blue wrote: > > +1 (binding) > > On Wed, Apr 30, 2025 at 10:34 AM Huang-Hsiang Cheng > wrote: >> +1 (non-binding) >> >> Huang-Hsiang >> >>> On Apr 30, 2025, at 10:06 AM, Yufei Gu >>

Re: [DISCUSS] Finalizing the v3 spec

Hi Jia I feel it would be nice to get that Parquet spec clarificiation https://github.com/apache/parquet-format/pull/494 into Iceberg V3 spec as well, once we finalize that. Thanks Szehon On Tue, Apr 29, 2025 at 10:55 PM Jia Yu wrote: > Hi Szehon, > > Thanks for clarifying it. > > We’re curren

Re: [VOTE] Add encryption keys to table metadata

+1 (binding) On Wed, Apr 30, 2025 at 10:34 AM Huang-Hsiang Cheng wrote: > +1 (non-binding) > > Huang-Hsiang > > On Apr 30, 2025, at 10:06 AM, Yufei Gu wrote: > > +1 > Yufei > > > On Wed, Apr 30, 2025 at 10:04 AM Russell Spitzer < > russell.spit...@gmail.com> wrote: > >> +1 >> >> On Wed, Apr 30,

Re: [VOTE] Add encryption keys to table metadata

+1 (non-binding) Huang-Hsiang > On Apr 30, 2025, at 10:06 AM, Yufei Gu wrote: > > +1 > Yufei > > > On Wed, Apr 30, 2025 at 10:04 AM Russell Spitzer > wrote: >> +1 >> >> On Wed, Apr 30, 2025 at 11:36 AM Szehon Ho > > wrote: >>

Re: [VOTE] Add encryption keys to table metadata

+1 Yufei On Wed, Apr 30, 2025 at 10:04 AM Russell Spitzer wrote: > +1 > > On Wed, Apr 30, 2025 at 11:36 AM Szehon Ho > wrote: > >> +1 >> >> Thanks >> Szehon >> >> On Wed, Apr 30, 2025 at 4:10 AM Eduard Tudenhöfner < >> etudenhoef...@apache.org> wrote: >> >>> +1 (binding) >>> >>> On Tue, Apr 29

Re: [VOTE] Add encryption keys to table metadata

+1 On Wed, Apr 30, 2025 at 11:36 AM Szehon Ho wrote: > +1 > > Thanks > Szehon > > On Wed, Apr 30, 2025 at 4:10 AM Eduard Tudenhöfner < > etudenhoef...@apache.org> wrote: > >> +1 (binding) >> >> On Tue, Apr 29, 2025 at 9:29 PM Ryan Blue wrote: >> >>> Hi everyone, >>> >>> I’d like to propose merg

Re: [VOTE] Add encryption keys to table metadata

+1 Thanks Szehon On Wed, Apr 30, 2025 at 4:10 AM Eduard Tudenhöfner wrote: > +1 (binding) > > On Tue, Apr 29, 2025 at 9:29 PM Ryan Blue wrote: > >> Hi everyone, >> >> I’d like to propose merging PR 12162 >> into the table spec >> for v3. The

Re: [ANNOUNCE] Apache PyIceberg release 0.9.1

Thanks for running the release, Fokko! It's great to see the number of fixes in this patch found and contributed by the community. PyIceberg is really taking off! :) Best, Kevin Liu On Wed, Apr 30, 2025 at 8:30 AM Fokko Driesprong wrote: > I'm pleased to announce the release of Apache PyIceberg

Re: [DISCUSS] Finalizing the v3 spec

Thanks JB and Fokko! I agree that we are good with multi-arg transform for v3. Best, Gang On Wed, Apr 30, 2025 at 2:12 PM Xuanwo wrote: > Hi Ryan. > > Thank for starting this. > > I share the same concern as Russell regarding the recent discussion about > `metadata.json.gz`. I think it's a good

[ANNOUNCE] Apache PyIceberg release 0.9.1

I'm pleased to announce the release of Apache PyIceberg 0.9.1! Apache Iceberg is an open table format for huge analytic datasets. Iceberg delivers high query performance for tables with tens of petabytes of data, along with atomic commits, concurrent writes, and SQL-compatible table evolution. Th

Re: [VOTE] Release Apache PyIceberg 0.9.1rc1

Thanks everyone for voting. We have 3 binding votes (Honah, Eduard, Fokko) and 4 non-binding votes (JB, Manu, Sung, Kevin). I'll release the artifacts shortly. Eduard, regarding to the errors. What version of Python are you using? Ray is still working on 3.13 support, but it passes the rest of th

Re: [VOTE] Release Apache PyIceberg 0.9.1rc1

+1 (binding) checked license and unit/integration tests. I've had a few failing tests related to ray. tests/integration/test_reads.py::test_ray_nan_rewritten[session_catalog_hive] FAILED tests/integration/test_reads.py::test_ray_nan_rewritten[session_catalog] FAILED tests/integration/test_read

Re: [VOTE] Add encryption keys to table metadata

+1 (binding) On Tue, Apr 29, 2025 at 9:29 PM Ryan Blue wrote: > Hi everyone, > > I’d like to propose merging PR 12162 > into the table spec > for v3. The changes are a minimal set of additions needed to support table > encryption schemes, incl