Re: Proposal for RESTful Data Operations

2024-01-26 Thread Drew
Hey everyone, I wanted to provide a quick update on the progress of the commit API proposal. Based on the feedback in the design doc and the Slack conversation with Dan and Jack, we've reached an agreement that this is more of a fine-grained metadata commit, rather than a data operation or commit.

Re: [DISCUSS] Release new Iceberg docs site in the main repository

2024-01-26 Thread Daniel Weeks
+1 I'm all for moving forward with this and fixing issues quickly as they're uncovered with smaller PRs. Thanks Bits! On Fri, Jan 26, 2024, 2:45 PM Ryan Blue wrote: > +1 > > On Fri, Jan 26, 2024 at 2:40 PM Brian Olsen > wrote: > >> Hey everyone, >> >> As discussed during the community sync, I

Subscribe

2024-01-26 Thread Sung Yun (BLOOMBERG/ 120 PARK)

Subscribe

2024-01-26 Thread Adrian Qin (BLOOMBERG/ 120 PARK)

Re: [DISCUSS] Release new Iceberg docs site in the main repository

2024-01-26 Thread Ryan Blue
+1 On Fri, Jan 26, 2024 at 2:40 PM Brian Olsen wrote: > Hey everyone, > > As discussed during the community sync, I'd like to get a vote on moving > forward with the documentation. I have created a PR ( > https://github.com/apache/iceberg/pull/9520) that references the changes > that have happen

[DISCUSS] Release new Iceberg docs site in the main repository

2024-01-26 Thread Brian Olsen
Hey everyone, As discussed during the community sync, I'd like to get a vote on moving forward with the documentation. I have created a PR ( https://github.com/apache/iceberg/pull/9520) that references the changes that have happened up to this point. - Simpler contribution by collocating the w

Re: [DISCUSS] PyIceberg 0.6.0 release

2024-01-26 Thread Daniel Weeks
I'm also strongly in favor of getting this release out even with the limitations as it's still a huge step forward and we can build incrementally on the write support. Incredible work everyone, I'm really excited about the progress here. -Dan On Fri, Jan 26, 2024 at 11:16 AM Fokko Driesprong wr

Re: [DISCUSS] PyIceberg 0.6.0 release

2024-01-26 Thread Fokko Driesprong
Thanks everyone for the responses and great to see everyone is as excited as I am :D I have some good news. The guys from Eventual have been working on integrating PyIceberg into their Daft dataframe

Re: Gravitino an Iceberg REST catalog service

2024-01-26 Thread Ryan Blue
I think I'd be fine exposing this through a test Jar, but it seems to me that if we were to put it into a normal package it would turn into the situation we want to avoid. People would use it for unintended purposes and it would become a distraction. What do you think about using the tests Jar for

Re: [DISCUSS] PyIceberg 0.6.0 release

2024-01-26 Thread Ryan Blue
Like the Java implementation, we've been building toward a library that can be used in distributed applications as well as directly on a single node. For example, job planning can produce a set of file scan tasks or a scan can be pushed to duckdb (to_duckdb) or pandas (to_pandas). The write side is

Re: [DISCUSS] PyIceberg 0.6.0 release

2024-01-26 Thread Steve Zhang
I am really excited to see the both append and overwrite support are finally checked and in favor of seeing it in 0.6.0 release. It’s a big milestone worth celebrating! I am happy to help with partitioned write and sort order. Thanks, Steve Zhang > On Jan 26, 2024, at 5:22 AM, Fokko Driespro

Re: [DISCUSS] PyIceberg 0.6.0 release

2024-01-26 Thread Pucheng Yang
I have similar questions as Yufei's. My organization has interest in Ray Iceberg integration and during the conversation with the Ray team, we know they would also like the have Iceberg integration as well. I think this is a good opportunity for both projects to collaborate. On Fri, Jan 26, 2024 a

Re: [DISCUSS] PyIceberg 0.6.0 release

2024-01-26 Thread Sung Yun
It’s so exciting to see the project take another step forward, Fokko!Really great job to everyone involved.Best,SungOn Jan 26, 2024, at 11:48 AM, Ryan Blue wrote:It's great to see all the progress in PyIceberg. Thanks to everyone that's been contributing!I'm all for getting a release out as soon

Re: [DISCUSS] PyIceberg 0.6.0 release

2024-01-26 Thread Yufei Gu
Thanks for the update on PyIceberg's new features. It's exciting to see the progress! I have a quick question: Currently, as I understand, PyIceberg operates within a single process. Are there any plans to expand its capabilities to support distributed computation, particularly for write operation

Re: [DISCUSS] PyIceberg 0.6.0 release

2024-01-26 Thread Ryan Blue
It's great to see all the progress in PyIceberg. Thanks to everyone that's been contributing! I'm all for getting a release out as soon as possible and following up with more features in the write path in 0.7.0. On Fri, Jan 26, 2024 at 5:22 AM Fokko Driesprong wrote: > Hey everyone, > > I want

[DISCUSS] PyIceberg 0.6.0 release

2024-01-26 Thread Fokko Driesprong
Hey everyone, I want to discuss the 0.6.0 release that will bring a lot of functionality to the public: - Write support for writing to unpartitioned tables - Includes snapshot generation - Constructing Avro writer trees - Support writing metadata which allows to commit support f