Re: [DISCUSS] Change iceberg-rust CI Settings to only require approval for new github users

+1! And I think we should also do that for iceberg-go -Jack On Wed, Jan 31, 2024, 5:42 PM Renjie Liu wrote: > +1 for this. > > It greatly improves the contributor's experience, especially for new and > non experienced contributors. > > On Thu, Feb 1, 2024 at 1:51 AM Fokko Driesprong wrote: > >

Re: [DISCUSS] Change iceberg-rust CI Settings to only require approval for new github users

+1 for this. It greatly improves the contributor's experience, especially for new and non experienced contributors. On Thu, Feb 1, 2024 at 1:51 AM Fokko Driesprong wrote: > Thanks for raising this Xuanwo, > > I dislike having to give Approval for the CI both from a contributor and a > committer

Re: [DISCUSS] iceberg-rust 0.2.0 release

Thanks everyone for the discussion. Since we have reached consensus on this, I'm happy to run the first release. On Thu, Feb 1, 2024 at 1:41 AM Daniel Weeks wrote: > +1 for 0.2.0 release as well. Really excited about the progress here. > > -Dan > > On Wed, Jan 31, 2024 at 9:36 AM Fokko Driespr

Re: Partition column order in rewrite manifests

Yeah I was also thinking about potentially exposing something more flexible. However, I don't think we can directly expose the logic for users to manipulate the data frame directly, because we want the RewriteManifests core API to be not engine-specific. In addition, I think we still need to expos

Re: [VOTE] Release Apache PyIceberg 0.6.0rc1

+1 (non binding) - verified signature and checksum - RAT license check - run install and basic tests in python 3.11 docker image - run s3 and integration tests in python 3.10.3 macOS - also did a few manual test to test write with pyarrow on s3 with REST catalog Looking forward to the release o

Re: Proposal for REST APIs for Iceberg table scans

Sure, I can look into adding this to the spec. Thanks to everyone for sharing their thoughts, appreciate it! From: Ryan Blue Reply-To: "dev@iceberg.apache.org" Date: Wednesday, January 31, 2024 at 10:22 AM To: "dev@iceberg.apache.org" Subject: RE: [EXTERNAL] Proposal for REST APIs for Iceberg t

RE: Partition column order in rewrite manifests

I love this idea. Instead of (or in addition to) inferring the desired sort order, I would propose that the ability for the user to define their own sorting/partitioning be exposed. That way the user could balance the metadata tree more specifically to their use case. Rough thinking - https://g

Re: Proposal for REST APIs for Iceberg table scans

Looks good to me! Should we get a PR up to add it to the OpenAPI spec? On Wed, Jan 31, 2024 at 10:16 AM Jack Ye wrote: > Sounds good. I don't really have any strong opinions here. So looks like > we are landing on this? > > > *PreplanTable: POST /v1/namespaces/ns/tables/t/preplan*{ "filter": { >

Re: Proposal for REST APIs for Iceberg table scans

Sounds good. I don't really have any strong opinions here. So looks like we are landing on this? *PreplanTable: POST /v1/namespaces/ns/tables/t/preplan*{ "filter": { "type": "in", "term": "x", "values": [1, 2, 3] }, "select": ["x", "a.b"] } { "plan-tasks": [ { ... }, { ... } ] } // opaque objec

Re: Proposal for REST APIs for Iceberg table scans

I agree with Dan. I'd rather have two endpoints instead of needing an option that changes the behavior entirely in the same route. I don't think that a `preplan` route would be too bad. On Wed, Jan 31, 2024 at 9:51 AM Daniel Weeks wrote: > I agree with the opaque tokens. > > However, I'm concern

Re: [DISCUSS] Change iceberg-rust CI Settings to only require approval for new github users

Thanks for raising this Xuanwo, I dislike having to give Approval for the CI both from a contributor and a committer standpoint. Often I see people raising PRs that cause the CI to fail but not knowing yet because it hasn't run yet. Having direct feedback from the CI makes the PR cycle much faster

Re: Proposal for REST APIs for Iceberg table scans

I agree with the opaque tokens. However, I'm concerned we're overloading the endpoint two perform two distinctly different operations: distribute a plan and scan a plan. Changing the task-type then changes the behavior and the result. I feel it would be more straightforward to separate the distr

Re: [DISCUSS] Change iceberg-rust CI Settings to only require approval for new github users

I agree with this change. The defaults are not very community friendly and make contributing hard. -Dan On Wed, Jan 31, 2024 at 6:01 AM Xuanwo wrote: > Hello, everyone > > I'm starting this thread to discuss the possibility of changing the CI > settings for > iceberg-rust to only require appro

Re: [DISCUSS] iceberg-rust 0.2.0 release

+1 for 0.2.0 release as well. Really excited about the progress here. -Dan On Wed, Jan 31, 2024 at 9:36 AM Fokko Driesprong wrote: > I'm all for the 0.2.0 release. Kudos to all the work so far. While the > functionality is limited today, a lot of things are already in progress and > it looks v

Re: [DISCUSS] iceberg-rust 0.2.0 release

I'm all for the 0.2.0 release. Kudos to all the work so far. While the functionality is limited today, a lot of things are already in progress and it looks very promising. Also, running a release now will help to streamline the release process. Kind regards, Fokko Driesprong Op wo 31 jan 2024 om

RE: Partition column order in rewrite manifests

I love this idea. Instead of (or in addition to) inferring the desired sort order, I would propose that the ability for the user to define their own sorting/partitioning be exposed. That way the user could balance the metadata tree more specifically to their use case. Rough thinking - https://gith

Re: [VOTE] Release Apache PyIceberg 0.6.0rc1

nvm, I was under the wrong impression it was released already. Thanks. On Wed, Jan 31, 2024 at 8:31 AM Pucheng Yang wrote: > 0.6.0 has been released already. Do you mean to release 0.6.1? > > On Wed, Jan 31, 2024 at 8:26 AM Sung Yun (BLOOMBERG/ 120 PARK) < > syu...@bloomberg.net> wrote: > >> Hi

Re: [VOTE] Release Apache PyIceberg 0.6.0rc1

0.6.0 has been released already. Do you mean to release 0.6.1? On Wed, Jan 31, 2024 at 8:26 AM Sung Yun (BLOOMBERG/ 120 PARK) < syu...@bloomberg.net> wrote: > Hi Everyone, > > I propose that we release the following RC as the official PyIceberg 0.6.0 > release. > > A summary of the high level fea

Re: [DISCUSS] iceberg-rust 0.2.0 release

Excited about the progress in Rust! +1 for releasing 0.2.0 -Jack On Wed, Jan 31, 2024 at 8:26 AM Ryan Blue wrote: > Thanks, Renjie! It's great to see all of the progress in Rust. I agree > with getting the code released and I'm looking forward to testing it out! > > On Wed, Jan 31, 2024 at 12:3

Re: [DISCUSS] iceberg-rust 0.2.0 release

Great to see the progress in Rust. Thanks Renjie! Yufei On Wed, Jan 31, 2024 at 8:24 AM Ryan Blue wrote: > Thanks, Renjie! It's great to see all of the progress in Rust. I agree > with getting the code released and I'm looking forward to testing it out! > > On Wed, Jan 31, 2024 at 12:35 AM Xuan

[VOTE] Release Apache PyIceberg 0.6.0rc1

Hi Everyone, I propose that we release the following RC as the official PyIceberg 0.6.0 release. A summary of the high level features: * Write support for writing to unpartitioned tables * Includes snapshot generation * Constructing Avro writer trees * Support writing metadata which allows to c

Re: [DISCUSS] iceberg-rust 0.2.0 release

Thanks, Renjie! It's great to see all of the progress in Rust. I agree with getting the code released and I'm looking forward to testing it out! On Wed, Jan 31, 2024 at 12:35 AM Xuanwo wrote: > We have been working on the iceberg-rust project for a while. Although > there is still much work to b

[DISCUSS] Change iceberg-rust CI Settings to only require approval for new github users

Hello, everyone I'm starting this thread to discuss the possibility of changing the CI settings for iceberg-rust to only require approval for new GitHub users. This will improve the experience for our contributors. We do not have self-hosted runners, so there is no attack surface from the act

Re: [DISCUSS] iceberg-rust 0.2.0 release

We have been working on the iceberg-rust project for a while. Although there is still much work to be done, I believe it is important to release it in order to attract more users and developers to join us. At current stage, we have implemented basic features that users need to read a table. It

Re: [DISCUSS] iceberg-rust 0.2.0 release

Thanks for raising this discussion! Current features look great for me to release the new version for iceberg-rust. Renjie Liu 于2024年1月31日周三 14:46写道： > Hi, everyone: > > iceberg-rust has been under > active development for several months, and it now has