Re: Meeting Minutes from 2023-11-22 Iceberg Sync

Apologies, this should fix the formatting issues. Iceberg Community Sync (Recorded) - November 22 VIEW RECORDING: https://www.youtube.com/watch?v=iz0Oex1hQA0 Meeting Purpose: Weekly Iceberg dev sync meeting to discuss recent updates, issues, and next steps Key Takeaways - No urgent need for 1

Re: Meeting Minutes from 2023-11-01 Iceberg Sync

Apologies, this should fix the formatting issues. Iceberg Community Sync (Recorded) - November 01 VIEW RECORDING: https://www.youtube.com/watch?v=1yljcXTAOuA Meeting Purpose: Weekly Iceberg community sync up meeting to discuss recent developments, upcoming releases, and proposals. Key Takeaways

RFC: Control flink upsert sink’s memory usage of insertedRowMap

Hi: I want to raise a discussion about controlling flink's upsert sink's memory usage: https://toys-flash-4hl.craft.me/3VHrdWbV30QMk6 Welcome to comment and share your thoughts.

Re: Iceberg Logo Fix and Iceberg Swag Shop

+1 On Wed, Dec 6, 2023, 9:35 AM Ryan Blue wrote: > +1 > > On Wed, Dec 6, 2023 at 9:05 AM Russell Spitzer > wrote: > >> Ah got it! For some reason I kept looking for a circle, but in the link >> you sent I can see the obvious polygon that is missing. >> >> I'm +1 on switching the image to the on

Re: Why manifest rewrite only touches files that have latest spec id?

Got it, thank you Ryan. The PR is just merged btw. On Thu, Dec 7, 2023 at 3:22 PM Ryan Blue wrote: > It's not my concern about this, it's why I think the rewrite procedure > does what it does currently. What I mean is that I don't think the initial > implementation didn't rewrite files across di

Re: Why manifest rewrite only touches files that have latest spec id?

It's not my concern about this, it's why I think the rewrite procedure does what it does currently. What I mean is that I don't think the initial implementation didn't rewrite files across different partition specs because the manifest files themselves would have a different schema for the partitio

Meeting Minutes from 2023-11-01 Iceberg Sync

Key Takeaways 0:00 Introduction 11:07 Logo update proposed to fix artifact issue 10:38 Release process improvements proposed to better track c

Meeting Minutes from 2023-11-22 Iceberg Sync

Key Takeaways 0:00 Introduction 4:33 No urgent need for 1.4.3 release currently, will wait for meaningful bug fixes 25:43 Branches should ref

Re: Community Meeting Minutes ?

Hey Wing Yew, Sorry about this. I am just about to publish the last two. Me and the other person that is responsible for these were hit by a series of family and medical issues so apologies. I will put some better backups into place in the unlikely event we are both out of commission. Thanks for

Re: Why manifest rewrite only touches files that have latest spec id?

Ryan, got it. Can you speak more for your first concern? If we are to rewrite manifests of the same spec at a time, how will that lead to inconsistent schema? As of now, we have a PR ready to be merged https://github.com/apache/iceberg/pull/9242 please comment if you suggest this to be put on hold

Re: Why manifest rewrite only touches files that have latest spec id?

It sounds like a feature that we could add. I think there are two concerns. First, we don't want to process the files for multiple specs at the same time because we probably want a consistent schema. Second, there's probably some confusion over how to select a spec since we don't like users to need

Re: [DISCUSS] Run GC with Catalog or Tables

Regarding the 2 points Yufei brought up, in the inventory list way, I think it would offer the following experience: *1. Do we retry from the beginning or from the middle if the procedure failed in the middle while expiring snapshots for one catalog? If we started from the beginning, some tables m

Re: [DISCUSS] Run GC with Catalog or Tables

Running GC across the entire catalog has always been something I want to explore, because of one particular benefit: People typically use one S3 bucket for all tables in a catalog, and you can run a JOIN of the union of all files metadata table against the S3 inventory list

Re: [DISCUSS] Run GC with Catalog or Tables

Error report and Retry are tricky. 1. Do we retry from the beginning or from the middle if the procedure failed in the middle while expiring snapshots for one catalog? If we started from the beginning, some tables may never get GCed. 2. Users may be more interested in some tables getting GCed inste

Re: [DISCUSS] JUnit5 and parameterized testing

Looking at the referenced open issue on Junit5 side, it seems like at least the community is actively working on a @ParameterizedContainer solution. In that case I would +1 for option 2, since we have an easy path for moving to the official class-level annotation when that is ready. -Jack On Thu,

Re: Community Meeting Minutes ?

Hi, Good point. I'm sure Brian will share soon. I think we should do that "systematically" on the website after the meeting. Regards JB On Wed, Dec 6, 2023 at 10:05 PM Wing Yew Poon wrote: > > The meeting minutes and a link to the recording used to be sent out to this > list regularly soon aft

Re: [DISCUSS] JUnit5 and parameterized testing

Hi Ed, I like option 2, it simplifies the parameterized support. Just my $0.01 Regards JB On Thu, Dec 7, 2023 at 5:16 PM Eduard Tudenhoefner wrote: > Hey everyone, > > a while ago we (thanks to many contributors that helped here) started > migrating tests from JUnit4 to JUnit5. While most thi

[DISCUSS] JUnit5 and parameterized testing

Hey everyone, a while ago we (thanks to many contributors that helped here) started migrating tests from JUnit4 to JUnit5. While most things are straightforward to migrate, parameterized tests are not. JUnit5 doesn't support parameterization at the class level yet