Re: Flink table maintenance

2024-03-28 Thread Péter Váry
What would the community think of exploiting tags for preventing concurrent maintenance loop executions. The issue: Some maintenance tasks couldn't run parallel, like DeleteOrphanFiles vs. ExpireSnapshots, or RewriteDataFiles vs. RewriteManifestFiles. We make sure, not to run tasks started by a si

Re: Materialized view integration with REST spec

2024-03-28 Thread Jean-Baptiste Onofré
Yes, correct, thanks Manu for pointing it out. Thanks ! Regards JB On Thu, Mar 28, 2024 at 9:55 AM Manu Zhang wrote: > > I think Jan already created it > https://github.com/apache/iceberg/issues/10043 > > Jean-Baptiste Onofré 于2024年3月28日 周四16:46写道: >> >> Hi Walaa, >> >> Yes, I think it would be

Flink table maintenance

2024-03-28 Thread Péter Váry
Hi Team, As discussed on yesterday's community sync, I am working on adding a possibility to the Flink Iceberg connector to run maintenance tasks on the Iceberg tables. This will fix the small files issues and in the long run help compacting the high number of positional and equality deletes creat

Re: Materialized view integration with REST spec

2024-03-28 Thread Manu Zhang
I think Jan already created it https://github.com/apache/iceberg/issues/10043 Jean-Baptiste Onofré 于2024年3月28日 周四16:46写道: > Hi Walaa, > > Yes, I think it would be great to create the GH Issue with the > proposal template, it would allow us to track the proposal and link > the doc (the comments sh

Re: Materialized view integration with REST spec

2024-03-28 Thread Renjie Liu
Hi, Walaa: +1 for creating a github issue for tracking it with a new template. On Thu, Mar 28, 2024 at 4:47 PM Jean-Baptiste Onofré wrote: > Hi Walaa, > > Yes, I think it would be great to create the GH Issue with the > proposal template, it would allow us to track the proposal and link > the d

Re: Materialized view integration with REST spec

2024-03-28 Thread Jean-Baptiste Onofré
Hi Walaa, Yes, I think it would be great to create the GH Issue with the proposal template, it would allow us to track the proposal and link the doc (the comments should go in the doc directly). Please, let me know if I can help on that. I'm working on a PR to list the proposals on the website an