Re: FlinkFileIO implementation

I think that the FileIO API is very limited for a good reason. The goal is to allow a wide variety of implementations, and this is what we are seeing here. All of the implementations has their pros and cons, and the actual use-case defines the best one the user should use. For example, my first te

Re: [Proposal] Add support for Materialized Views in Iceberg

+1 for separate table and view objects. On Thu, Apr 25, 2024 at 12:33 PM Russell Spitzer wrote: > +1 to separate. > > > On Apr 25, 2024, at 2:08 PM, Jean-Baptiste Onofré > wrote: > > > > +1 to separate, it makes sense to me. > > > > Regards > > JB > > > > On Thu, Apr 18, 2024 at 11:50 AM Walaa

Re: [Proposal] Add support for Materialized Views in Iceberg

+1 to separate. > On Apr 25, 2024, at 2:08 PM, Jean-Baptiste Onofré wrote: > > +1 to separate, it makes sense to me. > > Regards > JB > > On Thu, Apr 18, 2024 at 11:50 AM Walaa Eldin Moustafa > wrote: >> >> Hi everyone, >> >> I would like to make a proposal for issue [1] to support material

Re: [Proposal] Add support for Materialized Views in Iceberg

+1 to separate, it makes sense to me. Regards JB On Thu, Apr 18, 2024 at 11:50 AM Walaa Eldin Moustafa wrote: > > Hi everyone, > > I would like to make a proposal for issue [1] to support materialized views > in Iceberg. The support leverages two separate objects, an Iceberg view and > an Iceb

Re: FlinkFileIO implementation

Good point about the schemas. That's true, it would be more complicated. Agree to have more discussion about that. Personally, I think it's not a bad idea to have the catalog as the "source" for FileIO, and let the engine/client deal with that. I think it's an engine/client responsibility (I remem

Re: [Proposal] Add support for Materialized Views in Iceberg

Thanks everyone for participating in the vote/recommendation so far. Let us plan to close the vote by the end of Sunday April 28. Thanks, Walaa. On Tue, Apr 23, 2024 at 12:46 PM Daniel Weeks wrote: > +1 as well for separate objects. I think Netflix has proven this model > works well. Exposu

Re: FlinkFileIO implementation

agree with Dan that ResolvingFileIO solves a different problem to resolve FileIO based on storage schema (like s3) and is probably not a good fit for what Peter is trying to do. I also mentioned some downsides of FlinkFileSystemFileIO in the PR. It doesn't support batch deletes or progressive uplo

Re: FlinkFileIO implementation

JB, The ResolvingFIleIO is somewhat a different issue and more complicated with a concept like FlinkFileIO because the schemes would overlap. The main issue here is around how Flink handles file system operations outside of the Iceberg space (e.g. checkpointing) and the confusion it causes for pe

Re: FlinkFileIO implementation

Hi Peter, On a similar topic, I created a PR to support custom schema in ResolvingFileIO (https://github.com/apache/iceberg/pull/9884). Maybe the FlinkIO can be a new schema/extension in the ResolvingFileIO. If I agree that it would be interesting to have support for FlinkFileIO, I'm not sure it'

[ANNOUNCE] Apache Iceberg release 1.5.1

I'm pleased to announce the release of Apache Iceberg 1.5.1! Apache Iceberg is an open table format for huge analytic datasets. Iceberg delivers high query performance for tables with tens of petabytes of data, along with atomic commits, concurrent writes, and SQL-compatible table evolution. This

Re: [VOTE] Release Apache PyIceberg 0.6.1rc3

+1 (non-binding) Thanks, Fokko for the docutils fix. * Verified signatures, checksums & license. * Ran unit tests and integration tests. Warm regards, Mehul Batra On Mon, Apr 22, 2024 at 12:41 AM Honah J. wrote: > +1 (non-binding) > > - Verified signatures and checksums > - Verified license >