[VOTE] Release Apache Iceberg 1.1.0 RC1

Hi Everyone, I propose that we release the following RC as the official Apache Iceberg 1.1.0 release. The commit ID is 1d10c53906847893b45c6acb0137dcb55a15353d * This corresponds to the tag: apache-iceberg-1.1.0-rc1 * https://github.com/apache/iceberg/commits/apache-iceberg-1.1.0-rc1 * https://g

Spark Views in Iceberg Catalog

Hi Iceberg folks, I'm working on a project where we're migrating tables from hive to iceberg. We are revamping our ingestion pipeline in parallel from batch to stream. Originally, our plan was to have two separate tables, a backfill table and a live table, that would be stitched together via a vie

Re: Spark Views in Iceberg Catalog

Hi Marc, This is expected. Although the ViewCatalog SPIP was approved by the Spark community, the implementation hasn't made it in yet for v2. Ryan On Tue, Nov 15, 2022 at 11:38 AM Marc Laforet wrote: > Hi Iceberg folks, > > I'm working on a project where we're migrating tables from hive to >

Re: Spark Views in Iceberg Catalog

Hi Marc, Could you clarify where you store the view definitions in this case, and how the syntax looks like? Thanks, Walaa. On Tue, Nov 15, 2022 at 2:34 PM Ryan Blue wrote: > Hi Marc, > > This is expected. Although the ViewCatalog SPIP was approved by the Spark > community, the implementation

Re: Spark Views in Iceberg Catalog

In our case, we store the view definitions in HMS, have them access both Hive and Iceberg tables, and if they are expressed in Hive/Spark SQL, then they should be accessible from both Spark and Trino with the same name (Trino uses Coral to query HiveQL views in T

Re: Spark Views in Iceberg Catalog

Hey guys, Thanks for the responses. Ryan - Thanks for confirming the behaviour. I'm wondering if you'd have any recommendations of how to approach this barring maintaining our own spark fork? Walaa - I tried creating the view using spark sql's standard `create view as select` statement (trying w

Re: Spark Views in Iceberg Catalog

I have added more details just before you sent the last message :) Please let me know if it answers your question. On Tue, Nov 15, 2022 at 3:21 PM Marc Laforet wrote: > Hey guys, > > Thanks for the responses. > > Ryan - Thanks for confirming the behaviour. I'm wondering if you'd have > any recom

Re: Spark Views in Iceberg Catalog

Hey Walaa, I read your latest response and it makes sense, thanks for sharing your approach! - Marc On Tue, Nov 15, 2022 at 6:24 PM Walaa Eldin Moustafa wrote: > I have added more details just before you sent the last message :) Please > let me know if it answers your question. > > On Tue, Nov

Re: Spark Views in Iceberg Catalog

> I'm wondering if you'd have any recommendations of how to approach this barring maintaining our own spark fork? You can probably get everything working in Spark SQL extensions. That allows you to add rules to certain batches in the analyzer, which is all you need. The implementation that John ha

Re: [VOTE] Release Apache Iceberg 1.1.0 RC1

+1 (non binding) Quickly check build and "Apache legal related", not deep dive tests as I'm still learning Iceberg ;) Regards JB On Tue, Nov 15, 2022 at 12:28 PM Gabor Kaszab wrote: > > Hi Everyone, > > I propose that we release the following RC as the official Apache Iceberg > 1.1.0 release.

Re: [VOTE] Release Apache Iceberg 1.1.0 RC1

-1 (non-binding) 1. Downloaded the source tarball. 1. Verified signature and checksum. 2. Untar the source archive. 3. ./gradlew build. There are unit test failures (details in the end). 2. Flink SQL testing (both 1.15 and 1.16) 1. Downloaded Iceberg-flink-runtime j