Re: Spark Views in Iceberg Catalog

> I'm wondering if you'd have any recommendations of how to approach this barring maintaining our own spark fork? You can probably get everything working in Spark SQL extensions. That allows you to add rules to certain batches in the analyzer, which is all you need. The implementation that John ha

Re: Spark Views in Iceberg Catalog

Hey Walaa, I read your latest response and it makes sense, thanks for sharing your approach! - Marc On Tue, Nov 15, 2022 at 6:24 PM Walaa Eldin Moustafa wrote: > I have added more details just before you sent the last message :) Please > let me know if it answers your question. > > On Tue, Nov

Re: Spark Views in Iceberg Catalog

I have added more details just before you sent the last message :) Please let me know if it answers your question. On Tue, Nov 15, 2022 at 3:21 PM Marc Laforet wrote: > Hey guys, > > Thanks for the responses. > > Ryan - Thanks for confirming the behaviour. I'm wondering if you'd have > any recom

Re: Spark Views in Iceberg Catalog

Hey guys, Thanks for the responses. Ryan - Thanks for confirming the behaviour. I'm wondering if you'd have any recommendations of how to approach this barring maintaining our own spark fork? Walaa - I tried creating the view using spark sql's standard `create view as select` statement (trying w

Re: Spark Views in Iceberg Catalog

In our case, we store the view definitions in HMS, have them access both Hive and Iceberg tables, and if they are expressed in Hive/Spark SQL, then they should be accessible from both Spark and Trino with the same name (Trino uses Coral to query HiveQL views in T

Re: Spark Views in Iceberg Catalog

Hi Marc, Could you clarify where you store the view definitions in this case, and how the syntax looks like? Thanks, Walaa. On Tue, Nov 15, 2022 at 2:34 PM Ryan Blue wrote: > Hi Marc, > > This is expected. Although the ViewCatalog SPIP was approved by the Spark > community, the implementation

Re: Spark Views in Iceberg Catalog

Hi Marc, This is expected. Although the ViewCatalog SPIP was approved by the Spark community, the implementation hasn't made it in yet for v2. Ryan On Tue, Nov 15, 2022 at 11:38 AM Marc Laforet wrote: > Hi Iceberg folks, > > I'm working on a project where we're migrating tables from hive to >

Spark Views in Iceberg Catalog

Hi Iceberg folks, I'm working on a project where we're migrating tables from hive to iceberg. We are revamping our ingestion pipeline in parallel from batch to stream. Originally, our plan was to have two separate tables, a backfill table and a live table, that would be stitched together via a vie