Hey guys, Thanks for the responses.
Ryan - Thanks for confirming the behaviour. I'm wondering if you'd have any recommendations of how to approach this barring maintaining our own spark fork? Walaa - I tried creating the view using spark sql's standard `create view as select` statement (trying with the fully qualified table name as well as first setting catalog & namespace). Our iceberg tables are backed by a HMS so it would presumably be stored there? Thanks again for your responses! On Tue, Nov 15, 2022 at 5:38 PM Walaa Eldin Moustafa <wa.moust...@gmail.com> wrote: > Hi Marc, > > Could you clarify where you store the view definitions in this case, and > how the syntax looks like? > > Thanks, > Walaa. > > > On Tue, Nov 15, 2022 at 2:34 PM Ryan Blue <b...@tabular.io> wrote: > >> Hi Marc, >> >> This is expected. Although the ViewCatalog SPIP was approved by the Spark >> community, the implementation hasn't made it in yet for v2. >> >> Ryan >> >> On Tue, Nov 15, 2022 at 11:38 AM Marc Laforet <mlafor...@gmail.com> >> wrote: >> >>> Hi Iceberg folks, >>> >>> I'm working on a project where we're migrating tables from hive to >>> iceberg. We are revamping our ingestion pipeline in parallel from batch to >>> stream. Originally, our plan was to have two separate tables, a backfill >>> table and a live table, that would be stitched together via a view for >>> downstream consumers. This is proving rather difficult. In the absence of >>> engine agnostic views we were going to prepend views with the engine type >>> (ie trino_my_table and spark_my_table) but I receive a >>> org.apache.spark.sql.AnalysisException: >>> Catalog iceberg_catalog does not support views error when trying to >>> create the spark view. With the ongoing work towards engine agnostic views >>> I'm unsure if this limitation is expected or easily surpassed with some >>> config/spark change? >>> >>> Thank you for your time, >>> >>> Marc >>> >> >> >> -- >> Ryan Blue >> Tabular >> >
