Re: Proposal: Support for views in Iceberg

Hi All, Please see the spec in markdown format at the PR here to facilitate adding/responding to comments. Please review. thanks, Anjali On Tue, Sep 7, 2021 at 9:31 PM Jack Ye wrote: > Hi everyone, > > I have been thinking about the view support du

Re: Proposal: Support for views in Iceberg

Hi everyone, I have been thinking about the view support during the weekend, and I realize there is a conflict that Trino today already claims to support Iceberg view through Hive metastore. I believe we need to figure out a path forward around this issue before voting to pass the current proposa

Re: Proposal: Support for views in Iceberg

On Thu, Aug 26, 2021 at 6:07 PM Jacques Nadeau wrote: > > On Thu, Aug 26, 2021 at 2:44 PM Ryan Blue wrote: > >> Would a physical plan be portable for the purpose of an engine-agnostic >> view? >> > > My goal is it would be. There may be optional "hints" that a particular > engine could leverage

Re: Proposal: Support for views in Iceberg

On Thu, Aug 26, 2021 at 2:44 PM Ryan Blue wrote: > Would a physical plan be portable for the purpose of an engine-agnostic > view? > My goal is it would be. There may be optional "hints" that a particular engine could leverage and others wouldn't but I think the goal should be that the IR is ent

Re: Proposal: Support for views in Iceberg

Would a physical plan be portable for the purpose of an engine-agnostic view? Physical plan details seem specific to an engine to me, but maybe I'm thinking too much about how Spark is implemented. My inclination would be to accept only logical IR, which could just mean accepting a subset of the s

Re: Proposal: Support for views in Iceberg

The original proposal was more focused on executable plans (as opposed to the logical plans). However, I'm arguing that it should include both logical and physical since the two are fairly heavily overlapping. (See how much logic is in the base classes for most Calcite operators as opposed to the c

Re: Proposal: Support for views in Iceberg

I'll point a link to this thread on dev@arrow so people not subscribed to both can way in? Should we maybe start a new thread about IR? https://docs.google.com/document/d/1C_XVOG7iFkl6cgWWMyzUoIjfKt-X2UxqagPJrla0bAE/edit is the document that Wes wrote up that started the conversation. On Thu, Au

Re: Proposal: Support for views in Iceberg

Micah or someone more familiar (Jacques?), could you summarize some of the details of the Arrow IR proposal? What are the intended use cases and goals? We'd want to see whether the goals align with our own. On Thu, Aug 26, 2021 at 9:24 AM Micah Kornfield wrote: > Small tangent: > > Regarding an

Re: Proposal: Support for views in Iceberg

Small tangent: Regarding an IR, Apache Arrow is looking at this now [1] with several RFC proposals. It would be nice to coordinate, at least to make sure translation from one IR to another isn't too onerous (in an ideal world we could maybe share the same one). Either way external feedback would

Re: Proposal: Support for views in Iceberg

I think that the current proposal is looking good, but it is always a good idea to give people a few days to review it and bring up any issues or further discussion on the topics in this thread. I'll also add this to the next sync agenda so we can farm for dissent next week. Sometimes you can get

Re: Proposal: Support for views in Iceberg

Hello All, I answered all the comments and made changes to the spec to reflect them .. the doc now shows the new revision (link here: https://docs.google.com/document/d/1wQt57EWylluNFdnxVxaCkCSvnWlI8vVtwfnPjQ6Y7aw/edit# ). Please diff with the version named 'Spec v1' in order to see the deltas. Ple

Re: Proposal: Support for views in Iceberg

+1 On Sun, Aug 22, 2021 at 10:02 AM Ryan Blue wrote: > Thanks for working on this, Anjali! It’s great to see the thorough > discussion here with everyone. > > For the discussion about SQL dialect, I think that the right first step is > to capture the SQL or query dialect. That will give us the m

Re: Proposal: Support for views in Iceberg

Thanks for working on this, Anjali! It’s great to see the thorough discussion here with everyone. For the discussion about SQL dialect, I think that the right first step is to capture the SQL or query dialect. That will give us the most flexibility. Engines that can use Coral for translation can a

Re: Proposal: Support for views in Iceberg

Hi Anjali, That's a nice summary. re dialect field. It shouldn't be a bit trouble to have it (or any other way to identify application that created the view), and it might be useful. Why not make it required from the start? re "expanded/resolved SQL" -- i don't understand yet what we would put t

Re: Proposal: Support for views in Iceberg

Hi, Thank you for all the comments. I will try to address them all here together. - @all Cross engine compatibility of view definition: Multiple options such as engine agnostic SQL or IR of some form have been mentioned. We can all agree that all of these options are non-trivial to desi

Re: Proposal: Support for views in Iceberg

Hi, Thanks Jack and Jacques for sharing your thoughts. I agree that tracking dialect/origin is better than nothing. I think having a Map {dialect: sql} is not going to buy us much. I.e. it would be useful if there was some external app (or a human being) that would write those alternative SQLs f

Re: Proposal: Support for views in Iceberg

Some thoughts... - In general, many engines want (or may require) a resolved sql field. This--at minimum--typically includes star expansion since traditional view behavior is stars are expanded at view creation time (since this is the only way to guarantee that the view returns the sam

Re: Proposal: Support for views in Iceberg

Did not notice that we are also discussing cross-engine interoperability here, I will add my response in the design doc here. I would personally prefer cross-engine interoperability as a goal and get the spec in the right structure in the initial release, because: 1. I believe that cross-engine c

Re: Proposal: Support for views in Iceberg

Hi, First of all thank you for this discussion and all the view-related work! I agree that solving cross-engine compatibility problem may not be primary feature today, I am concerned that not thinking about this from the start may "tunnel" us into a wrong direction. Cross-engine compatible views

Re: Proposal: Support for views in Iceberg

Hey Anjali, I am definitely happy to help with implementing 1-3 in your first list once the spec has been approved by the community. My hope is that the final version of the view spec will make it easy to re-use existing rollback/time travel/metadata etc functionalities. Regarding SQL dialects.My

Re: Proposal: Support for views in Iceberg

Thank you Ryan (M), Piotr and Vivekanand for the comments. I have and will continue to address them in the doc. Great to know about Trino views, Piotr! Thanks to everybody who has offered help with implementation. The spec as it is proposed in the doc has been implemented and is in use at Netflix

Re: Proposal: Support for views in Iceberg

Hi, FWIW, in Trino we just added Trino views support. https://github.com/trinodb/trino/pull/8540 Of course, this is by no means usable by other query engines. Anjali, your document does not talk much about compatibility between query engines. How do you plan to address that? For example, I am fa

Re: Proposal: Support for views in Iceberg

Thanks Anjali! I have left some comments on the document. I unfortunately have to miss the community meetup tomorrow but would love to chat more/help w/ implementation. Best, Ryan On Tue, Jul 20, 2021 at 7:42 AM Anjali Norwood wrote: > Hello, > > John Zhuge and I would like to propose the foll

Proposal: Support for views in Iceberg

Hello, John Zhuge and I would like to propose the following spec for storing view metadata in Iceberg. The proposal has been implemented [1] and is in production at Netflix for over 15 months. https://docs.google.com/document/d/1wQt57EWylluNFdnxVxaCkCSvnWlI8vVtwfnPjQ6Y7aw/edit?usp=sharing [1] ht