Re: Proposal: Support for views in Iceberg

On Thu, Aug 26, 2021 at 6:07 PM Jacques Nadeau wrote: > > On Thu, Aug 26, 2021 at 2:44 PM Ryan Blue wrote: > >> Would a physical plan be portable for the purpose of an engine-agnostic >> view? >> > > My goal is it would be. There may be optional "hints" that a particular > engine could leverage

Re: Proposal: Support for views in Iceberg

On Thu, Aug 26, 2021 at 2:44 PM Ryan Blue wrote: > Would a physical plan be portable for the purpose of an engine-agnostic > view? > My goal is it would be. There may be optional "hints" that a particular engine could leverage and others wouldn't but I think the goal should be that the IR is ent

Re: Proposal: Support for views in Iceberg

Would a physical plan be portable for the purpose of an engine-agnostic view? Physical plan details seem specific to an engine to me, but maybe I'm thinking too much about how Spark is implemented. My inclination would be to accept only logical IR, which could just mean accepting a subset of the s

Re: Proposal: Support for views in Iceberg

The original proposal was more focused on executable plans (as opposed to the logical plans). However, I'm arguing that it should include both logical and physical since the two are fairly heavily overlapping. (See how much logic is in the base classes for most Calcite operators as opposed to the c

Re: Proposal: Support for views in Iceberg

I'll point a link to this thread on dev@arrow so people not subscribed to both can way in? Should we maybe start a new thread about IR? https://docs.google.com/document/d/1C_XVOG7iFkl6cgWWMyzUoIjfKt-X2UxqagPJrla0bAE/edit is the document that Wes wrote up that started the conversation. On Thu, Au

Re: Proposal: Support for views in Iceberg

Micah or someone more familiar (Jacques?), could you summarize some of the details of the Arrow IR proposal? What are the intended use cases and goals? We'd want to see whether the goals align with our own. On Thu, Aug 26, 2021 at 9:24 AM Micah Kornfield wrote: > Small tangent: > > Regarding an

Matching iceberg data types to Parquet data types

Hello Iceberg devs! I was looking through the Parquet Logical types and Iceberg data types and I have a couple of questions. 1. Parquet allows storing nano second precision data. If I convert an existing Parquet file with nanos

Re: Proposal: Support for views in Iceberg

Small tangent: Regarding an IR, Apache Arrow is looking at this now [1] with several RFC proposals. It would be nice to coordinate, at least to make sure translation from one IR to another isn't too onerous (in an ideal world we could maybe share the same one). Either way external feedback would

Re: Proposal: Support for views in Iceberg

I think that the current proposal is looking good, but it is always a good idea to give people a few days to review it and bring up any issues or further discussion on the topics in this thread. I'll also add this to the next sync agenda so we can farm for dissent next week. Sometimes you can get