> 1. If `default-catalog`, `default-namespace` are not specified I would guess it is assumed that any non-fully qualified tables should be resolved against the same catalog and namespace that the view is registered in?
I think that names should be fully qualified if these are not set. I don't think that we want to have the engine guess based on the location of the view, but I'd be open to discussion to clarify the spec on this. > 2. It seems like there are two optional methods for specifying documentation for columns in the view, one is field-docs, and one is the doc field on schema. Is the assumption that only one or the other would be set? The field-docs take precedence. The format we're using follows the SQL syntax, where you can create a view with top-level column names and field docs. We store everything that comes through to avoid losing information. > 3. Are field-aliases intended to be further documentation or consumed programmatically (i.e. can they be referenced in queries that use the view)? These are also from the SQL syntax and should be applied like field comments to rename fields. > 4. Are there any thoughts on standardization of dialects, at least for common OSS query engines? Should there be an option to encode dialect version encoded at least informationally in case a view uses a SQL feature that was only introduced in later versions? We have a "dialect" field on the SQL view representation, and we should document what goes into that field. I think it makes sense to have a version of the dialect as well. On Tue, Mar 14, 2023 at 11:48 PM Micah Kornfield <emkornfi...@gmail.com> wrote: > I apologize if some of these items are already discussed or too obvious > but a few questions after reading the spec in a little more detail: > 1. If `default-catalog`, `default-namespace` are not specified I would > guess it is assumed that any non-fully qualified tables should be resolved > against the same catalog and namespace that the view is registered in? > 2. It seems like there are two optional methods for specifying > documentation for columns in the view, one is field-docs, and one is the > doc field on schema. Is the assumption that only one or the other would be > set? > 3. Are field-aliases intended to be further documentation or consumed > programmatically (i.e. can they be referenced in queries that use the view)? > 4. Are there any thoughts on standardization of dialects, at least for > common OSS query engines? Should there be an option to encode > dialect version encoded at least informationally in case a view uses a SQL > feature that was only introduced in later versions? > > Thanks, > Micah > > On Tue, Mar 14, 2023 at 8:02 PM John Zhuge <jzh...@apache.org> wrote: > >> +1 >> >> On Tue, Mar 14, 2023 at 7:31 PM Walaa Eldin Moustafa < >> wa.moust...@gmail.com> wrote: >> >>> +1 to get a basic implementation in. Some of the discussions/feedback on >>> the API PR slightly changed the API from the initial proposed API that >>> probably more closely resembled Netflix's implementation. Getting an >>> implementation going on the finalized APIs could give some good feedback to >>> the spec or the finalized APIs. >>> >>> On Tue, Mar 14, 2023 at 7:10 PM Ryan Blue <b...@tabular.io> wrote: >>> >>>> Thanks for the detailed update, Amogh! >>>> >>>> I think that the view spec is close, but it will likely change as we >>>> get the implementation done. Netflix has a working implementation that >>>> we're basing this off of, so I think it won't change significantly, but we >>>> don't want to vote to adopt the spec before it's ready. >>>> >>>> The next steps are to get the implementation done, which I think is a >>>> high priority now that branches and tags are being released in 1.2.0. We'd >>>> love more help implementing this, and especially help implementing the spec >>>> in other languages because that will definitely catch things we haven't >>>> seen yet. >>>> >>>> Let's talk about this at the sync tomorrow. >>>> >>>> Ryan >>>> >>>> On Tue, Mar 14, 2023 at 9:41 AM Jahagirdar, Amogh >>>> <jaham...@amazon.com.invalid> wrote: >>>> >>>>> Hi Micah, >>>>> >>>>> >>>>> >>>>> Thanks for starting this discussion. I also am not aware of a formal >>>>> vote thread for the View Specification. I’m open to starting a formal vote >>>>> if one wasn’t already done to ensure we have made all the right >>>>> considerations across the community. >>>>> >>>>> >>>>> >>>>> Currently, we are working on the view metadata parser implementations >>>>> based on the spec as it’s defined today >>>>> <https://iceberg.apache.org/view-spec/>. A few components have >>>>> already been merged but the root level metadata parser is still WIP. So if >>>>> there’s more discussion on aspects of the spec which we want to revisit, >>>>> we >>>>> should do that before exposing the root level metadata parser. >>>>> >>>>> >>>>> >>>>> For blockers to adoption, there’s a few implementations we need to >>>>> complete after we conclude on the spec: >>>>> >>>>> 1.) Completing the parser implementations. Right now, the view version >>>>> parser implementation <https://github.com/apache/iceberg/pull/6861> >>>>> is still in progress, and after this we’d do the overall view metadata >>>>> parser, and expose that in the library. >>>>> 2.) A “BaseMetastoreViewCatalog” or something of that sort where >>>>> common view operation implementations can be performed for metastore based >>>>> catalogs. >>>>> >>>>> Then I think the community can progress on the different view catalog >>>>> implementations and engine integration to further help adoption. >>>>> >>>>> >>>>> Open to suggestions and ideas here! >>>>> >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> >>>>> >>>>> Amogh Jahagirdar >>>>> >>>>> >>>>> >>>>> *From: *Micah Kornfield <emkornfi...@gmail.com> >>>>> *Reply-To: *"dev@iceberg.apache.org" <dev@iceberg.apache.org> >>>>> *Date: *Monday, March 13, 2023 at 6:17 PM >>>>> *To: *Iceberg Dev List <dev@iceberg.apache.org> >>>>> *Subject: *[EXTERNAL] Current Status of View Specification >>>>> >>>>> >>>>> >>>>> *CAUTION*: This email originated from outside of the organization. Do >>>>> not click links or open attachments unless you can confirm the sender and >>>>> know the content is safe. >>>>> >>>>> >>>>> >>>>> Hi Iceberg Dev, >>>>> >>>>> I see the spec has been checked in but I couldn't find a vote thread >>>>> ratifying it as a final V1 version (I might have been using the wrong >>>>> search terms) but for other additions of things like puffin it seemed like >>>>> there was an official vote. >>>>> >>>>> >>>>> >>>>> Should the spec be considered finalized as a V1 version now? Was >>>>> there a vote held? Will there be one? Are there any blockers to >>>>> adoption? >>>>> >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> Micah >>>>> >>>> >>>> >>>> -- >>>> Ryan Blue >>>> Tabular >>>> >>> >> >> -- >> John Zhuge >> > -- Ryan Blue Tabular
