Given that I think we are all in agreement, and Micah has signed off on the PR I think we can go ahead and merge this one.
On Tue, Dec 10, 2024 at 12:16 AM Aihua Xu <aihu...@gmail.com> wrote: > Thank you all for the discussion. Micah raised a valid concern about > including a specification that has not yet been finalized in Parquet. As we > discussed earlier, the community has shown interest in introducing the > basic variant type and shredding in V3. > > From my perspective, while the change is relatively simple and largely > references the Parquet variant spec, it will still take time to get it > thoroughly reviewed. My approach is to update the Iceberg spec to align > with the current version, iteratively refine it as we make changes to the > Parquet variant spec, and proceed with implementation in parallel. Hope > that makes sense. > > Thanks, > > Aihua > > On Tue, Nov 26, 2024 at 11:10 AM Jack Ye <yezhao...@gmail.com> wrote: > >> I am +1 on adding it to the spec and not waiting for Parquet. It feels >> like a better 2-way door decision compared to being blocked by Parquet >> ratification timeline. >> >> -Jack >> >> On Tue, Nov 26, 2024 at 10:05 AM Micah Kornfield <emkornfi...@gmail.com> >> wrote: >> >>> 2. We aren't going to formally close V3 Spec yet, so if we do end up in >>>> a situation where we want to close the spec and Parquet has not removed the >>>> tag, we can remove the variant from the spec then. (I think that scenario >>>> is unlikely) >>> >>> >>> This seems like it puts the effort at the wrong side of things. While I >>> agree it is probably low probability reviewing the whole V3 spec for >>> completeness and making sure there are no loose ends makes it more likely >>> to miss things like this. And if Variant ends up being the long poll of >>> the release, it seems like we are just adding effort to shipping V3 (which >>> already has a lot of other valuable additions). >>> >>> >>>> 3. There is very little in our change set here that specifically >>>> references the Parquet spec except for our reference link to it. >>> >>> >>> This cuts both ways? What is the rush to get this into V3 if it can >>> easily be merged once the Parquet side is official? >>> >>> Cheers, >>> Micah >>> >>> On Mon, Nov 25, 2024 at 9:21 AM Russell Spitzer < >>> russell.spit...@gmail.com> wrote: >>> >>>> I'm +1, >>>> >>>> 1. I don't think we are going to change our decision on whether to >>>> include variants based on the timing of Parquet ratification >>>> 2. We aren't going to formally close V3 Spec yet, so if we do end up in >>>> a situation where we want to close the spec and Parquet has not removed the >>>> tag, we can remove the variant from the spec then. (I think that scenario >>>> is unlikely) >>>> 3. There is very little in our change set here that specifically >>>> references the Parquet spec except for our reference link to it. >>>> >>>> I don't think there is anything that will happen in the spec that will >>>> change what we would include in the Iceberg Spec (especially in this PR) >>>> >>>> On Fri, Nov 22, 2024 at 5:10 PM Micah Kornfield <emkornfi...@gmail.com> >>>> wrote: >>>> >>>>> My (non-binding) vote is -1 until the variant spec is formally >>>>> adopted in Parquet. >>>>> >>>>> On Fri, Nov 22, 2024 at 2:51 PM Aihua Xu <aihu...@gmail.com> wrote: >>>>> >>>>>> Hi everyone, >>>>>> >>>>>> I've updated the Iceberg spec to include the new Variant type as >>>>>> part of #10831 <https://github.com/apache/iceberg/pull/10831>. The >>>>>> changes are basically complete. This is a heads-up about the upcoming >>>>>> change. Please review and +1 to acknowledge, so we will merge. >>>>>> >>>>>> Thanks, >>>>>> Aihua >>>>>> >>>>>
