Thank you all for the discussion. Micah raised a valid concern about including a specification that has not yet been finalized in Parquet. As we discussed earlier, the community has shown interest in introducing the basic variant type and shredding in V3.
>From my perspective, while the change is relatively simple and largely references the Parquet variant spec, it will still take time to get it thoroughly reviewed. My approach is to update the Iceberg spec to align with the current version, iteratively refine it as we make changes to the Parquet variant spec, and proceed with implementation in parallel. Hope that makes sense. Thanks, Aihua On Tue, Nov 26, 2024 at 11:10 AM Jack Ye <yezhao...@gmail.com> wrote: > I am +1 on adding it to the spec and not waiting for Parquet. It feels > like a better 2-way door decision compared to being blocked by Parquet > ratification timeline. > > -Jack > > On Tue, Nov 26, 2024 at 10:05 AM Micah Kornfield <emkornfi...@gmail.com> > wrote: > >> 2. We aren't going to formally close V3 Spec yet, so if we do end up in a >>> situation where we want to close the spec and Parquet has not removed the >>> tag, we can remove the variant from the spec then. (I think that scenario >>> is unlikely) >> >> >> This seems like it puts the effort at the wrong side of things. While I >> agree it is probably low probability reviewing the whole V3 spec for >> completeness and making sure there are no loose ends makes it more likely >> to miss things like this. And if Variant ends up being the long poll of >> the release, it seems like we are just adding effort to shipping V3 (which >> already has a lot of other valuable additions). >> >> >>> 3. There is very little in our change set here that specifically >>> references the Parquet spec except for our reference link to it. >> >> >> This cuts both ways? What is the rush to get this into V3 if it can >> easily be merged once the Parquet side is official? >> >> Cheers, >> Micah >> >> On Mon, Nov 25, 2024 at 9:21 AM Russell Spitzer < >> russell.spit...@gmail.com> wrote: >> >>> I'm +1, >>> >>> 1. I don't think we are going to change our decision on whether to >>> include variants based on the timing of Parquet ratification >>> 2. We aren't going to formally close V3 Spec yet, so if we do end up in >>> a situation where we want to close the spec and Parquet has not removed the >>> tag, we can remove the variant from the spec then. (I think that scenario >>> is unlikely) >>> 3. There is very little in our change set here that specifically >>> references the Parquet spec except for our reference link to it. >>> >>> I don't think there is anything that will happen in the spec that will >>> change what we would include in the Iceberg Spec (especially in this PR) >>> >>> On Fri, Nov 22, 2024 at 5:10 PM Micah Kornfield <emkornfi...@gmail.com> >>> wrote: >>> >>>> My (non-binding) vote is -1 until the variant spec is formally >>>> adopted in Parquet. >>>> >>>> On Fri, Nov 22, 2024 at 2:51 PM Aihua Xu <aihu...@gmail.com> wrote: >>>> >>>>> Hi everyone, >>>>> >>>>> I've updated the Iceberg spec to include the new Variant type as part >>>>> of #10831 <https://github.com/apache/iceberg/pull/10831>. The changes >>>>> are basically complete. This is a heads-up about the upcoming change. >>>>> Please review and +1 to acknowledge, so we will merge. >>>>> >>>>> Thanks, >>>>> Aihua >>>>> >>>>
