Thanks Gang for initiating the discussion. On Fri, Aug 23, 2024 at 2:22 AM Gang Wu <ust...@gmail.com> wrote:
> Thanks Aihua! > > I've started the discussion in dev@parquet: > https://lists.apache.org/thread/6h58hj39lhqtcyd2hlsyvqm4lzdh4b9z > > Best, > Gang > > On Fri, Aug 23, 2024 at 12:53 PM Aihua Xu <aihua...@snowflake.com> wrote: > >> From this thread >> https://lists.apache.org/thread/0k5oj3mn0049fcxoxm3gx3d7r28gw4rj, seems >> Spark community is leaning toward moving to Parquet. >> >> Gang, can you help start a discussion in the parquet community on >> adopting and maintaining such Variant spec? >> >> On Thu, Aug 22, 2024 at 8:08 AM Curt Hagenlocher <c...@hagenlocher.org> >> wrote: >> >>> This seems to straddle that line, in that you can also view this as a >>> way to represent semi-structured data in a manner that allows for more >>> efficient querying and computation by breaking out some of its components >>> into a more structured form. >>> >>> (I also happen to want a canonical Arrow representation for variant >>> data, as this type occurs in many databases but doesn't have a great >>> representation today in ADBC results. That's why I filed [Format] >>> Consider adding an official variant type to Arrow · Issue #42069 · >>> apache/arrow (github.com) <https://github.com/apache/arrow/issues/42069>. >>> Of course, there's no specific reason why a canonical Arrow >>> representation for variants must align with Spark and/or Iceberg.) >>> >>> -Curt >>> >>> On Thu, Aug 22, 2024 at 2:01 AM Antoine Pitrou <anto...@python.org> >>> wrote: >>> >>>> >>>> Ah, thanks. I've tried to find a rationale and ended up on >>>> https://lists.apache.org/thread/xnyo1k66dxh0ffpg7j9f04xgos0kwc34 . Is >>>> it >>>> a good description of what you're after? >>>> >>>> If so, then I don't think Arrow is a good match. This seems mostly to >>>> be >>>> a marshalling format for semi-structured data (like Avro?). Arrow data >>>> types are meant to be in a representation ideal for querying and >>>> computation, rather than transport and storage. >>>> >>>> This could be developed separately and then be represented in Arrow >>>> using an extension type (perhaps a canonical one as in >>>> https://arrow.apache.org/docs/dev/format/CanonicalExtensions.html). >>>> >>>> What do other Arrow developers think? >>>> >>>> Regards >>>> >>>> Antoine. >>>> >>>> >>>> Le 22/08/2024 à 10:45, Gang Wu a écrit : >>>> > Sorry for the inconvenience. >>>> > >>>> > This is the permalink for the discussion: >>>> > https://lists.apache.org/thread/hopkr2f0ftoywwt9zo3jxb7n0ob5s5bw >>>> > >>>> > On Thu, Aug 22, 2024 at 3:51 PM Antoine Pitrou <anto...@python.org> >>>> wrote: >>>> > >>>> >> >>>> >> Hi Gang, >>>> >> >>>> >> Sorry, but can you give a pointer to the start of this discussion >>>> thread >>>> >> in a readable format (for example a mailing-list archive)? It appears >>>> >> that dev@arrow wasn't cc'ed from the start and that can make it >>>> >> difficult to understand what this is about. >>>> >> >>>> >> Regards >>>> >> >>>> >> Antoine. >>>> >> >>>> >> >>>> >> Le 22/08/2024 à 08:32, Gang Wu a écrit : >>>> >>> It seems that we have reached a consensus to some extent that there >>>> >>> should be a new home for the variant spec. The pending question >>>> >>> is whether Parquet or Arrow is a better choice. As a committer from >>>> >> Arrow, >>>> >>> Parquet and ORC communities, I am neutral to choose any and happy to >>>> >>> help with the movement once a decision has been made. >>>> >>> >>>> >>> Should we start a vote to move forward? >>>> >>> >>>> >>> Best, >>>> >>> Gang >>>> >>> >>>> >>> On Sat, Aug 17, 2024 at 8:34 AM Micah Kornfield < >>>> emkornfi...@gmail.com> >>>> >>> wrote: >>>> >>> >>>> >>>>> >>>> >>>>> That being said, I think the most important consideration for now >>>> is >>>> >>>> where >>>> >>>>> are the current maintainers / contributors to the variant type. >>>> If most >>>> >>>> of >>>> >>>>> them are already PMC members / committers on a project, it >>>> becomes a >>>> >> bit >>>> >>>>> easier. Otherwise if there isn't much overlap with a project's >>>> existing >>>> >>>>> governance, I worry there could be a bit of friction. How many >>>> active >>>> >>>>> contributors are there from Iceberg? And how about from Arrow? >>>> >>>> >>>> >>>> >>>> >>>> I think this is the key question. What are the requirements around >>>> >>>> governance? I've seen some tangential messaging here but I'm not >>>> clear >>>> >> on >>>> >>>> what everyone expects. >>>> >>>> >>>> >>>> I think for a lot of the other concerns my view is that the exact >>>> >> project >>>> >>>> does not really matter (and choosing a project with mature cross >>>> >> language >>>> >>>> testing infrastructure or committing to building it is critical). >>>> IIUC >>>> >> we >>>> >>>> are talking about following artifacts: >>>> >>>> >>>> >>>> 1. A stand alone specification document (this can be hosted >>>> anyplace) >>>> >>>> 2. A set of language bindings with minimal dependencies can be >>>> consumed >>>> >>>> downstream (again, as long as dependencies are managed carefully >>>> any >>>> >>>> project can host these) >>>> >>>> 3. Potential integration where appropriate into file format >>>> libraries >>>> >> to >>>> >>>> support shredding (but as of now this is being bypassed by using >>>> >>>> conventions anyways). My impression is that at least for Parquet >>>> there >>>> >> has >>>> >>>> been a proliferation of vectorized readers across different >>>> projects, so >>>> >>>> I'm not clear how much standardization in parquet-java could help >>>> here. >>>> >>>> >>>> >>>> To respond to some other questions: >>>> >>>> >>>> >>>> Arrow is not used as Spark's in-memory model, nor Trino and others >>>> so >>>> >> those >>>> >>>>> existing relationships aren't there. I also worry that >>>> differences in >>>> >>>>> approaches would make it difficult later on. >>>> >>>> >>>> >>>> >>>> >>>> While Arrow is not in the core memory model, for Spark I believe >>>> it is >>>> >>>> still used for IPC for things like Java<->Python. Trino also >>>> consumes >>>> >> Arrow >>>> >>>> libraries today to support things like Snowflake/Bigquery >>>> federation. >>>> >> But I >>>> >>>> think this is minor because as mentioned above I think the >>>> functional >>>> >>>> libraries would be relatively stand-alone. >>>> >>>> >>>> >>>> Do we think it could be introduced as a canonical extension arrow >>>> type? >>>> >>>> >>>> >>>> >>>> >>>> I believe it can be, I think there are probably different >>>> layouts >>>> >> that can >>>> >>>> be supported: >>>> >>>> >>>> >>>> 1. A struct with two variable width bytes columns (metadata and >>>> value >>>> >> data >>>> >>>> are stored separately and each entry has a 1:1 relationship). >>>> >>>> 2. Shredded (shredded according to the same convention as >>>> parquet), I >>>> >>>> would need to double check but I don't think Arrow would have >>>> problems >>>> >> here >>>> >>>> but REE would likely be required to make this efficient (i.e. >>>> sparse >>>> >> value >>>> >>>> support is important). >>>> >>>> >>>> >>>> In both cases the main complexity is providing the necessary >>>> functions >>>> >> for >>>> >>>> manipulation. >>>> >>>> >>>> >>>> Thanks, >>>> >>>> Micah >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> On Fri, Aug 16, 2024 at 3:58 PM Will Jones < >>>> will.jones...@gmail.com> >>>> >>>> wrote: >>>> >>>> >>>> >>>>> In being more engine and format agnostic, I agree the Arrow >>>> project >>>> >> might >>>> >>>>> be a good host for such a specification. It seems like we want to >>>> move >>>> >>>> away >>>> >>>>> from hosting in Spark to make it engine agnostic. But moving into >>>> >> Iceberg >>>> >>>>> might make it less format agnostic, as I understand multiple >>>> formats >>>> >>>> might >>>> >>>>> want to implement this. I'm not intimately familiar with the >>>> state of >>>> >>>> this, >>>> >>>>> but I believe Delta Lake would like to be aligned with the same >>>> format >>>> >> as >>>> >>>>> Iceberg. In addition, the Lance format (which I work on), will >>>> >> eventually >>>> >>>>> be interesting as well. It seems equally bad to me to attach this >>>> >>>>> specification to a particular table format as it does a particular >>>> >> query >>>> >>>>> engine. >>>> >>>>> >>>> >>>>> That being said, I think the most important consideration for now >>>> is >>>> >>>> where >>>> >>>>> are the current maintainers / contributors to the variant type. >>>> If most >>>> >>>> of >>>> >>>>> them are already PMC members / committers on a project, it >>>> becomes a >>>> >> bit >>>> >>>>> easier. Otherwise if there isn't much overlap with a project's >>>> existing >>>> >>>>> governance, I worry there could be a bit of friction. How many >>>> active >>>> >>>>> contributors are there from Iceberg? And how about from Arrow? >>>> >>>>> >>>> >>>>> BTW, I'd add I'm interested in helping develop an Arrow extension >>>> type >>>> >>>> for >>>> >>>>> the binary variant type. I've been experimenting with a DataFusion >>>> >>>>> extension that operates on this [1], and already have some ideas >>>> on how >>>> >>>>> such an extension type might be defined. I'm not yet caught up on >>>> the >>>> >>>>> shredded specification, but I think having just the binary format >>>> would >>>> >>>> be >>>> >>>>> beneficial for in-memory analytics, which are most relevant to >>>> Arrow. >>>> >>>> I'll >>>> >>>>> be creating a seperate thread on the Arrow ML about this soon. >>>> >>>>> >>>> >>>>> Best, >>>> >>>>> >>>> >>>>> Will Jones >>>> >>>>> >>>> >>>>> [1] >>>> >>>>> >>>> >>>> >>>> >> >>>> https://github.com/datafusion-contrib/datafusion-functions-variant/issues >>>> >>>>> >>>> >>>>> >>>> >>>>> On Thu, Aug 15, 2024 at 7:39 PM Gang Wu <ust...@gmail.com> wrote: >>>> >>>>> >>>> >>>>>> + dev@arrow >>>> >>>>>> >>>> >>>>>> Thanks for all the valuable suggestions! I am inclined to >>>> Micah's idea >>>> >>>>> that >>>> >>>>>> Arrow might be a better host compared to Parquet. >>>> >>>>>> >>>> >>>>>> To give more context, I am taking the initiative to add the >>>> geometry >>>> >>>> type >>>> >>>>>> to both Parquet and ORC. I'd like to do the same thing for >>>> variant >>>> >> type >>>> >>>>> in >>>> >>>>>> that variant type is engine and file format agnostic. This does >>>> mean >>>> >>>> that >>>> >>>>>> Parquet might not be the neutral place to hold the variant spec. >>>> >>>>>> >>>> >>>>>> Best, >>>> >>>>>> Gang >>>> >>>>>> >>>> >>>>>> On Fri, Aug 16, 2024 at 10:00 AM Jingsong Li < >>>> jingsongl...@gmail.com> >>>> >>>>>> wrote: >>>> >>>>>> >>>> >>>>>>> Thanks all for your discussion. >>>> >>>>>>> >>>> >>>>>>> The Apache Paimon community is also considering support for this >>>> >>>>>>> Variant type, without a doubt, we hope to maintain consistency >>>> with >>>> >>>>>>> Iceberg. >>>> >>>>>>> >>>> >>>>>>> Not only the Paimon community, but also various computing >>>> engines >>>> >>>> need >>>> >>>>>>> to adapt to this type, such as Flink and StarRocks. We also >>>> hope to >>>> >>>>>>> promote them to adapt to this type. >>>> >>>>>>> >>>> >>>>>>> It is worth noting that we also need to standardize many >>>> functions >>>> >>>>>>> related to it. >>>> >>>>>>> >>>> >>>>>>> A neutral place to maintain it is a great choice. >>>> >>>>>>> >>>> >>>>>>> - As Gang Wu said, a standalone project is good, just like >>>> >>>>> RoaringBitmap >>>> >>>>>>> [1]. >>>> >>>>>>> - As Ryan said, Parquet community is a neutral option too. >>>> >>>>>>> - As Micah said, Arrow is also an option too. >>>> >>>>>>> >>>> >>>>>>> [1] https://github.com/RoaringBitmap >>>> >>>>>>> >>>> >>>>>>> Best, >>>> >>>>>>> Jingsong >>>> >>>>>>> >>>> >>>>>>> On Fri, Aug 16, 2024 at 7:18 AM Micah Kornfield < >>>> >>>> emkornfi...@gmail.com >>>> >>>>>> >>>> >>>>>>> wrote: >>>> >>>>>>>>> >>>> >>>>>>>>> Thats fair @Micah, so far all the discussions have been >>>> direct and >>>> >>>>> off >>>> >>>>>>> the dev list. Would you like to make the request on the public >>>> Spark >>>> >>>>> Dev >>>> >>>>>>> list? I would be glad to co-sign, I can also draft up a quick >>>> email >>>> >>>> if >>>> >>>>>> you >>>> >>>>>>> don't have time. >>>> >>>>>>>> >>>> >>>>>>>> >>>> >>>>>>>> I think once we come to consensus, if you have bandwidth, I >>>> think >>>> >>>> the >>>> >>>>>>> message might be better coming from you, as you have more >>>> context on >>>> >>>>> some >>>> >>>>>>> of the non-public conversations, the requirements from an >>>> Iceberg >>>> >>>>>>> perspective on governance and the blockers that were >>>> encountered. If >>>> >>>>>>> details on the conversations can't be shared, (i.e. we are >>>> starting >>>> >>>>> from >>>> >>>>>>> scratch) it seems like suggesting a new project via SPIP might >>>> be the >>>> >>>>> way >>>> >>>>>>> forward. I'm happy to help with that if it is useful but I >>>> would >>>> >>>> guess >>>> >>>>>>> Aihua or Tyler might be in a better place to start as it seems >>>> they >>>> >>>>> have >>>> >>>>>>> done more serious thinking here. >>>> >>>>>>>> >>>> >>>>>>>> If we decide to try to standardize on Parquet or Arrow I'm >>>> happy to >>>> >>>>>> help >>>> >>>>>>> support the effort in those communities. >>>> >>>>>>>> >>>> >>>>>>>> Thanks, >>>> >>>>>>>> Micah >>>> >>>>>>>> >>>> >>>>>>>> On Thu, Aug 15, 2024 at 8:09 AM Russell Spitzer < >>>> >>>>>>> russell.spit...@gmail.com> wrote: >>>> >>>>>>>>> >>>> >>>>>>>>> Thats fair @Micah, so far all the discussions have been >>>> direct and >>>> >>>>> off >>>> >>>>>>> the dev list. Would you like to make the request on the public >>>> Spark >>>> >>>>> Dev >>>> >>>>>>> list? I would be glad to co-sign, I can also draft up a quick >>>> email >>>> >>>> if >>>> >>>>>> you >>>> >>>>>>> don't have time. >>>> >>>>>>>>> >>>> >>>>>>>>> On Thu, Aug 15, 2024 at 10:04 AM Micah Kornfield < >>>> >>>>>> emkornfi...@gmail.com> >>>> >>>>>>> wrote: >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> I agree that it would be beneficial to make a sub-project, >>>> the >>>> >>>>> main >>>> >>>>>>> problem is political and not logistic. I've been asking for >>>> movement >>>> >>>>> from >>>> >>>>>>> other relative projects for a month and we simply haven't gotten >>>> >>>>>> anywhere. >>>> >>>>>>>>>> >>>> >>>>>>>>>> >>>> >>>>>>>>>> I just wanted to double check that these issues were brought >>>> >>>>> directly >>>> >>>>>>> to the spark community (i.e. a discussion thread on the Spark >>>> >>>> developer >>>> >>>>>>> mailing list) and not via backchannels. >>>> >>>>>>>>>> >>>> >>>>>>>>>> I'm not sure the outcome would be different and I don't think >>>> >>>> this >>>> >>>>>>> should block forking the spec, but we should make sure that the >>>> >>>>> decision >>>> >>>>>> is >>>> >>>>>>> publicly documented within both communities. >>>> >>>>>>>>>> >>>> >>>>>>>>>> Thanks, >>>> >>>>>>>>>> Micah >>>> >>>>>>>>>> >>>> >>>>>>>>>> On Thu, Aug 15, 2024 at 7:47 AM Russell Spitzer < >>>> >>>>>>> russell.spit...@gmail.com> wrote: >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> @Gang Wu >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> I agree that it would be beneficial to make a sub-project, >>>> the >>>> >>>>> main >>>> >>>>>>> problem is political and not logistic. I've been asking for >>>> movement >>>> >>>>> from >>>> >>>>>>> other relative projects for a month and we simply haven't gotten >>>> >>>>>> anywhere. >>>> >>>>>>> I don't think there is anything that would stop us from moving >>>> to a >>>> >>>>> joint >>>> >>>>>>> project in the future and if you know of some way of >>>> encouraging that >>>> >>>>>>> movement from other relevant parties I would be glad to >>>> collaborate >>>> >>>> in >>>> >>>>>>> doing that. One thing that I don't want to do is have the >>>> Iceberg >>>> >>>>> project >>>> >>>>>>> stay in a holding pattern without any clear roadmap as to how to >>>> >>>>> proceed. >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> On Wed, Aug 14, 2024 at 11:12 PM Yufei Gu < >>>> flyrain...@gmail.com >>>> >>>>> >>>> >>>>>>> wrote: >>>> >>>>>>>>>>>> >>>> >>>>>>>>>>>> I’m on board with copying the spec into our repository. >>>> >>>> However, >>>> >>>>> as >>>> >>>>>>> we’ve talked about, it’s not just a straightforward copy—there >>>> are >>>> >>>>>> already >>>> >>>>>>> some divergences. Some of them are under discussion. Iceberg is >>>> >>>>>> definitely >>>> >>>>>>> the best place for these specs. Engines like Trino and Flink >>>> can then >>>> >>>>>> rely >>>> >>>>>>> on the Iceberg specs as a solid foundation. >>>> >>>>>>>>>>>> >>>> >>>>>>>>>>>> Yufei >>>> >>>>>>>>>>>> >>>> >>>>>>>>>>>> On Wed, Aug 14, 2024 at 7:51 PM Gang Wu <ust...@gmail.com> >>>> >>>>> wrote: >>>> >>>>>>>>>>>>> >>>> >>>>>>>>>>>>> Sorry for chiming in late. >>>> >>>>>>>>>>>>> >>>> >>>>>>>>>>>>> From the discussion in >>>> >>>>>>> >>>> https://lists.apache.org/thread/xcyytoypgplfr74klg1z2rgjo6k5b0sq, I >>>> >>>>>> don't >>>> >>>>>>> quite understand why it is logistically complicated to create a >>>> >>>>>> sub-project >>>> >>>>>>> to hold the variant spec and impl. >>>> >>>>>>>>>>>>> >>>> >>>>>>>>>>>>> IMHO, coping the variant type spec into Apache Iceberg has >>>> >>>> some >>>> >>>>>>> deficiencies: >>>> >>>>>>>>>>>>> - It is a burden to update two repos if there is a variant >>>> >>>> type >>>> >>>>>>> spec change and will likely result in deviation if some changes >>>> do >>>> >>>> not >>>> >>>>>>> reach agreement from both parties. >>>> >>>>>>>>>>>>> - Implementers are required to keep an eye on both specs >>>> >>>>>>> (considering proprietary engines where both Iceberg and Delta >>>> are >>>> >>>>>>> supported). >>>> >>>>>>>>>>>>> - Putting the spec and impl of variant type in Iceberg >>>> repo >>>> >>>> does >>>> >>>>>>> lose the opportunity for better native support from file >>>> formats like >>>> >>>>>>> Parquet and ORC. >>>> >>>>>>>>>>>>> >>>> >>>>>>>>>>>>> I'm not sure if it is possible to create a separate >>>> project >>>> >>>>> (e.g. >>>> >>>>>>> apache/variant-type) to make it a single point of truth. We can >>>> learn >>>> >>>>>> from >>>> >>>>>>> the experience of Apache Arrow. In this fashion, different >>>> engines, >>>> >>>>> table >>>> >>>>>>> formats and file formats can follow the same spec and are free >>>> to >>>> >>>>> depend >>>> >>>>>> on >>>> >>>>>>> the reference implementations from apache/variant-type or >>>> implement >>>> >>>>> their >>>> >>>>>>> own. >>>> >>>>>>>>>>>>> >>>> >>>>>>>>>>>>> Best, >>>> >>>>>>>>>>>>> Gang >>>> >>>>>>>>>>>>> >>>> >>>>>>>>>>>>> >>>> >>>>>>>>>>>>> >>>> >>>>>>>>>>>>> >>>> >>>>>>>>>>>>> On Thu, Aug 15, 2024 at 10:07 AM Jack Ye < >>>> yezhao...@gmail.com >>>> >>>>> >>>> >>>>>>> wrote: >>>> >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> +1 for copying the spec into our repository, I think we >>>> need >>>> >>>> to >>>> >>>>>>> own it fully as a part of the table spec, and we can build >>>> >>>>> compatibility >>>> >>>>>>> through tests. >>>> >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> -Jack >>>> >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> On Wed, Aug 14, 2024 at 12:52 PM Russell Spitzer < >>>> >>>>>>> russell.spit...@gmail.com> wrote: >>>> >>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>> I'm not really in favor of linking and annotating as >>>> that >>>> >>>> just >>>> >>>>>>> makes things more complicated and still is essentially forking >>>> just >>>> >>>>> with >>>> >>>>>>> more steps. If we just track our annotations / modifications >>>> to a >>>> >>>>> single >>>> >>>>>>> commit/version then we have the same issue again but now you >>>> have to >>>> >>>> go >>>> >>>>>> to >>>> >>>>>>> multiple sources to get the actual Spec. In addition, our very >>>> copy >>>> >>>> of >>>> >>>>>> the >>>> >>>>>>> Spec is going to require new types which don't exist in the >>>> Spark >>>> >>>> Spec >>>> >>>>>>> which necessarily means diverging. We will need to take up new >>>> >>>>> primitive >>>> >>>>>>> id's (as noted in my first email) >>>> >>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>> The other issue I have is I don't think the Spark Spec >>>> is >>>> >>>>> really >>>> >>>>>>> going through a thorough review process from all members of the >>>> Spark >>>> >>>>>>> community, I believe it probably should have gone through the >>>> SPIP >>>> >>>> but >>>> >>>>>>> instead seems to have been merged without broad community >>>> >>>> involvement. >>>> >>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>> The only way to truly avoid diverging is to only have a >>>> >>>> single >>>> >>>>>>> copy of the spec, in our previous discussions the vast majority >>>> of >>>> >>>>> Apache >>>> >>>>>>> Iceberg community want it to exist here. >>>> >>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>> On Wed, Aug 14, 2024 at 2:19 PM Daniel Weeks < >>>> >>>>> dwe...@apache.org >>>> >>>>>>> >>>> >>>>>>> wrote: >>>> >>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>> I'm really excited about the introduction of variant >>>> type >>>> >>>> to >>>> >>>>>>> Iceberg, but I want to raise concerns about forking the spec. >>>> >>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>> I feel like preemptively forking would create the >>>> situation >>>> >>>>>>> where we end up diverging because there's little reason to work >>>> with >>>> >>>>> both >>>> >>>>>>> communities to evolve in a way that benefits everyone. >>>> >>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>> I would much rather point to a specific version of the >>>> spec >>>> >>>>> and >>>> >>>>>>> annotate any variance in Iceberg's handling. This would allow >>>> us to >>>> >>>>>>> continue without dividing the communities. >>>> >>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>> If at any point there are irreconcilable differences, I >>>> >>>> would >>>> >>>>>>> support forking, but I don't feel like that should be the >>>> initial >>>> >>>> step. >>>> >>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>> No one is excited about the possibility that the >>>> physical >>>> >>>>>>> representations end up diverging, but it feels like we're >>>> setting >>>> >>>>>> ourselves >>>> >>>>>>> up for that exact scenario. >>>> >>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>> -Dan >>>> >>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>> On Wed, Aug 14, 2024 at 6:54 AM Fokko Driesprong < >>>> >>>>>>> fo...@apache.org> wrote: >>>> >>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>> +1 to what's already being said here. It is good to >>>> copy >>>> >>>> the >>>> >>>>>>> spec to Iceberg and add context that's specific to Iceberg, but >>>> at >>>> >>>> the >>>> >>>>>> same >>>> >>>>>>> time, we should maintain compatibility. >>>> >>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>> Kind regards, >>>> >>>>>>>>>>>>>>>>> Fokko >>>> >>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>> Op wo 14 aug 2024 om 15:30 schreef Manu Zhang < >>>> >>>>>>> owenzhang1...@gmail.com>: >>>> >>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>> +1 to copy the spec into our repository. I think the >>>> best >>>> >>>>> way >>>> >>>>>>> to keep compatibility is building integration tests. >>>> >>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>> Thanks, >>>> >>>>>>>>>>>>>>>>>> Manu >>>> >>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>> On Wed, Aug 14, 2024 at 8:27 PM Péter Váry < >>>> >>>>>>> peter.vary.apa...@gmail.com> wrote: >>>> >>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>> Thanks Russell and Aihua for pushing Variant >>>> support! >>>> >>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>> Given the differences between the supported types >>>> and >>>> >>>> the >>>> >>>>>>> lack of interest from the other project, I think it is >>>> reasonable to >>>> >>>>>>> duplicate the specification to our repository. >>>> >>>>>>>>>>>>>>>>>>> I would give very strong emphasis on sticking to the >>>> >>>> Spark >>>> >>>>>>> spec as much as possible, to keep compatibility as much as >>>> possible. >>>> >>>>>> Maybe >>>> >>>>>>> even revert to a shared specification if the situation changes. >>>> >>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>> Thanks, >>>> >>>>>>>>>>>>>>>>>>> Peter >>>> >>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>> Aihua Xu <aihu...@gmail.com> ezt írta (időpont: >>>> 2024. >>>> >>>>> aug. >>>> >>>>>>> 13., K, 19:52): >>>> >>>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>>> Thanks Russell for bringing this up. >>>> >>>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>>> This is the main blocker to move forward with the >>>> >>>> Variant >>>> >>>>>>> support in Iceberg and hopefully we can have a consensus. To >>>> me, I >>>> >>>> also >>>> >>>>>>> feel it makes more sense to move the spec into Iceberg rather >>>> than >>>> >>>>> Spark >>>> >>>>>>> engine owns it and we try to keep it compatible with Spark spec. >>>> >>>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>>> Thanks, >>>> >>>>>>>>>>>>>>>>>>>> Aihua >>>> >>>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>>> On Mon, Aug 12, 2024 at 6:50 PM Russell Spitzer < >>>> >>>>>>> russell.spit...@gmail.com> wrote: >>>> >>>>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>>>> Hi Y’all, >>>> >>>>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>>>> We’ve hit a bit of a roadblock with the Variant >>>> >>>>> Proposal, >>>> >>>>>>> while we were hoping to move the Variant and Shredding >>>> specifications >>>> >>>>>> from >>>> >>>>>>> Spark into Iceberg there doesn’t seem to be a lot of interest in >>>> >>>> that. >>>> >>>>>>> Unfortunately, I think we have a number of issues with just >>>> linking >>>> >>>> to >>>> >>>>>> the >>>> >>>>>>> Spark project directly from within Iceberg and I believe we >>>> need to >>>> >>>>> copy >>>> >>>>>>> the specifications into our repository. >>>> >>>>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>>>> There are a few reasons why i think this is >>>> necessary >>>> >>>>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>>>> First, we have a divergence of types already. The >>>> >>>> Spark >>>> >>>>>>> Specification already includes types which Iceberg has no >>>> definition >>>> >>>>> for >>>> >>>>>>> (19, 20 - Interval Types) and Iceberg already has a type which >>>> is not >>>> >>>>>>> included within the Spark Specification (Time) and will soon >>>> have >>>> >>>> more >>>> >>>>>> with >>>> >>>>>>> TimestampNS, and Geo. >>>> >>>>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>>>> Second, We would like to make sure that Spark is >>>> not a >>>> >>>>>> hard >>>> >>>>>>> dependency for other engines. We are working with several >>>> >>>> implementers >>>> >>>>> of >>>> >>>>>>> the Iceberg spec and it has previously been agreed that it >>>> would be >>>> >>>>> best >>>> >>>>>> if >>>> >>>>>>> the source of truth for Variant existed in an engine and file >>>> format >>>> >>>>>>> neutral location. The Iceberg project has a good open model of >>>> >>>>> governance >>>> >>>>>>> and, as we have seen so far discussing Variant, open and active >>>> >>>>>>> collaboration. This would also help as we can strictly version >>>> our >>>> >>>>>> changes >>>> >>>>>>> in-line with the rest of the Iceberg spec. >>>> >>>>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>>>> Third, The Shredding spec is not quite finished >>>> and >>>> >>>>>>> requires some group analysis and discussion before we commit >>>> it. I >>>> >>>>> think >>>> >>>>>>> again the Iceberg community is probably the right place for >>>> this to >>>> >>>>>> happen >>>> >>>>>>> as we have already started discussions here on these topics. >>>> >>>>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>>>> For these reasons I think we should go with a >>>> direct >>>> >>>>> copy >>>> >>>>>>> of the existing specification from the Spark Project and move >>>> ahead >>>> >>>>> with >>>> >>>>>>> our discussions and modifications within Iceberg. That said, I >>>> do not >>>> >>>>>> want >>>> >>>>>>> to diverge if possible from the Spark proposal. For example, >>>> although >>>> >>>>> we >>>> >>>>>> do >>>> >>>>>>> not use the Interval types above, I think we should not reuse >>>> those >>>> >>>>> type >>>> >>>>>>> ids within our spec. Iceberg's Variant Spec types 19 and 20 >>>> would >>>> >>>>> remain >>>> >>>>>>> unused along with any other types we think are not applicable. >>>> We >>>> >>>>> should >>>> >>>>>>> strive whenever possible to allow for compatibility. >>>> >>>>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>>>> In the interest of moving forward with this >>>> proposal I >>>> >>>>> am >>>> >>>>>>> hoping to see if anyone in the community objects to this plan >>>> going >>>> >>>>>> forward >>>> >>>>>>> or has a better alternative. >>>> >>>>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>>>> As always I am thankful for your time and am >>>> eager to >>>> >>>>> hear >>>> >>>>>>> back from everyone, >>>> >>>>>>>>>>>>>>>>>>>>> Russ >>>> >>>>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>>>> >>>> >>>>>>> >>>> >>>>>> >>>> >>>>> >>>> >>>> >>>> >>> >>>> >> >>>> > >>>> >>>
