Hi community, Thanks for joining the meeting to discuss variant shredding. For those who were unable to attend the meeting, please check out the recorded meeting <https://drive.google.com/file/d/1kiwv29nxxOqMCbxXn-NRoz-x2E9yIMlJ/view?usp=drive_link> if you are interested. Also to follow up on the meeting to converge on lossiness discussion from shredding offline, I have converted the spark shredding proposal by David into google doc <https://docs.google.com/document/d/1JeBt4NIju08jQ2AbludiK-U0M9ISIgysP7fUDWtv7rg/edit> and please comment.
Thanks, Aihua On Thu, Jul 25, 2024 at 10:14 AM Aihua Xu <aihu...@gmail.com> wrote: > Yes. This time I was able to record it and I will share it when it’s > processed. > > > On Jul 25, 2024, at 10:01 AM, Amogh Jahagirdar <2am...@gmail.com> wrote: > > > Any chance this meeting was recorded? I couldn't make it but would be > interested in catching up on the discussion. > > Thanks, > > Amogh Jahagirdar > > On Tue, Jul 23, 2024 at 11:30 AM Aihua Xu <aihu...@gmail.com> wrote: > >> Thanks folks for additional discussion. >> >> There are some questions related to subcolumniziation (spark shredding - >> see the discussion <https://github.com/apache/spark/pull/46831>) and we >> would like to host another meeting to mainly discuss that since we plan to >> adopt it. We can also follow up the Spark variant topics (I can see that >> mostly we are aligned with the exception to find a place for the spec and >> implementation). Look forward to meeting with you. BTW: should I include >> dev@iceberg.apache.org in the email invite? >> >> Sync up on Variant subcolumnization (shredding) >> Thursday, July 25 · 8:00 – 9:00am >> Time zone: America/Los_Angeles >> Google Meet joining info >> Video call link: https://meet.google.com/mug-dvnv-hnq >> Or dial: (US) +1 904-900-0730 PIN: 671 997 419# >> More phone numbers: https://tel.meet/mug-dvnv-hnq?pin=1654043233422 >> >> Thanks, >> Aihua >> >> On Tue, Jul 23, 2024 at 6:36 AM Amogh Jahagirdar <2am...@gmail.com> >> wrote: >> >>> I'm late replying to this but I'm also in agreement with 1 (adopting the >>> spark variant encoding), 3 (specifically only having a variant type), and 4 >>> (ensuring we are thinking through subcolumnarization upfront since without >>> it the variant type may not be that useful). >>> >>> I'd also support having the spec, and reference implementation in >>> Iceberg; as others have said, it centralizes improvements in a single, >>> agnostic dependency for engines, rather than engines having to take >>> dependencies on other engine modules. >>> >>> Thanks, >>> >>> Amogh Jahagirdar >>> >>> On Tue, Jul 23, 2024 at 12:15 AM Péter Váry <peter.vary.apa...@gmail.com> >>> wrote: >>> >>>> I have been looking around, how can we map Variant type in Flink. I >>>> have not found any existing type which we could use, but Flink already have >>>> some JSON parsing capabilities [1] for string fields. >>>> >>>> So until we have native support in Flink for something similar to >>>> Vartiant type, I expect that we need to map it to JSON strings in RowData. >>>> >>>> Based on that, here are my preferences: >>>> 1. I'm ok with adapting Spark Variant type, if we build our own Iceberg >>>> serializer/deserializer module for it >>>> 2. I prefer to move the spec to Iceberg, so we own it, and extend it, >>>> if needed. This could be important in the first phase. Later when it is >>>> more stable we might donate it to some other project, like Parquet >>>> 3. I would prefer to support only a single type, and Variant is more >>>> expressive, but having a standard way to convert between JSON and Variant >>>> would be useful for Flink users. >>>> 4. On subcolumnarization: I think Flink will only use this feature as >>>> much as the Iceberg readers implement this, so I would like to see as much >>>> as possible of it in the common Iceberg code >>>> >>>> Thanks, >>>> Peter >>>> >>>> [1] - >>>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/systemfunctions/#json-functions >>>> >>>> >>>> On Tue, Jul 23, 2024, 06:36 Micah Kornfield <emkornfi...@gmail.com> >>>> wrote: >>>> >>>>> Sorry for the late reply. I agree with the sentiments on 1 and 3 that >>>>> have already been posted (adopt the Spark encoding, and only have the >>>>> Variant type). As mentioned on the doc for 3, I think it would be good to >>>>> specify how to map scalar types to a JSON representation so there can be >>>>> consistency between engines that don't support variant. >>>>> >>>>> >>>>>> Regarding point 2, I also feel Iceberg is more natural to host such a >>>>>> subproject for variant spec and implementation. But let me reach out to >>>>>> the >>>>>> Spark community to discuss. >>>>> >>>>> >>>>> The only other place I can think of that might be a good home for >>>>> Variant spec could be in Apache Arrow as a canonical extension type. There >>>>> is an issue for this [1]. I think the main thing on where this is housed >>>>> is which types are intended to be supported. I believe Arrow is currently >>>>> a superset of the Iceberg type system (UUID is supported as a canonical >>>>> extension type [2]). >>>>> >>>>> For point 4 subcolumnarization, I think ideally this belongs in >>>>> Iceberg (and if Iceberg and Delta Lake can agree on how to do it that >>>>> would >>>>> be great) with potential consultation with Parquet/ORC communities to >>>>> potentially add better native support. >>>>> >>>>> Thanks, >>>>> Micah >>>>> >>>>> >>>>> >>>>> [1] https://github.com/apache/arrow/issues/42069 >>>>> [2] https://arrow.apache.org/docs/format/CanonicalExtensions.html >>>>> >>>>> On Sat, Jul 20, 2024 at 5:54 PM Aihua Xu <aihu...@gmail.com> wrote: >>>>> >>>>>> Thanks for the discussion and feedback. >>>>>> >>>>>> Do we have the consensus on point 1 and point 3 to move forward with >>>>>> Spark variant encoding and support Variant type only? Or let me know how >>>>>> to >>>>>> proceed from here. >>>>>> >>>>>> Regarding point 2, I also feel Iceberg is more natural to host such a >>>>>> subproject for variant spec and implementation. But let me reach out to >>>>>> the >>>>>> Spark community to discuss. >>>>>> >>>>>> Thanks, >>>>>> Aihua >>>>>> >>>>>> >>>>>> On Fri, Jul 19, 2024 at 9:35 AM Yufei Gu <flyrain...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Agreed with point 1. >>>>>>> >>>>>>> For point 2, I also prefer to hold the spec and reference >>>>>>> implementation under Iceberg. Here are the reasons: >>>>>>> 1. It is unconventional and impractical for one engine to depend on >>>>>>> another for data types. For instance, it is not ideal for Trino to rely >>>>>>> on >>>>>>> data types defined by the Spark engine. >>>>>>> 2. Iceberg serves as a bridge between engines and file formats. By >>>>>>> centralizing the specification in Iceberg, any future optimizations or >>>>>>> updates to file formats can be referred to within Iceberg, ensuring >>>>>>> consistency and reducing dependencies. >>>>>>> >>>>>>> For point 3, I'd prefer to support the variant type only at this >>>>>>> moment. >>>>>>> >>>>>>> Yufei >>>>>>> >>>>>>> >>>>>>> On Thu, Jul 18, 2024 at 12:55 PM Ryan Blue >>>>>>> <b...@databricks.com.invalid> wrote: >>>>>>> >>>>>>>> Similarly, I'm aligned with point 1 and I'd choose to support only >>>>>>>> variant for point 3. >>>>>>>> >>>>>>>> We'll need to work with the Spark community to find a good place >>>>>>>> for the library and spec, since it touches many different projects. I'd >>>>>>>> also prefer Iceberg as the home. >>>>>>>> >>>>>>>> I also think it's a good idea to get subcolumnarization into our >>>>>>>> spec when we update. Without that I think the feature will be fairly >>>>>>>> limited. >>>>>>>> >>>>>>>> On Thu, Jul 18, 2024 at 10:56 AM Russell Spitzer < >>>>>>>> russell.spit...@gmail.com> wrote: >>>>>>>> >>>>>>>>> I'm aligned with point 1. >>>>>>>>> >>>>>>>>> For point 2 I think we should choose quickly, I honestly do think >>>>>>>>> this would be fine as part of the Iceberg Spec directly but >>>>>>>>> understand it >>>>>>>>> may be better for the more broad community if it was a sub project. >>>>>>>>> As a >>>>>>>>> sub-project I would still prefer it being an Iceberg Subproject since >>>>>>>>> we >>>>>>>>> are engine/file-format agnostic. >>>>>>>>> >>>>>>>>> 3. I support adding just Variant. >>>>>>>>> >>>>>>>>> On Thu, Jul 18, 2024 at 12:54 AM Aihua Xu <aihu...@apache.org> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hello community, >>>>>>>>>> >>>>>>>>>> It’s great to sync up with some of you on Variant and >>>>>>>>>> SubColumarization support in Iceberg again. Apologize that I didn’t >>>>>>>>>> record >>>>>>>>>> the meeting but here are some key items that we want to follow up >>>>>>>>>> with the >>>>>>>>>> community. >>>>>>>>>> >>>>>>>>>> 1. Adopt Spark Variant encoding >>>>>>>>>> Those present were in favor of adopting the Spark variant >>>>>>>>>> encoding for Iceberg Variant with extensions to support other Iceberg >>>>>>>>>> types. We would like to know if anyone has an objection to this to >>>>>>>>>> reuse an >>>>>>>>>> open source encoding. >>>>>>>>>> >>>>>>>>>> 2. Movement of the Spark Variant Spec to another project >>>>>>>>>> To avoid introducing Apache Spark as a dependency for the engines >>>>>>>>>> and file formats, we discussed separating Spark Variant encoding >>>>>>>>>> spec and >>>>>>>>>> implementation from the Spark Project to a neutral location. We >>>>>>>>>> thought up >>>>>>>>>> several solutions but didn’t have consensus on any of them. We are >>>>>>>>>> looking >>>>>>>>>> for more feedback on this topic from the community either in terms of >>>>>>>>>> support for one of these options or another idea on how to support >>>>>>>>>> the spec. >>>>>>>>>> >>>>>>>>>> Options Proposed: >>>>>>>>>> * Leave the Spec in Spark (Difficult for versioning and other >>>>>>>>>> engines) >>>>>>>>>> * Copying the Spec into Iceberg Project Directly (Difficult for >>>>>>>>>> other Table Formats) >>>>>>>>>> * Creating a Sub-Project of Apache Iceberg and moving the spec >>>>>>>>>> and reference implementation there (Logistically complicated) >>>>>>>>>> * Creating a Sub-Project of Apache Spark and moving the spec and >>>>>>>>>> reference implementation there (Logistically complicated) >>>>>>>>>> >>>>>>>>>> 3. Add Variant type vs. Variant and JSON types >>>>>>>>>> Those who were present were in favor of adding only the Variant >>>>>>>>>> type to Iceberg. We are looking for anyone who has an objection to >>>>>>>>>> going >>>>>>>>>> forward with just the Variant Type and no Iceberg JSON Type. We were >>>>>>>>>> favoring adding Variant type only because: >>>>>>>>>> * Introducing a JSON type would require engines that only support >>>>>>>>>> VARIANT to do write time validation of their input to a JSON column. >>>>>>>>>> If >>>>>>>>>> they don’t have a JSON type an engine wouldn’t support this. >>>>>>>>>> * Engines which don’t support Variant will work most of the time >>>>>>>>>> but can have fallback strings defined in the spec for reading >>>>>>>>>> unsupported >>>>>>>>>> types. Writing a JSON into a Variant will always work. >>>>>>>>>> >>>>>>>>>> 4. Support for Subcolumnization spec (shredding in Spark) >>>>>>>>>> We have no action items on this but would like to follow up on >>>>>>>>>> discussions on Subcolumnization in the future. >>>>>>>>>> * We had general agreement that this should be included in >>>>>>>>>> Iceberg V3 or else adding variant may not be useful. >>>>>>>>>> * We are interested in also adopting the shredding spec from >>>>>>>>>> Spark and would like to move it to whatever place we decided the >>>>>>>>>> Variant >>>>>>>>>> spec is going to live. >>>>>>>>>> >>>>>>>>>> Let us know if missed anything and if you have any additional >>>>>>>>>> thoughts or suggestions. >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> Aihua >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 2024/07/15 18:32:22 Aihua Xu wrote: >>>>>>>>>> > Thanks for the discussion. >>>>>>>>>> > >>>>>>>>>> > I will move forward to work on spec PR. >>>>>>>>>> > >>>>>>>>>> > Regarding the implementation, we will have module for Variant >>>>>>>>>> support in Iceberg so we will not have to bring in Spark libraries. >>>>>>>>>> > >>>>>>>>>> > I'm reposting the meeting invite in case it's not clear in my >>>>>>>>>> original email since I included in the end. Looks like we don't have >>>>>>>>>> major >>>>>>>>>> objections/diverges but let's sync up and have consensus. >>>>>>>>>> > >>>>>>>>>> > Meeting invite: >>>>>>>>>> > >>>>>>>>>> > Wednesday, July 17 · 9:00 – 10:00am >>>>>>>>>> > Time zone: America/Los_Angeles >>>>>>>>>> > Google Meet joining info >>>>>>>>>> > Video call link: https://meet.google.com/pbm-ovzn-aoq >>>>>>>>>> > Or dial: (US) +1 650-449-9343 PIN: 170 576 525# >>>>>>>>>> > More phone numbers: >>>>>>>>>> https://tel.meet/pbm-ovzn-aoq?pin=4079632691790 >>>>>>>>>> > >>>>>>>>>> > Thanks, >>>>>>>>>> > Aihua >>>>>>>>>> > >>>>>>>>>> > On 2024/07/12 20:55:01 Micah Kornfield wrote: >>>>>>>>>> > > I don't think this needs to hold up the PR but I think coming >>>>>>>>>> to a >>>>>>>>>> > > consensus on the exact set of types supported is worthwhile >>>>>>>>>> (and if the >>>>>>>>>> > > goal is to maintain the same set as specified by the Spark >>>>>>>>>> Variant type or >>>>>>>>>> > > if divergence is expected/allowed). From a fragmentation >>>>>>>>>> perspective it >>>>>>>>>> > > would be a shame if they diverge, so maybe a next step is >>>>>>>>>> also suggesting >>>>>>>>>> > > support to the Spark community on the missing existing >>>>>>>>>> Iceberg types? >>>>>>>>>> > > >>>>>>>>>> > > Thanks, >>>>>>>>>> > > Micah >>>>>>>>>> > > >>>>>>>>>> > > On Fri, Jul 12, 2024 at 1:44 PM Russell Spitzer < >>>>>>>>>> russell.spit...@gmail.com> >>>>>>>>>> > > wrote: >>>>>>>>>> > > >>>>>>>>>> > > > Just talked with Aihua and he's working on the Spec PR now. >>>>>>>>>> We can get >>>>>>>>>> > > > feedback there from everyone. >>>>>>>>>> > > > >>>>>>>>>> > > > On Fri, Jul 12, 2024 at 3:41 PM Ryan Blue >>>>>>>>>> <b...@databricks.com.invalid> >>>>>>>>>> > > > wrote: >>>>>>>>>> > > > >>>>>>>>>> > > >> Good idea, but I'm hoping that we can continue to get >>>>>>>>>> their feedback in >>>>>>>>>> > > >> parallel to getting the spec changes started. Piotr didn't >>>>>>>>>> seem to object >>>>>>>>>> > > >> to the encoding from what I read of his comments. >>>>>>>>>> Hopefully he (and others) >>>>>>>>>> > > >> chime in here. >>>>>>>>>> > > >> >>>>>>>>>> > > >> On Fri, Jul 12, 2024 at 1:32 PM Russell Spitzer < >>>>>>>>>> > > >> russell.spit...@gmail.com> wrote: >>>>>>>>>> > > >> >>>>>>>>>> > > >>> I just want to make sure we get Piotr and Peter on board >>>>>>>>>> as >>>>>>>>>> > > >>> representatives of Flink and Trino engines. Also make >>>>>>>>>> sure we have anyone >>>>>>>>>> > > >>> else chime in who has experience with Ray if possible. >>>>>>>>>> > > >>> >>>>>>>>>> > > >>> Spec changes feel like the right next step. >>>>>>>>>> > > >>> >>>>>>>>>> > > >>> On Fri, Jul 12, 2024 at 3:14 PM Ryan Blue >>>>>>>>>> <b...@databricks.com.invalid> >>>>>>>>>> > > >>> wrote: >>>>>>>>>> > > >>> >>>>>>>>>> > > >>>> Okay, what are the next steps here? This proposal has >>>>>>>>>> been out for >>>>>>>>>> > > >>>> quite a while and I don't see any major objections to >>>>>>>>>> using the Spark >>>>>>>>>> > > >>>> encoding. It's quite well designed and fits the need >>>>>>>>>> well. It can also be >>>>>>>>>> > > >>>> extended to support additional types that are missing if >>>>>>>>>> that's a priority. >>>>>>>>>> > > >>>> >>>>>>>>>> > > >>>> Should we move forward by starting a draft of the >>>>>>>>>> changes to the table >>>>>>>>>> > > >>>> spec? Then we can vote on committing those changes and >>>>>>>>>> get moving on an >>>>>>>>>> > > >>>> implementation (or possibly do the implementation in >>>>>>>>>> parallel). >>>>>>>>>> > > >>>> >>>>>>>>>> > > >>>> On Fri, Jul 12, 2024 at 1:08 PM Russell Spitzer < >>>>>>>>>> > > >>>> russell.spit...@gmail.com> wrote: >>>>>>>>>> > > >>>> >>>>>>>>>> > > >>>>> That's fair, I'm sold on an Iceberg Module. >>>>>>>>>> > > >>>>> >>>>>>>>>> > > >>>>> On Fri, Jul 12, 2024 at 2:53 PM Ryan Blue >>>>>>>>>> <b...@databricks.com.invalid> >>>>>>>>>> > > >>>>> wrote: >>>>>>>>>> > > >>>>> >>>>>>>>>> > > >>>>>> > Feels like eventually the encoding should land in >>>>>>>>>> parquet proper >>>>>>>>>> > > >>>>>> right? >>>>>>>>>> > > >>>>>> >>>>>>>>>> > > >>>>>> What about using it in ORC? I don't know where it >>>>>>>>>> should end up. >>>>>>>>>> > > >>>>>> Maybe Iceberg should make a standalone module from it? >>>>>>>>>> > > >>>>>> >>>>>>>>>> > > >>>>>> On Fri, Jul 12, 2024 at 12:38 PM Russell Spitzer < >>>>>>>>>> > > >>>>>> russell.spit...@gmail.com> wrote: >>>>>>>>>> > > >>>>>> >>>>>>>>>> > > >>>>>>> Feels like eventually the encoding should land in >>>>>>>>>> parquet proper >>>>>>>>>> > > >>>>>>> right? I'm fine with us just copying into Iceberg >>>>>>>>>> though for the time >>>>>>>>>> > > >>>>>>> being. >>>>>>>>>> > > >>>>>>> >>>>>>>>>> > > >>>>>>> On Fri, Jul 12, 2024 at 2:31 PM Ryan Blue >>>>>>>>>> > > >>>>>>> <b...@databricks.com.invalid> wrote: >>>>>>>>>> > > >>>>>>> >>>>>>>>>> > > >>>>>>>> Oops, it looks like I missed where Aihua brought >>>>>>>>>> this up in his >>>>>>>>>> > > >>>>>>>> last email: >>>>>>>>>> > > >>>>>>>> >>>>>>>>>> > > >>>>>>>> > do we have an issue to directly use Spark >>>>>>>>>> implementation in >>>>>>>>>> > > >>>>>>>> Iceberg? >>>>>>>>>> > > >>>>>>>> >>>>>>>>>> > > >>>>>>>> Yes, I think that we do have an issue using the >>>>>>>>>> Spark library. What >>>>>>>>>> > > >>>>>>>> do you think about a Java implementation in Iceberg? >>>>>>>>>> > > >>>>>>>> >>>>>>>>>> > > >>>>>>>> Ryan >>>>>>>>>> > > >>>>>>>> >>>>>>>>>> > > >>>>>>>> On Fri, Jul 12, 2024 at 12:28 PM Ryan Blue < >>>>>>>>>> b...@databricks.com> >>>>>>>>>> > > >>>>>>>> wrote: >>>>>>>>>> > > >>>>>>>> >>>>>>>>>> > > >>>>>>>>> I raised the same point from Peter's email in a >>>>>>>>>> comment on the doc >>>>>>>>>> > > >>>>>>>>> as well. There is a spark-variant_2.13 artifact >>>>>>>>>> that would be a much >>>>>>>>>> > > >>>>>>>>> smaller scope than relying on large portions of >>>>>>>>>> Spark, but I even then I >>>>>>>>>> > > >>>>>>>>> doubt that it is a good idea for Iceberg to depend >>>>>>>>>> on that because it is a >>>>>>>>>> > > >>>>>>>>> Scala artifact and we would need to bring in a ton >>>>>>>>>> of Scala libs. I think >>>>>>>>>> > > >>>>>>>>> what makes the most sense is to have an independent >>>>>>>>>> implementation of the >>>>>>>>>> > > >>>>>>>>> spec in Iceberg. >>>>>>>>>> > > >>>>>>>>> >>>>>>>>>> > > >>>>>>>>> On Fri, Jul 12, 2024 at 11:51 AM Péter Váry < >>>>>>>>>> > > >>>>>>>>> peter.vary.apa...@gmail.com> wrote: >>>>>>>>>> > > >>>>>>>>> >>>>>>>>>> > > >>>>>>>>>> Hi Aihua, >>>>>>>>>> > > >>>>>>>>>> Long time no see :) >>>>>>>>>> > > >>>>>>>>>> Would this mean, that every engine which plans to >>>>>>>>>> support Variant >>>>>>>>>> > > >>>>>>>>>> data type needs to add Spark as a dependency? Like >>>>>>>>>> Flink/Trino/Hive etc? >>>>>>>>>> > > >>>>>>>>>> Thanks, Peter >>>>>>>>>> > > >>>>>>>>>> >>>>>>>>>> > > >>>>>>>>>> >>>>>>>>>> > > >>>>>>>>>> On Fri, Jul 12, 2024, 19:10 Aihua Xu < >>>>>>>>>> aihu...@apache.org> wrote: >>>>>>>>>> > > >>>>>>>>>> >>>>>>>>>> > > >>>>>>>>>>> Thanks Ryan. >>>>>>>>>> > > >>>>>>>>>>> >>>>>>>>>> > > >>>>>>>>>>> Yeah. That's another reason we want to pursue >>>>>>>>>> Spark encoding to >>>>>>>>>> > > >>>>>>>>>>> keep compatibility for the open source engines. >>>>>>>>>> > > >>>>>>>>>>> >>>>>>>>>> > > >>>>>>>>>>> One more question regarding the encoding >>>>>>>>>> implementation: do we >>>>>>>>>> > > >>>>>>>>>>> have an issue to directly use Spark >>>>>>>>>> implementation in Iceberg? Russell >>>>>>>>>> > > >>>>>>>>>>> pointed out that Trino doesn't have Spark >>>>>>>>>> dependency and that could be a >>>>>>>>>> > > >>>>>>>>>>> problem? >>>>>>>>>> > > >>>>>>>>>>> >>>>>>>>>> > > >>>>>>>>>>> Thanks, >>>>>>>>>> > > >>>>>>>>>>> Aihua >>>>>>>>>> > > >>>>>>>>>>> >>>>>>>>>> > > >>>>>>>>>>> On 2024/07/12 15:02:06 Ryan Blue wrote: >>>>>>>>>> > > >>>>>>>>>>> > Thanks, Aihua! >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>>>>> > > >>>>>>>>>>> > I think that the encoding choice in the current >>>>>>>>>> doc is a good >>>>>>>>>> > > >>>>>>>>>>> one. I went >>>>>>>>>> > > >>>>>>>>>>> > through the Spark encoding in detail and it >>>>>>>>>> looks like a >>>>>>>>>> > > >>>>>>>>>>> better choice than >>>>>>>>>> > > >>>>>>>>>>> > the other candidate encodings for quickly >>>>>>>>>> accessing nested >>>>>>>>>> > > >>>>>>>>>>> fields. >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>>>>> > > >>>>>>>>>>> > Another reason to use the Spark type is that >>>>>>>>>> this is what >>>>>>>>>> > > >>>>>>>>>>> Delta's variant >>>>>>>>>> > > >>>>>>>>>>> > type is based on, so Parquet files in tables >>>>>>>>>> written by Delta >>>>>>>>>> > > >>>>>>>>>>> could be >>>>>>>>>> > > >>>>>>>>>>> > converted or used in Iceberg tables without >>>>>>>>>> needing to rewrite >>>>>>>>>> > > >>>>>>>>>>> variant >>>>>>>>>> > > >>>>>>>>>>> > data. (Also, note that I work at Databricks and >>>>>>>>>> have an >>>>>>>>>> > > >>>>>>>>>>> interest in >>>>>>>>>> > > >>>>>>>>>>> > increasing format compatibility.) >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>>>>> > > >>>>>>>>>>> > Ryan >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>>>>> > > >>>>>>>>>>> > On Thu, Jul 11, 2024 at 11:21 AM Aihua Xu < >>>>>>>>>> > > >>>>>>>>>>> aihua...@snowflake.com.invalid> >>>>>>>>>> > > >>>>>>>>>>> > wrote: >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>>>>> > > >>>>>>>>>>> > > [Discuss] Consensus for Variant Encoding >>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>>> > > It’s great to be able to present the Variant >>>>>>>>>> type proposal >>>>>>>>>> > > >>>>>>>>>>> in the >>>>>>>>>> > > >>>>>>>>>>> > > community sync yesterday and I’m looking to >>>>>>>>>> host a meeting >>>>>>>>>> > > >>>>>>>>>>> next week >>>>>>>>>> > > >>>>>>>>>>> > > (targeting for 9am, July 17th) to go over any >>>>>>>>>> further >>>>>>>>>> > > >>>>>>>>>>> concerns about the >>>>>>>>>> > > >>>>>>>>>>> > > encoding of the Variant type and any other >>>>>>>>>> questions on the >>>>>>>>>> > > >>>>>>>>>>> first phase of >>>>>>>>>> > > >>>>>>>>>>> > > the proposal >>>>>>>>>> > > >>>>>>>>>>> > > < >>>>>>>>>> > > >>>>>>>>>>> >>>>>>>>>> https://docs.google.com/document/d/1QjhpG_SVNPZh3anFcpicMQx90ebwjL7rmzFYfUP89Iw/edit >>>>>>>>>> > > >>>>>>>>>>> >. >>>>>>>>>> > > >>>>>>>>>>> > > We are hoping that anyone who is interested >>>>>>>>>> in the proposal >>>>>>>>>> > > >>>>>>>>>>> can either join >>>>>>>>>> > > >>>>>>>>>>> > > or reply with their comments so we can >>>>>>>>>> discuss them. Summary >>>>>>>>>> > > >>>>>>>>>>> of the >>>>>>>>>> > > >>>>>>>>>>> > > discussion and notes will be sent to the >>>>>>>>>> mailing list for >>>>>>>>>> > > >>>>>>>>>>> further comment >>>>>>>>>> > > >>>>>>>>>>> > > there. >>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>>> > > - >>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>>> > > What should be the underlying binary >>>>>>>>>> representation >>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>>> > > We have evaluated a few encodings in the doc >>>>>>>>>> including ION, >>>>>>>>>> > > >>>>>>>>>>> JSONB, and >>>>>>>>>> > > >>>>>>>>>>> > > Spark encoding.Choosing the underlying >>>>>>>>>> encoding is an >>>>>>>>>> > > >>>>>>>>>>> important first step >>>>>>>>>> > > >>>>>>>>>>> > > here and we believe we have general support >>>>>>>>>> for Spark’s >>>>>>>>>> > > >>>>>>>>>>> Variant encoding. >>>>>>>>>> > > >>>>>>>>>>> > > We would like to hear if anyone else has >>>>>>>>>> strong opinions in >>>>>>>>>> > > >>>>>>>>>>> this space. >>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>>> > > - >>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>>> > > Should we support multiple logical types >>>>>>>>>> or just Variant? >>>>>>>>>> > > >>>>>>>>>>> Variant vs. >>>>>>>>>> > > >>>>>>>>>>> > > Variant + JSON. >>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>>> > > This is to discuss what logical data type(s) >>>>>>>>>> to be supported >>>>>>>>>> > > >>>>>>>>>>> in Iceberg - >>>>>>>>>> > > >>>>>>>>>>> > > Variant only vs. Variant + JSON. Both types >>>>>>>>>> would share the >>>>>>>>>> > > >>>>>>>>>>> same underlying >>>>>>>>>> > > >>>>>>>>>>> > > encoding but would imply different >>>>>>>>>> limitations on engines >>>>>>>>>> > > >>>>>>>>>>> working with >>>>>>>>>> > > >>>>>>>>>>> > > those types. >>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>>> > > From the sync up meeting, we are more >>>>>>>>>> favoring toward >>>>>>>>>> > > >>>>>>>>>>> supporting Variant >>>>>>>>>> > > >>>>>>>>>>> > > only and we want to have a consensus on the >>>>>>>>>> supported >>>>>>>>>> > > >>>>>>>>>>> type(s). >>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>>> > > - >>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>>> > > How should we move forward with >>>>>>>>>> Subcolumnization? >>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>>> > > Subcolumnization is an optimization for >>>>>>>>>> Variant type by >>>>>>>>>> > > >>>>>>>>>>> separating out >>>>>>>>>> > > >>>>>>>>>>> > > subcolumns with their own metadata. This is >>>>>>>>>> not critical for >>>>>>>>>> > > >>>>>>>>>>> choosing the >>>>>>>>>> > > >>>>>>>>>>> > > initial encoding of the Variant type so we >>>>>>>>>> were hoping to >>>>>>>>>> > > >>>>>>>>>>> gain consensus on >>>>>>>>>> > > >>>>>>>>>>> > > leaving that for a follow up spec. >>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>>> > > Thanks >>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>>> > > Aihua >>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>>> > > Meeting invite: >>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>>> > > Wednesday, July 17 · 9:00 – 10:00am >>>>>>>>>> > > >>>>>>>>>>> > > Time zone: America/Los_Angeles >>>>>>>>>> > > >>>>>>>>>>> > > Google Meet joining info >>>>>>>>>> > > >>>>>>>>>>> > > Video call link: >>>>>>>>>> https://meet.google.com/pbm-ovzn-aoq >>>>>>>>>> > > >>>>>>>>>>> > > Or dial: (US) +1 650-449-9343 PIN: 170 576 >>>>>>>>>> 525# >>>>>>>>>> > > >>>>>>>>>>> > > More phone numbers: >>>>>>>>>> > > >>>>>>>>>>> https://tel.meet/pbm-ovzn-aoq?pin=4079632691790 >>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>>> > > On Tue, May 28, 2024 at 9:21 PM Aihua Xu < >>>>>>>>>> > > >>>>>>>>>>> aihua...@snowflake.com> wrote: >>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>>> > >> Hello, >>>>>>>>>> > > >>>>>>>>>>> > >> >>>>>>>>>> > > >>>>>>>>>>> > >> We have drafted the proposal >>>>>>>>>> > > >>>>>>>>>>> > >> < >>>>>>>>>> > > >>>>>>>>>>> >>>>>>>>>> https://docs.google.com/document/d/1QjhpG_SVNPZh3anFcpicMQx90ebwjL7rmzFYfUP89Iw/edit >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>>>>> > > >>>>>>>>>>> > >> for Variant data type. Please help review >>>>>>>>>> and comment. >>>>>>>>>> > > >>>>>>>>>>> > >> >>>>>>>>>> > > >>>>>>>>>>> > >> Thanks, >>>>>>>>>> > > >>>>>>>>>>> > >> Aihua >>>>>>>>>> > > >>>>>>>>>>> > >> >>>>>>>>>> > > >>>>>>>>>>> > >> On Thu, May 16, 2024 at 12:45 PM Jack Ye < >>>>>>>>>> > > >>>>>>>>>>> yezhao...@gmail.com> wrote: >>>>>>>>>> > > >>>>>>>>>>> > >> >>>>>>>>>> > > >>>>>>>>>>> > >>> +10000 for a JSON/BSON type. We also had >>>>>>>>>> the same >>>>>>>>>> > > >>>>>>>>>>> discussion internally >>>>>>>>>> > > >>>>>>>>>>> > >>> and a JSON type would really play well with >>>>>>>>>> for example >>>>>>>>>> > > >>>>>>>>>>> the SUPER type in >>>>>>>>>> > > >>>>>>>>>>> > >>> Redshift: >>>>>>>>>> > > >>>>>>>>>>> > >>> >>>>>>>>>> > > >>>>>>>>>>> >>>>>>>>>> https://docs.aws.amazon.com/redshift/latest/dg/r_SUPER_type.html, >>>>>>>>>> > > >>>>>>>>>>> and >>>>>>>>>> > > >>>>>>>>>>> > >>> can also provide better integration with >>>>>>>>>> the Trino JSON >>>>>>>>>> > > >>>>>>>>>>> type. >>>>>>>>>> > > >>>>>>>>>>> > >>> >>>>>>>>>> > > >>>>>>>>>>> > >>> Looking forward to the proposal! >>>>>>>>>> > > >>>>>>>>>>> > >>> >>>>>>>>>> > > >>>>>>>>>>> > >>> Best, >>>>>>>>>> > > >>>>>>>>>>> > >>> Jack Ye >>>>>>>>>> > > >>>>>>>>>>> > >>> >>>>>>>>>> > > >>>>>>>>>>> > >>> >>>>>>>>>> > > >>>>>>>>>>> > >>> On Wed, May 15, 2024 at 9:37 AM Tyler Akidau >>>>>>>>>> > > >>>>>>>>>>> > >>> <tyler.aki...@snowflake.com.invalid> wrote: >>>>>>>>>> > > >>>>>>>>>>> > >>> >>>>>>>>>> > > >>>>>>>>>>> > >>>> On Tue, May 14, 2024 at 7:58 PM Gang Wu < >>>>>>>>>> ust...@gmail.com> >>>>>>>>>> > > >>>>>>>>>>> wrote: >>>>>>>>>> > > >>>>>>>>>>> > >>>> >>>>>>>>>> > > >>>>>>>>>>> > >>>>> > We may need some guidance on just how >>>>>>>>>> many we need to >>>>>>>>>> > > >>>>>>>>>>> look at; >>>>>>>>>> > > >>>>>>>>>>> > >>>>> > we were planning on Spark and Trino, >>>>>>>>>> but weren't sure >>>>>>>>>> > > >>>>>>>>>>> how much >>>>>>>>>> > > >>>>>>>>>>> > >>>>> > further down the rabbit hole we needed >>>>>>>>>> to go。 >>>>>>>>>> > > >>>>>>>>>>> > >>>>> >>>>>>>>>> > > >>>>>>>>>>> > >>>>> There are some engines living outside the >>>>>>>>>> Java world. It >>>>>>>>>> > > >>>>>>>>>>> would be >>>>>>>>>> > > >>>>>>>>>>> > >>>>> good if the proposal could cover the >>>>>>>>>> effort it takes to >>>>>>>>>> > > >>>>>>>>>>> integrate >>>>>>>>>> > > >>>>>>>>>>> > >>>>> variant type to them (e.g. velox, >>>>>>>>>> datafusion, etc.). >>>>>>>>>> > > >>>>>>>>>>> This is something >>>>>>>>>> > > >>>>>>>>>>> > >>>>> that >>>>>>>>>> > > >>>>>>>>>>> > >>>>> some proprietary iceberg vendors also >>>>>>>>>> care about. >>>>>>>>>> > > >>>>>>>>>>> > >>>>> >>>>>>>>>> > > >>>>>>>>>>> > >>>> >>>>>>>>>> > > >>>>>>>>>>> > >>>> Ack, makes sense. We can make sure to >>>>>>>>>> share some >>>>>>>>>> > > >>>>>>>>>>> perspective on this. >>>>>>>>>> > > >>>>>>>>>>> > >>>> >>>>>>>>>> > > >>>>>>>>>>> > >>>> > Not necessarily, no. As long as there's >>>>>>>>>> a binary type >>>>>>>>>> > > >>>>>>>>>>> and Iceberg and >>>>>>>>>> > > >>>>>>>>>>> > >>>>> > the query engines are aware that the >>>>>>>>>> binary column >>>>>>>>>> > > >>>>>>>>>>> needs to be >>>>>>>>>> > > >>>>>>>>>>> > >>>>> > interpreted as a variant, that should >>>>>>>>>> be sufficient. >>>>>>>>>> > > >>>>>>>>>>> > >>>>> >>>>>>>>>> > > >>>>>>>>>>> > >>>>> From the perspective of interoperability, >>>>>>>>>> it would be >>>>>>>>>> > > >>>>>>>>>>> good to support >>>>>>>>>> > > >>>>>>>>>>> > >>>>> native >>>>>>>>>> > > >>>>>>>>>>> > >>>>> type from file specs. Life will be easier >>>>>>>>>> for projects >>>>>>>>>> > > >>>>>>>>>>> like Apache >>>>>>>>>> > > >>>>>>>>>>> > >>>>> XTable. >>>>>>>>>> > > >>>>>>>>>>> > >>>>> File format could also provide >>>>>>>>>> finer-grained statistics >>>>>>>>>> > > >>>>>>>>>>> for variant >>>>>>>>>> > > >>>>>>>>>>> > >>>>> type which >>>>>>>>>> > > >>>>>>>>>>> > >>>>> facilitates data skipping. >>>>>>>>>> > > >>>>>>>>>>> > >>>>> >>>>>>>>>> > > >>>>>>>>>>> > >>>> >>>>>>>>>> > > >>>>>>>>>>> > >>>> Agreed, there can definitely be additional >>>>>>>>>> value in >>>>>>>>>> > > >>>>>>>>>>> native file format >>>>>>>>>> > > >>>>>>>>>>> > >>>> integration. Just wanted to highlight that >>>>>>>>>> it's not a >>>>>>>>>> > > >>>>>>>>>>> strict requirement. >>>>>>>>>> > > >>>>>>>>>>> > >>>> >>>>>>>>>> > > >>>>>>>>>>> > >>>> -Tyler >>>>>>>>>> > > >>>>>>>>>>> > >>>> >>>>>>>>>> > > >>>>>>>>>>> > >>>> >>>>>>>>>> > > >>>>>>>>>>> > >>>>> >>>>>>>>>> > > >>>>>>>>>>> > >>>>> Gang >>>>>>>>>> > > >>>>>>>>>>> > >>>>> >>>>>>>>>> > > >>>>>>>>>>> > >>>>> On Wed, May 15, 2024 at 6:49 AM Tyler >>>>>>>>>> Akidau >>>>>>>>>> > > >>>>>>>>>>> > >>>>> <tyler.aki...@snowflake.com.invalid> >>>>>>>>>> wrote: >>>>>>>>>> > > >>>>>>>>>>> > >>>>> >>>>>>>>>> > > >>>>>>>>>>> > >>>>>> Good to see you again as well, JB! >>>>>>>>>> Thanks! >>>>>>>>>> > > >>>>>>>>>>> > >>>>>> >>>>>>>>>> > > >>>>>>>>>>> > >>>>>> -Tyler >>>>>>>>>> > > >>>>>>>>>>> > >>>>>> >>>>>>>>>> > > >>>>>>>>>>> > >>>>>> >>>>>>>>>> > > >>>>>>>>>>> > >>>>>> On Tue, May 14, 2024 at 1:04 PM >>>>>>>>>> Jean-Baptiste Onofré < >>>>>>>>>> > > >>>>>>>>>>> j...@nanthrax.net> >>>>>>>>>> > > >>>>>>>>>>> > >>>>>> wrote: >>>>>>>>>> > > >>>>>>>>>>> > >>>>>> >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> Hi Tyler, >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> Super happy to see you there :) It >>>>>>>>>> reminds me our >>>>>>>>>> > > >>>>>>>>>>> discussions back in >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> the start of Apache Beam :) >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> Anyway, the thread is pretty >>>>>>>>>> interesting. I remember >>>>>>>>>> > > >>>>>>>>>>> some discussions >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> about JSON datatype for spec v3. The >>>>>>>>>> binary data type >>>>>>>>>> > > >>>>>>>>>>> is already >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> supported in the spec v2. >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> I'm looking forward to the proposal and >>>>>>>>>> happy to help >>>>>>>>>> > > >>>>>>>>>>> on this ! >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> Regards >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> JB >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> On Sat, May 11, 2024 at 7:06 AM Tyler >>>>>>>>>> Akidau >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> <tyler.aki...@snowflake.com.invalid> >>>>>>>>>> wrote: >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> > >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> > Hello, >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> > >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> > We (Tyler, Nileema, Selcuk, Aihua) >>>>>>>>>> are working on a >>>>>>>>>> > > >>>>>>>>>>> proposal for >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> which we’d like to get early feedback >>>>>>>>>> from the >>>>>>>>>> > > >>>>>>>>>>> community. As you may know, >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> Snowflake has embraced Iceberg as its >>>>>>>>>> open Data Lake >>>>>>>>>> > > >>>>>>>>>>> format. Having made >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> good progress on our own adoption of >>>>>>>>>> the Iceberg >>>>>>>>>> > > >>>>>>>>>>> standard, we’re now in a >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> position where there are features not >>>>>>>>>> yet supported in >>>>>>>>>> > > >>>>>>>>>>> Iceberg which we >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> think would be valuable for our users, >>>>>>>>>> and that we >>>>>>>>>> > > >>>>>>>>>>> would like to discuss >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> with and help contribute to the Iceberg >>>>>>>>>> community. >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> > >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> > The first two such features we’d like >>>>>>>>>> to discuss are >>>>>>>>>> > > >>>>>>>>>>> in support of >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> efficient querying of dynamically typed, >>>>>>>>>> > > >>>>>>>>>>> semi-structured data: variant data >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> types, and subcolumnarization of >>>>>>>>>> variant columns. In >>>>>>>>>> > > >>>>>>>>>>> more detail, for >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> anyone who may not already be familiar: >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> > >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> > 1. Variant data types >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> > Variant types allow for the efficient >>>>>>>>>> binary >>>>>>>>>> > > >>>>>>>>>>> encoding of dynamic >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> semi-structured data such as JSON, >>>>>>>>>> Avro, etc. By >>>>>>>>>> > > >>>>>>>>>>> encoding semi-structured >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> data as a variant column, we retain the >>>>>>>>>> flexibility of >>>>>>>>>> > > >>>>>>>>>>> the source data, >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> while allowing query engines to more >>>>>>>>>> efficiently >>>>>>>>>> > > >>>>>>>>>>> operate on the data. >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> Snowflake has supported the variant >>>>>>>>>> data type on >>>>>>>>>> > > >>>>>>>>>>> Snowflake tables for many >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> years [1]. As more and more users >>>>>>>>>> utilize Iceberg >>>>>>>>>> > > >>>>>>>>>>> tables in Snowflake, >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> we’re hearing an increasing chorus of >>>>>>>>>> requests for >>>>>>>>>> > > >>>>>>>>>>> variant support. >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> Additionally, other query engines such >>>>>>>>>> as Apache Spark >>>>>>>>>> > > >>>>>>>>>>> have begun adding >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> variant support [2]. As such, we >>>>>>>>>> believe it would be >>>>>>>>>> > > >>>>>>>>>>> beneficial to the >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> Iceberg community as a whole to >>>>>>>>>> standardize on the >>>>>>>>>> > > >>>>>>>>>>> variant data type >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> encoding used across Iceberg tables. >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> > >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> > One specific point to make here is >>>>>>>>>> that, since an >>>>>>>>>> > > >>>>>>>>>>> Apache OSS >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> version of variant encoding already >>>>>>>>>> exists in Spark, >>>>>>>>>> > > >>>>>>>>>>> it likely makes sense >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> to simply adopt the Spark encoding as >>>>>>>>>> the Iceberg >>>>>>>>>> > > >>>>>>>>>>> standard as well. The >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> encoding we use internally today in >>>>>>>>>> Snowflake is >>>>>>>>>> > > >>>>>>>>>>> slightly different, but >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> essentially equivalent, and we see no >>>>>>>>>> particular value >>>>>>>>>> > > >>>>>>>>>>> in trying to clutter >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> the space with another >>>>>>>>>> equivalent-but-incompatible >>>>>>>>>> > > >>>>>>>>>>> encoding. >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> > >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> > >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> > 2. Subcolumnarization >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> > Subcolumnarization of variant columns >>>>>>>>>> allows query >>>>>>>>>> > > >>>>>>>>>>> engines to >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> efficiently prune datasets when >>>>>>>>>> subcolumns (i.e., >>>>>>>>>> > > >>>>>>>>>>> nested fields) within a >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> variant column are queried, and also >>>>>>>>>> allows optionally >>>>>>>>>> > > >>>>>>>>>>> materializing some >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> of the nested fields as a column on >>>>>>>>>> their own, >>>>>>>>>> > > >>>>>>>>>>> affording queries on these >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> subcolumns the ability to read less >>>>>>>>>> data and spend >>>>>>>>>> > > >>>>>>>>>>> less CPU on extraction. >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> When subcolumnarizing, the system >>>>>>>>>> managing table >>>>>>>>>> > > >>>>>>>>>>> metadata and data tracks >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> individual pruning statistics (min, >>>>>>>>>> max, null, etc.) >>>>>>>>>> > > >>>>>>>>>>> for some subset of the >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> nested fields within a variant, and >>>>>>>>>> also manages any >>>>>>>>>> > > >>>>>>>>>>> optional >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> materialization. Without >>>>>>>>>> subcolumnarization, any query >>>>>>>>>> > > >>>>>>>>>>> which touches a >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> variant column must read, parse, >>>>>>>>>> extract, and filter >>>>>>>>>> > > >>>>>>>>>>> every row for which >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> that column is non-null. Thus, by >>>>>>>>>> providing a >>>>>>>>>> > > >>>>>>>>>>> standardized way of tracking >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> subcolum metadata and data for variant >>>>>>>>>> columns, >>>>>>>>>> > > >>>>>>>>>>> Iceberg can make >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> subcolumnar optimizations accessible >>>>>>>>>> across various >>>>>>>>>> > > >>>>>>>>>>> catalogs and query >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> engines. >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> > >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> > Subcolumnarization is a non-trivial >>>>>>>>>> topic, so we >>>>>>>>>> > > >>>>>>>>>>> expect any >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> concrete proposal to include not only >>>>>>>>>> the set of >>>>>>>>>> > > >>>>>>>>>>> changes to Iceberg >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> metadata that allow compatible query >>>>>>>>>> engines to >>>>>>>>>> > > >>>>>>>>>>> interopate on >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> subcolumnarization data for variant >>>>>>>>>> columns, but also >>>>>>>>>> > > >>>>>>>>>>> reference >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> documentation explaining >>>>>>>>>> subcolumnarization principles >>>>>>>>>> > > >>>>>>>>>>> and recommended best >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> practices. >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> > >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> > >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> > It sounds like the recent Geo >>>>>>>>>> proposal [3] may be a >>>>>>>>>> > > >>>>>>>>>>> good starting >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> point for how to approach this, so our >>>>>>>>>> plan is to >>>>>>>>>> > > >>>>>>>>>>> write something up in >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> that vein that covers the proposed spec >>>>>>>>>> changes, >>>>>>>>>> > > >>>>>>>>>>> backwards compatibility, >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> implementor burdens, etc. But we wanted >>>>>>>>>> to first reach >>>>>>>>>> > > >>>>>>>>>>> out to the community >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> to introduce ourselves and the idea, >>>>>>>>>> and see if >>>>>>>>>> > > >>>>>>>>>>> there’s any early feedback >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> we should incorporate before we spend >>>>>>>>>> too much time on >>>>>>>>>> > > >>>>>>>>>>> a concrete proposal. >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> > >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> > Thank you! >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> > >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> > [1] >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> >>>>>>>>>> > > >>>>>>>>>>> >>>>>>>>>> https://docs.snowflake.com/en/sql-reference/data-types-semistructured >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> > [2] >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> >>>>>>>>>> > > >>>>>>>>>>> >>>>>>>>>> https://github.com/apache/spark/blob/master/common/variant/README.md >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> > [3] >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> >>>>>>>>>> > > >>>>>>>>>>> >>>>>>>>>> https://docs.google.com/document/d/1iVFbrRNEzZl8tDcZC81GFt01QJkLJsI9E2NBOt21IRI/edit >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> > >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> > -Tyler, Nileema, Selcuk, Aihua >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> > >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>> >>>>>>>>>> > > >>>>>>>>>>> > >>>>>> >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>>>>> > > >>>>>>>>>>> > -- >>>>>>>>>> > > >>>>>>>>>>> > Ryan Blue >>>>>>>>>> > > >>>>>>>>>>> > Databricks >>>>>>>>>> > > >>>>>>>>>>> > >>>>>>>>>> > > >>>>>>>>>>> >>>>>>>>>> > > >>>>>>>>>> >>>>>>>>>> > > >>>>>>>>> >>>>>>>>>> > > >>>>>>>>> -- >>>>>>>>>> > > >>>>>>>>> Ryan Blue >>>>>>>>>> > > >>>>>>>>> Databricks >>>>>>>>>> > > >>>>>>>>> >>>>>>>>>> > > >>>>>>>> >>>>>>>>>> > > >>>>>>>> >>>>>>>>>> > > >>>>>>>> -- >>>>>>>>>> > > >>>>>>>> Ryan Blue >>>>>>>>>> > > >>>>>>>> Databricks >>>>>>>>>> > > >>>>>>>> >>>>>>>>>> > > >>>>>>> >>>>>>>>>> > > >>>>>> >>>>>>>>>> > > >>>>>> -- >>>>>>>>>> > > >>>>>> Ryan Blue >>>>>>>>>> > > >>>>>> Databricks >>>>>>>>>> > > >>>>>> >>>>>>>>>> > > >>>>> >>>>>>>>>> > > >>>> >>>>>>>>>> > > >>>> -- >>>>>>>>>> > > >>>> Ryan Blue >>>>>>>>>> > > >>>> Databricks >>>>>>>>>> > > >>>> >>>>>>>>>> > > >>> >>>>>>>>>> > > >> >>>>>>>>>> > > >> -- >>>>>>>>>> > > >> Ryan Blue >>>>>>>>>> > > >> Databricks >>>>>>>>>> > > >> >>>>>>>>>> > > > >>>>>>>>>> > > >>>>>>>>>> > >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Ryan Blue >>>>>>>> Databricks >>>>>>>> >>>>>>>
