Great . Thank you Neelesh. On Thursday, November 5, 2026, Neelesh Salian <[email protected]> wrote:
> Hi folks, > I’ve updated the document to track the Variant work [here > <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?usp=sharing>]. > Please refer to the “Next milestones” section to see what is coming. > I’ve cancelled the call for December due to the holidays, we’ll catch up > in January 2027. > > Regardless of the sync, I’ll continue to keep that document updated as I > go across the communities that work on Variant. > Please don’t hesitate to reach out to me if you have questions around > Variant and how we are progressing. > > On Sun, Sep 6, 2026 at 10:44 AM Neelesh Salian <[email protected]> > wrote: > >> Hi folks, >> >> Here is the Meeting Notes and the Recording link from the Variant Sync on >> September 03, 2026: [Notes >> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?tab=t.g06kaw1fbwhz> >> ]. >> Please reach out if you have any questions. >> Thanks. >> >> On Sun, Aug 9, 2026 at 5:49 PM Neelesh Salian <[email protected]> >> wrote: >> >>> Hi folks, >>> >>> Here is the Meeting Notes and Recording link from the Variant Sync on >>> August 06, 2026: [Notes >>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?tab=t.g06kaw1fbwhz> >>> ]. >>> Please reach out if you have any questions. >>> Thanks. >>> >>> On Thu, Jul 2, 2026 at 8:21 PM Neelesh Salian <[email protected]> >>> wrote: >>> >>>> Hi folks, >>>> >>>> Here is the Meeting Notes and Recording link from the Variant Sync >>>> today July 02, 2026: [Notes >>>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?tab=t.g06kaw1fbwhz> >>>> ]. >>>> There are some action items and active work. >>>> Please reach out if you have any questions. >>>> Thanks >>>> >>>> On Fri, Jun 5, 2026 at 12:22 PM Neelesh Salian < >>>> [email protected]> wrote: >>>> >>>>> Hi folks, >>>>> >>>>> Here is the Meeting Notes and Recording link from the Variant Sync on >>>>> June 4, 2026: [Notes >>>>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?tab=t.g06kaw1fbwhz#heading=h.r977qio1wsv2> >>>>> ]. >>>>> There are some action items and active work. >>>>> Please reach out if you have any questions. >>>>> Thanks. >>>>> >>>>> On Fri, May 8, 2026 at 3:20 PM Neelesh Salian < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi folks, >>>>>> >>>>>> Here is the Meeting Notes and Recording link from the Variant Sync on >>>>>> May 7, 2026: [Notes >>>>>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?tab=t.g06kaw1fbwhz> >>>>>> ]. >>>>>> There are some action items and active work. >>>>>> Please reach out if you have any questions. >>>>>> Thanks. >>>>>> >>>>>> On Thu, Apr 30, 2026 at 1:36 PM Neelesh Salian < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi folks, >>>>>>> >>>>>>> I've set up a time starting next week on Thursday (May 7, 2026) at >>>>>>> 10 am Pacific time for a sync for the active work on Variant. >>>>>>> This will be a monthly sync (on the first Thursday of every month). >>>>>>> You can find it on the dev calendar. >>>>>>> Here is the calendar invite: https://calendar.app. >>>>>>> google/b8ykdTV3EaNnVnkv8 >>>>>>> I'll be recording the call and capturing notes in the sync document: >>>>>>> Iceberg >>>>>>> - Variant Community Update >>>>>>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?usp=sharing> >>>>>>> (Meeting >>>>>>> Notes tab). >>>>>>> Thanks. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Mon, Apr 20, 2026 at 1:49 PM Steve Loughran <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> + regarding the rust, go and cpp impls, a status from each team >>>>>>>> would be great! >>>>>>>> >>>>>>>> I've been reviewing arrow parquet variant stuff and it is all >>>>>>>> there, including with some benchmarks and optimisations. Which may put >>>>>>>> it >>>>>>>> ahead of the others. >>>>>>>> >>>>>>>> It also has some special handling for sorted variants, as key >>>>>>>> search there is straightforward. AFAIK I don't think the others do >>>>>>>> that, >>>>>>>> and nor do I see them going to any effort to sort fields in an object. >>>>>>>> I >>>>>>>> think sorting would be good, but you would have to handle the case >>>>>>>> where >>>>>>>> there are duplicate keys. It's allowed in the spec, and seems like >>>>>>>> itcould >>>>>>>> creep in from nested variants. Has anyone looked at this? >>>>>>>> >>>>>>>> Also: has anyone created malformed parquet files with a shredded >>>>>>>> variant and a metadata entry of the same name. The requirement is >>>>>>>> "ignore >>>>>>>> the metadata one", but that's something to test. You'd have to write a >>>>>>>> shredded file and then edit the binary content to achieve this, or >>>>>>>> manually >>>>>>>> create one and put it into the parquet-testing repository under >>>>>>>> bad-data/ >>>>>>>> >>>>>>>> >>>>>>>> On Mon, 20 Apr 2026 at 19:08, Qiegang Long <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Thanks for the doc to track the status! +1 on the dedicated >>>>>>>>> sync—definitely feels like there’s a lot of work before we see >>>>>>>>> Variant’s >>>>>>>>> full potential. >>>>>>>>> >>>>>>>>> Qiegang >>>>>>>>> >>>>>>>>> On Mon, Apr 20, 2026 at 11:09 AM Steve Loughran < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> This is great, we need that tracker as it is cross-project. piece >>>>>>>>>> of work to say "this is readly >>>>>>>>>> >>>>>>>>>> I did have an agenda item from last month's community call which >>>>>>>>>> didn't get through. If we can retain that open time slot we could do >>>>>>>>>> a very >>>>>>>>>> quick summary of where we are (summarly slides of Qiegang's results >>>>>>>>>> and >>>>>>>>>> mine, key outstanding issues and next steps, then we can start that >>>>>>>>>> monthly session on it. >>>>>>>>>> >>>>>>>>>> Meanwhile, I have both parquet and iceberg PRs for >>>>>>>>>> benchmarks which I think are ready for review -please take a look >>>>>>>>>> >>>>>>>>>> Finally, I'm thinking about interop of those many, many variant >>>>>>>>>> readers out there. Has anyone explicitly cross-tested their >>>>>>>>>> implementations >>>>>>>>>> of variant? what about consistent handling of invalid data? That >>>>>>>>>> includes >>>>>>>>>> iceberg-rust, parquet-cpp and more... >>>>>>>>>> >>>>>>>>>> Steve >>>>>>>>>> >>>>>>>>>> On Sun, 19 Apr 2026 at 21:57, Neelesh Salian < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hi everyone, >>>>>>>>>>> >>>>>>>>>>> The Variant umbrella issue (#10392 >>>>>>>>>>> <https://github.com/apache/iceberg/issues/10392>) hasn't been >>>>>>>>>>> updated in a while, and with active work happening across multiple >>>>>>>>>>> PRs in >>>>>>>>>>> Iceberg, Spark, and Parquet, it's been hard to keep track of where >>>>>>>>>>> things >>>>>>>>>>> stand. >>>>>>>>>>> >>>>>>>>>>> Since a few of us are actively working on variant features, I >>>>>>>>>>> thought it would help to put together a tracking document so the >>>>>>>>>>> community >>>>>>>>>>> has a single place to see the current state, open work, and >>>>>>>>>>> benchmark >>>>>>>>>>> findings. I plan to update this on a weekly basis to keep track of >>>>>>>>>>> the >>>>>>>>>>> issues and PRs that are updated. >>>>>>>>>>> >>>>>>>>>>> Iceberg Variant Community Document >>>>>>>>>>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?usp=sharing> >>>>>>>>>>> >>>>>>>>>>> The document has three tabs: >>>>>>>>>>> >>>>>>>>>>> 1. Overview - what shipped in 1.10, what's merged to main, >>>>>>>>>>> open work areas, and the dependency graph across Iceberg, Spark, >>>>>>>>>>> and Parquet >>>>>>>>>>> 2. Tracker - all open variant issues and PRs across Iceberg, >>>>>>>>>>> Parquet-Java, Parquet-Format, and Spark with authors and status >>>>>>>>>>> 3. Benchmarks - summary of three independent benchmark >>>>>>>>>>> efforts (details below) >>>>>>>>>>> >>>>>>>>>>> *Benchmark findings* >>>>>>>>>>> >>>>>>>>>>> Three independent benchmarks have measured variant performance. >>>>>>>>>>> All converge on the same picture: variant is a modest improvement >>>>>>>>>>> over JSON >>>>>>>>>>> strings today (1.1-1.7x faster reads), but 15-17x slower than typed >>>>>>>>>>> columns. >>>>>>>>>>> >>>>>>>>>>> 1. Qiegang Long - 14 queries on GitHub Archive, 5 configs: >>>>>>>>>>> https://qlong.github.io/posts/2026-03-30-variant-early- >>>>>>>>>>> results >>>>>>>>>>> <https://qlong.github.io/posts/2026-03-30-variant-early-results> >>>>>>>>>>> 2. Steve Loughran - JMH microbenchmarks, profiler-driven >>>>>>>>>>> optimization: https://steveloughran.github. >>>>>>>>>>> io/benchmarking-variants/benchmarking-variants.html >>>>>>>>>>> >>>>>>>>>>> <https://steveloughran.github.io/benchmarking-variants/benchmarking-variants.html> >>>>>>>>>>> 3. Neelesh Salian - Controlled baseline, 10M+100M rows, >>>>>>>>>>> write + read: https://github.com/nssalian/ >>>>>>>>>>> iceberg/tree/iceberg-variant-benchmark/benchmark >>>>>>>>>>> >>>>>>>>>>> <https://github.com/nssalian/iceberg/tree/iceberg-variant-benchmark/benchmark> >>>>>>>>>>> >>>>>>>>>>> If you're working on variant-related changes, please chime in or >>>>>>>>>>> let me know and I'll add it to the tracker. Feedback on the >>>>>>>>>>> benchmarks or >>>>>>>>>>> anything else is welcome. >>>>>>>>>>> >>>>>>>>>>> I've been giving variant updates during the Iceberg Spark Sync >>>>>>>>>>> (Tuesdays, 10 AM PT), but given that this work now spans Iceberg, >>>>>>>>>>> Spark, >>>>>>>>>>> Parquet, and Flink, I think it deserves its own forum. I'd like to >>>>>>>>>>> propose >>>>>>>>>>> a monthly Variant Sync; a short call where contributors can share >>>>>>>>>>> progress, >>>>>>>>>>> surface blockers, and coordinate across repos. If there's interest, >>>>>>>>>>> I'll >>>>>>>>>>> set one up and share an invite on this thread. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Neelesh Salian. >>>>>>>>>>> >>>>>>>>>>
