Fewer commit conflicts meaning the tables representing column families are updated independently, rather than having to serialize commits to a single table. Perhaps with a wide table solution the commit logic could be enhanced to support things like concurrent overwrites to independent column families, but it seems like it would be fairly involved.
> On May 29, 2025, at 7:16 PM, Steven Wu <stevenz...@gmail.com> wrote: > > Bryan, interesting approach to split horizontally across multiple tables. > > A few potential down sides > * operational overhead. tables need to be managed consistently and probably > in some coordinated way > * complex read > * maybe fragile to enforce correctness (during join). It is robust to enforce > the stitching correctness at file group level in file reader and writer if > built in the table format. > > > fewer commit conflicts > > Can you elaborate on this one? Are those tables populated by streaming or > batch pipelines? > > On Thu, May 29, 2025 at 5:03 PM Bryan Keller <brya...@gmail.com > <mailto:brya...@gmail.com>> wrote: >> Hi everyone, >> >> We have been investigating a wide table format internally for a similar use >> case, i.e. we have wide ML tables with features generated by different >> pipelines and teams but want a unified view of the data. We are comparing >> that against separate tables joined together using a shuffle-less join (e.g. >> storage partition join), along with a corresponding view. >> >> The join/view approach seems to give us much of we need, with some added >> benefits like splitting up the metadata, fewer commit conflicts, and ability >> to share, nest, and swap "column families". The downsides are table >> management is split across multiple tables, it requires engine support of >> shuffle-less joins for best performance, and even then, scans probably won't >> be as optimal. >> >> I'm curious if anyone had further thoughts on the two? >> >> -Bryan >> >> >> >>> On May 29, 2025, at 8:18 AM, Péter Váry <peter.vary.apa...@gmail.com >>> <mailto:peter.vary.apa...@gmail.com>> wrote: >>> >>> I received feedback from Alkis regarding their Parquet optimization work. >>> Their internal testing shows promising results for reducing metadata size >>> and improving parsing performance. They plan to formalize a proposal for >>> these Parquet enhancements in the near future. >>> >>> Meanwhile, I'm putting together our horizontal sharding proposal as a >>> complementary approach. Even with the Parquet metadata improvements, >>> horizontal sharding would provide additional benefits for: >>> More efficient column-level updates >>> Streamlined column additions >>> Better handling of dominant columns that can cause RowGroup size imbalances >>> (placing these in separate files could significantly improve performance) >>> Thanks, Peter >>> >>> >>> >>> Péter Váry <peter.vary.apa...@gmail.com >>> <mailto:peter.vary.apa...@gmail.com>> ezt írta (időpont: 2025. máj. 28., >>> Sze, 15:39): >>>> I would be happy to put together a proposal based on the inputs got here. >>>> >>>> Thanks everyone for your thoughts! >>>> I will try to incorporate all of this. >>>> >>>> Thanks, Peter >>>> >>>> Daniel Weeks <dwe...@apache.org <mailto:dwe...@apache.org>> ezt írta >>>> (időpont: 2025. máj. 27., K, 20:07): >>>>> I feel like we have two different issues we're talking about here that >>>>> aren't necessarily tied (though solutions may address both): 1) wide >>>>> tables, 2) adding columns >>>>> >>>>> Wide tables are definitely a problem where parquet has limitations. I'm >>>>> optimistic about the ongoing work to help improve parquet footers/stats >>>>> in this area that Fokko mentioned. There are always limitations in how >>>>> this scales as wide rows lead to small row groups and the cost to >>>>> reconstitute a row gets more expensive, but for cases that are read heavy >>>>> and projecting subsets of columns should significantly improve >>>>> performance. >>>>> >>>>> Adding columns to an existing dataset is something that comes up >>>>> periodically, but there's a lot of complexity involved in this. Parquet >>>>> does support referencing columns in separate files per the spec, but >>>>> there's no implementation that takes advantage of this to my knowledge. >>>>> This does allow for approaches where you separate/rewrite just the >>>>> footers or various other tricks, but these approaches get complicated >>>>> quickly and the number of readers that can consume those representations >>>>> would initially be very limited. >>>>> >>>>> A larger problem for splitting columns across files is that there are a >>>>> lot of assumptions about how data is laid out in both readers and >>>>> writers. For example, aligning row groups and correctly handling split >>>>> calculation is very complicated if you're trying to split rows across >>>>> files. Other features are also impacted like deletes, which reference >>>>> the file to which they apply and would need to account for deletes >>>>> applying to multiple files and needing to update those references if >>>>> columns are added. >>>>> >>>>> I believe there are a lot of interesting approaches to addressing these >>>>> use cases, but we'd really need a thorough proposal that explores all of >>>>> these scenarios. The last thing we would want is to introduce >>>>> incompatibilities within the format that result in incompatible features. >>>>> >>>>> -Dan >>>>> >>>>> On Tue, May 27, 2025 at 10:02 AM Russell Spitzer >>>>> <russell.spit...@gmail.com <mailto:russell.spit...@gmail.com>> wrote: >>>>>> Point definitely taken. We really should probably POC some of these >>>>>> ideas and see what we are actually dealing with. (He said without >>>>>> volunteering to do the work :P) >>>>>> >>>>>> On Tue, May 27, 2025 at 11:55 AM Selcuk Aya >>>>>> <selcuk....@snowflake.com.invalid> wrote: >>>>>>> Yes having to rewrite the whole file is not ideal but I believe most of >>>>>>> the cost of rewriting a file comes from decompression, encoding, stats >>>>>>> calculations etc. If you are adding new values for some columns but are >>>>>>> keeping the rest of the columns the same in the file, then a bunch of >>>>>>> rewrite cost can be optimized away. I am not saying this is better than >>>>>>> writing to a separate file, I am not sure how much worse it is though. >>>>>>> >>>>>>> On Tue, May 27, 2025 at 9:40 AM Russell Spitzer >>>>>>> <russell.spit...@gmail.com <mailto:russell.spit...@gmail.com>> wrote: >>>>>>>> I think that "after the fact" modification is one of the requirements >>>>>>>> here, IE: Updating a single column without rewriting the whole file. >>>>>>>> If we have to write new metadata for the file aren't we in the same >>>>>>>> boat as having to rewrite the whole file? >>>>>>>> >>>>>>>> On Tue, May 27, 2025 at 11:27 AM Selcuk Aya >>>>>>>> <selcuk....@snowflake.com.invalid> wrote: >>>>>>>>> If files represent column projections of a table rather than the >>>>>>>>> whole columns in the table, then any read that reads across these >>>>>>>>> files needs to identify what constitutes a row. Lance DB for example >>>>>>>>> has vertical partitioning across columns but also horizontal >>>>>>>>> partitioning across rows such that in each horizontal >>>>>>>>> partitioning(fragment), the same number of rows exist in each >>>>>>>>> vertical partition, which I think is necessary to make whole/partial >>>>>>>>> row construction cheap. If this is the case, there is no reason not >>>>>>>>> to achieve the same data layout inside a single columnar file with a >>>>>>>>> lean header. I think the only valid argument for a separate file is >>>>>>>>> adding a new set of columns to an existing table, but even then I am >>>>>>>>> not sure a separate file is absolutely necessary for good performance. >>>>>>>>> >>>>>>>>> Selcuk >>>>>>>>> >>>>>>>>> On Tue, May 27, 2025 at 9:18 AM Devin Smith >>>>>>>>> <devinsm...@deephaven.io.invalid> wrote: >>>>>>>>>> There's a `file_path` field in the parquet ColumnChunk structure, >>>>>>>>>> https://github.com/apache/parquet-format/blob/apache-parquet-format-2.11.0/src/main/thrift/parquet.thrift#L959-L962 >>>>>>>>>> >>>>>>>>>> I'm not sure what tooling actually supports this though. Could be >>>>>>>>>> interesting to see what the history of this is. >>>>>>>>>> https://lists.apache.org/thread/rcv1cxndp113shjybfcldh6nq1t3lcq3, >>>>>>>>>> https://lists.apache.org/thread/k5nv310yp315fttcz213l8o0vmnd7vyw >>>>>>>>>> >>>>>>>>>> On Tue, May 27, 2025 at 8:59 AM Russell Spitzer >>>>>>>>>> <russell.spit...@gmail.com <mailto:russell.spit...@gmail.com>> wrote: >>>>>>>>>>> I have to agree that while there can be some fixes in Parquet, we >>>>>>>>>>> fundamentally need a way to split a "row group" >>>>>>>>>>> or something like that between separate files. If that's something >>>>>>>>>>> we can do in the parquet project that would be great >>>>>>>>>>> but it feels like we need to start exploring more drastic options >>>>>>>>>>> than footer encoding. >>>>>>>>>>> >>>>>>>>>>> On Mon, May 26, 2025 at 8:42 PM Gang Wu <ust...@gmail.com >>>>>>>>>>> <mailto:ust...@gmail.com>> wrote: >>>>>>>>>>>> I agree with Steven that there are limitations that Parquet cannot >>>>>>>>>>>> do. >>>>>>>>>>>> >>>>>>>>>>>> In addition to adding new columns by rewriting all files, files of >>>>>>>>>>>> wide tables may suffer from bad performance like below: >>>>>>>>>>>> - Poor compression of row groups because there are too many >>>>>>>>>>>> columns and even a small number of rows can reach the row group >>>>>>>>>>>> threshold. >>>>>>>>>>>> - Dominating columns (e.g. blobs) may contribute to 99% size of a >>>>>>>>>>>> row group, leading to unbalanced column chunks and deteriorate the >>>>>>>>>>>> row group compression. >>>>>>>>>>>> - Similar to adding new columns, partial update also requires >>>>>>>>>>>> rewriting all columns of the affected rows. >>>>>>>>>>>> >>>>>>>>>>>> IIRC, some table formats already support splitting columns into >>>>>>>>>>>> different files: >>>>>>>>>>>> - Lance manifest splits a fragment [1] into one or more data files. >>>>>>>>>>>> - Apache Hudi has the concept of column family [2]. >>>>>>>>>>>> - Apache Paimon supports sequence groups [3] for partial update. >>>>>>>>>>>> >>>>>>>>>>>> Although Parquet can introduce the concept of logical file and >>>>>>>>>>>> physical file to manage the columns to file mapping, this looks >>>>>>>>>>>> like yet another manifest file design which duplicates the purpose >>>>>>>>>>>> of Iceberg. >>>>>>>>>>>> These might be something worth exploring in Iceberg. >>>>>>>>>>>> >>>>>>>>>>>> [1] https://lancedb.github.io/lance/format.html#fragments >>>>>>>>>>>> [2] https://github.com/apache/hudi/blob/master/rfc/rfc-80/rfc-80.md >>>>>>>>>>>> [3] >>>>>>>>>>>> https://paimon.apache.org/docs/0.9/primary-key-table/merge-engine/partial-update/#sequence-group >>>>>>>>>>>> >>>>>>>>>>>> Best, >>>>>>>>>>>> Gang >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 27, 2025 at 7:03 AM Steven Wu <stevenz...@gmail.com >>>>>>>>>>>> <mailto:stevenz...@gmail.com>> wrote: >>>>>>>>>>>>> The Parquet metadata proposal (linked by Fokko) is mainly >>>>>>>>>>>>> addressing the read performance due to bloated metadata. >>>>>>>>>>>>> >>>>>>>>>>>>> What Peter described in the description seems useful for some ML >>>>>>>>>>>>> workload of feature engineering. A new set of features/columns >>>>>>>>>>>>> are added to the table. Currently, Iceberg would require >>>>>>>>>>>>> rewriting all data files to combine old and new columns (write >>>>>>>>>>>>> amplification). Similarly, in the past the community also talked >>>>>>>>>>>>> about the use cases of updating a single column, which would >>>>>>>>>>>>> require rewriting all data files. >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, May 26, 2025 at 2:42 PM Péter Váry >>>>>>>>>>>>> <peter.vary.apa...@gmail.com >>>>>>>>>>>>> <mailto:peter.vary.apa...@gmail.com>> wrote: >>>>>>>>>>>>>> Do you have the link at hand for the thread where this was >>>>>>>>>>>>>> discussed on the Parquet list? >>>>>>>>>>>>>> The docs seem quite old, and the PR stale, so I would like to >>>>>>>>>>>>>> understand the situation better. >>>>>>>>>>>>>> If it is possible to do this in Parquet, that would be great, >>>>>>>>>>>>>> but Avro, ORC would still suffer. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Amogh Jahagirdar <2am...@gmail.com <mailto:2am...@gmail.com>> >>>>>>>>>>>>>> ezt írta (időpont: 2025. máj. 26., H, 22:07): >>>>>>>>>>>>>>> Hey Peter, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks for bringing this issue up. I think I agree with Fokko; >>>>>>>>>>>>>>> the issue of wide tables leading to Parquet metadata bloat and >>>>>>>>>>>>>>> poor Thrift deserialization performance is a long standing >>>>>>>>>>>>>>> issue that I believe there's motivation in the community to >>>>>>>>>>>>>>> address. So to me it seems better to address it in Parquet >>>>>>>>>>>>>>> itself rather than Iceberg library facilitate a pattern which >>>>>>>>>>>>>>> works around the limitations. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> Amogh Jahagirdar >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Mon, May 26, 2025 at 1:42 PM Fokko Driesprong >>>>>>>>>>>>>>> <fo...@apache.org <mailto:fo...@apache.org>> wrote: >>>>>>>>>>>>>>>> Hi Peter, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks for bringing this up. Wouldn't it make more sense to >>>>>>>>>>>>>>>> fix this in Parquet itself? It has been a long-running issue >>>>>>>>>>>>>>>> on Parquet, and there is still active interest from the >>>>>>>>>>>>>>>> community. There is a PR to replace the footer with >>>>>>>>>>>>>>>> FlatBuffers, which dramatically improves performance >>>>>>>>>>>>>>>> <https://github.com/apache/arrow/pull/43793>. The underlying >>>>>>>>>>>>>>>> proposal can be found here >>>>>>>>>>>>>>>> <https://docs.google.com/document/d/1PQpY418LkIDHMFYCY8ne_G-CFpThK15LLpzWYbc7rFU/edit?tab=t.0#heading=h.atbrz9ch6nfa>. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Kind regards, >>>>>>>>>>>>>>>> Fokko >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Op ma 26 mei 2025 om 20:35 schreef yun zou >>>>>>>>>>>>>>>> <yunzou.colost...@gmail.com >>>>>>>>>>>>>>>> <mailto:yunzou.colost...@gmail.com>>: >>>>>>>>>>>>>>>>> +1, I am really interested in this topic. Performance has >>>>>>>>>>>>>>>>> always been a problem when dealing with wide tables, not just >>>>>>>>>>>>>>>>> read/write, but also during compilation. Most of the ML use >>>>>>>>>>>>>>>>> cases typically exhibit a vectorized read/write pattern, I am >>>>>>>>>>>>>>>>> also wondering if there is any way at the metadata level to >>>>>>>>>>>>>>>>> help the whole compilation and execution process. I do not >>>>>>>>>>>>>>>>> have any answer fo this yet, but I would be really interested >>>>>>>>>>>>>>>>> in exploring this further. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>>>>> Yun >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Mon, May 26, 2025 at 9:14 AM Pucheng Yang >>>>>>>>>>>>>>>>> <py...@pinterest.com.invalid> wrote: >>>>>>>>>>>>>>>>>> Hi Peter, I am interested in this proposal. What's more, I >>>>>>>>>>>>>>>>>> am curious if there is a similar story on the write side as >>>>>>>>>>>>>>>>>> well (how to generate these splitted files) and >>>>>>>>>>>>>>>>>> specifically, are you targeting feature backfill use cases >>>>>>>>>>>>>>>>>> in ML use? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Mon, May 26, 2025 at 6:29 AM Péter Váry >>>>>>>>>>>>>>>>>> <peter.vary.apa...@gmail.com >>>>>>>>>>>>>>>>>> <mailto:peter.vary.apa...@gmail.com>> wrote: >>>>>>>>>>>>>>>>>>> Hi Team, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> In machine learning use-cases, it's common to encounter >>>>>>>>>>>>>>>>>>> tables with a very high number of columns - sometimes even >>>>>>>>>>>>>>>>>>> in the range of several thousand. I've seen cases with up >>>>>>>>>>>>>>>>>>> to 15,000 columns. Storing such wide tables in a single >>>>>>>>>>>>>>>>>>> Parquet file is often suboptimal, as Parquet can become a >>>>>>>>>>>>>>>>>>> bottleneck, even when only a subset of columns is queried. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> A common approach to mitigate this is to split the data >>>>>>>>>>>>>>>>>>> across multiple Parquet files. With the upcoming File >>>>>>>>>>>>>>>>>>> Format API, we could introduce a layer that combines these >>>>>>>>>>>>>>>>>>> files into a single iterator, enabling efficient reading of >>>>>>>>>>>>>>>>>>> wide and very wide tables. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> To support this, we would need to revise the metadata >>>>>>>>>>>>>>>>>>> specification. Instead of the current `_file` column, we >>>>>>>>>>>>>>>>>>> could introduce a _files column containing: >>>>>>>>>>>>>>>>>>> - `_file_column_ids`: the column IDs present in each file >>>>>>>>>>>>>>>>>>> - `_file_path`: the path to the corresponding file >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Has there been any prior discussion around this idea? >>>>>>>>>>>>>>>>>>> Is anyone else interested in exploring this further? >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Best regards, >>>>>>>>>>>>>>>>>>> Peter >>
