> All we have to do is add the parquet module as a test dependency, working on a poc now.
This will be a circular dependency on the core module. That's why I suggested abstracting out the test cases and executing them in a parquet module. Partition stats writing (as parquet) from the core module uses `InternalData` and does the same now. So, I guess it will be a similar work (but on a larger scale due to testcase refactoring). Let me know the results of your POC and happy to collaborate on this work. - Ajantha On Fri, Jun 13, 2025 at 3:16 AM Russell Spitzer <russell.spit...@gmail.com> wrote: > All we have to do is add the parquet module as a test dependency, working > on a poc now. I don't think we really need to block on any other projects > although I'll probably hold off on any work on manifest-list since I hope > it won't be needed. > > On Thu, May 29, 2025 at 8:37 PM Ajantha Bhat <ajanthab...@gmail.com> > wrote: > >> I am interested in working on this proposal. >> I would assume it is to use `InternalData` with the format as >> `parquet`. But the challenge will be the test cases, the core module cannot >> write the parquet metadata due to circular dependency. We need to abstract >> out the test cases in the core module and run them from the parquet module >> I guess. >> >> I can work on a design doc as well. So, add me as a collaborator for the >> document. >> But should this work be done after we complete the work on "single file >> commit in v4" ? because metadata structure can change? >> >> - Ajantha >> >> On Thu, May 29, 2025 at 11:37 PM Russell Spitzer < >> russell.spit...@gmail.com> wrote: >> >>> Hi Y'all >>> >>> As discussed in the last community sync, we are beginning to gather up >>> folks who are interested in various efforts for Iceberg V4. To that end, >>> I'd like to use this thread as a gathering point for folks interested in >>> the metadata file format shift to Parquet. I wrote a quick abstract to >>> describe the purpose of this group. >>> >>> Following this I'll be working on a full design document or if someone >>> has one in prod please let us know and we can start discussing/working on >>> it there. >>> >>> *Abstract: Parquet as Metadata File Format* >>> >>> Currently the Iceberg SDK and Spec use Avro file format files for all >>> Manifest Lists and Manifests. The row oriented format was selected >>> because it was assumed that most metadata would be read in its entirety. >>> This has turned out to seldom be the case and the ability to read >>> single elements of the metrics would be very useful for query planning. >>> To address this we propose switching the underlying manifest format >>> from Avro to Parquet. In V4, Avro files would still be readable but all >>> new metadata files would be written in Parquet instead of Avro. >>> >>
