Hi Radu, This is a conversation best had on dev@parquet. It came up recently [1] and I cross-posted there as well.
[1] https://lists.apache.org/thread.html/re4fe4bc80c9eadd446761588f9b03d827193f91269a7c14ce0c444dd%40%3Cdev.arrow.apache.org%3E On Thu, Sep 3, 2020 at 3:20 PM Radu Teodorescu <radukay...@yahoo.com.invalid> wrote: > Hello, > What is the current thinking around allowing the logical content of a > parquet file to be split across multiple files? > I see that in theory there is support for reading files where different > row groups are in separate files but I cannot see any features that allow > that for writing. > > On a somewhat related note, what are the thoughts on supporting parquet > file append mode? > Specifically if the meatadata is stored in a standalone file one can > easily add new row groups to an existing file and create a new version of > the metadata file without affecting potential consumers of the existing > data. > > >
