Understood, thank you for the quick response -----Original Message----- From: Micah Kornfield <emkornfi...@gmail.com> Sent: Wednesday, June 16, 2021 4:13 PM To: dev <dev@arrow.apache.org> Cc: Shamis, Michael <michael.sha...@cubistsystematic.com> Subject: Re: [C++] Parquet streaming
Correct, you cannot recover a partially written parquet file. This is only really feasible with the Arrow Streaming format and even there you might run into issues if the data is not synced at the appropriate place. The arrow file format requires a footer be written so it has the same issue. -Micah On Wed, Jun 16, 2021 at 1:04 PM Ambalu, Robert <robert.amb...@point72.com> wrote: > Apache community, I just want to confirm my understanding of parquet files. > I have a streaming set of data that may be produced in realtime. Ideally > I would stream it into a parquet file ( and if the process crashes, still > be able to read some part of what was streamed ). > I can do this with arrow output ( last written batch would be readable ), > but as far as I can tell its not possible with parquet writers. > Is my understanding correct? > > Thanks > - Rob > > > > > > DISCLAIMER: This e-mail message and any attachments are intended solely > for the use of the individual or entity to which it is addressed and may > contain information that is confidential or legally privileged. If you are > not the intended recipient, you are hereby notified that any dissemination, > distribution, copying or other use of this message or its attachments is > strictly prohibited. If you have received this message in error, please > notify the sender immediately and permanently delete this message and any > attachments. > > > > DISCLAIMER: This e-mail message and any attachments are intended solely for the use of the individual or entity to which it is addressed and may contain information that is confidential or legally privileged. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, copying or other use of this message or its attachments is strictly prohibited. If you have received this message in error, please notify the sender immediately and permanently delete this message and any attachments.
