GitHub user gitmodimo added a comment to the discussion: Safe way to periodically add arrow RecordBatch to a file
I have similar use-case. If you use IPC file and there is a crash you can recover most(flushed) data by skipping file header and reading it as if IPC stream. That is what I do to recover missing footer. For 1 batch=1epoch this should work for you. GitHub link: https://github.com/apache/arrow/discussions/48124#discussioncomment-16670034 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
