hi Vinay, I think these are application-level concerns -- Flight provides an efficient means to transport datasets on a network. The result of a DoGet or DoPut provides an iterator of record batches which generally may not be that big. How you handle the chunks of data in memory (or put them on disk, etc.) seems out of scope for Flight itself.
- Wes On Mon, Dec 23, 2019 at 3:27 AM Vinay Kesarwani <vnkesarw...@gmail.com> wrote: > > Flight is great way to share data among processes, and holding shared data > in-memory. > > I couldn't find any api or design to handle fall back mechanism in case > data is not fitting in memory. > > Cases: > 1- Once memory buffer is nearing full, data should spill over to disk. > 2- Spilling over disk or memory mapped file. > 3- Should it be .arrow file or feather format on Disk. > 4- Should it be compressed? any design suggestion? > > Can we have a feasibility check and performance impact in terms of > benchmark #.
