Hi Anthony,
Le 26/06/2018 à 17:55, Anthony Scopatz a écrit : > > 2. Should arrow compute assume that everything fits in memory? Arrow > seem to handle data that is larger than memory via the Buffer API. Are > there restrictions that using Buffers imply that we should be aware of? A Buffer certainly assumes that your data can be accessed through its linear address range; it doesn't assume that everything is present at a given point in physical memory. So the solution to larger than memory data would be to use virtual memory facilities, which is how Plasma works. As for CudaBuffer, I *think* it assumes that everything is in the GPU-addressable range of memory it describes. Whether or not the GPU supports some notion of virtual memory depends on the GPU model, I guess? On the topic of Buffers and GPUs, I'd like to point out the following issue with the current APIs: https://issues.apache.org/jira/browse/ARROW-2447 Regards Antoine.
