XiangpengHao commented on issue #26682: URL: https://github.com/apache/datafusion/issues/26682#issuecomment-6478891743
fwiw, I'm working on a very related thing as part of my work at [SpiralDB](https://spiraldb.com): https://github.com/XiangpengHao/t4 t4 is not ready to use yet, will make blog posts & announcements when it's ready in a few weeks. The high level idea is that `object_store` trait already give us a lot of opportunity to do io optimizations, and t4 especially focus on super fast SSD caching (using io_uring), with good enough SSDs (which you can rent from AWS), t4 can serve Parquet/Vortex data from disk as fast as from memory. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
