cetra3 commented on issue #22758: URL: https://github.com/apache/datafusion/issues/22758#issuecomment-4919831831
> If a single query is causing the memory pressure, you'd end up cancelling all the other queries running on that node too. Have you found this problematic? Yes, and because the memory story in DF is not great, even prior to this we isolate queries to be one per process at a time to prevent any out of memory events having a high blast radius. > I agree, ideally this would live in Arrow, although the community has expressed concern over how breaking it would be to integrate a claim() API into each Arrow buffer creation call and/or to adopt the still-unstable Allocator API Yeah it sucks that there isn't a standard trait we can rely on yet and the risk is that the allocator trait looks *very* different when standardised so would create churn. I still think pursuing it is worthwhile and is actually a pretty well trodden path. [PostgreSQL basically highly suggests if not mandates](https://github.com/postgres/postgres/blob/master/src/backend/utils/mmgr/README) you use their `palloc` for allocation for everything. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
