Hey Alex, This is a great proposal. Thanks for writing it up. I gave it a read and have a couple questions: 1. About the way operators are identified, I think it should use TaskId unless I'm mistaken, which itself contains an ActivityId and an OperatorDescriptorId. I'm sure you've seen TAID:ANID:OID:9:0:0 or similar identifiers in the debugger. I think this encapsulates the intent, which is to uniquely identify an operator within a job. I might be mistaken though, but I think it's worth a try.
2. I am thinking of a case where let's say there are 2 HHJ operators. Let's say they both need to spill. One starts first and tries to allocate all the memory it can, but it's not enough to do it all in-memory anyway. The next one starts but most of the memory is used up now by the first. Will the second be starved and have to go into recursive spilling to fit in a tiny budget? Or can they both share to fit optimally into 1 level of spilling? 3. At what frequency does an operator ask for new memory? I wonder about the overhead of having an operator call out to something that is NC-wide, even if it is a concurrent structure. - Ian On Tue, Aug 18, 2026 at 12:31 PM Alexander Anokhin <[email protected]> wrote: > > Hi everyone, > > I would love to start discussion on the initial dynamic memory resource > broker prototype. > AsterixDB currently assigns fixed memory budgets to memory-intensive > operators (join, sort, group-by) at compile time, and these budgets never > change during execution. If one operator is holding unused memory while > another is spilling to disk, there is no mechanism to redistribute it. > > This proposal introduces a per-NC memory resource broker that dynamically > reallocates memory between operators at runtime. The design covers the > broker interface, a two-stage victimization system (query and operator > level), and integration plans for existing operators. > > APE: > https://cwiki.apache.org/confluence/spaces/ASTERIXDB/pages/449282523/APE+35+Dynamic+Operator+Memory+Broker > > Looking forward to your feedback! > > Regards, > Alexander Anokhin
