Trying the dev list instead. I can't find any mention in the documentation whether `merge_accumulators` is called with all the accumulators at once or with an iterator. Perusing some of the combinefns in the python sdk, it looks like all of them work with an iterator and don't try to iterate over it twice.
I'm considering implementing CombineFn execution with my runner such that only two accumulators are ever loaded into memory at a time. Not sure if CombineFn contract supports this though. On Fri, Mar 14, 2025 at 8:20 AM Joey Tran <joey.t...@schrodinger.com> wrote: > My intuition says no says all the accumulators will just grouped together > into a single key/accumulators element and that entire element will get > loaded into memory, but not sure if there's any kind of special magic to > handle this > > On Tue, Mar 11, 2025 at 11:16 PM Joey Tran <joey.t...@schrodinger.com> > wrote: > >> Hey all, >> >> Just wondering if CombineFn.merge_accumulators is called with a lazy >> iterator of accumulators or if it's expected to be called with all the >> accumulators in memory >> >> Best, >> Joey >> >
