Pratyoy,
Attached is a description of the system's result management framework in
its initial days - I don't think there have been huge changes since,
surprisingly, but there have been some (e.g., to support management of
results that are range-partitioned results of things like parallel
sorts). Shiva Jahangiri may have some expertise related to this due to
the work that she has done on staged execution of queries - not sure...
Historically the reason we don't have a pull-based API - instead, having
our push-based API - is because one of our design criteria was that an
application shouldn't be able to impede the progress of a query if its
consumption rate was lower than the query's production rate,
particularly in the post client-server era where the consumer is on the
wrong side of an internet connection. (Our goal was for the system to
be able to get a query done ASAP to free up the resources it uses for
other queries.)
Cheers,
Mike
On 1/28/25 11:33 AM, Pratyoy Das wrote:
Hi all,
I am trying to get Asterix DB to return eager results instead of returning
all the results at the end when the job is completed. Currently the logic
of result generation waits for the job to complete, but I feel like there
should be a way to make the process eager, if we can figure out where the
results are being stored and keep polling the location. Any pointers that
can help me identify the correct code points to look at and potentially
modify would be appreciated.
Thanks!
