2022 10:59 AM
To: dev@arrow.apache.org
Subject: Re: ExecutionContext, batch ordering clarification
Thanks Weston, two follow up questions:
(1) What is the threading model when passing "exector=nullptr" to
"ExecContext" ? (Does it only uses one thread?)
(2) For the file rea
Thanks Weston, two follow up questions:
(1) What is the threading model when passing "exector=nullptr" to
"ExecContext" ? (Does it only uses one thread?)
(2) For the file reader, if we want to ensure batches coming out of the
reader are ordered but also have parallelism, I'd imagine doing sth like
If you are using a source node (which it appears you are) then it will
be creating new thread tasks for each batch. So, in theory, these
could get out of order. My guess is that the file reader is slow
enough that by the time you load batch N from disk and decode it, you
have a pretty good chance
Hi all,
I am doing some investigations of the AsOfJoinNode, and consequently have
come across some strange behavior when experimenting with the
ExecutionContext and in-memory / file streaming source nodes.
Our AsOfJoin algorithm requires that the input be in chronological order
with respect to on
