2022 10:59 AM
To: dev@arrow.apache.org
Subject: Re: ExecutionContext, batch ordering clarification
Thanks Weston, two follow up questions:
(1) What is the threading model when passing "exector=nullptr" to
"ExecContext" ? (Does it only uses one thread?)
(2) For the file rea
Thanks Weston, two follow up questions:
(1) What is the threading model when passing "exector=nullptr" to
"ExecContext" ? (Does it only uses one thread?)
(2) For the file reader, if we want to ensure batches coming out of the
reader are ordered but also have parallelism, I'd imagine doing sth like
If you are using a source node (which it appears you are) then it will
be creating new thread tasks for each batch. So, in theory, these
could get out of order. My guess is that the file reader is slow
enough that by the time you load batch N from disk and decode it, you
have a pretty good chance
