Jack is right. We evaluate the residual for each task lazily, so it happens on the node that is reading data for the task. The residual is the part of the original filter expression that needs to be run in the task to produce the correct filtered rows. Since that is specific to the task, we don't want to build it during job planning.
On Thu, Jun 3, 2021 at 2:39 PM Jack Ye <yezhao...@gmail.com> wrote: > Not sure if it is the only case, but the top thing that comes to my mind > is that: BaseFileScanTask has ResidualEvaluator which has Expression. > Because ScanTask is serializable and passed around workers, expressions > have to be serializable. > -Jack > > On Thu, Jun 3, 2021 at 2:33 PM QH Yan <mailto...@gmail.com> wrote: > >> Curious about the use-case where Expression get serialized and passed >> around. >> >> Thank you! >> Qinhua >> -- >> *Qinhua* >> >> -- Ryan Blue
