alamb commented on issue #7957: URL: https://github.com/apache/datafusion/issues/7957#issuecomment-2246257591
I made some pictures for what I had in mind
This is what happens today (`FilterExec` + `RepartitionExec`):
```
┌────────────────────┐ Filter
│ │ ┌────────────────────┐
Coalesce
│ │ ─ ─ ─ ─ ─ ─ ▶ │ RecordBatch │
Batches
│ RecordBatch │ │ num_rows = 234 │─ ─ ─ ─ ─ ┐
│ num_rows = 8000 │ └────────────────────┘
│ │ │
│ │
┌────────────────────┐
└────────────────────┘ │
│ │
┌────────────────────┐ ┌────────────────────┐
│ │
│ │ Filter │ │ │
│ │
│ │ │ RecordBatch │ ─
─ ─ ─ ─ ▶│ │
│ RecordBatch │ ─ ─ ─ ─ ─ ─ ▶ │ num_rows = 500 │─ ─ ─ ─ ─ ┐
│ │
│ num_rows = 8000 │ │ │
│ RecordBatch │
│ │ │ │ └
─ ─ ─ ─ ─▶│ num_rows = 8000 │
│ │ └────────────────────┘
│ │
└────────────────────┘
│ │
... ─
─ ─ ─ ─ ▶│ │
... ... │
│ │
│ │
┌────────────────────┐ │
└────────────────────┘
│ │ ┌────────────────────┐
│ │ Filter │ │ │
│ RecordBatch │ │ RecordBatch │
│ num_rows = 8000 │ ─ ─ ─ ─ ─ ─ ▶ │ num_rows = 333 │─ ─ ─ ─ ─ ┘
│ │ │ │
│ │ └────────────────────┘
└────────────────────┘
FilterExec
RepartitonExec copies the data
creates output batches with copies
*again* to form final large
of the matching rows (calls take()
RecordBatches
to make a copy)
```
Instead do it like this (only buffer the filter mask results and create the
final output in one go):
```
┌────────────────────┐ Filter
│ │ ┌────────────────────┐
Filter output
│ │ ─ ─ ─ ─ ─ ─ ▶ │ mask │
│ RecordBatch │ │ (BooleanArray) │─ ─ ─ ─ ─ ┐
│ num_rows = 8000 │ └────────────────────┘
│ │ │
│ │
┌────────────────────┐
└────────────────────┘ │
│ │
┌────────────────────┐ ┌────────────────────┐
│ │
│ │ Filter │ │ │
│ │
│ │ │ mask │ ─
─ ─ ─ ─ ▶│ │
│ RecordBatch │ ─ ─ ─ ─ ─ ─ ▶ │ (BooleanArray) │─ ─ ─ ─ ─ ┐
│ │
│ num_rows = 8000 │ │ │
│ RecordBatch │
│ │ │ │ └
─ ─ ─ ─ ─▶│ num_rows = 8000 │
│ │ └────────────────────┘
│ │
└────────────────────┘
│ │
... ─
─ ─ ─ ─ ▶│ │
... ... │
│ │
│ │
┌────────────────────┐ │
└────────────────────┘
│ │ ┌────────────────────┐
│ │ Filter │ │ │
│ RecordBatch │ │ mask │
│ num_rows = 8000 │ ─ ─ ─ ─ ─ ─ ▶ │ (BooleanArray) │─ ─ ─ ─ ─ ┘
│ │ │ │
│ │ └────────────────────┘
└────────────────────┘
FilterExec et all The Exec
then copies rows from
internally buffers RecordBatches and multiple
input batches (based on the
filter results until enough rows are masks)
into a single new output
ready
RecordBatch avoiding the second data
copy
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
