andygrove opened a new issue, #2675: URL: https://github.com/apache/datafusion-ballista/issues/2675
## Describe the issue `PipelinedShuffleReaderExec::partition` (#2670) is a `Vec<Vec<PipelinedPartitionLocation>>`, and each location carries a full `ExecutorMetadata`: id, host, port, gRPC port, specification and OS info. A stage that reads P partitions from T producer tasks serializes P × T copies of it, and `execute` clones a partition's whole `Vec` for each output partition it starts. This follows `ShuffleReaderExec`, whose `PartitionLocation` carries the same `ExecutorMetadata`, so the blocking reader has the same shape. On wide jobs the plan size grows quadratically, and every task of the consumer stage receives the whole thing. ## Proposed change - Carry an executor table once per reader node and have each location index into it, or - slim the location down to what the reader uses: executor id, host and Flight port. Either applies to both readers. The second is the smaller change and the executor id, host and port are all the readers look at. ## Context Raised in the review of #2670. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
