comphead commented on PR #3703:
URL:
https://github.com/apache/datafusion-comet/pull/3703#issuecomment-5330342170
> Does this always coalesce many record batches into a single Arrow batch?
Spark sometimes allows very large BHJs, with several GB of broadcast data. It
might be better to use a single compressed Arrow stream containing multiple
bounded-size record batches. We may also need to update DataFusion’s hash join
implementation to support multiple `RecordBatch`es.
Thanks @sunchao looks like you proposing to introduce something like
```
private[spark] val BROADCAST_BLOCKSIZE =
ConfigBuilder("spark.broadcast.blockSize")
.doc("Size of each piece of a block for TorrentBroadcastFactory, in " +
"KiB unless otherwise specified. Too large a value decreases " +
"parallelism during broadcast (makes it slower); however, " +
"if it is too small, BlockManager might take a performance hit")
.version("0.5.0")
.bytesConf(ByteUnit.KiB)
```
but for native side, or respect this param on native to keep broadcasted
batches with respected size?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
