zhangfengcdt commented on PR #7316: URL: https://github.com/apache/datafusion-comet/pull/7316#issuecomment-6529904779
Thanks @comphead and @sunchao ! Agreed that capping the input batch size cannot bound what reaches the expanding operator, and it is not a optimal solution and fix. So I reworked the fix to enforce the bound at the operator, with the following changes: - A new `CometBatchSplitExec` over DataFusion's `BatchSplitStream` is placed directly under a projection, filter, hash aggregate or hash shuffle writer, only when that operator's string scalars would exceed 1 GiB expanded. Plans without a large scalar are unchanged. - Scalars are measured at plan time by evaluating each column-free string subtree once, so `repeat(<subquery>, 8)` counts at its final width. `IN` list members, volatile expressions and JVM UDFs are skipped. - The fixture covers the `WHERE`, window and `repeat` cases, all in `query` mode so it passes strict testing. The grouped count uses a literal; the `missingInput` issue with a grouped count of a subquery is separate. Appreciate if you both could take another look and let me know if you'd refine it or go with other directions. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
