sunchao opened a new issue, #7318: URL: https://github.com/apache/datafusion-comet/issues/7318
### What is the problem the feature request solves? PR #7190 reduces repeated integer membership keys within individual decoded shuffle batches. It passes through batches smaller than 1,024 rows, so a large partition fed by many small map/reducer frames can receive no reduction even when its keys are highly duplicated. Both native shuffle reader paths currently emit decoded IPC frames without regrouping. For balanced data, `N / (maps * reducers)` is a useful average-frame estimate, but skew and writer flushes mean it is not a hard workload-size threshold. ### Describe the potential solution Evaluate bounded cross-batch reduction for small frames as a separate change from #7190. A key set capped at `MAX_BATCH_ROWS.div_ceil(3)` entries and cleared at the cap is one candidate; it is not a selected implementation. Compare it with other approaches using many-map/reducer controls, unique-key controls and duplicate-heavy inputs. Keeping the set across input pulls changes the current synchronous per-batch scratch lifetime. Validate retained memory, same-task JVM consumer interactions, cancellation/cleanup, row/null semantics, and whole-query cost before choosing that design. A missed optimization by itself does not justify widening #7190's memory scope. Review context: https://github.com/apache/datafusion-comet/pull/7190#discussion_r4414769431 ### Willingness to contribute I can help evaluate the alternatives and controls. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
