coderfender opened a new issue, #1960: URL: https://github.com/apache/datafusion-ballista/issues/1960
Follow-up to #1916 (part of #1827). #1916 collects per-column null counts on the executor and stores them aggregated on `SuccessfulStage.output_column_stats` in the scheduler. They are not yet consumed: `ShuffleReaderExec::partition_statistics` (`stats_for_partition` in `shuffle_reader.rs`) still returns `column_statistics: vec![ColumnStatistics::new_unknown(); num_fields]`, so the optimizer never sees them. Threading the stored stats to the reader touches ~5 sites: 1. `finalise_stage_internal` (`aqe/planner.rs`) — carry `output_column_stats` alongside partitions. 2. `resolve_shuffle_partitions` (`aqe/execution_plan/exchange.rs` + `adaptive.rs`) — store stats next to partition locations. 3. `BallistaAdapter::transform_children` (`aqe/adapter.rs`) — pass into `ShuffleReaderExec::try_new*`. 4. `ShuffleReaderExec` — new field plus proto/serde changes (serialized core plan node). 5. `stats_for_partition` (`shuffle_reader.rs`) — use the carried stats instead of `new_unknown()`. Also: - [ ] Collect stats on the sort-based shuffle-write path (`sort_shuffle/writer.rs`, currently empty; TODO in code). - [ ] Collect stats on the single-partition write path (`shuffle_writer.rs`, currently empty; TODO in code). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
