andygrove opened a new issue, #2833: URL: https://github.com/apache/datafusion-ballista/issues/2833
**Describe the bug** `RunningStage::update_task_metrics` (`ballista/scheduler/src/state/execution_stage.rs`) takes the metrics shape of whichever task reports first. Every later task whose metrics count differs is rejected. After #2828 those rejected tasks are logged and skipped instead of hanging the job, which is the right call. But the choice of which tasks win is still arbitrary. Tasks of one stage can legitimately report different numbers of metric sets. Restricting a task's plan to its partitions can swap an `EmptyExec` (no metrics) for a 0-partition `DataSourceExec` stand-in (with metrics), see #2827. If one of those odd tasks reports first, its shape wins. All the normal tasks after it are then dropped, and the stage ends up with metrics that don't line up with its plan. The plan display and REST API then show no metrics for the stage, which hurts the stage/task observability used to diagnose skew. **Suggested fix** Check each task's metrics count against the stage plan (`collect_plan_metrics(&self.plan).len()`) instead of the first reporter. Tasks that match the stage plan always merge, and only the odd ones get skipped. A smaller related point: the warning added in #2828 fires once per task. In a wide stage where many tasks don't match, that can be thousands of identical lines. Warning once per stage and logging the rest at `debug!` would keep the log readable. **Additional context** The root cause of the mismatch is the `MemorySourceConfig` stand-in in `task_builder.rs`, which exists because a 0-partition `EmptyExec` / `PlaceholderRowExec` doesn't survive a protobuf round trip (apache/datafusion#23643 decodes with `partitions.max(1)`). Fixing that upstream would remove the mismatch entirely. This issue is about making the scheduler pick the right tasks' metrics either way. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
