akshaychitneni commented on code in PR #2633:
URL:
https://github.com/apache/datafusion-ballista/pull/2633#discussion_r4395360894
##########
ballista/scheduler/src/state/execution_stage.rs:
##########
@@ -1379,6 +1380,48 @@ fn stage_input_partitions(plan: &Arc<dyn ExecutionPlan>)
-> usize {
.max(1)
}
+/// The inputs `plan` reads from outside its job, e.g. a cache read: a complete
+/// [`StageOutput`] per external [`ShuffleReaderExec`]. As stage inputs,
rollback
+/// and re-resolution restore them like any other input.
+pub(crate) fn external_inputs(plan: &dyn ExecutionPlan) -> HashMap<usize,
StageOutput> {
+ fn collect(plan: &dyn ExecutionPlan, inputs: &mut HashMap<usize,
StageOutput>) {
+ if let Some(reader) = plan.downcast_ref::<ShuffleReaderExec>()
+ && is_external_stage_id(reader.stage_id)
+ {
+ let output = inputs.entry(reader.stage_id).or_default();
+ for (partition, locations) in reader.partition.iter().enumerate() {
+ output
+ .partition_locations
+ .entry(partition)
+ .or_default()
+ .extend(locations.iter().cloned());
+ }
+ output.complete = true;
+ }
+ for child in plan.children() {
+ collect(child.as_ref(), inputs);
+ }
+ }
+ let mut inputs = HashMap::new();
+ collect(plan, &mut inputs);
+ inputs
+}
+
+/// Whether `inputs` hold external (cache) data on `executor_id`.
+pub(crate) fn reads_external_data_on(
Review Comment:
Fixed: a failed fetch now counts as lost cache data only when it matches a
cache location's executor, build stage and partition (an own-input match too is
treated as cache loss, as you suggested). Your repro is a regression test in
a_failed_fetch_fails_the_job_only_for_cache_data
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
