zentol opened a new pull request #15019: URL: https://github.com/apache/flink/pull/15019
Introduces a data structure to store the attempt numbers outside the ExecutionGraph. It's really just a simple Map<ExecutionVertexID, Integer> which ties a specific vertex+subtask to an attempt count. Counts are set when an execution is registered at the EG, and retrieved when the ExecutionVertex creates a new Execution. The current attempt count is also still stored in the Execution, making the change less invasive (for example, resetForNewExecution continues to work without modifications). One thing is that, as is, the semantics when it comes to rescaling are a bit funky. ScaleUp: If you begin with p=1 and an attempt count of 4, and then rescale to p=2, then what should the attempt count be for both subtasks? In this version the attempt count for subtask 1 would be retained, while subtask 2 starts at 0. Setting both to 0 would also make sense, but if we downscale again to p=1 then it would be nice if the attempt count had some relation to the original count. Alternatively we could try to derive the attempt count for subtask 2 from other subtasks; in this example the obvious choice would be 2, because we're just replicating subtask 1. ScaleDown: The main issue arises when scaling down where the subtask with the largest index has the highest attempt count; currently this count would be lost. So you have p=2, and subtask 2 has an attempt count of 4, and now you scale down to p=1. The attempt count would now be solely determined by subtask 1, although we in essence just merged the two. Overall, I don't think resetting attempt counts to 0 is an option, because they can be used to gauge the health of a vertex, and we'd run into collisions within metrics if we ever re-use a subtask+attempt combination. The current approach is by far the simplest, and is the only option iff we want to adhere to these rules: * every combination of subtask + attempt count is only used once * the attempt counts for a given subtask over time always form a continuous series starting at 0 But I'm quite interested in what other people think about this. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
