pltbkd commented on issue #1109: URL: https://github.com/apache/flink-agents/issues/1109#issuecomment-5602052216
Thanks for the detailed report. I think it's the correct direction to use a callback-style submit instead of polling the actions. During my survey I found that, because the `actionTasksKState` is used both as the in-flight requests state and the execution queue, the in-flight async tasks have no correct place: they should be in the `actionTasksKState` as the in-flight requests, while they shouldn't because they should not be scheduled by the execution loop. I think this is also the reason why the wakeup needs a single owner, as you pointed in the second point: as long as the suspended task stays in the queue that the execution loop polls, both the re-submit of the loop and the callback may wake it up. The poll&push also makes the execution order during the recovery depend on how many poll rounds happened before the checkpoint, instead of the order in which the tasks were created. So I'm wondering if we should redesign the execution loop instead of making the coordination of the mailbox submission and the callback submission: separate the in-flight-request list (as state, like the AsyncWaitOperator) and the execution queue (as a pure in-memory queue, which is rebuilt during recovery according to the in-flight-request list). Of course, we can also fix the busy-wait based on the current code first, and do the refactoring later if it's needed. We can discuss the details in the PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
