xuyangzhong commented on PR #26051: URL: https://github.com/apache/flink/pull/26051#issuecomment-2639758425
@pnowojski I believe you’re correct. 1. I think the current implementation of StreamCommonSubGraphBasedOptimizer has a flaw in how it handles the removal of mini-batch configs. The rank/dedup node is generated during optimization via specific patterns involving project and filter nodes. Upon further reflection, it might be more effective to remove unnecessary mini-batch assigner nodes only after the exec graph is fully generated. 2. Regarding the second point: if different key configs are used during compiling plan and execute plan, it can indeed lead to issues. This applies not only to configs related to mini-batch, but also to any changes in configs marked in `ExecNodeMetadata#consumedOptions`. It would be advisable for users to recompile the plan in such cases. To avoid blocking this PR, my suggestion is to make a slight improvement to `FlinkRexUtil#shouldSkipMiniBatch` to identify projects that include the `rank_number` function, allowing to temporarily bypass the issue. We can then create a separate jira to address this problem more thoroughly at a later time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
