zhijiangW commented on issue #8646: [FLINK-12735][network] Make shuffle environment implementation independent with IOManager URL: https://github.com/apache/flink/pull/8646#issuecomment-506310994 @StephanEwen thanks for sharing the background and bringing this concern! The abstract `IOManager` actually has two roles: - One is like the utility tool for getting file channels which is currently used for some streaming component, like `StateBackend, `BarrierBuffer`, `ShuffleService`. - The other one is real abstract methods for handing read/write operations which is mainly used for batch in sort-merge. If we assume that one component should only focus on one single function, it is reasonable to break down `IOManager` into two sub components as now. The other components could only rely on `FileChannelManager` instead of heavy-weight `IOManager`. Currently the shuffle service would only need internal `FileChannelManager`, I agree with that it might rely on the functions of write-behind or pre-fetching threads provided by `IOManager` future. If so, the other shuffle service implementations could implement a new `IOManager` instance or using current `IOManagerAsync` directly if needed future. Another option as you suggested is implementing a new `IOManager` instance without spawning threads, which also makes sense. If the shuffle service would need the read/writer functions future, it has to replace another `IOManager` implementation. But I still prefer the current way in PR, because the individual two sub components are both light-weight and they could be used together or separately.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
