On Wed, Oct 10, 2018 at 8:15 AM Aljoscha Krettek <aljos...@apache.org> wrote:
> I think the two things (shared state and new source interface) are > somewhat orthogonal. The new source interface itself alone doesn't solve > the problem, we would still need some mechanism for sharing the event-time > information between different subtasks. This could be the state sharing > mechanism. Therefore I would say we should not block one on the other and > therefore should go ahead with state sharing. > Is that really the case? The reason Thomas gave for the request to share state among subtasks was to implement stream alignment. If streams can be aligned, then the given reason for state sharing disappears. Not to say there aren't other situations where state sharing could be useful. It would have been handy in a number of our jobs. Also, it's not clear to me that if sources (and multiple streams operators) were performing time alignment, you'd need some mechanism for sharing even-time information between subtasks. Each source and multiple input operator can perform its own local alignment and back-pressure can take care of squelching sources that are advancing too fast.
