Hi Nicholas, Thanks for taking a look at the PR!
1. Regarding switching mechanism: There has been previous discussion in this thread regarding the pros and cons of how the switching can be exposed to the user. With fixed start positions, no special switching interface to transfer information between enumerators is required. Sources are configured as they would be when used standalone and just plugged into HybridSource. I expect that to be a common use case. You can find an example for this in the ITCase: https://github.com/apache/flink/pull/15924/files#diff-fe1407d135d7b7b3a72aeb4471ab53ccd2665a58ff0129a83db1ec19cea06f1bR101 For dynamic start position, the checkpoint state is used to transfer information from old to new enumerator. An example for that can be found here: https://github.com/apache/flink/pull/15924/files#diff-fe1407d135d7b7b3a72aeb4471ab53ccd2665a58ff0129a83db1ec19cea06f1bR112-R136 That may look verbose, but the code to convert from one state to another can be factored out into a utility and the function becomes a one-liner. For common sources like files and Kafka we can potentially (later) implement the conversion logic as part of the respective connector's checkpoint and split classes. I hope that with the PR up for review, we can soon reach a conclusion on how we want to expose this to the user. Following is an example for Files -> Files -> Kafka that I'm using for e2e testing. It exercises both ways of setting the start position. https://gist.github.com/tweise/3139d66461e87986f6eddc70ff06ef9a 2. Regarding the events used to implement the actual switch between enumerator and readers: I updated the PR with javadoc to clarify the intent. Please let me know if that helps or let's continue to discuss those details on the PR? Thanks, Thomas On Mon, May 17, 2021 at 1:03 AM Nicholas Jiang <programg...@163.com> wrote: > > Hi Thomas, > > Sorry for later reply for your POC. I have reviewed the based abstract > implementation of your pull request: > https://github.com/apache/flink/pull/15924. IMO, for the switching > mechanism, this level of abstraction is not concise enough, which doesn't > make connector contribution easier. In theory, it is necessary to introduce > a set of interfaces to support the switching mechanism. The SwitchableSource > and SwitchableSplitEnumerator interfaces are needed for connector > expansibility. > In other words, the whole switching process of above mentioned PR is > different from that mentioned in FLIP-150. In the above implementation, the > source reading switching is executed after receving the SwitchSourceEvent, > which could be before the sending SourceReaderFinishEvent. This timeline of > source reading switching could be discussed here. > @Stephan @Becket, if you are available, please help to review the > abstract implementation, and compare with the interfaces mentioned in > FLIP-150. > > Thanks, > Nicholas Jiang > > > > -- > Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/
